• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# The Art Of Scripting HTTP Requests Using Curl
2
3## Background
4
5 This document assumes that you are familiar with HTML and general networking.
6
7 The increasing amount of applications moving to the web has made "HTTP
8 Scripting" more frequently requested and wanted. To be able to automatically
9 extract information from the web, to fake users, to post or upload data to
10 web servers are all important tasks today.
11
12 Curl is a command line tool for doing all sorts of URL manipulations and
13 transfers, but this particular document will focus on how to use it when
14 doing HTTP requests for fun and profit. This documents assumes that you know
15 how to invoke `curl --help` or `curl --manual` to get basic information about
16 it.
17
18 Curl is not written to do everything for you. It makes the requests, it gets
19 the data, it sends data and it retrieves the information. You probably need
20 to glue everything together using some kind of script language or repeated
21 manual invokes.
22
23## The HTTP Protocol
24
25 HTTP is the protocol used to fetch data from web servers. It is a simple
26 protocol that is built upon TCP/IP. The protocol also allows information to
27 get sent to the server from the client using a few different methods, as will
28 be shown here.
29
30 HTTP is plain ASCII text lines being sent by the client to a server to
31 request a particular action, and then the server replies a few text lines
32 before the actual requested content is sent to the client.
33
34 The client, curl, sends an HTTP request. The request contains a method (like
35 GET, POST, HEAD etc), a number of request headers and sometimes a request
36 body. The HTTP server responds with a status line (indicating if things went
37 well), response headers and most often also a response body. The "body" part
38 is the plain data you requested, like the actual HTML or the image etc.
39
40## See the Protocol
41
42 Using curl's option [`--verbose`](https://curl.se/docs/manpage.html#-v)
43 (`-v` as a short option) will display what kind of commands curl sends to the
44 server, as well as a few other informational texts.
45
46 `--verbose` is the single most useful option when it comes to debug or even
47 understand the curl<->server interaction.
48
49 Sometimes even `--verbose` is not enough. Then
50 [`--trace`](https://curl.se/docs/manpage.html#-trace) and
51 [`--trace-ascii`](https://curl.se/docs/manpage.html#--trace-ascii)
52 offer even more details as they show **everything** curl sends and
53 receives. Use it like this:
54
55    curl --trace-ascii debugdump.txt http://www.example.com/
56
57## See the Timing
58
59 Many times you may wonder what exactly is taking all the time, or you just
60 want to know the amount of milliseconds between two points in a transfer. For
61 those, and other similar situations, the
62 [`--trace-time`](https://curl.se/docs/manpage.html#--trace-time) option
63 is what you need. It will prepend the time to each trace output line:
64
65    curl --trace-ascii d.txt --trace-time http://example.com/
66
67## See which Transfer
68
69 When doing parallel transfers, it is relevant to see which transfer is
70 doing what. When response headers are received (and logged) you need to
71 know which transfer these are for.
72 [`--trace-ids`](https://curl.se/docs/manpage.html#--trace-ids) option
73 is what you need. It will prepend the transfer and connection identifier
74 to each trace output line:
75
76    curl --trace-ascii d.txt --trace-ids http://example.com/
77
78## See the Response
79
80 By default curl sends the response to stdout. You need to redirect it
81 somewhere to avoid that, most often that is done with `-o` or `-O`.
82
83# URL
84
85## Spec
86
87 The Uniform Resource Locator format is how you specify the address of a
88 particular resource on the Internet. You know these, you have seen URLs like
89 https://curl.se or https://example.com a million times. RFC 3986 is the
90 canonical spec. And yeah, the formal name is not URL, it is URI.
91
92## Host
93
94 The host name is usually resolved using DNS or your /etc/hosts file to an IP
95 address and that is what curl will communicate with. Alternatively you specify
96 the IP address directly in the URL instead of a name.
97
98 For development and other trying out situations, you can point to a different
99 IP address for a host name than what would otherwise be used, by using curl's
100 [`--resolve`](https://curl.se/docs/manpage.html#--resolve) option:
101
102    curl --resolve www.example.org:80:127.0.0.1 http://www.example.org/
103
104## Port number
105
106 Each protocol curl supports operates on a default port number, be it over TCP
107 or in some cases UDP. Normally you do not have to take that into
108 consideration, but at times you run test servers on other ports or
109 similar. Then you can specify the port number in the URL with a colon and a
110 number immediately following the host name. Like when doing HTTP to port
111 1234:
112
113    curl http://www.example.org:1234/
114
115 The port number you specify in the URL is the number that the server uses to
116 offer its services. Sometimes you may use a proxy, and then you may
117 need to specify that proxy's port number separately from what curl needs to
118 connect to the server. Like when using an HTTP proxy on port 4321:
119
120    curl --proxy http://proxy.example.org:4321 http://remote.example.org/
121
122## User name and password
123
124 Some services are setup to require HTTP authentication and then you need to
125 provide name and password which is then transferred to the remote site in
126 various ways depending on the exact authentication protocol used.
127
128 You can opt to either insert the user and password in the URL or you can
129 provide them separately:
130
131    curl http://user:password@example.org/
132
133 or
134
135    curl -u user:password http://example.org/
136
137 You need to pay attention that this kind of HTTP authentication is not what
138 is usually done and requested by user-oriented websites these days. They tend
139 to use forms and cookies instead.
140
141## Path part
142
143 The path part is just sent off to the server to request that it sends back
144 the associated response. The path is what is to the right side of the slash
145 that follows the host name and possibly port number.
146
147# Fetch a page
148
149## GET
150
151 The simplest and most common request/operation made using HTTP is to GET a
152 URL. The URL could itself refer to a web page, an image or a file. The client
153 issues a GET request to the server and receives the document it asked for.
154 If you issue the command line
155
156    curl https://curl.se
157
158 you get a web page returned in your terminal window. The entire HTML document
159 that that URL holds.
160
161 All HTTP replies contain a set of response headers that are normally hidden,
162 use curl's [`--include`](https://curl.se/docs/manpage.html#-i) (`-i`)
163 option to display them as well as the rest of the document.
164
165## HEAD
166
167 You can ask the remote server for ONLY the headers by using the
168 [`--head`](https://curl.se/docs/manpage.html#-I) (`-I`) option which
169 will make curl issue a HEAD request. In some special cases servers deny the
170 HEAD method while others still work, which is a particular kind of annoyance.
171
172 The HEAD method is defined and made so that the server returns the headers
173 exactly the way it would do for a GET, but without a body. It means that you
174 may see a `Content-Length:` in the response headers, but there must not be an
175 actual body in the HEAD response.
176
177## Multiple URLs in a single command line
178
179 A single curl command line may involve one or many URLs. The most common case
180 is probably to just use one, but you can specify any amount of URLs. Yes
181 any. No limits. You will then get requests repeated over and over for all the
182 given URLs.
183
184 Example, send two GET requests:
185
186    curl http://url1.example.com http://url2.example.com
187
188 If you use [`--data`](https://curl.se/docs/manpage.html#-d) to POST to
189 the URL, using multiple URLs means that you send that same POST to all the
190 given URLs.
191
192 Example, send two POSTs:
193
194    curl --data name=curl http://url1.example.com http://url2.example.com
195
196
197## Multiple HTTP methods in a single command line
198
199 Sometimes you need to operate on several URLs in a single command line and do
200 different HTTP methods on each. For this, you will enjoy the
201 [`--next`](https://curl.se/docs/manpage.html#-:) option. It is basically
202 a separator that separates a bunch of options from the next. All the URLs
203 before `--next` will get the same method and will get all the POST data
204 merged into one.
205
206 When curl reaches the `--next` on the command line, it will sort of reset the
207 method and the POST data and allow a new set.
208
209 Perhaps this is best shown with a few examples. To send first a HEAD and then
210 a GET:
211
212    curl -I http://example.com --next http://example.com
213
214 To first send a POST and then a GET:
215
216    curl -d score=10 http://example.com/post.cgi --next http://example.com/results.html
217
218# HTML forms
219
220## Forms explained
221
222 Forms are the general way a website can present an HTML page with fields for
223 the user to enter data in, and then press some kind of 'OK' or 'Submit'
224 button to get that data sent to the server. The server then typically uses
225 the posted data to decide how to act. Like using the entered words to search
226 in a database, or to add the info in a bug tracking system, display the
227 entered address on a map or using the info as a login-prompt verifying that
228 the user is allowed to see what it is about to see.
229
230 Of course there has to be some kind of program on the server end to receive
231 the data you send. You cannot just invent something out of the air.
232
233## GET
234
235 A GET-form uses the method GET, as specified in HTML like:
236
237```html
238<form method="GET" action="junk.cgi">
239  <input type=text name="birthyear">
240  <input type=submit name=press value="OK">
241</form>
242```
243
244 In your favorite browser, this form will appear with a text box to fill in
245 and a press-button labeled "OK". If you fill in '1905' and press the OK
246 button, your browser will then create a new URL to get for you. The URL will
247 get `junk.cgi?birthyear=1905&press=OK` appended to the path part of the
248 previous URL.
249
250 If the original form was seen on the page `www.example.com/when/birth.html`,
251 the second page you will get will become
252 `www.example.com/when/junk.cgi?birthyear=1905&press=OK`.
253
254 Most search engines work this way.
255
256 To make curl do the GET form post for you, just enter the expected created
257 URL:
258
259    curl "http://www.example.com/when/junk.cgi?birthyear=1905&press=OK"
260
261## POST
262
263 The GET method makes all input field names get displayed in the URL field of
264 your browser. That is generally a good thing when you want to be able to
265 bookmark that page with your given data, but it is an obvious disadvantage if
266 you entered secret information in one of the fields or if there are a large
267 amount of fields creating a long and unreadable URL.
268
269 The HTTP protocol then offers the POST method. This way the client sends the
270 data separated from the URL and thus you will not see any of it in the URL
271 address field.
272
273 The form would look similar to the previous one:
274
275```html
276<form method="POST" action="junk.cgi">
277  <input type=text name="birthyear">
278  <input type=submit name=press value=" OK ">
279</form>
280```
281
282 And to use curl to post this form with the same data filled in as before, we
283 could do it like:
284
285    curl --data "birthyear=1905&press=%20OK%20" http://www.example.com/when/junk.cgi
286
287 This kind of POST will use the Content-Type
288 `application/x-www-form-urlencoded` and is the most widely used POST kind.
289
290 The data you send to the server MUST already be properly encoded, curl will
291 not do that for you. For example, if you want the data to contain a space,
292 you need to replace that space with `%20`, etc. Failing to comply with this will
293 most likely cause your data to be received wrongly and messed up.
294
295 Recent curl versions can in fact url-encode POST data for you, like this:
296
297    curl --data-urlencode "name=I am Daniel" http://www.example.com
298
299 If you repeat `--data` several times on the command line, curl will
300 concatenate all the given data pieces - and put a `&` symbol between each
301 data segment.
302
303## File Upload POST
304
305 Back in late 1995 they defined an additional way to post data over HTTP. It
306 is documented in the RFC 1867, why this method sometimes is referred to as
307 RFC 1867-posting.
308
309 This method is mainly designed to better support file uploads. A form that
310 allows a user to upload a file could be written like this in HTML:
311
312```html
313<form method="POST" enctype='multipart/form-data' action="upload.cgi">
314  <input type=file name=upload>
315  <input type=submit name=press value="OK">
316</form>
317```
318
319 This clearly shows that the Content-Type about to be sent is
320 `multipart/form-data`.
321
322 To post to a form like this with curl, you enter a command line like:
323
324    curl --form upload=@localfilename --form press=OK [URL]
325
326## Hidden Fields
327
328 A common way for HTML based applications to pass state information between
329 pages is to add hidden fields to the forms. Hidden fields are already filled
330 in, they are not displayed to the user and they get passed along just as all
331 the other fields.
332
333 A similar example form with one visible field, one hidden field and one
334 submit button could look like:
335
336```html
337<form method="POST" action="foobar.cgi">
338  <input type=text name="birthyear">
339  <input type=hidden name="person" value="daniel">
340  <input type=submit name="press" value="OK">
341</form>
342```
343
344 To POST this with curl, you will not have to think about if the fields are
345 hidden or not. To curl they are all the same:
346
347    curl --data "birthyear=1905&press=OK&person=daniel" [URL]
348
349## Figure Out What A POST Looks Like
350
351 When you are about to fill in a form and send it to a server by using curl
352 instead of a browser, you are of course interested in sending a POST exactly
353 the way your browser does.
354
355 An easy way to get to see this, is to save the HTML page with the form on
356 your local disk, modify the 'method' to a GET, and press the submit button
357 (you could also change the action URL if you want to).
358
359 You will then clearly see the data get appended to the URL, separated with a
360 `?`-letter as GET forms are supposed to.
361
362# HTTP upload
363
364## PUT
365
366 Perhaps the best way to upload data to an HTTP server is to use PUT. Then
367 again, this of course requires that someone put a program or script on the
368 server end that knows how to receive an HTTP PUT stream.
369
370 Put a file to an HTTP server with curl:
371
372    curl --upload-file uploadfile http://www.example.com/receive.cgi
373
374# HTTP Authentication
375
376## Basic Authentication
377
378 HTTP Authentication is the ability to tell the server your username and
379 password so that it can verify that you are allowed to do the request you are
380 doing. The Basic authentication used in HTTP (which is the type curl uses by
381 default) is **plain text** based, which means it sends username and password
382 only slightly obfuscated, but still fully readable by anyone that sniffs on
383 the network between you and the remote server.
384
385 To tell curl to use a user and password for authentication:
386
387    curl --user name:password http://www.example.com
388
389## Other Authentication
390
391 The site might require a different authentication method (check the headers
392 returned by the server), and then
393 [`--ntlm`](https://curl.se/docs/manpage.html#--ntlm),
394 [`--digest`](https://curl.se/docs/manpage.html#--digest),
395 [`--negotiate`](https://curl.se/docs/manpage.html#--negotiate) or even
396 [`--anyauth`](https://curl.se/docs/manpage.html#--anyauth) might be
397 options that suit you.
398
399## Proxy Authentication
400
401 Sometimes your HTTP access is only available through the use of an HTTP
402 proxy. This seems to be especially common at various companies. An HTTP proxy
403 may require its own user and password to allow the client to get through to
404 the Internet. To specify those with curl, run something like:
405
406    curl --proxy-user proxyuser:proxypassword curl.se
407
408 If your proxy requires the authentication to be done using the NTLM method,
409 use [`--proxy-ntlm`](https://curl.se/docs/manpage.html#--proxy-ntlm), if
410 it requires Digest use
411 [`--proxy-digest`](https://curl.se/docs/manpage.html#--proxy-digest).
412
413 If you use any one of these user+password options but leave out the password
414 part, curl will prompt for the password interactively.
415
416## Hiding credentials
417
418 Do note that when a program is run, its parameters might be possible to see
419 when listing the running processes of the system. Thus, other users may be
420 able to watch your passwords if you pass them as plain command line
421 options. There are ways to circumvent this.
422
423 It is worth noting that while this is how HTTP Authentication works, many
424 websites will not use this concept when they provide logins etc. See the Web
425 Login chapter further below for more details on that.
426
427# More HTTP Headers
428
429## Referer
430
431 An HTTP request may include a 'referer' field (yes it is misspelled), which
432 can be used to tell from which URL the client got to this particular
433 resource. Some programs/scripts check the referer field of requests to verify
434 that this was not arriving from an external site or an unknown page. While
435 this is a stupid way to check something so easily forged, many scripts still
436 do it. Using curl, you can put anything you want in the referer-field and
437 thus more easily be able to fool the server into serving your request.
438
439 Use curl to set the referer field with:
440
441    curl --referer http://www.example.come http://www.example.com
442
443## User Agent
444
445 Similar to the referer field, all HTTP requests may set the User-Agent
446 field. It names what user agent (client) that is being used. Many
447 applications use this information to decide how to display pages. Silly web
448 programmers try to make different pages for users of different browsers to
449 make them look the best possible for their particular browsers. They usually
450 also do different kinds of JavaScript etc.
451
452 At times, you will see that getting a page with curl will not return the same
453 page that you see when getting the page with your browser. Then you know it
454 is time to set the User Agent field to fool the server into thinking you are
455 one of those browsers.
456
457 To make curl look like Internet Explorer 5 on a Windows 2000 box:
458
459    curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
460
461 Or why not look like you are using Netscape 4.73 on an old Linux box:
462
463    curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
464
465## Redirects
466
467## Location header
468
469 When a resource is requested from a server, the reply from the server may
470 include a hint about where the browser should go next to find this page, or a
471 new page keeping newly generated output. The header that tells the browser to
472 redirect is `Location:`.
473
474 Curl does not follow `Location:` headers by default, but will simply display
475 such pages in the same manner it displays all HTTP replies. It does however
476 feature an option that will make it attempt to follow the `Location:`
477 pointers.
478
479 To tell curl to follow a Location:
480
481    curl --location http://www.example.com
482
483 If you use curl to POST to a site that immediately redirects you to another
484 page, you can safely use
485 [`--location`](https://curl.se/docs/manpage.html#-L) (`-L`) and
486 `--data`/`--form` together. Curl will only use POST in the first request, and
487 then revert to GET in the following operations.
488
489## Other redirects
490
491 Browsers typically support at least two other ways of redirects that curl
492 does not: first the html may contain a meta refresh tag that asks the browser
493 to load a specific URL after a set number of seconds, or it may use
494 JavaScript to do it.
495
496# Cookies
497
498## Cookie Basics
499
500 The way the web browsers do "client side state control" is by using
501 cookies. Cookies are just names with associated contents. The cookies are
502 sent to the client by the server. The server tells the client for what path
503 and host name it wants the cookie sent back, and it also sends an expiration
504 date and a few more properties.
505
506 When a client communicates with a server with a name and path as previously
507 specified in a received cookie, the client sends back the cookies and their
508 contents to the server, unless of course they are expired.
509
510 Many applications and servers use this method to connect a series of requests
511 into a single logical session. To be able to use curl in such occasions, we
512 must be able to record and send back cookies the way the web application
513 expects them. The same way browsers deal with them.
514
515## Cookie options
516
517 The simplest way to send a few cookies to the server when getting a page with
518 curl is to add them on the command line like:
519
520    curl --cookie "name=Daniel" http://www.example.com
521
522 Cookies are sent as common HTTP headers. This is practical as it allows curl
523 to record cookies simply by recording headers. Record cookies with curl by
524 using the [`--dump-header`](https://curl.se/docs/manpage.html#-D) (`-D`)
525 option like:
526
527    curl --dump-header headers_and_cookies http://www.example.com
528
529 (Take note that the
530 [`--cookie-jar`](https://curl.se/docs/manpage.html#-c) option described
531 below is a better way to store cookies.)
532
533 Curl has a full blown cookie parsing engine built-in that comes in use if you
534 want to reconnect to a server and use cookies that were stored from a
535 previous connection (or hand-crafted manually to fool the server into
536 believing you had a previous connection). To use previously stored cookies,
537 you run curl like:
538
539    curl --cookie stored_cookies_in_file http://www.example.com
540
541 Curl's "cookie engine" gets enabled when you use the
542 [`--cookie`](https://curl.se/docs/manpage.html#-b) option. If you only
543 want curl to understand received cookies, use `--cookie` with a file that
544 does not exist. Example, if you want to let curl understand cookies from a
545 page and follow a location (and thus possibly send back cookies it received),
546 you can invoke it like:
547
548    curl --cookie nada --location http://www.example.com
549
550 Curl has the ability to read and write cookie files that use the same file
551 format that Netscape and Mozilla once used. It is a convenient way to share
552 cookies between scripts or invokes. The `--cookie` (`-b`) switch
553 automatically detects if a given file is such a cookie file and parses it,
554 and by using the `--cookie-jar` (`-c`) option you will make curl write a new
555 cookie file at the end of an operation:
556
557    curl --cookie cookies.txt --cookie-jar newcookies.txt \
558      http://www.example.com
559
560# HTTPS
561
562## HTTPS is HTTP secure
563
564 There are a few ways to do secure HTTP transfers. By far the most common
565 protocol for doing this is what is generally known as HTTPS, HTTP over
566 SSL. SSL encrypts all the data that is sent and received over the network and
567 thus makes it harder for attackers to spy on sensitive information.
568
569 SSL (or TLS as the current version of the standard is called) offers a set of
570 advanced features to do secure transfers over HTTP.
571
572 Curl supports encrypted fetches when built to use a TLS library and it can be
573 built to use one out of a fairly large set of libraries - `curl -V` will show
574 which one your curl was built to use (if any!). To get a page from an HTTPS
575 server, simply run curl like:
576
577    curl https://secure.example.com
578
579## Certificates
580
581 In the HTTPS world, you use certificates to validate that you are the one
582 you claim to be, as an addition to normal passwords. Curl supports client-
583 side certificates. All certificates are locked with a pass phrase, which you
584 need to enter before the certificate can be used by curl. The pass phrase
585 can be specified on the command line or if not, entered interactively when
586 curl queries for it. Use a certificate with curl on an HTTPS server like:
587
588    curl --cert mycert.pem https://secure.example.com
589
590 curl also tries to verify that the server is who it claims to be, by
591 verifying the server's certificate against a locally stored CA cert
592 bundle. Failing the verification will cause curl to deny the connection. You
593 must then use [`--insecure`](https://curl.se/docs/manpage.html#-k)
594 (`-k`) in case you want to tell curl to ignore that the server cannot be
595 verified.
596
597 More about server certificate verification and ca cert bundles can be read in
598 the [`SSLCERTS` document](https://curl.se/docs/sslcerts.html).
599
600 At times you may end up with your own CA cert store and then you can tell
601 curl to use that to verify the server's certificate:
602
603    curl --cacert ca-bundle.pem https://example.com/
604
605# Custom Request Elements
606
607## Modify method and headers
608
609 Doing fancy stuff, you may need to add or change elements of a single curl
610 request.
611
612 For example, you can change the POST method to `PROPFIND` and send the data
613 as `Content-Type: text/xml` (instead of the default `Content-Type`) like
614 this:
615
616    curl --data "<xml>" --header "Content-Type: text/xml" \
617      --request PROPFIND example.com
618
619 You can delete a default header by providing one without content. Like you
620 can ruin the request by chopping off the `Host:` header:
621
622    curl --header "Host:" http://www.example.com
623
624 You can add headers the same way. Your server may want a `Destination:`
625 header, and you can add it:
626
627    curl --header "Destination: http://nowhere" http://example.com
628
629## More on changed methods
630
631 It should be noted that curl selects which methods to use on its own
632 depending on what action to ask for. `-d` will do POST, `-I` will do HEAD and
633 so on. If you use the
634 [`--request`](https://curl.se/docs/manpage.html#-X) / `-X` option you
635 can change the method keyword curl selects, but you will not modify curl's
636 behavior. This means that if you for example use -d "data" to do a POST, you
637 can modify the method to a `PROPFIND` with `-X` and curl will still think it
638 sends a POST . You can change the normal GET to a POST method by simply
639 adding `-X POST` in a command line like:
640
641    curl -X POST http://example.org/
642
643 ... but curl will still think and act as if it sent a GET so it will not send
644 any request body etc.
645
646# Web Login
647
648## Some login tricks
649
650 While not strictly just HTTP related, it still causes a lot of people
651 problems so here's the executive run-down of how the vast majority of all
652 login forms work and how to login to them using curl.
653
654 It can also be noted that to do this properly in an automated fashion, you
655 will most certainly need to script things and do multiple curl invokes etc.
656
657 First, servers mostly use cookies to track the logged-in status of the
658 client, so you will need to capture the cookies you receive in the
659 responses. Then, many sites also set a special cookie on the login page (to
660 make sure you got there through their login page) so you should make a habit
661 of first getting the login-form page to capture the cookies set there.
662
663 Some web-based login systems feature various amounts of JavaScript, and
664 sometimes they use such code to set or modify cookie contents. Possibly they
665 do that to prevent programmed logins, like this manual describes how to...
666 Anyway, if reading the code is not enough to let you repeat the behavior
667 manually, capturing the HTTP requests done by your browsers and analyzing the
668 sent cookies is usually a working method to work out how to shortcut the
669 JavaScript need.
670
671 In the actual `<form>` tag for the login, lots of sites fill-in
672 random/session or otherwise secretly generated hidden tags and you may need
673 to first capture the HTML code for the login form and extract all the hidden
674 fields to be able to do a proper login POST. Remember that the contents need
675 to be URL encoded when sent in a normal POST.
676
677# Debug
678
679## Some debug tricks
680
681 Many times when you run curl on a site, you will notice that the site does not
682 seem to respond the same way to your curl requests as it does to your
683 browser's.
684
685 Then you need to start making your curl requests more similar to your
686 browser's requests:
687
688 - Use the `--trace-ascii` option to store fully detailed logs of the requests
689   for easier analyzing and better understanding
690
691 - Make sure you check for and use cookies when needed (both reading with
692   `--cookie` and writing with `--cookie-jar`)
693
694 - Set user-agent (with [`-A`](https://curl.se/docs/manpage.html#-A)) to
695   one like a recent popular browser does
696
697 - Set referer (with [`-E`](https://curl.se/docs/manpage.html#-E)) like
698   it is set by the browser
699
700 - If you use POST, make sure you send all the fields and in the same order as
701   the browser does it.
702
703## Check what the browsers do
704
705 A good helper to make sure you do this right, is the web browsers' developers
706 tools that let you view all headers you send and receive (even when using
707 HTTPS).
708
709 A more raw approach is to capture the HTTP traffic on the network with tools
710 such as Wireshark or tcpdump and check what headers that were sent and
711 received by the browser. (HTTPS forces you to use `SSLKEYLOGFILE` to do
712 that.)
713