How can I send cookies using PHP curl in addition to CURLOPT_COOKIEFILE?

php curl get cookie
php curl cookie
curlopt_cookiejar
php curlopt_cookie
php curl cookie jar
php curl login get cookie
php read curl cookie file
extract cookie from curl response php

I am scraping some content from a website after a form submission. The problem is that the script is failing every now and then, say 2 times out of 5 the script fails. I am using php curl, COOKIEFILE and COOKIEJAR to handle the cookie. However when I observed the sent headers of my browser (when visiting the target website from my browser and using live http headers) and the headers sent by php and saw there are many differences.

My browser sent a lot more cookie variables than php curl. I think this difference might be because javascript is resposible for setting most of the cookies, however I'm not sure about this.

I am using the below code to do the scraping and I am showing the sent headers of my browser and of php curl:

$ckfile = tempnam ("/tmp", 'cookiename');

$url = 'https://www.domain.com/firststep';
$poststring = 'variable1=4&variable2=5';
$ch = curl_init ($url);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $poststring);
$output = curl_exec ($ch);
curl_close($ch);



$url = 'https://www.domain.com/nextstep';
$poststring = 'variableB1=4&variableB2=5';
$ch = curl_init ($url);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $poststring);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
$output = curl_exec ($ch);
$headers = curl_getinfo($ch, CURLINFO_HEADER_OUT);
curl_close($ch);

print_r($headers);

// Gives:
POST /d-cobs-web/doffers.html;jsessionid=7BC2A5277A4EB07D9A7237A707BE1366 HTTP/1.1
User-Agent: Mozilla
Host: domain.subdomain.nl
Accept: */*
Cookie: JSESSIONID=7BC2A5277A4EB07D9A7237A707BE1366; www-20480=MIFBNLFDFAAA
Content-Length: 187
Content-Type: application/x-www-form-urlencoded

// Where live http headers gives:
POST /d-cobs-web/doffers.html;jsessionid=7BC2A5277A4EB07D9A7237A707BE1366 HTTP/1.1
Host: domain.subdomain.nl
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: nl,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: https://domain.subdomain.nl/dd/doffers.html?returnUrl=https%3A%2F%2Fttcc.subdomain.nl%2Fdd%2Fpreferences.html%3FValueChanged%3Dfalse&BEGBA=&departureDate=13-06-2013&extChangeTime=&pax2=0&bp=&pax1=1&pax4=0&bk=&pax3=0&shopId=&xtpage=&partner=NSINT&bc=&xt_pc=&ov=&departureTime=&comfortClass=2&destination=DEBHF&thalysTicketless=&beneUser=&debugDOffer=&logonId=&valueChanged=&iDomesticOrigin=&rp=&returnTime=&locale=nl_NL&vu=&thePassWeekend=false&returnDate=&xtsite=&pax=A&lc2=&lc1=&lc4=&lc3=&lc6=&lc5=&BECRA=&passType2=&custId=&lc9=&iDomesticDestination=&passType1=A&lc7=&lc8=&origin=NLASC&toporef=&pid=&passType4=&returnTimeType=1&passType3=&departureTimeType=1&socusId=&idr3=&xtn2=&loyaltyCard=&idr2=&idr1=&thePassBusiness=false&cid=14812
Content-Length: 219
Cookie: subdomainPARTNER=NSINT; JSESSIONID=CB3FEB3AC72AD61A80BFED91D3FD96CA; www-20480=MHFBNLFDFAAA; campaignPos=5; www-47873=MGFBNLFDFAAA; __utma=1.993399624.1370027094.1370040145.1370082133.5; __utmc=1; __utmz=1.1370027094.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BCSessionID=5dc05787-c2c8-43e1-9abe-93989970b087; BCPermissionLevel=PERSONAL; __utmb=1.1.10.1370082133
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
AJAXREQUEST=_viewRoot&doffersForm=doffersForm&doffersForm%3AvalueChanged=&doffersForm%3ArequestValid=true&javax.faces.ViewState=j_id3&doffersForm%3Aj_id937=doffersForm%3Aj_id937&valueChanged=false&AJAX%3AEVENTS_COUNT=1&

I would like to use:

$headers   = array();
$headers[] = 'Cookie: ' . $cookie;

and:

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

where:

$cookie = 'subdomainPARTNER=NSINT; JSESSIONID=CB3FEB3AC72AD61A80BFED91D3FD96CA; www-20480=MHFBNLFDFAAA; campaignPos=5; www-47873=MGFBNLFDFAAA; __utma=1.993399624.1370027094.1370040145.1370082133.5; __utmc=1; __utmz=1.1370027094.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BCSessionID=5dc05787-c2c8-43e1-9abe-93989970b087; BCPermissionLevel=PERSONAL; __utmb=1.1.10.1370082133';

Some of the parameters in the cookie above I might be able to scrape from the content of the website, but not all. Some of them I might be able to read from the $ckfile, but I don't know how to do that. Especially the utma utmc, utmz, utmcsr, utmccn, utmcmd I am not able to get from anywhere, I think these are generated by the javascript.

Question 1: Am I doing something wrong with the cookie handling in the current code as very few cookie variables are sent by php curl and a lot more by the browser? Further: can other differences between sent headers by browser and php curl be a problem to return the right content?

Question 2: Are the missing cookie variables due to the javascript setting those cookies?

Question 3: What is the best way to handle the cookies to make sure that all required cookies are being sent to the remote server?

Your help is very welcome!

If the cookie is generated from script, then you can send the cookie manually along with the cookie from the file(using cookie-file option). For example:

# sending manually set cookie
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Cookie: test=cookie"));

# sending cookies from file
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);

In this case curl will send your defined cookie along with the cookies from the file.

If the cookie is generated through javascrript, then you have to trace it out how its generated and then you can send it using the above method(through http-header).

The utma utmc, utmz are seen when cookies are sent from Mozilla. You shouldn't bet worry about these things anymore.

Finally, the way you are doing is alright. Just make sure you are using absolute path for the file names(i.e. /var/dir/cookie.txt) instead of relative one.

Always enable the verbose mode when working with curl. It will help you a lot on tracing the requests. Also it will save lot of your times.

curl_setopt($ch, CURLOPT_VERBOSE, true);

How can I send cookies using PHP curl in addition , I'm not clear on what you are expecting to have happen with cookies. However, you might try this: How can I send cookies using PHP curl in addition to CURLOPT_COOKIEFILE? (2) I think the only cookie you need is JSESSIONID=xxx.. Also NEVER share your cookies, becasuse someone may access your personal data that way. Specially when the cookies are session.

Try below code,

$cookieFile = "cookies.txt";
if(!file_exists($cookieFile)) {
    $fh = fopen($cookieFile, "w");
    fwrite($fh, "");
    fclose($fh);
}


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $apiCall);
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonDataEncoded);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile); // Cookie aware
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile); // Cookie aware
curl_setopt($ch, CURLOPT_VERBOSE, true);
if(!curl_exec($ch)){
    die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}
else{
    $response = curl_exec($ch); 
}
curl_close($ch);
$result = json_decode($response, true);

echo '<pre>';
var_dump($result);
echo'</pre>';

I hope this will help you.

Best regards, Dasitha.

setting a cookie in PHP CURL, So, I tried adding Cookie management. curl_setopt($ch, CURLOPT_COOKIEFILE, COOKIE_FILE); taken from the “LIB_http.php” file available from here: from client? How can I create cookie file on the server to send in cURL requests? If a cookie is set with NO expiration date, the browser should use that cookie until the browser session is closed, or the user logs out and the cookie gets unset. That said, CURLOPT_COOKIESESSION is a way to get cURL to simulate having closed the browser.

Here is a list of examples for sending cookies - https://github.com/andriichuk/php-curl-cookbook#cookies

$curlHandler = curl_init();

curl_setopt_array($curlHandler, [
CURLOPT_URL => 'https://httpbin.org/cookies',
CURLOPT_RETURNTRANSFER => true,

CURLOPT_COOKIEFILE  => $cookieFile,
CURLOPT_COOKIE => 'foo=bar;baz=foo',

/**
 * Or set header
 * CURLOPT_HTTPHEADER => [
       'Cookie: foo=bar;baz=foo',
   ]
 */
]);

$response = curl_exec($curlHandler);
curl_close($curlHandler);

echo $response;

Sending cookie in cURL request - PHP, PHP scripts have the ability to request content from other servers using To allow cURL to store and send cookies when contacting this server, Once it's accessible, adding a couple of new lines to the topmost example will yield cookie curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile); // Cookie  Guzzle version(s) affected: 6.3.3 Description I want to save the cookie file with Netscape HTTP Cookie File format not json format. So I use the CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR option to save cookie.

I think the only cookie you need is JSESSIONID=xxx..

Also NEVER share your cookies, becasuse someone may access your personal data that way. Specially when the cookies are session. These cookies will stop working once you logout the site.

PHP cURL with Sessions – HotSource, This is done using the CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE options as shown in this post. CURLOPT_COOKIEJAR. CURLOPT_COOKIEJAR​  I have some problem with PHP Curl and Cookies Authentication. I have a file Connector.php which authenticates users on another server and returns cookie with current user.

PHP CURL and Cookies, A Guide to Developing Internet Agents with PHP/CURL Michael Schrenk CURLOPT_COOKIEJAR options to tell PHP/CURL where the cookies should be written Tell PHP/CURL which cookies to send curl_setopt($ch, CURLOPT_POST,  CURLcode curl_easy_setopt(CURL *handle, CURLOPT_COOKIEFILE, char *filename); DESCRIPTION. Pass a pointer to a zero terminated string as parameter. It should point to the file name of your file holding cookie data to read. The cookie data can be in either the old Netscape / Mozilla cookie data format or just regular HTTP headers (Set-Cookie style) dumped to a file.

Webbots, Spiders, and Screen Scrapers, 2nd Edition: A Guide to , To manage cookies with curl, we will use CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE options in addition to the other options. CURL stands for client URL, it is a tool for getting and sending data using various protocols, like - GET,  A request to the server will send both the 'foo' held by the cookie engine and the 'foo' held by this option. To set a cookie that is instead held by the cookie engine and can be modified by the server use CURLOPT_COOKIELIST. Using this option multiple times will only make the latest string override the previous ones.

PHP CURL Cookie Jar, 5, PHP 7). curl_setopt — Set an option for a cURL transfer "RELOAD" loads all cookies from the files specified by CURLOPT_COOKIEFILE . Available I used several cURL options, but the only two (other than URL) that mattered were: I did this On Debian 7.9 with cURL 7.26.0-1+wheezy13 and PHP 5.3.3-7+squeeze19 we can succesfully login into Siemens C470/N300 DECT base stations. After upgrading to Debian 8 and cURL 7.38.0 this no longer works.

Comments
  • Have you gotten anywhere with this? I am experiencing the same issue - wondering why the browser sends 3 cookies and curl only sends 1, although the cookie file contains all 3.
  • Hmmm...sounds like a problem I'm having scraping backpage.com.
  • Will the saved cookie file include the test=cookie? If not, then how would I get cURL to save the cookies from the custom header request?
  • @brant cookie file only write based on Set-cookie response header from server. If the server doesn't return that cookie with that header that case it will be ignored.
  • The order of curl_setopt calls is very important. set CURLOPT_POST after CURLOPT_POSTFIELDS and you're lost in a hard to find bug... Similarly, the order in the solution above is important to keep. Swap it and you override everything in the cookie file with just "test=cookie"
  • For me, it does not work. You have to pass a pair of key&value to work curl_setopt($ch, CURLOPT_HTTPHEADER, array("Cookie" => "test=cookie"));