How to handle recaptcha on third-party site in my client application

google recaptcha
recaptcha v2 example
recaptcha v3
recaptcha code
google recaptcha v3 example
recaptcha error
recaptcha v2 vs v3
this site key is not enabled for the invisible captcha.

I was curious about how people build third-party apps for sites with NO public APIs, but I could not really find any tutorials on this topic. So I decided to just give it a try. I created a simple desktop application, which uses HttpClient to send GET requests to the site I frequently use, and then parses the response and displays the data in my WPF window. This approach worked pretty well (probably because the site is fairly simple).

However, today I tried to run my application from a different place, and I kept getting 403 errors in response to my application's requests. It turned out, that the network I was using went through a VPN server, while the site I was trying to access used CloudFlare as protection layer, which apparently forces VPN users to enter reCaptcha in order to access the target site.

var baseAddress = new Uri("http://www.cloudflare.com");
using (var client = new HttpClient() { BaseAddress = baseAddress })
{
   var message = new HttpRequestMessage(HttpMethod.Get, "/");
   //this line returns CloudFlare home page if I use regualr network and reCaptcha page, when I use VPN
   var result = await client.SendAsync(message);
   //this line throws if I use VPN (403 Forbidden)
   result.EnsureSuccessStatusCode();
}

Now the question is: what is the proper way to deal with CloudFlare protection in client application? Do I have to display the reCaptcha in my application just like the web browser does? Do I have to set any particular headers in order to get a proper response instead of 403? Any tips are welcome, as this is a completely new area to me.

P.S. I write in C# because this is the laguage I'm most comfortable with, but I don't mind aswers using any other language as long as they answer the question.

I guess, one way to go about it is to handle captcha in web browser, outside the client application.

  1. Parse the response to see if it is a captcha page.
  2. If it is - open this page in browser.
  3. Let user solve the captcha there.
  4. Fetch the CloudFlare cookies form browser's cookie storage. You gonna need __cfduid (user ID) and cf_clearance (proof of solving the captcha).
  5. Attach those cookies to requests sent by client application.
  6. Use application as normal for the next 24 hours (until CloudFlare cookies expire).

Now the hard part here is (4). It's easy to manually copy-paste the cookies to make the code snippet in my question work with VPN:

var baseAddress = new Uri("http://www.cloudflare.com");
var cookieContainer = new CookieContainer();
using (var client = new HttpClient(new HttpClientHandler() { CookieContainer = cookieContainer } , true) { BaseAddress = baseAddress })
{
    var message = new HttpRequestMessage(HttpMethod.Get, "/");
    //I've also copy-pasted all the headers from browser
    //some of those might be optional
    message.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0");
    message.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    message.Headers.Add("Accept-Encoding", "gzip, deflate" });
    message.Headers.Add("Accept-Language", "en-US;q=0.5,en;q=0.3");
    //adding CloudFlare cookies
    cookieContainer.Add(new Cookie("__cfduid", "copy-pasted-cookie-value", "/", "cloudflare.com"));
    cookieContainer.Add(new Cookie("cf_clearance", "copy-pasted-cookie-value", "/", "cloudflare.com"));
    var result = await client.SendAsync(message);
    result.EnsureSuccessStatusCode();
}

But I think its going to be a tricky task to automate the process of fetching the cookies, due to different browsers storing cookies in different places and/or formats. Not to metion the fact that you need to use external browser for this approach to work, which is really annoying. Still, its something to consider.

Integrate ReCaptcha in Flask Web App without Plugin, Learn how Google's reCAPTCHA service keeps sites clean at While the latter can be shared with third parties, ensure that the Its results are then analysed – if an error is thrown, a failure will be returned to the client-side application. Stop the bots with Google reCAPTCHA: Handle CAPTCHA events. With each version of reCAPTCHA, Google has strengthened its capabilities to detect and filter out spam. Specifically, reCAPTCHA v2 is one of the best among the different third-party anti-spam solutions. You can integrate reCAPTCHA v2 on your website in two different ways. The first is the famous "I’m not a robot" checkbox.

Answer to "build third-party apps for sites with NO public APIs" is that even though some Software Vendors don't have a public api's they have partner programs.

Good example is Netflix, they used to have a public api. Some of the Apps developed when the Public Api was enabled allowed to continue api usage.

In your scenario, your client app acts as a web crawler (downloading html content and trying to parse information). What you are trying to do is to Crawl the Cloudfare data which is not meant to be crawled by a third party app (bot). From the cloudfare side, they have done the correct thing to have a Captcha which prevents automated requests.

Further, if you try to send requests at a high frequency (requests/sec), and if the Cloudfare has Threat detection mechanisms, your ip address will be blocked. I assume that they already identified the VPN server IP address you are trying to use and blacklisted that, that's why you are getting a 403.

Basically you solely depend on security holes in Cloudfare pages you try to access via the client app. This is sort of hacking Cloudfare (doing something cloudfare has restricted) which I would not recommend.

If you have a cool idea, better to contact their developer team and discuss about that.

Stop the bots with Google reCAPTCHA, Welcome to the reCAPTCHA developer documentation. reCAPTCHA protects automated abuse. Here, we explain how to add reCAPTCHA to your site or application. mobile application: Choose the client side integration:. A Refresher on ReCaptcha v1. Back when we originally implemented ReCaptcha, we opted not to use any of the third party libraries that mask the details of how ReCaptcha works. Implementing ReCaptcha typically involves writing two separate pieces of code within the application: the Rendering Logic and the Validation Logic. Rendering Logic

Developer's Guide | reCAPTCHA, Disable third-party cookies in your browser and try passing reCaptcha's This repo is for the client code for developer that need to verify their response to the API. You might want to consider contacting the sites where you're This kind of poorly coded application with google sponsorship can cause real� My client is facing a similar problem. We created a firewall rule that includes the range of IP addresses for reCAPTCHA but Google periodically changes the IP addresses. This means we are relying on the customer to report a problem before we are aware of the IP address change and can update the rule.

reCaptcha seems to loop indefinitely when third party cookies are , Bypass reCAPTCHA, solve image captchas, hCaptcha, FunCaptcha, GeeTest for the cheapest price. From the reCaptcha to detect if the cookies are not enabled, and if so, render a string to tell the user that the reCaptcha is not going to work, and in order to have it working, the user should enable the third party cookies in the browser. Steps to reproduce. Disable the third party cookies in chrome or FF; Clean the cookies; Reload the browser

Anti Captcha: captcha solving service. Bypass reCAPTCHA , On the other hand, if you have used third-party anti-spam solutions for the form data to the server, the reCAPTCHA v3 code on the client makes an In the next section, we'll see how to register your site with Google to The subscribe_newsletter_submit.php file handles the form Mobile App Templates. To handle this case, we can check if any window client is focused and send the message to every client using the postMessage method. This method is a common way to organize a communication channel

Example of How to Add Google reCAPTCHA v3 to a PHP Form, In recent times the types of CAPTCHA that appear on web sites and mobile It should also be noted that answers may need to be handled flexibly, if they require free-form text. It requires the client to execute JavaScript code to solve the However, while this may prevent the third party site from collecting� First, let's see why personal information is collected through reCAPTCHA. How "Invisible Captcha" Works. Invisible Captcha, or reCAPTCHA, requires end-users to click a button that says "I'm not a robot" and Google can determine whether to prompt the user with additional question (i.e. select pictures that best describe X) to verify if that person is in fact not a robot.

Comments
  • I don't think there is anything you should be doing. Whoever is publishing the api needs to correct the matter(support.cloudflare.com/hc/en-us/articles/…). As requests to an API are typically made outside of a browser, we advise creating a page rule for your API's URL pattern to ensure that these features do not limit access to your API.
  • @ErikPhilips, there is no public API available. Site in question probably does not have the private API either. So there is nothing to ask about. :) I still want to fetch the data from this site and process it in my app. And I am able do that, except for when I connect to the site through VPN. I could just ignore that, as normally I dont use VPN and I am doing this whole thing for educational purposes anyway. But I am kinda curious, how people deal with such things. :)
  • You're connecting to a site that has no public or private API... how on earth are you connecting to them...
  • Let me figure this out- You are trying to connect to the CloudFlare API and are receiving the challenge page because your client does not support javascript correct?
  • @ErikPhilips the same way your browser does, when you navigate to this page. :) I send a request via HTTP and parse the response using C# code (see the code snippet in my question).
  • I am not building an automated software in sense, that it sends (or spams) HTTP requests on its own. It only sends requests when user interacts with it, i.e. presses a button. It is not that different, from clicking on hyperlink in browser, it's just that I also filter all the gibberish from response, and only grab the data I am interested in and display it in a format, that is convenient to me. I don't see how this is hacking. I do as much hacking as, say, AdBlock extension does when it filters ads and other unwanted content in a browser. :)
  • Also from what I've read, CloudFlare "bans" entire VPN networks when someone from a network is attacking CloudFlare-protected sites. So it's not that my IP got blacklisted because I did something bad, like spamming requests, but someone else on the same network did. :) I mean, my application works just fine, when I use my home IP. While my browser hits the same protection, my application does, when I use VPN. Still this whole situation with recaptchas got me curious. Oh, And its nice to get a detailed response, thanks.
  • Even though this is not quite the answer I was hoping for, I will still award it. Because I can (and bounty period has expired anyway). :)
  • Once I was on the "Cloudfare" side of the world fighting with bots. If there is a genuine idea, and it it benefits both parties, "CLoudfare" will be more than happy to talk to you, coz they also do business.