Can a website detect when you are using selenium with chromedriver?

how to make selenium undetectable
selenium bot
chrome headless selenium
selenium hide automation
execute_cdp_cmd
python selenium hide chrome browser
southwest selenium
chromedriver source code

I've been testing out Selenium with Chromedriver and I noticed that some pages can detect that you're using Selenium even though there's no automation at all. Even when I'm just browsing manually just using chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected. I've checked my user agent, and my browser fingerprint, and they are all exactly identical to the normal chrome browser.

When I browse to these sites in normal chrome everything works fine, but the moment I use Selenium I'm detected.

In theory chromedriver and chrome should look literally exactly the same to any webserver, but somehow they can detect it.

If you want some testcode try out this:

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=1, size=(1600, 902))
display.start()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('--profile-directory=Default')
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-plugins-discovery");
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.delete_all_cookies()
driver.set_window_size(800,800)
driver.set_window_position(0,0)
print 'arguments done'
driver.get('http://stubhub.com')

If you browse around stubhub you'll get redirected and 'blocked' within one or two requests. I've been investigating this and I can't figure out how they can tell that a user is using Selenium.

How do they do it?

EDIT UPDATE:

I installed the Selenium IDE plugin in Firefox and I got banned when I went to stubhub.com in the normal firefox browser with only the additional plugin.

EDIT:

When I use Fiddler to view the HTTP requests being sent back and forth I've noticed that the 'fake browser\'s' requests often have 'no-cache' in the response header.

EDIT:

results like this Is there a way to detect that I'm in a Selenium Webdriver page from Javascript suggest that there should be no way to detect when you are using a webdriver. But this evidence suggests otherwise.

EDIT:

The site uploads a fingerprint to their servers, but I checked and the fingerprint of selenium is identical to the fingerprint when using chrome.

EDIT:

This is one of the fingerprint payloads that they send to their servers

{"appName":"Netscape","platform":"Linuxx86_64","cookies":1,"syslang":"en-US","userlang":"en-US","cpu":"","productSub":"20030107","setTimeout":1,"setInterval":1,"plugins":{"0":"ChromePDFViewer","1":"ShockwaveFlash","2":"WidevineContentDecryptionModule","3":"NativeClient","4":"ChromePDFViewer"},"mimeTypes":{"0":"application/pdf","1":"ShockwaveFlashapplication/x-shockwave-flash","2":"FutureSplashPlayerapplication/futuresplash","3":"WidevineContentDecryptionModuleapplication/x-ppapi-widevine-cdm","4":"NativeClientExecutableapplication/x-nacl","5":"PortableNativeClientExecutableapplication/x-pnacl","6":"PortableDocumentFormatapplication/x-google-chrome-pdf"},"screen":{"width":1600,"height":900,"colorDepth":24},"fonts":{"0":"monospace","1":"DejaVuSerif","2":"Georgia","3":"DejaVuSans","4":"TrebuchetMS","5":"Verdana","6":"AndaleMono","7":"DejaVuSansMono","8":"LiberationMono","9":"NimbusMonoL","10":"CourierNew","11":"Courier"}}

Its identical in selenium and in chrome

EDIT:

VPNs work for a single use but get detected after I load the first page. Clearly some javascript is being run to detect Selenium.

For Mac Users

Replacing cdc_ variable using Vim or Perl

You can use vim, or as @Vic Seedoubleyew has pointed out in the answer by @Erti-Chris Eelmaa, perl, to replace the cdc_ variable in chromedriver(See post by @Erti-Chris Eelmaa to learn more about that variable). Using vim or perl prevents you from having to recompile source code or use a hex-editor. Make sure to make a copy of the original chromedriver before attempting to edit it. Also, the methods below were tested on chromedriver version 2.41.578706.


Using Vim
vim /path/to/chromedriver

After running the line above, you'll probably see a bunch of gibberish. Do the following:

  1. Search for cdc_ by typing /cdc_ and pressing return.
  2. Enable editing by pressing a.
  3. Delete any amount of $cdc_lasutopfhvcZLmcfl and replace what was deleted with an equal amount characters. If you don't, chromedriver will fail.
  4. After you're done editing, press esc.
  5. To save the changes and quit, type :wq! and press return.
  6. If you don't want to save the changes, but you want to quit, type :q! and press return.
  7. You're done.

Go to the altered chromedriver and double click on it. A terminal window should open up. If you don't see killed in the output, you successfully altered the driver.


Using Perl

The line below replaces cdc_ with dog_:

perl -pi -e 's/cdc_/dog_/g' /path/to/chromedriver

Make sure that the replacement string has the same number of characters as the search string, otherwise the chromedriver will fail.

Perl Explanation

s///g denotes that you want to search for a string and replace it globally with another string (replaces all occurrences).

e.g., s/string/replacment/g

So,

s/// denotes searching for and replacing a string.

cdc_ is the search string.

dog_ is the replacement string.

g is the global key, which replaces every occurrence of the string.

How to check if the Perl replacement worked

The following line will print every occurrence of the search string cdc_:

perl -ne 'while(/cdc_/g){print "$&\n";}' /path/to/chromedriver

If this returns nothing, then cdc_ has been replaced.

Conversely, you can use the this:

perl -ne 'while(/dog_/g){print "$&\n";}' /path/to/chromedriver

to see if your replacement string, dog_, is now in the chromedriver binary. If it is, the replacement string will be printed to the console.

Go to the altered chromedriver and double click on it. A terminal window should open up. If you don't see killed in the output, you successfully altered the driver.


Wrapping Up

After altering the chromedriver binary, make sure that the name of the altered chromedriver binary is chromedriver, and that the original binary is either moved from its original location or renamed.


My Experience With This Method

I was previously being detected on a website while trying to log in, but after replacing cdc_ with an equal sized string, I was able to log in. Like others have said though, if you've already been detected, you might get blocked for a plethora of other reasons even after using this method. So you may have to try accessing the site that was detecting you using a VPN, different network, or what have you.

Can a website detect when you are using selenium , I am using Selenium with Chrome driver and I noticed that some web pages can detect that I am fine, but whenever I use Selenium I'm  I've been testing out Selenium with Chromedriver and I noticed that some pages can detect that you're using Selenium even though there's no automation at all. Even when I'm just browsing manually just using chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected.

Basically the way the selenium detection works, is that they test for pre-defined javascript variables which appear when running with selenium. The bot detection scripts usually look anything containing word "selenium" / "webdriver" in any of the variables (on window object), and also document variables called $cdc_ and $wdc_. Of course, all of this depends on which browser you are on. All the different browsers expose different things.

For me, I used chrome, so, all that I had to do was to ensure that $cdc_ didn't exist anymore as document variable, and voila (download chromedriver source code, modify chromedriver and re-compile $cdc_ under different name.)

this is the function I modified in chromedriver:

call_function.js:

function getPageCache(opt_doc) {
  var doc = opt_doc || document;
  //var key = '$cdc_asdjflasutopfhvcZLmcfl_';
  var key = 'randomblabla_';
  if (!(key in doc))
    doc[key] = new Cache();
  return doc[key];
}

(note the comment, all I did I turned $cdc_ to randomblabla_.

Here is a pseudo-code which demonstrates some of the techniques that bot networks might use:

runBotDetection = function () {
    var documentDetectionKeys = [
        "__webdriver_evaluate",
        "__selenium_evaluate",
        "__webdriver_script_function",
        "__webdriver_script_func",
        "__webdriver_script_fn",
        "__fxdriver_evaluate",
        "__driver_unwrapped",
        "__webdriver_unwrapped",
        "__driver_evaluate",
        "__selenium_unwrapped",
        "__fxdriver_unwrapped",
    ];

    var windowDetectionKeys = [
        "_phantom",
        "__nightmare",
        "_selenium",
        "callPhantom",
        "callSelenium",
        "_Selenium_IDE_Recorder",
    ];

    for (const windowDetectionKey in windowDetectionKeys) {
        const windowDetectionKeyValue = windowDetectionKeys[windowDetectionKey];
        if (window[windowDetectionKeyValue]) {
            return true;
        }
    };
    for (const documentDetectionKey in documentDetectionKeys) {
        const documentDetectionKeyValue = documentDetectionKeys[documentDetectionKey];
        if (window['document'][documentDetectionKeyValue]) {
            return true;
        }
    };

    for (const documentKey in window['document']) {
        if (documentKey.match(/\$[a-z]dc_/) && window['document'][documentKey]['cache_']) {
            return true;
        }
    }

    if (window['external'] && window['external'].toString() && (window['external'].toString()['indexOf']('Sequentum') != -1)) return true;

    if (window['document']['documentElement']['getAttribute']('selenium')) return true;
    if (window['document']['documentElement']['getAttribute']('webdriver')) return true;
    if (window['document']['documentElement']['getAttribute']('driver')) return true;

    return false;
};

according to user @szx, it is also possible to simply open chromedriver.exe in hex editor, and just do the replacement manually, without actually doing any compiling.

Is it possible for a website to detect that we are using Selenium with , In theory chromedriver and chrome should look literally exactly the same to any webserver, but somehow they can detect it. If you want some testcode try out this: I’ve been testing out Selenium with Chromedriver and I noticed that some pages can detect that you’re using Selenium even though there’s no automation at all. Even when I’m just browsing manually just using chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected.

Can a website detect when you are using selenium with , According to the WebDriver spec I found word of this interface (which may be the /can-a-website-detect-when-you-are-using-selenium-with-chromedriver. I am using Selenium with Chrome driver and I noticed that some web pages can detect that I am using Selenium. I always get a page like "suspicious activity was detected". I have checked it with my user agent and it is exactly identical to chrome. When I normally use the chrome browse everything is fine, but whenever I use Selenium I'm detected.

Example of how it's implemented on wellsfargo.com:

try {
 if (window.document.documentElement.getAttribute("webdriver")) return !+[]
} catch (IDLMrxxel) {}
try {
 if ("_Selenium_IDE_Recorder" in window) return !+""
} catch (KknKsUayS) {}
try {
 if ("__webdriver_script_fn" in document) return !+""

How would a website know you're using selenium? : selenium, Does anyone know how to bypass selenium detection on websites? But when I use normal browser all is ok I tried using different browsers(ie,chrome,Firefox)  A ChromeDriver is a standalone server or a separate executable that is used by Selenium WebDriver to control Chrome. It is impossible to run Selenium test scripts on the Google Chrome browser without ChromeDriver. One can easily initialize the object of ChromeDriver using the following command: WebDriver driver = new ChromeDriver.

Selenium detection bypass : learnpython, I've been testing out Selenium with Chromedriver and I noticed that some pages can detect that you're using Selenium even though there's no automation at all. Yes selenium is detectable.check Can a website detect when you are using selenium with chromedriver? If some one is using Firefox driver for automation then it is easy to detect if you put this code at your client side

Can a website detect when you are using selenium with Chrome , Websites can detect the automation using JavaScript experimental technology navigator.webdriver in navigator interface. If the website is loaded with automation  As far as I know there is no cross-browser method that Selenium provides to detect that it is driving the browser. In FF, webdriver sets the webdriver attribute on the html element but apparently not in other browsers.

Making Chrome Headless Undetectable, A short article titled Detecting Chrome Headless popped up on Hacker News over the weekend and It's like DRM; the best you can hope for is to universally give your users a worse if you're using Python, Selenium, and ChromeDriver. If you're doing serious web scraping, then using proxies is a must. Download Selenium ChromeDriver. We would recommend you to download the latest version of ChromeDriver, mainly because it supports the latest versions of Chrome, and secondly it contains all the bug fixes. Let us see the steps that you can follow to download Selenium ChromeDriver – 1.

Detecting Selenium – Edmund Martin, When using Chrome, the Selenium driver injects a webdriver property into I have never seen this technique used in the wild, but I can confirm that it Opera which are much less commonly used by those scraping the web. More generally, websites can detect Selenium using a variety of methods (one of which is discussed here. On Mon, 6 Apr 2020 at 16:49, Chrissy Freiboth < ccf@gmail.com > wrote: I'm using using selenium with chromedriver in c#.

Comments
  • @RyanWeinstein: It is not traffic. My guess is that Selenium needs to expose some JavaScript hooks which can be detected on the client-side JavaScript.
  • Or if it is traffic then it is a traffic pattern.... you are browsing pages too fast.
  • I'm not browsing too fast. I only load a single page and I navigate through it normally using my mouse and keyboard. Also it doesn't make sense that Selenium needs to expose hooks, because its literally running chrome.exe. It just runs normal chrome and allows you to get data from it. Any other ideas? I was thinking maybe it has something to do with cookies. This is driving me crazy.
  • This site uses distill bot detection technology and delivers content using akamaitechnologies.com CDN from diffrent ips e.g. 95.100.59.245 , 104.70.243.66 , 23.202.161.241
  • I am experiencing the same issue with Selenium and the firefox driver. The interesting thing to note is I am running Selenium in a VMWare Workstation Virtual Machine that is accessing the internet through a NAT. The host machine is able to access stubhub, while the VM is unable to access when using Selenium, or even the browser instance Selenium launched. I had the VM Browser instance Blocked and stubhub still recognizes the machine and has it blocked. So it must be performing a fingerprint of the browser and machine in some manner.
  • @LekaBaper Thanks for the heads up. The chromedriver version that I used was version 2.41.578706.
  • Did not worked even when I used this chromedriver.exe modification on new physical computer on different network.
  • it's give an error says, this version cannot work in this computer :(
  • @colossatr0n Is there any undetectable open-source fork which you are aware of ?
  • Note that the chromedriver people have declared this issue won't-fix, so you can expect to have to use a fork or edit the binary for the indefinite future. bugs.chromium.org/p/chromedriver/issues/detail?id=3220
  • yes it worked without probs, note one problem is if you fell into the "blacklist" BEFORE this change, it's quite hard to get out. if you want to get out of the existing black list, you need to implement fake canvas fingerprinting, disable flash, change IP, and change request header order (swap language and Accept headers). Once you fell into the blacklist, they have very good measures to track you, even if you change IP, even if you open chrome in incognito, etc
  • I found the file "/Users/your_username/chromium/src/chrome/test/chromedriver/js"
  • I simply replaced $cdc with xxxx in chromedriver.exe in a hex editor and it worked! I also noticed that if you maximize the browser window (rather than use a predefined size) it's detected less often.