Detecting 'stealth' web-crawlers

detecting synonym
detecting meaning
detect antonym
detecting antonyms
detect sentence
synonym for the word detecting
metal detecting
how to pronounce detect

What options are there to detect web-crawlers that do not want to be detected?

(I know that listing detection techniques will allow the smart stealth-crawler programmer to make a better spider, but I do not think that we will ever be able to block smart stealth-crawlers anyway, only the ones that make mistakes.)

I'm not talking about the nice crawlers such as googlebot and Yahoo! Slurp. I consider a bot nice if it:

  1. identifies itself as a bot in the user agent string
  2. reads robots.txt (and obeys it)

I'm talking about the bad crawlers, hiding behind common user agents, using my bandwidth and never giving me anything in return.

There are some trapdoors that can be constructed updated list (thanks Chris, gs):

  1. Adding a directory only listed (marked as disallow) in the robots.txt,
  2. Adding invisible links (possibly marked as rel="nofollow"?),
    • style="display: none;" on link or parent container
    • placed underneath another element with higher z-index
  3. detect who doesn't understand CaPiTaLiSaTioN,
  4. detect who tries to post replies but always fail the Captcha.
  5. detect GET requests to POST-only resources
  6. detect interval between requests
  7. detect order of pages requested
  8. detect who (consistently) requests https resources over http
  9. detect who does not request image file (this in combination with a list of user-agents of known image capable browsers works surprisingly nice)

Some traps would be triggered by both 'good' and 'bad' bots. you could combine those with a whitelist:

  1. It trigger a trap
  2. It request robots.txt?
  3. It doest not trigger another trap because it obeyed robots.txt

One other important thing here is: Please consider blind people using a screen readers: give people a way to contact you, or solve a (non-image) Captcha to continue browsing.

What methods are there to automatically detect the web crawlers trying to mask themselves as normal human visitors.

Update The question is not: How do I catch every crawler. The question is: How can I maximize the chance of detecting a crawler.

Some spiders are really good, and actually parse and understand html, xhtml, css javascript, VB script etc... I have no illusions: I won't be able to beat them.

You would however be surprised how stupid some crawlers are. With the best example of stupidity (in my opinion) being: cast all URLs to lower case before requesting them.

And then there is a whole bunch of crawlers that are just 'not good enough' to avoid the various trapdoors.

A while back, I worked with a smallish hosting company to help them implement a solution to this. The system I developed examined web server logs for excessive activity from any given IP address and issued firewall rules to block offenders. It included whitelists of IP addresses/ranges based on http://www.iplists.com/, which were then updated automatically as needed by checking claimed user-agent strings and, if the client claimed to be a legitimate spider but not on the whitelist, it performed DNS/reverse-DNS lookups to verify that the source IP address corresponds to the claimed owner of the bot. As a failsafe, these actions were reported to the admin by email, along with links to black/whitelist the address in case of an incorrect assessment.

I haven't talked to that client in 6 months or so, but, last I heard, the system was performing quite effectively.

Side point: If you're thinking about doing a similar detection system based on hit-rate-limiting, be sure to use at least one-minute (and preferably at least five-minute) totals. I see a lot of people talking about these kinds of schemes who want to block anyone who tops 5-10 hits in a second, which may generate false positives on image-heavy pages (unless images are excluded from the tally) and will generate false positives when someone like me finds an interesting site that he wants to read all of, so he opens up all the links in tabs to load in the background while he reads the first one.

Detecting Synonyms, Detecting Antonyms, 24 synonyms of detecting from the Merriam-Webster Thesaurus, plus 6 related words, definitions, and antonyms. Find another word for detecting. Detect definition is - to discover the true character of. How to use detect in a sentence.

See Project Honeypot - they're setting up bot traps on large scale (and have DNSRBL with their IPs).

Use tricky URLs and HTML:

<a href="//example.com/"> = http://example.com/ on http pages.
<a href="page&amp;&#x23;hash"> = page& + #hash

In HTML you can use plenty of tricks with comments, CDATA elements, entities, etc:

<a href="foo<!--bar-->"> (comment should not be removed)
<script>var haha = '<a href="bot">'</script>
<script>// <!-- </script> <!--><a href="bot"> <!-->

DETECTING, detecting meaning: 1. present participle of detect 2. to notice something that is partly hidden or not clear, or to…. Learn more. Detect definition, to discover or catch (a person) in the performance of some act: to detect someone cheating. See more.

An easy solution is to create a link and make it invisible

<a href="iamabot.script" style="display:none;">Don't click me!</a>

Of course you should expect that some people who look at the source code follow that link just to see where it leads. But you could present those users with a captcha...

Valid crawlers would, of course, also follow the link. But you should not implement a rel=nofollow, but look for the sign of a valid crawler. (like the user agent)

Detect, Define detecting. detecting synonyms, detecting pronunciation, detecting translation, English dictionary definition of detecting. tr.v. de�tect�ed , de�tect�ing� de·tec·tion (dĭ-tĕk′shən) n. 1. The act or process of detecting; discovery: detection of a crime; detection of radiation from a distant galaxy. 2. See demodulation

One thing you didn't list, that are used commonly to detect bad crawlers.

Hit speed, good web crawlers will break their hits up so they don't deluge a site with requests. Bad ones will do one of three things:

  1. hit sequential links one after the other
  2. hit sequential links in some paralell sequence (2 or more at a time.)
  3. hit sequential links at a fixed interval

Also, some offline browsing programs will slurp up a number of pages, I'm not sure what kind of threshold you'd want to use, to start blocking by IP address.

This method will also catch mirroring programs like fmirror or wget.

If the bot randomizes the time interval, you could check to see if the links are traversed in a sequential or depth-first manner, or you can see if the bot is traversing a huge amount of text (as in words to read) in a too-short period of time. Some sites limit the number of requests per hour, also.

Actually, I heard an idea somewhere, I don't remember where, that if a user gets too much data, in terms of kilobytes, they can be presented with a captcha asking them to prove they aren't a bot. I've never seen that implemented though.

Update on Hiding Links

As far as hiding links goes, you can put a div under another, with CSS (placing it first in the draw order) and possibly setting the z-order. A bot could not ignore that, without parsing all your javascript to see if it is a menu. To some extent, links inside invisible DIV elements also can't be ignored without the bot parsing all the javascript.

Taking that idea to completion, uncalled javascript which could potentially show the hidden elements would possilby fool a subset of javascript parsing bots. And, it is not a lot of work to implement.

Detecting, detect meaning, definition, what is detect: to notice or discover something, especia Word family (noun) detection detective (adjective) detectable ≠ indetectible� 24 synonyms of detecting from the Merriam-Webster Thesaurus, plus 6 related words, definitions, and antonyms. Find another word for detecting. Detecting: to come upon after searching, study, or effort.

One simple bot detection method I've heard of for forms is the hidden input technique. If you are trying to secure a form put a input in the form with an id that looks completely legit. Then use css in an external file to hide it. Or if you are really paranoid, setup something like jquery to hide the input box on page load. If you do this right I imagine it would be very hard for a bot to figure out. You know those bots have it in there nature to fill out everything on a page especially if you give your hidden input an id of something like id="fname", etc.

detect, Synonyms for detect at Thesaurus.com with free online thesaurus, antonyms, and definitions. Find descriptive alternatives for detect. detect definition: 1. to notice something that is partly hidden or not clear, or to discover something, especially…. Learn more.

Detect Synonyms, Detect Antonyms, In the history of radio communications, the term "detector" was first used for a device that detected the simple presence or absence of a radio signal, since all� Site that offers information and links to the Bounty Hunter range of machines.

Detection, Uses include detecting land mines, the detection of weapons such as knives and guns (especially in airport security), geophysical prospecting, archaeology and� Synonyms for detect at Thesaurus.com with free online thesaurus, antonyms, and definitions. Find descriptive alternatives for detect.

Metal detector, Devices that use millimeter-wave radar can detect both metal and nonmetal objects and displays them on a screen. Note for Metal Detectorists and Collectors: Discuss plans for detecting finds before your detect. On private hunts, leave all finds and copies of your notes at the property, including GPS coordinates, depth, photographs, etc. in case it can assist future Archaeological work.

Comments
  • I find that false positive due to blocking web crawler absolutely kills web traffic. You are basically pissing off the 99.8% of your user in a poor attempt to hinder crawlers that can easily bypass this naive method described. Never a good idea to deny user access or hinder it because it destroys the user experience with your site.
  • Unless the bot checks the CSS attributes of the link and doesn't follow the link because it's not visible to a human user...
  • Labelling the link "DO NOT click me" would be a better idea.. If someone has CSS disabled (or no CSS support), the link will be visible..
  • Good idea. Perhaps change the text to "." and the css style to match the background - making it invisible to most users? Or, run a script to hide it after 1 second leaving it only visible to a bot who can't link the javascript hide command to the link?
  • Beware of black hat penalty from SEO perspective.
  • Major flaw with "ignoring JavaScript means you're a bot" methods: Some of us use the NoScript plugin. No site runs JavaScript on me unless I whitelist the site and I'm pretty sure I'm not a bot.
  • bots can execute Javascript now...it's 2013 for christ sakes. so there goes the whole argument. who says web crawlers visits sites in sequential selections? another huge assumption.
  • The javascript was only for the showing of a honeypot link. The idea is that the bots will parse the javascript that will make a honeypot link visible, making them more likely to follow the link. However for a real user, the code that makes the link visible would never be executed. Thus NoScript users, along with anyone that doesn't go randomly executing functions would be fine. That said, I'm not sure why/how a bot would randomly be executing code, and if it was doing a static analysis to determine if an element might become visible, that would be one fancy bot.
  • not if the bots are able to wait for the jquery to finish, just like a regular browser can. This would've worked well in the early 00s
  • What exactly and why would it get you black listed?
  • I'm sorry, but if you run a crawler that does not obey robots.txt, you are not obeying the rules. By not obeying the rules, you yourself are putting yourself in some really bad company. By suggesting that enforcing the rules as set by the owner of the website (in robots.txt) is bad practice, you are wrongfully flipping the issue upside down. You basically state that you do not understand who the rightfull owner of content is.
  • @Jacco: If a crawler looking for malware obeyed the rules, it would never find any. Go talk to the malware authors.
  • @Jacco: Percentage of legit sites that try to block non-compliant crawlers? Under 1%. Malware sites that try? Over 60%. So yeah, it is suspicious.
  • @Jacco actually no, there are no owners of a content on the web if it's public. Someone who is doing this without copying and pasting manually should be given an award not punishment. This whole copyright concept needs to be abolished on the internet. Only creative innovation and trust can build value and worthy of people's attention, not by some threat of opaque legal veil.
  • All this outlier indicates is that the person running the site put a lot of time and thought into the site, and they have some degree of technical skill. So of course that rules out most of the bell curve.