How does Bot detection work?

Content of this page

• # Ethical Bots? What does it mean?
• # The Bot database

Ethical Bots? What does it mean?

When Google is indexing your website it means time to time a Google Bot visits your pages in order to index them. When Google Bot comes he does not hide the fact he is an automated mechanism - it correctly identifies himself by sending a User-Agent, something like Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). So he is a "good bot".

There are many other Bots who might visit your site for different purposes. Not just search index, but also when you copy-paste your URL into a LinkedIn post, LinkedIn also visits to generate a preview picture, etc etc. They are also identifying themselves with a User-Agent string which is publicly announced.

The Bot database

We have integrated Keytiles with WhatIsMyBrowser.com who maintains a huge database of User-Agent strings. So when your pages are visited by an ethical bot, Keytiles can detect that from the User-Agent string, and in this case automatically classifies it a bot visit.

Note: your real Bot traffic can be higher in reality! Keytiles is only tracking visits from JavaScript capable devices (like Web Browsers people using)