Search Bots Filtering

Allow basic search bots filtering through a single switch in the Request Filtering Dashboard section

Search bots generate legitimate internet traffic that allows various search engines to index web pages and build a searchable database. This traffic might be unwanted in some cases, one of them being when the customer does not want to trigger the identification or the Smart Signals processing.

Although Fingerprint already provides configurable request filtering solution based on User-Agent, every customer would need to manage their list of all different search bot User-Agent strings. Instead, we keep and update our own list of search bot filtering rules that can be turned on or off through a simple configuration in the Dashboard.

Search bots that are filtered through this setting then behave in the same way as other request filtering rules and you don't get billed for them. Keep in mind that we won't return any information except requestId in those cases.

📘

Search bots filtering based on User-Agent is a basic mechanism we provide for free to filter out commonly encountered and generally well behaved search bots.

Because of its simplicity, it might introduce both false positive and false negative cases. If you need better protection against sophisticated bots, check out our Smart Signals that contain a paid Bot Detection product.

Configuration

Search Bots Filtering can be turned on or off in our Dashboard in App Settings -> Request Filtering section in the Bots tab.

Turn on the toggle for Exclude Search Bots to start filtering search bots from your billable API calls.

Dashboard - Search Bots Filtering

Dashboard - Search Bots Filtering

List of Supported Search Bots User-Agent Strings

The list of search bot User-Agent strings we support is public and we keep it updated based on the currently supported User-Agent strings. The following table contains all supported User-Agent strings with their respective information sources.

OwnerUser-Agent (Substrings)Public doc
Mail.rumail.rusource
Microsoftbingbot
AdIdxBot
MicrosoftPreview
source
Appleapplebotsource
Huaweipetalbotsource
Navernaver.me/spdsource
Yahooslurpsource
Baidubaiduspidersource
Sogousogou web spider
sogou wap spider
sogou pic spider
source
Exaleadexabotsource
Coccoccoccocbotsource
SeznamseznamBotsource
Googlegooglebot
AdsBot-Google
Storebot-Google
APIs-Google
AdsBot-Google-Mobile
Mediapartners-Google
Googlebot-Image
Googlebot-News
Googlebot-Video
AdsBot-Google-Mobile-Apps
FeedFetcher-Google
Google-Read-Aloud
Google-Site-Verification
source
Yandexyandex.com/botssource
Bytedancebytespidersource
Duckduckgoduckduckbotsource
AhrefsAhrefsBot
AhrefsSiteAudit
source
Facebookfacebookexternalhit
facebookcatalog
source
PinterestPinterestbotsource
CincrawCincrawsource
Semrushhttp://www.semrush.com/bot.html
SeekportSeekportBotsource
AmazonAmazonbot/0.1source
DataDogdatadogsource