AI bot request filtering

Allow basic AI bots filtering through a single switch in the Request Filtering Dashboard section

AI bots use scrapers to gather content for model training as well as offering tools that process a webpage for additional information. The bots can generate a significant amount of traffic and cause a surge in billable API calls. To prevent this scenario, Fingerprint offers AI Bots Filtering toggle in the Dashboard.

AI bots that are filtered through this setting then behave in the same way as other request filtering rules and you don't get billed for them. Keep in mind that we won't return any information except requestId in those cases.

📘

AI bots filtering based on User-Agent is a basic mechanism we provide for free to filter out commonly encountered and generally well behaved search bots.

Because of its simplicity, it might introduce both false positive and false negative cases. If you need better protection against sophisticated bots, check out our Smart Signals that contain a paid Bot Detection product.

Configuration

AI Bots Filtering can be turned on or off in our Dashboard.

  1. Navigate to Dashboard > Security and switch to the Bots tab.
  2. Enable Exclude AI Bots to start filtering AI bots from your billable API calls.
Dashboard - Search Bots Filtering

Dashboard - AI Bots Filtering

List of supported search bots User-Agent strings

The list of AI botsUser-Agent strings we support is public and we keep it updated based on the currently supported User-Agent strings. The following table contains all supported User-Agent strings with their respective information sources.

Owner

User-Agent (Substrings)

OpenAI

ChatGPT-User
GPTBot

Duckduckgo

DuckAssistbot

Google

GoogleOther

Huawei

PetalBot

LinerBot

LINER Bot

Factset

Factset_spyderbot

Amazon

Amazonbot

Apple

Applebot

OAISearchBot

OAI-SearchBot

Qualified

QualifiedBot

ICC

ICC Crawler

Anthropic

ClaudeBot