Fingerprint shipped a public, free directory on 18 September 2026 that lets any site owner check whether a bot hitting their server is a legitimate AI crawler or something pretending to be one. It runs on a new authentication standard called Web Bot Auth, and it matters because most robots.txt setups still can't tell ChatGPT's crawler from a scraper wearing its name.
What is Fingerprint's Bot Directory, exactly?#
It's a searchable, continuously updated catalog (Fingerprint calls it the Robodex) of AI tools, bots, and crawlers, organized by provider and purpose. Look up a bot, get a detection snippet, decide what to do with it. Fingerprint already detects 1+ billion unique devices a month across 6,000+ customers, so this isn't starting from zero.
The company's co-founder and CTO, Valentin Vasilyev, put it plainly: "As AI agents act on behalf of people, businesses must be able to accurately distinguish helpful automation from malicious activity." Fingerprint's launch announcement frames this as a response to nearly half of tech leaders now reporting AI-driven fraud, using bot lists built before generative AI existed.
What is Web Bot Auth, and why does it matter more than the directory itself?#
Web Bot Auth is the part that actually proves identity. A bot signs its requests with Ed25519 cryptography, and a verifier checks that signature against a public key the bot publishes at a fixed address. No guessing from a User-Agent string. No trusting an IP range that got rented out six months ago.
That's a real gap. Old bot detection leaned on circumstantial evidence: reverse DNS, known IP blocks, a User-Agent that says "Googlebot" and hopes nobody checks. Any of that breaks the moment someone spoofs it. Web Bot Auth, built on the HTTP Message Signatures standard, RFC 9421, makes the identity claim cryptographically unforgeable instead.
Cloudflare has driven this, shipping Web Bot Auth into its Verified Bots Program back in mid-2025 alongside a default block on unverified AI crawlers: a different move from Cloudflare's own AI training opt-out setting we covered a few days ago, which controls scraping intent, not identity. Fingerprint's directory is the next layer: a free, vendor-neutral place to check any bot, not just the ones one company chose to allowlist.
One honest limit. A verified signature tells you who's asking. It says nothing about whether they should be let in. Identity and permission stay two separate decisions, and Fingerprint's own directory admits it isn't exhaustive.
Does this affect my site if I'm not running a big content operation?#
If your site is five pages and a contact form, not really. Not yet, anyway. Nobody's spoofing a bot to hit your booking page.
If you publish anything worth stealing (original research, pricing pages, case studies, a blog people actually read, the kind of work our answer engine optimization clients build) this is worth ten minutes this week. You want to know which crawlers hitting your server are ChatGPT or Perplexity looking to cite you, and which are scraping your prices for a competitor's dashboard. Those two look identical in a raw access log.
The honest tradeoffs#
Free access is a real plus: no credit card, no sales call, set up through a package manager or CDN. That lowers the bar for a small team to actually check this instead of ignoring it.
The catch is coverage and pace. Web Bot Auth is still an individual IETF draft, not a ratified standard, and its own header shapes have shifted between revisions this year. Not every crawler worth watching has adopted it yet, including legitimate ones, so a bot missing from the directory doesn't automatically mean it's fake. My honest read: this is a good second opinion, not a firewall you install and forget.
How we're handling it#
We run every client site self-hosted, on our own VPS behind Nginx rather than a managed platform, so bot policy is our call, not a vendor default. At WebEpex we pulled our Nginx access logs the same week this shipped and checked the AI crawler user-agents already hitting client sites against Fingerprint's directory, flagging two we didn't recognize. Neither turned out malicious. We'd still rather check than assume.
We build for GCC, European, US, Canadian, and Indian clients, and none of them can afford to guess whether a hit on their pricing page is a citation or a scrape. Our default now: allow known citation-worthy crawlers explicitly in robots.txt, log everything else, and only hard-block at the Nginx level once a pattern looks like abuse rather than curiosity. We're not bolting on a paid bot-management layer for this. Directory plus logs is doing the job, the same instinct behind protecting the citations AI Overviews are already pulling from publishers rather than blocking a crawler that's actually sending us credit.
What I'd tell a client asking about this#
Don't touch your robots.txt today because of this alone. Pull a week of access logs, list any crawler hits you don't recognize, and check the unfamiliar ones against Fingerprint's directory before blocking anything. If you're already working with us on SEO or AEO, this folds into that conversation, not a separate project. If you don't publish anything worth scraping yet, put this on the list for later and go finish the thing actually making you money this month.
If you want a second opinion on whether your own site is quietly leaking data to the wrong crawlers, send me what you're running and I'll tell you straight, two minutes, no pitch. cal.com/webepex/growth-review
