AI Bot Crawler Log Analyzer
Paste your server access log and see exactly which AI bots crawl you — GPTBot, ClaudeBot, PerplexityBot, CCBot and 20+ more. Runs 100% in your browser; nothing is uploaded.
Your logs never leave your browser — parsing runs 100% client-side.
No upload, no server call, no storage. Files are read in-browser with FileReader and discarded when you close the tab.
Paste or upload your access log
Common or combined log format (Apache / Nginx). 20+ AI crawlers detected.
Paste your server access log above, or load the sample to see how it works.
We'll show which AI bots crawled you, how often, and exactly which pages they read.
How to use this analyzer
- 1
Grab your access log
Export a slice of your Apache or Nginx access log (e.g. /var/log/nginx/access.log). A few days to a few weeks of traffic gives the clearest picture of AI crawl activity.
- 2
Paste or upload it
Paste the lines into the box, or click "Upload .log / .txt" to read a file in-browser. Nothing is uploaded — parsing runs entirely on your machine. No log handy? Hit "Load sample log".
- 3
Read the breakdown
See total AI-bot hits, the % of traffic that is AI, every distinct crawler with its vendor and category, a crawl-over-time chart, and the top 10 pages AI bots read.
- 4
Act on the gaps
Missing GPTBot or ClaudeBot? Your content is not being discovered for AI citations. Export the CSV, then fix crawlability with robots.txt, llms.txt, and an AI-bot policy on your CDN.
Frequently asked questions
Which AI bots does the analyzer detect?
It matches 20+ known AI crawler user-agents: GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI); ClaudeBot, anthropic-ai, and Claude-Web (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Google-Extended and GoogleOther (Google); Applebot-Extended (Apple); meta-externalagent and FacebookBot (Meta); Bytespider (ByteDance); CCBot (Common Crawl); Amazonbot (Amazon); cohere-ai (Cohere); YouBot (You.com); Diffbot; ImagesiftBot; and Timpibot. Each bot is tagged as training, search, or both, so you can tell apart crawlers that feed model training from ones that power live AI-search citations.
Do my logs get uploaded anywhere?
No. Parsing runs 100% client-side in your browser. When you paste text or upload a .log/.txt file, the file is read in-browser with the FileReader API and analyzed in JavaScript on your own machine — there is no server call, no storage, and nothing leaves your computer. Close the tab and the data is gone. That makes it safe to run on production access logs that may contain visitor IP addresses.
What log format does it expect?
The common and combined log formats produced by Apache and Nginx by default — lines like `203.0.113.7 - - [08/Oct/2026:03:11:42 +0000] "GET /page HTTP/1.1" 200 5312 "-" "user-agent"`. The analyzer extracts the user-agent from the last quoted field, the request path from the request quote, and the date from the bracketed timestamp. Junk or non-matching lines are counted but skipped, so a messy log will not break the report.
Why should I care which AI bots crawl my site?
AI assistants can only cite pages they have crawled. If GPTBot, ClaudeBot, and PerplexityBot are not in your logs, you are invisible inside ChatGPT, Claude, and Perplexity answers — even if you rank #1 in Google. Seeing the crawl pattern tells you whether AI engines have discovered your content, which pages they prioritise, and whether an aggressive crawler like Bytespider is hammering your origin. It is the ground-truth complement to robots.txt and llms.txt: those declare your policy; the logs show what bots actually did.
The crawl-over-time chart is missing — why?
The daily bar chart only renders when the analyzer can parse dates from your log lines. It reads the bracketed timestamp in `[10/Oct/2026:03:11:42 +0000]` format and maps the three-letter month to a number. If your log uses a non-standard date format or strips timestamps, the panel hides itself gracefully and the rest of the report (per-bot table, top paths, summary) still works.
Can I export the results?
Yes. The per-bot table has an Export CSV button that builds the file in-browser via a Blob and downloads it directly — bot name, vendor, category, hit count, and share of AI traffic, one row per crawler. Like everything else in the tool, the CSV is generated on your machine and never touches a server.
Want AI-bot rules configured on your CDN?
Reading the logs is diagnosis. We do the fix: AI-bot allow / block rules at the edge (Cloudflare, Fastly, Vercel), rate limits on aggressive crawlers like Bytespider, plus the robots.txt, llms.txt, and schema plumbing that gets you cited in ChatGPT and Perplexity. The same playbook we run on our in-house product TalkDrill.
Configure my CDNNot showing up in AI answers?
If AI crawlers are missing from your logs, our SEO team makes your content discoverable and citable — technical SEO, llms.txt curation, schema markup, and the structured-data work that wins both Google and AI-search engines.
Talk to our SEO teamTurn AI-crawler data into AI-search visibility
We audit your logs, configure AI-bot rules at the edge, and ship the robots.txt + llms.txt + schema markup that gets your pages cited in ChatGPT, Claude, and Perplexity answers.