AI crawler log, 2026-09-16
9
crawler requests, down from 34 the day before — 0.2× the 7-day average of 36.4
77.8%
took the markdown twin — 7 of 9, and 100% of those found one
Ignored robots.txt
1 request for paths this site’s robots.txt disallows for every crawler.
/api/cron/lock1By crawler
| Crawler | Requests | Markdown | Found | 404 |
|---|---|---|---|---|
| meta | 4 | 4 | 4 | 0 |
| amazon | 2 | 2 | 2 | 0 |
| other | 2 | 0 | 0 | 0 |
| seo-tools | 1 | 1 | 1 | 0 |
Most fetched pathsall 8
/blog/build-in-public-marketing-numbers.md2/blog/bidsurvivor-vs-google-ads.md1/blog/the-first-hundred-brands-on-the-board.md1/blog/what-500-followers-actually-buys-you.md1/v21/api/cron/lock1/b.md1/blog/sponsored-nofollow-dofollow-paid-placement-links.md1How this is counted
Every request that reaches the edge is checked against a list of known crawler user-agent tokens and recorded with the path it asked for. Nothing is sampled and nothing is extrapolated: this is the count, and the raw rows are one click away so that is a claim you can check rather than take.
A crawler is grouped by family, so GPTBot, OAI-SearchBot and ChatGPT-User all read as one row. An agent nobody here recognises is recorded as other rather than guessed at, and a user-agent string is a claim rather than proof — anything can say it is anything.
“Found” and “404” split the markdown requests by whether the .md twin actually existed when it was asked for. A crawler asking for twins that are not there is the clearest signal on this page about whether offering them is worth it. The seven-day average counts only days that were reported, and is not shown until there are at least two of them.
One small site is not the internet and one day is not a trend. What it is, is real, dated and checkable, which is more than most of what gets written about AI crawlers. Every day so far is on the index, and there is an RSS feed.