AI crawler log, 2026-09-15
34
crawler requests, up from 3 the day before
100%
took the markdown twin — 34 of 34 · found-or-404 not recorded for this day
By crawler
| Crawler | Requests | Markdown |
|---|---|---|
| meta | 19 | 19 |
| openai | 11 | 11 |
| amazon | 2 | 2 |
| other | 1 | 1 |
| 1 | 1 |
Most fetched paths
/blog/bidsurvivor-vs-carbon-ads.md2/b.md2/blog/how-many-clicks-before-a-conversion-rate-means-anything.md1/blog/the-two-channel-rule.md1/blog/hourly-ad-slots.md1/blog/contextual-advertising-for-small-sites.md1/blog/cheap-alternative-to-google-ads-for-startups.md1/blog/best-free-marketing-tools-for-bootstrapped-startups.md1/blog/best-advertising-platforms-for-startups.md1/blog/attention-marketplace-startups.md1How this is counted
Every request that reaches the edge is checked against a list of known crawler user-agent tokens and recorded with the path it asked for. Nothing is sampled and nothing is extrapolated: this is the count, and the raw rows are one click away so that is a claim you can check rather than take.
A crawler is grouped by family, so GPTBot, OAI-SearchBot and ChatGPT-User all read as one row. An agent nobody here recognises is recorded as other rather than guessed at, and a user-agent string is a claim rather than proof — anything can say it is anything.
“Found” and “404” split the markdown requests by whether the .md twin actually existed when it was asked for. A crawler asking for twins that are not there is the clearest signal on this page about whether offering them is worth it. The seven-day average counts only days that were reported, and is not shown until there are at least two of them.
One small site is not the internet and one day is not a trend. What it is, is real, dated and checkable, which is more than most of what gets written about AI crawlers. Every day so far is on the index, and there is an RSS feed.