agentic-demand-logs

analyze.py reads HTTP access logs and answers two questions: which machine-readable paths AI agents actually request (/llms.txt, /.well-known/*), and whether the agents making those requests are who their user-agent string claims to be — checked against confirmed reverse DNS and against the address ranges the operators publish themselves.

Read the write-up

The measurement this tool produced, and what it does and does not support, is written up here:

Measuring the agentic web without trusting the user agent

That page is the canonical version of the analysis. This repository is the tool and the data note (DATANOTE.md), not a second copy of the article.

Install

There is nothing to install. Python 3.9+ and the standard library, no third-party dependencies, no virtualenv, no lockfile.

git clone https://github.com/NessFlow/agentic-demand-logs
cd agentic-demand-logs
python3 analyze.py --help

Subcommands

selftest — check the harness before believing a number

Runs the verifier against known-good and known-bad witnesses (a real Googlebot address, a documentation address from RFC 5737 that must come back as spoofed, an operator that publishes no verification means at all and must therefore come back unverifiable rather than accused). Run this first.

python3 analyze.py selftest

fetch-ranges — pull the operator-published address ranges

Downloads each operator's own published range file into ranges/, and records what it got in ranges/_fetched.json. It fails loudly rather than quietly accepting a file that does not parse.

python3 analyze.py fetch-ranges

cdn-check — does the log carry the client's address or the CDN's?

Before any verification means anything, you have to know whose address is in the log. This compares the addresses in the log against a frontal CDN's published prefixes. A high share means the log records the edge, not the client, and every identity verdict downstream is worthless.

python3 analyze.py cdn-check \
    --corpus A --input /var/log/nginx/access.log \
    --ranges-file cloudflare-v4.txt cloudflare-v6.txt \
    --out data/cdn-check.json

analyze — measure one corpus

Reads one or more log files (.gz accepted), classifies agents, verifies identities, and writes a single JSON. No IP address is written to that JSON, in any form: addresses live in memory for the length of the reverse lookup and only the verdict survives.

python3 analyze.py analyze \
    --corpus A --description "SaaS product site at launch, low volume" \
    --pays TN --secteur "B2B SaaS" --stack "Laravel, nginx, PHP-FPM" \
    --cdn no --pages-indexables "135 sitemap URLs" \
    --input /var/log/nginx/access.log /var/log/nginx/access.log.*.gz \
    --out data/A.json

report — assemble the data note

Merges the per-corpus JSONs into the comparison tables and fills a template. --public omits the landing-page table (see Anonymised corpora below).

python3 analyze.py report --public \
    --input data/A.json data/B.json data/C.json \
    --cdn-check data/cdn-check.json \
    --template datanote-template.md --out DATANOTE.md

check-note — refuse a note that publishes an address or a name

A release gate, not a linter. It refuses the document if any address-shaped token sits outside a code fragment, or if any forbidden token appears anywhere. The check is on placement, not on shape: a user-agent string legitimately contains Chrome/145.0.0.0, and those strings are a deliverable.

python3 analyze.py check-note --input DATANOTE.md --forbid <site-name-1> <site-name-2>

What the measurement found

Figures below are taken verbatim from DATANOTE.md; read it for the denominators and the observation windows, which differ per corpus and are not comparable raw.

• Three corpora, three non-overlapping windows: A 3 763 requests over 1.7687 d, B 36 476 875 over 7.4238 d, C 2 838 859 over 14.3676 d.
• /llms.txt was requested on all three — 3 / 45 / 14 requests for A / B / C — and served 200 on A only; /.well-known/mcp and /.well-known/agent-skills drew 0 requests on all three, a measured zero rather than an absence of measurement.
• /robots.txt still dominates by orders of magnitude: 61 / 19 704 / 5 289 requests, against those 3 / 45 / 14 for /llms.txt.
• Identity verification splits the agents rather than clearing them: on B, 1.5 % (17 872/1 178 321) of Googlebot requests and 4.0 % (26 969/666 819) of Bingbot requests came back presumed-spoofed, while 100 % (133 195/133 195) of Meta-ExternalAgent requests are merely unverifiable — Meta publishes no verification means, so no accusation is pronounceable.
• AI referral traffic is negligible next to crawling: ChatGPT accounts for 60 browser visits on B (8.08/d) and 38 on C (2.64/d).

What this does not measure

The claims this data cannot support are enumerated in DATANOTE.md §9 “Ce que ces données ne disent pas” and the known holes in §10 “Trous”. Read both before quoting any figure. The short version: the three windows do not overlap, so no cross-corpus difference separates the shape of the site from the period; two of the three corpora sit behind a frontal cache with logs taken at origin, so their volumes are floors and never totals; and three sites are not a sample of anything.

Anonymised corpora

The three corpora are anonymised by their shape — sector, volume, stack, order of magnitude of the catalogue — never by name. report --public therefore omits the section 4 landing-page table: a real landing path is an article slug or a product reference, it can be searched verbatim, and one search names the site. The landing_pages key in data/*.json is emptied for the same reason, with landing_pages_withheld next to it recording how many paths were removed. Everything that carries the referral measurement is published unchanged: visits per source, the per-day trend, and the non-browser count.

About ranges/

The files in ranges/ are not data produced by this repository. They are the address ranges the operators publish themselves — claude.com/crawling/bots.json, openai.com/gptbot.json, www.bing.com/toolbox/bingbot.json, search.developer.apple.com/applebot.json, Google's and DuckDuckGo's equivalents, and so on. They are fetched by analyze.py fetch-ranges, and ranges/_fetched.json records for each one the source URL, the HTTP status, the fetch timestamp, the prefix count, the creation time the operator declared, and a SHA-256 of what was received.

The snapshot committed here holds 13 range files, 2 975 prefixes, fetched 2026-08-23. It goes stale. Operators add and retire ranges without notice, and a stale file turns a legitimate crawler into a false spoofing accusation. Re-run fetch-ranges before trusting a verdict, and read _fetched.json to see how old what you have actually is.

Status

A measurement tool published alongside an article, not a maintained product. No CI, no release process, no support. It is here so the numbers in the write-up can be checked and so the method can be re-run against someone else's logs.

License

MIT — see LICENSE.