Currently, api/negotiation.py handles the detection of scrapers vs. AI agents. We need to expand our detection patterns to ensure modern AI crawlers (like O1, ClaudeBot, etc.) are correctly identified and steered into the 402 flow.
Task:
Refactor the detection logic in api/negotiation.py.
Add a comprehensive list of common AI User-Agent patterns or regex to catch these crawlers reliably.
Currently,
api/negotiation.pyhandles the detection of scrapers vs. AI agents. We need to expand our detection patterns to ensure modern AI crawlers (like O1, ClaudeBot, etc.) are correctly identified and steered into the 402 flow.Task:
Refactor the detection logic in
api/negotiation.py.Add a comprehensive list of common AI User-Agent patterns or regex to catch these crawlers reliably.