GEO Improvement: Add robots.txt with explicit AI-bot allow rules to the docs site
Source audit: GEO Audit Report — 2026-08-14
Audit date: 2026-08-14
Run: https://github.com/github/gh-aw/actions/runs/31820072668
Finding
"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)"
The docs site (https://github.github.com/gh-aw/) scores 0/robots in the GEO audit, and this same gap propagates across the sitemap-wide audit (average robots: 0.0 across all 20 sampled pages out of 220 total). This is the single largest fixable category gap — every other scored category (schema, meta, content, signals) already has partial credit, but robots is a hard zero site-wide.
Why this matters
Without a robots.txt that explicitly allows AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.), AI engines have no clear, low-friction signal that they are permitted to crawl and cite the documentation. Since this is a Jekyll/GitHub Pages site (github.github.com/gh-aw) with no robots.txt currently, adding one is a single-file, low-risk change that recovers the entire robots scoring category (up to the category's max weight) on every page of the site — a much larger aggregate improvement than any single per-page content fix, since it applies uniformly across all 220 sitemap URLs.
Suggested fix
- Add a
robots.txt file to the docs site source (likely under the Jekyll docs/ root that publishes to github.github.com/gh-aw/).
- Include explicit
Allow: / (or targeted allow rules) for AI-related user-agents such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and anthropic-ai.
- Keep a general
User-agent: * / Allow: / fallback so standard search crawlers are unaffected.
- Optionally reference the sitemap (
Sitemap: https://github.github.com/gh-aw/sitemap.xml) in the same file to reinforce discoverability.
- Re-run the GEO docs-site audit after deployment to confirm the
robots score moves from 0 to full credit.
Generated by 🌍 GEO Optimizer Daily Audit · auto · 21.4 AIC · ⌖ 4.01 AIC · ⊞ 7K · ◷
GEO Improvement: Add
robots.txtwith explicit AI-bot allow rules to the docs siteSource audit: GEO Audit Report — 2026-08-14
Audit date: 2026-08-14
Run: https://github.com/github/gh-aw/actions/runs/31820072668
Finding
The docs site (
https://github.github.com/gh-aw/) scores 0/robots in the GEO audit, and this same gap propagates across the sitemap-wide audit (averagerobots: 0.0across all 20 sampled pages out of 220 total). This is the single largest fixable category gap — every other scored category (schema, meta, content, signals) already has partial credit, butrobotsis a hard zero site-wide.Why this matters
Without a
robots.txtthat explicitly allows AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.), AI engines have no clear, low-friction signal that they are permitted to crawl and cite the documentation. Since this is a Jekyll/GitHub Pages site (github.github.com/gh-aw) with norobots.txtcurrently, adding one is a single-file, low-risk change that recovers the entirerobotsscoring category (up to the category's max weight) on every page of the site — a much larger aggregate improvement than any single per-page content fix, since it applies uniformly across all 220 sitemap URLs.Suggested fix
robots.txtfile to the docs site source (likely under the Jekylldocs/root that publishes togithub.github.com/gh-aw/).Allow: /(or targeted allow rules) for AI-related user-agents such asGPTBot,ClaudeBot,PerplexityBot,Google-Extended,CCBot, andanthropic-ai.User-agent: */Allow: /fallback so standard search crawlers are unaffected.Sitemap: https://github.github.com/gh-aw/sitemap.xml) in the same file to reinforce discoverability.robotsscore moves from 0 to full credit.