# ============================================================================== # Vucense robots.txt — 2026 SEO + GEO (Generative Engine Optimization) Standard # Last updated: 2026-08-01 # # STRATEGY: # - Allow all editorial content (Phase 1 articles, category hubs, author pages) # - Allow AI SEARCH bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, # DuckAssistBot, Google-Extended, and the user-triggered fetchers) for # citation visibility # - Block AI TRAINING bots (GPTBot, ClaudeBot, Anthropic-AI, cohere-ai, # Applebot-Extended, meta-externalagent) to protect original content from # being used in model training without consent # # The two lists are deliberately split per vendor: Anthropic separates # Claude-SearchBot (search) from ClaudeBot (training), and Apple separates # Applebot (search) from Applebot-Extended (training). Naming only the # training token blocks training without giving up the citation surface. # - Block non-editorial paths: /api/, /drafts/, /src/ # - /utilities/ is crawlable; per-page robots meta controls indexing # - Block query-parameter facets: ?search=, ?filter=, ?sort= # - Two sitemaps declared: main + Google News # ============================================================================== # ── GOOGLE & ALL COMPLIANT CRAWLERS ─────────────────────────────────────────── User-agent: * Allow: / # Non-editorial paths (not editorial content) Disallow: /api/ Disallow: /drafts/ Disallow: /search/ Disallow: /src/ # /tags/ and /archive/ are deliberately NOT disallowed. # # Both are `noindex, follow`. A Disallow prevents the crawl, which prevents the # noindex from ever being read — so those URLs could still surface as URL-only # results from the links pointing at them — and it also prevents the `follow`, # which is the whole point of them: tag pages and the archive exist to pass # authority back out to the articles they list. Blocking the crawl turned a # distribution network into a sink. # # Crawl volume is controlled at the link layer instead: articles now link only # to tag pages with 3+ articles (see src/utils/tagHubs.ts), so the 2,322 # single-article tag pages have no inbound links and will not be discovered. # Cloudflare internals. These two lines used to sit at the very bottom of this # file, after the Sitemap: declarations and with no User-agent line of their own, # which binds them to whichever group was declared last (allenai-dolma). They # were therefore not applied to Google, Bing or anything else. Optimised images # are served from /cdn-cgi/image/ and must stay fetchable. Allow: /cdn-cgi/image/ Disallow: /cdn-cgi/ # NOTE: no `Disallow: /_astro/` here. Built assets go to /assets/ (see # build.assets in astro.config.mjs), so /_astro/ does not exist — and blocking # the real asset directory would stop Google fetching the CSS and JS it needs to # render the page. # Query parameter patterns (duplicate content / crawl budget) Disallow: /*?search= Disallow: /*?filter= Disallow: /*?sort= Disallow: /*?page= Disallow: /*?ref= Disallow: /*?utm_ # ── AI SEARCH BOTS (CITATION / SEARCH INDEXING — ALLOWED) ──────────────────── # These bots index content for AI-powered search and citation engines. # Allowing them increases Vucense's visibility in AI-generated answers # (Perplexity, SearchGPT, ChatGPT Browse). They do NOT use content for training. # OpenAI SearchGPT — powers search results in ChatGPT and SearchGPT # https://platform.openai.com/docs/bots User-agent: OAI-SearchBot Allow: / Disallow: /api/ Disallow: /search/ # Perplexity AI search indexer — powers search results in Perplexity.ai # https://docs.perplexity.ai/guides/perplexitybot User-agent: PerplexityBot Allow: / Disallow: /api/ Disallow: /search/ # Perplexity real-time user fetcher — retrieves content when user asks a question # This agent does not follow robots.txt by design but we declare intent here User-agent: Perplexity-User Allow: / # Google-Extended — used for Google AI features (AI Overviews, Gemini) User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /search/ # ChatGPT-User — user-triggered fetch when ChatGPT browses the web User-agent: ChatGPT-User Allow: / # Claude search indexer — powers citations in Claude's web search # Distinct from ClaudeBot below, which is the training crawler and stays blocked. User-agent: Claude-SearchBot Allow: / Disallow: /api/ Disallow: /search/ # Claude user-triggered fetch — retrieves a page because a user asked about it User-agent: Claude-User Allow: / # DuckDuckGo AI answers — citation surface for DuckAssist User-agent: DuckAssistBot Allow: / # Mistral user-triggered fetch User-agent: MistralAI-User Allow: / # ── AI TRAINING BOTS (SCRAPING FOR MODEL TRAINING — BLOCKED) ───────────────── # These bots crawl content specifically to train AI models. Blocking them # protects Vucense's original journalism from unauthorized AI model training. # OpenAI GPTBot — used for GPT model training (NOT search) # https://platform.openai.com/docs/bots User-agent: GPTBot Disallow: / # Anthropic AI training scraper User-agent: Anthropic-AI Disallow: / # Claude scraping bot (Anthropic) User-agent: ClaudeBot Disallow: / # Cohere AI training scraper User-agent: cohere-ai Disallow: / # Diffbot — commercial data extraction for AI training User-agent: Diffbot Disallow: / # Common Crawl — often used for pre-training large language models User-agent: CCBot Disallow: / # AI2 Open Language Model scraper User-agent: allenai-dolma Disallow: / # Apple's training opt-out token. Applebot itself (Apple's search crawler) is # allowed via the wildcard group above; Applebot-Extended controls whether that # crawl may also feed model training, so it is blocked to match the policy # applied to every other training crawler here. User-agent: Applebot-Extended Disallow: / # Meta's training crawler User-agent: meta-externalagent Disallow: / # ── SITEMAP DECLARATIONS ────────────────────────────────────────────────────── # Declare both sitemaps for maximum Google visibility. # Main sitemap: Phase 1 editorial articles + category hubs # News sitemap: Last 48-hour articles for Google News eligibility Sitemap: https://vucense.com/sitemap.xml Sitemap: https://vucense.com/sitemap-news.xml