# Green Pizza Calgary - Robots.txt # https://www.greenpizzayyc.ca # =========================================== # Default rules for all crawlers # =========================================== User-agent: * Allow: / # Disallow system and configuration directories Disallow: /config/ Disallow: /api/ Disallow: /static/ Disallow: /components/ # Disallow query string variations Disallow: /*?*author=* Disallow: /*?*tag=* Disallow: /*?*month=* Disallow: /*?*view=* Disallow: /*?*format=* Disallow: /*?*sort=* Disallow: /*?*filter=* # Disallow legal pages from indexing Disallow: /privacy-policy.html Disallow: /terms-of-use.html # =========================================== # Search engine specific rules # =========================================== # Google User-agent: Googlebot Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html # Noindex for llms.txt (search engines only) User-agent: Googlebot Disallow: /llms.txt User-agent: Bingbot Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html Disallow: /llms.txt User-agent: Slurp Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html Disallow: /llms.txt User-agent: DuckDuckBot Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html Disallow: /llms.txt User-agent: Yandex Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html Disallow: /llms.txt # =========================================== # AI Bot specific rules - Allow llms.txt # =========================================== User-agent: GPTBot Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: ChatGPT-User Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: CCBot Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: anthropic-ai Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: Claude-Web Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: Google-Extended Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: cohere-ai Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: PerplexityBot Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html User-agent: Bytespider Allow: /llms.txt Allow: / Disallow: /privacy-policy.html Disallow: /terms-of-use.html # =========================================== # Google Ads Bot # =========================================== User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / User-agent: AdsBot-Google-Mobile-Apps Allow: / # =========================================== # Social Media Bots # =========================================== User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / # =========================================== # Crawl delay for heavy bots # =========================================== User-agent: Baiduspider Crawl-delay: 10 User-agent: MJ12bot Crawl-delay: 10 User-agent: AhrefsBot Crawl-delay: 10 User-agent: SemrushBot Crawl-delay: 10 # =========================================== # Sitemap location # =========================================== Sitemap: https://www.greenpizzayyc.ca/sitemap.xml