Optimized for search engines, social media crawlers, and AI language models
# robots.txt for divinetalesforkids.com # Optimized for search engines and AI crawlers # Default rules for all bots User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /*.json$ Disallow: /BlogEditor # Crawl-delay for respectful crawling Crawl-delay: 1 # High-priority content for all crawlers Allow: /Store Allow: /Blog Allow: /Quizzes Allow: /PersonalityQuiz Allow: /About Allow: /Product Allow: /BlogPost # Google-specific rules User-agent: Googlebot Allow: / Crawl-delay: 0 User-agent: Googlebot-Image Allow: / # Bing specific rules User-agent: Bingbot Allow: / Crawl-delay: 1 # Social media crawlers (for rich previews) User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / User-agent: WhatsApp Allow: / # AI Crawlers and LLMs User-agent: GPTBot Allow: / Crawl-delay: 2 User-agent: ChatGPT-User Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / User-agent: CCBot Allow: / User-agent: PerplexityBot Allow: / User-agent: YouBot Allow: / # Sitemaps Sitemap: https://divinetalesforkids.com/Sitemap Sitemap: https://divinetalesforkids.com/llms # Host (preferred domain) Host: https://divinetalesforkids.com