# robots.txt for theflip.app # Last Updated: January 28, 2026 # ============================================================================= # LLMs.txt Files for AI Training # ============================================================================= # Guidelines: https://www.theflip.app/llms.txt # Full content: https://www.theflip.app/llms-full.txt # ============================================================================= # Major Search Engine Crawlers # ============================================================================= User-agent: Googlebot Allow: / Disallow: /*?search= Disallow: /*?page= Disallow: /ar/blog/ Crawl-delay: 1 User-agent: Bingbot Allow: / Disallow: /*?search= Disallow: /*?page= Crawl-delay: 2 User-agent: DuckDuckBot Allow: / Crawl-delay: 1 # ============================================================================= # Social Media Crawlers # ============================================================================= User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / # ============================================================================= # AI Crawlers and LLMs - OpenAI # ============================================================================= User-agent: GPTBot Allow: / Crawl-delay: 5 User-agent: ChatGPT-User Allow: / Crawl-delay: 5 # ============================================================================= # AI Crawlers and LLMs - Anthropic # ============================================================================= User-agent: Claude-Web Allow: / Crawl-delay: 5 User-agent: anthropic-ai Allow: / Crawl-delay: 5 # ============================================================================= # AI Crawlers and LLMs - Google AI # ============================================================================= User-agent: Bard Allow: / Crawl-delay: 5 User-agent: Google-Extended Allow: / Crawl-delay: 5 # ============================================================================= # AI Crawlers and LLMs - Perplexity # ============================================================================= User-agent: PerplexityBot Allow: / Crawl-delay: 5 # ============================================================================= # AI Crawlers and LLMs - Cohere # ============================================================================= User-agent: cohere-ai Allow: / Crawl-delay: 5 # ============================================================================= # AI Crawlers and LLMs - You.com # ============================================================================= User-agent: YouBot Allow: / Crawl-delay: 5 # ============================================================================= # AI Crawlers and LLMs - Common Crawl (used for training) # ============================================================================= User-agent: CCBot Allow: / Crawl-delay: 10 # ============================================================================= # AI Crawlers and LLMs - Apple # ============================================================================= User-agent: Applebot Allow: / Crawl-delay: 3 # ============================================================================= # Blocked Crawlers # ============================================================================= # ByteDance/TikTok crawler - aggressive crawling User-agent: Bytespider Disallow: / # ============================================================================= # SEO Tool Crawlers (Limited Access) # ============================================================================= User-agent: SemrushBot Disallow: /admin/ Disallow: /api/ Allow: / User-agent: AhrefsBot Disallow: /admin/ Disallow: /api/ Allow: / User-agent: MJ12bot Disallow: /admin/ Disallow: /api/ Allow: / # ============================================================================= # Default Rules for All Other Agents # ============================================================================= User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /wp-admin/ Disallow: /wp-json/ Disallow: /wp-content/ Disallow: /*.json$ Disallow: /*?search= Disallow: /*?page= Disallow: /search Disallow: /ar/blog/ # ============================================================================= # Specific Pages to Prioritize # ============================================================================= Allow: /features Allow: /pricing Allow: /about Allow: /contact Allow: /blog Allow: /industries Allow: /solutions Allow: /faq # Arabic pages Allow: /ar/ Allow: /ar/features Allow: /ar/pricing Allow: /ar/about Allow: /ar/contact Disallow: /ar/blog/ Allow: /ar/industries Allow: /ar/solutions # UAE Hub pages Allow: /uae Allow: /uae/resources Allow: /uae/faq Allow: /uae/send-money/india Allow: /uae/send-money/pakistan Allow: /uae/send-money/philippines Allow: /uae/send-money/uk-europe # ============================================================================= # Rate Limiting # ============================================================================= Crawl-delay: 1 # ============================================================================= # Sitemap Location # ============================================================================= Sitemap: https://www.theflip.app/sitemap.xml # ============================================================================= # LLMs.txt Reference # ============================================================================= # For AI training transparency and content guidelines, see: # https://www.theflip.app/llms.txt (guidelines) # https://www.theflip.app/llms-full.txt (comprehensive content)