Blog

robots.txt for policy pages

Your legal pages should be easy for search engines, app stores, and ad crawlers to reach.

By FreePrivacyPolicy Editorial Team · Privacy compliance editors · Technical SEO · 6 min read

Generate crawlable policies Free · no signup · hosted public URL

Do not block public policy URLs

Privacy policies, terms, EULAs, and child safety standards are public trust pages. Blocking them in robots.txt can create review failures, indexing gaps, and weaker trust signals.

Submit the sitemap

Reference your XML sitemap from robots.txt and make sure it includes the blog, generators, guide pages, and hosted policy URLs that should be discovered.

Keep text files at the root

/ads.txt and /app-ads.txt are not blog posts or policy pages. They are root-level text files that ad platforms crawl directly. Do not redirect them through marketing pages.

Ready to publish?

Answer six questions, get a hosted public URL the App Store, Google Play, and ad networks accept. No credit card.

Generate crawlable policies

Frequently asked questions

Should app-ads.txt be in the sitemap?

No. It should be available at the root, but XML sitemaps are for crawlable web pages, not ad authorization text files.