Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)
Dev Communityvia NewsData.io
Short answer: in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article. Small sample: one day, one machine, plain curl with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out. robots.txt, per host host GPTBot OAI-SearchBot PerplexityBot Google
Headline and summary provided via NewsData.io. The full article and its copyright belong to the original publisher.
