Tech Tool

ROBOTS.TXT

Crawler allow/disallow file generator for SEO.

Disallowed Paths (One per line)

Block GPTBot

Prevents OpenAI from scraping your data for ChatGPT

Block Common Crawl

Stops the massive dataset used by many AI models

Block Google-Extended

Stops Gemini/Bard without affecting Google Search

XML Sitemap URL

robots.txt

User-agent: *
Disallow: /admin
Disallow: /login
Disallow: /private

Place this file at the root of your domain: yourwebsite.com/robots.txt