diff options
| author | Marc Coquand <marc@mccd.space> | 2024-03-17 15:25:21 -0600 |
|---|---|---|
| committer | Marc Coquand <marc@mccd.space> | 2024-03-17 15:25:21 -0600 |
| commit | 173959393620d1da434611f857c107c1e34f24e3 (patch) | |
| tree | 73c49803563de4518012eadf11a24be492a44980 | |
| parent | 418c2b3f09ce24fa6ff8fe587def3ce004fce2db (diff) | |
| download | mccd.space-173959393620d1da434611f857c107c1e34f24e3.tar.gz mccd.space-173959393620d1da434611f857c107c1e34f24e3.tar.bz2 mccd.space-173959393620d1da434611f857c107c1e34f24e3.zip | |
Update Robots.txt
Taken from: https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers/
TY!
| -rw-r--r-- | robots.txt | 74 |
1 files changed, 65 insertions, 9 deletions
@@ -1,17 +1,73 @@ -# OpenAI’s web crawler: GPT3.5, GPT4, ChatGPT -User-agent: GPTBot -# ChatGPT plugins -User-agent: ChatGPT-User +User-agent: * +Disallow: -# Google's web crawler: Bard, VertexAI, Gemini -User-agent: Google-Extended +User-agent: AdsBot-Google +Disallow: / + +User-agent: Amazonbot +Disallow: / -# Claude User-agent: anthropic-ai +Disallow: / + +User-agent: AwarioRssBot +Disallow: / + +User-agent: AwarioSmartBot +Disallow: / + +User-agent: Bytespider +Disallow: / -# Common Crawl User-agent: CCBot +Disallow: / + +User-agent: ChatGPT-User +Disallow: / + +User-agent: ClaudeBot +Disallow: / + +User-agent: Claude-Web +Disallow: / + +User-agent: cohere-ai +Disallow: / + +User-agent: DataForSeoBot +Disallow: / + +User-agent: FacebookBot +Disallow: / + +User-agent: Google-Extended +Disallow: / + +User-agent: GPTBot +Disallow: / + +User-agent: ImagesiftBot +Disallow: / + +User-agent: magpie-crawler +Disallow: / + +User-agent: omgili +Disallow: / + +User-agent: omgilibot +Disallow: / + +User-agent: peer39_crawler +Disallow: / + +User-agent: peer39_crawler/1.0 +Disallow: / -# Censorship area +User-agent: PerplexityBot Disallow: / + +User-agent: YouBot +Disallow: / + |