Star AlbumentationsX on GitHub — it powers this leaderboard

← Back to leaderboard

herrbischoff

robots.txt

herrbischoff/robots.txt

A sane, minimal robots.txt file (for the western world)

4stars

Forks

2

Open issues

0

Watchers

4

Size

0.0 MB

crawlersrobots-txtrobotstxt

Created: Feb 19, 2020

Updated: Aug 12, 2024

Last push: Jul 31, 2020