RobotsDisallowed
danielmiessler/RobotsDisallowed
A curated list of the most common and most interesting robots.txt disallowed directories.
1,490stars
Forks
302
Open issues
5
Watchers
1,490
Size
211.0 MB
Shell
Created: Sep 21, 2015
Updated: Aug 23, 2026
Last push: Aug 22, 2022