RobotsDisallowed
danielmiessler/RobotsDisallowed
A curated list of the most common and most interesting robots.txt disallowed directories.
1,492stars
Forks
303
Open issues
5
Watchers
1,492
Size
211.0 MB
Shell
Created: Sep 21, 2015
Updated: Jul 11, 2026
Last push: Aug 22, 2022