RobotsDisallowed
danielmiessler/RobotsDisallowed
A curated list of the most common and most interesting robots.txt disallowed directories.
1,488stars
Forks
304
Open issues
5
Watchers
1,488
Size
211.0 MB
Shell
Created: Sep 21, 2015
Updated: May 21, 2026
Last push: Aug 22, 2022