mirror of
https://github.com/allaunthefox/Research-Stack.git
synced 2026-08-13 11:30:34 +00:00
9 lines
215 B
Text
9 lines
215 B
Text
# source: https://github.com/Smerity/texting_robots
|
|
# Suggestions for the fuzzer targeting robots.txt
|
|
|
|
allow="allow:"
|
|
disallow="disallow:"
|
|
sitemap="sitemap:"
|
|
delay="crawl-delay:"
|
|
ua="user-agent:"
|
|
comment="#comment"
|