ftp protocol robots.txt

Description

Consider following robots.txt for ftp?

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
"Google-specific: Google also accepts and follows robots.txt files for FTP sites. FTP-based robots.txt files are accessed via the FTP protocol, using an anonymous login."

Environment

None

Status

Assignee

Unassigned

Reporter

Noah Levitt

Labels

None

Group Assignee

None

ZendeskID

None

Estimated Difficulty

None

Actual Difficulty

None

Affects versions

Heritrix 3.1.1

Priority

Major
Configure