Heritrix ignores robots.txt

Description

None

Environment

None

Status

Assignee

Unassigned

Reporter

Robert Jäschke

Labels

Group Assignee

None

ZendeskID

None

Estimated Difficulty

None

Actual Difficulty

None

Affects versions

Heritrix 3.2.0

Priority

Minor