Website in .au Heritrix gave SEVERE log message regarding public suffix list

Description

1 2 3 4 5 6 7 8 9 10 2012-03-27 01:18:10.784 SEVERE thread-15 org.archive.crawler.framework.ToeThread.recoverableProblem() Problem java.lang.IllegalStateException: Not under a public suffix: csiro.au occured when trying to process 'http://csiro.au/robots.txt' at step ABOUT_TO_BEGIN_PROCESSOR in java.lang.IllegalStateException: Not under a public suffix: csiro.au at com.google.common.base.Preconditions.checkState(Preconditions.java:176) at com.google.common.net.InternetDomainName.topPrivateDomain(InternetDomainName.java:443) at org.archive.modules.fetcher.FetchWhois.addWhoisLinks(FetchWhois.java:448) at org.archive.modules.fetcher.FetchWhois.innerProcessResult(FetchWhois.java:247) at org.archive.modules.Processor.process(Processor.java:142) at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131) at org.archive.crawler.framework.ToeThread.run(ToeThread.java:147)

Ends up in crawl log with -5

1 2012-03-27T01:18:10.787Z -5 - http://csiro.au/robots.txt P http://csiro.au/ unknown #001 - - - err=java.lang.IllegalStateException

Environment

None

Status

Assignee

Unassigned

Reporter

Noah Levitt

Labels

None

Group Assignee

None

ZendeskID

None

Estimated Difficulty

None

Actual Difficulty

None

Affects versions

Heritrix 3.1.0

Priority

Major
Configure