Issues

heritrix hitting non existent URLs in wix.com/app-market

Heritrix ignores robots.txt

appCtx.getBean() does no longer work in scripting console

heritrix is missing facility to shutdown from console

Improve feedback after specifying errornous command line arguments

RuntimeException in AMQPUrlReceiver kills StarterRestarter?

HTML extractor fails to extract CSS from a link tag

duplicate user agent records in robots.txt cause overwriting of rules

password

Link Analysis with Apache Giraph (Cluster mode)

url alone not sufficient to identify unique unit of web content, should be something like canonicalize(url+headers)

Enable configuration of log4j in libraries

Using 'sun.security.tools.KeyTool' restricts to Oracle-based JVM's.

Add option to prefer the non-DNS resolves

[Optionally?] accelerated transition to terminated state after STOP issued

Identify programs with minimal Closed Captioning

H3: manifest of all files (esp. W/ARCs) from a job, access to W/ARCs, ability to delete/clear

WorkQueueFrontier - add log of queue lifecycle

improved completion time estimates (queue/total)

H3: improve crawler capacity/state reporting for participation in pool of crawling machines

console rates off after checkpoint-resume

H3: offer web operations to delete job/dir/files (cleaning up local machine/crawler state)

crawl-manifest.txt not produced by H3; update and improve manifest functionality

evaluate H3 in context of IPv6