Possible deadlock

Description

All progress, on a large scale crawl, stopped. On inspection, I found that the job page would not load, but I could still trigger most (although not all) the reports.Further, the progress-statistics.log had stopped getting new lines when this condition was hit. This suggests that the statistics thread was being blocked by whatever was causing this.

I proceeded to dump the heap and thread state (thread dump attached).

I then tried to end the crawl and found that this must have shaken whatever was causing this loose as end of crawl reports were all generated normally and the 'job page' in the web UI become responsive again.

I also noted that three URLs that had been locked 'in progress' where terminated at that point with a -5 (Unexpected runtime exception). All three threads holding those URLs seemed to be waiting to update the relevant queue in the Frontier, leading me to suspect that this has something to do with BDB.

I have another, nearly identical except for seeds, crawl that has been running for much longer. So clearly this is not something that can be easily reproduced.

Environment

Heritrix 3.1.2-20130625.183735-163
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Red Hat Enterprise Linux Server release 6.4 (Santiago)

Status

Assignee

Unassigned

Reporter

Kristinn SigurĂ°sson

Labels

None

Group Assignee

None

ZendeskID

None

Estimated Difficulty

None

Actual Difficulty

None

Affects versions

Heritrix 3.2.0

Priority

Major
Configure