H3: improve crawler capacity/state reporting for participation in pool of crawling machines

Description

From the expert forum feedback in Singapore: GordonP of NLNZ made a number of suggestions under the theme of helping an individual H3 instance participate in a pool of machines that gets handed jobs from an outside system (as per WebCuratorTool, NetArchiveSuite, or ArchiveIt). In particular:

  • a node should report its current load/resource utilization, especially available disk space but also other system capacity/health measures, or other factors that help answer the question: "can it take a job of certain expected size?"

  • a node should have a marked state where it rejects jobs (even if it otherwise has capacity), for example if it is headed for off/reboot/upgrade and just needs to finish current tasks

Environment

None

Assignee

Unassigned

Reporter

Gordon Mohr

Labels

None

Issue Category

None

Group Assignee

None

ZendeskID

None

Estimated Difficulty

None

Actual Difficulty

None

Fix versions

Affects versions

Priority

Major
Configure