url alone not sufficient to identify unique unit of web content, should be something like canonicalize(url+headers)

Description

The url alone is not sufficient to identify a unique unit of web content. There are lots of cases where the same url returns different things depending on the request headers. We should include at least some of them in the identifier.

Fundamental change with much trickiness involved! How feasible?

Environment

None

Assignee

Unassigned

Reporter

Noah Levitt

Labels

None

Issue Category

None

Group Assignee

None

ZendeskID

None

Estimated Difficulty

None

Actual Difficulty

None

Fix versions

Affects versions

Priority

Major
Configure