Some odd-ball characters display as '?' in search results.

I looked at a few of the items in the results page linked above.

1. http://www.webarchive.org.uk/wayback/archive/20090605150409/http://bda.org.uk/Fish_Chips-i-358.html

It appears that on this page, the HTML entity » which is the character '»' is shown as '?' in the search results.

2. http://www.webarchive.org.uk/wayback/archive/20080531231756/http://www.petiteanglaise.com/archives/2008/02/15/tomorrows-fish-and-chips/

The search result snippet includes the page's title, which is:

<title>Tomorrow&#8217;s fish and chips&nbsp;by&nbsp;petite anglaise</title>

Again, it looks like the HTML entities are the problem: &#8217; and &nbsp;

I'm a bit surprised that &nbsp; would be a problem. It's so ubiquitous in HTML pages that if it was troublesome, we would have noticed it a long time ago, and it would be popping-up in all NutchWAX deployments.

Brad pointed out that we should double-check the manner in which the results are displayed on the BL results pages, specifically which encoding they are using.

I looked at their results page and it uses "windows-1252", in the HTML, you can see

<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">

This is confused a bit further by the HTTP content-type header in the response that serves the page specifies "ISO-8859-1".

In either case, whether using "windows-1252" or "iso-8859-1", there are going to be characters that aren't representable within those encodings.

We strongly suggest that UTF-8 is used throughout, which avoids these encoding mis-match problems.