Decode responses with the declared charset to fix unicode titles (#2)

[kochj23]

Closes #2

Problem

The switch to lxml regressed non-ascii page titles. Raw response bytes were handed straight to lxml.html.fromstring(), which assumes latin-1 when it isn't told the encoding. A utf-8 page such as GitHub then came back as Title: [GitHub · Social Coding] instead of GitHub · Social Coding (the ·, U+00B7, is 0xC2 0xB7 in utf-8, which decodes to · under latin-1).

Fix

In plugin.py, decode the response body using the charset declared in the Content-Type header before it reaches the HTML parsers. A new charset_from_contenttype() helper extracts the charset (text/html; charset=utf-8 -> utf-8, text/html -> None), and unknown/garbage charsets fall through untouched so nothing regresses.

Testing — please read

I could not execute the test suite locally. This is a Python 2 + Supybot plugin, and this environment has only Python 3 (no python2) and no Supybot runtime, so supybot.test/PluginTestCase cannot be imported or run here. I did not run the suite and am not claiming a passing run.

What I did instead:

• Verified the fix by inspection against the fetch_url() flow and the mojibake mechanics above.
• Fleshed out the previously empty DetrollTestCase stub with a real testCharsetFromContentType method that asserts the charset extraction (including quoted and parameterised forms) and the utf-8 decode round-trip that produces the correct ·. Its logic is verified by inspection; it should run under a real Py2/Supybot setup.

🤖 Generated with Claude Code