html.parser python_issue 41748: HTMLParser: parsing error - Python tracker

The html.parser follows the HTML 5 specs as closely as possible. There are a few corner cases where it behaves slightly differently but it's only while dealing with invalid markup, and the differences should be trivial and generally not worth the extra complexity to deal with them.

In this case, if I recall correctly, the way the comma is handled is just a left-over from the previous version of the parser, that predates the HTML 5 specs. In tags like there was an ambiguous situation and parsing it was deemed a reasonable interpretation, so the comma was treated as an attribute separator (and there should be test cases for this).

This likely caused the issue reported by the OP, and I think it should be fixed, even if technically it's a change in behavior and will break some of the tests.

If I'm reading the specs[0] correctly:

* should be parsed as , and

* should be parsed as , where ',baz' is the attribute name

> Also, there is no warning about security in the html.parser documentation?

I'm not aware of any specific security issues, since html.parser just implements the parser described by the HTML 5 specs. If there are any security issues caused by divergences from the specs, they should be fixed. I'm not sure why a warning would be needed.

> Is this module mature and maintained enough to be considered as reliable?

Even though it hasn't been updated to the latest version of the specs (5.2 at the time of writing), it has been updated to implement the parsing rules described by the HTML 5 specs. I don't know if the parsing rules changed between 5.0 and 5.2.

> Or should we warn users about possible issues on corner cases, and point to BeautilfulSoup for a more mature HTML parser?

BeautifulSoup is built on top of html.parser (and can also use other parses, like lxml). BS uses the underlying parsers to parse the HTML, then builds the tree and provides, among other things, functions to search and edit it.

When I upgraded html.parser to HTML 5 I worked with the author of BeautifulSoup (Leonard Richardson), to make sure that my changes were compatible with BS. We also discussed about some corner cases he found and other feature requests and issues he had with the old version of the parser. That said, a link to BS would be a nice addition, since it's a great library.

[0] starting from https://www.w3.org/TR/html52/syntax.html#tokenizer-before-attribute-name-state