veselosky / hypallage Goto Github PK

EDIT: Decreased the scope to include only identifying URLs as objects. Performing URL resolution and proper comparisons can be done on a separate issue as needed.

The microdata spec draws a distinction between plain string values and URL values.

Make the extractor aware of URLs as a separate type.

Create a first iteration of a basic microdata extractor.

Create some basic testing of the extractor to prove it does something useful. Full use case testing will require some additional test data, and so will be captured as a separate issue.

Microdata spec requires specific behavior when encoded microdata results in a circular reference. Currently the Extractor does not follow this instruction, but merely terminates the circular reference and continues.

Make the Extractor behave according to spec.

DateTime properties should be extracted as appropriate Python objects, rather than opaque strings.

BeautifulSoup supports several different parsers. The code seems to work with lxml and with html5lib, but when it falls back to the standard library's HTMLParser, it fails to find any itemscopes. That parser sets itemscope=None rather than itemscope="" as the others do.

If possible, figure out how to make it work with the fallback parser. Until then, it will only work with one of the 3rd party parsers. :(

Currently the Extractor takes a URL or filesystem path, which it loads and parses. This could be a pain if you have already parsed the document in question for some other reason.

Extractor's constructor should accept a BeautifulSoup object as well as a URL. The Extractor instance should expose a property allowing access to the bs4 object.

Microdata spec requires that properties be sorted in document order. A depth-first traversal normally suffices for this, but when an item includes itemrefs, it is possible for the nodes to be out of order.

Implement sorting of nodes in _extract_properties

The microdata spec details how a JSON data structure should be constructed. The Extractor should know how to serialize its extracted data as JSON.