html-similarity's Introduction

HTML Similarity

This package provides a set of functions to measure the similarity between web pages.

Install

The quick way:

pip install html-similarity

How it works?

Structural Similarity

Uses sequence comparison of the html tags to compute the similarity.

We not implement the similarity based on tree edit distance because it is slower than sequence comparison.

Style Similarity

Extracts css classes of each html document and calculates the jaccard similarity of the sets of classes.

Joint Similarity (Structural Similarity and Style Similarity)

The joint similarity metric is calculated as:

k * structural_similarity(document_1, document_2) + (1 - k) * style_similarity(document_1, document_2)

All the similarity metrics takes values between 0 and 1.

Recommendations for joint similarity

Using k=0.3 give use better results. The style similarity gives more information about the similarity rather than the structural similarity.

Examples

Here is a example:

In [1]: html_1 = '''
<h1 class="title">First Document</h1>
<ul class="menu">
    <li class="active">Documents</li>
    <li>Extra</li>
</ul>
'''

In [2]: html_2 = '''
<h1 class="title">Second document Document</h1>
<ul class="menu">
    <li class="active">Extra Documents</li>
</ul>
'''

In [3] from html_similarity import style_similarity, structural_similarity, similarity

In [4]: style_similarity(html_1, html_2)
Out[4]: 1.0

In [7]: structural_similarity(html_1, html_2)
Out[7]: 0.9090909090909091

In [8]: similarity(html_1, html_2)
Out[8]: 0.9545454545454546

References

The idea of sequence comparision was taken from Page Compare.
The other ideas were taken from T. Gowda and C. A. Mattmann, Clustering Web Pages Based on Structure and Style Similarity, 2016 IEEE 17th International Conference on Information Reuse and Integration (IRI), Pittsburgh, PA, 2016, pp. 175-180.
Use case Clustering web pages based on structure and style similarity

html-similarity's People

Contributors

Stargazers

Watchers

html-similarity's Issues

but i think sometime it not similar

i had to compare the url is:[http://bbs.gfan.com/forum-22-1.html,http://bbs.gfan.com/android-9172442-2-1.html] by using similarity function,i got 0.3444178082191781,but when i compare [http://bbs.gfan.com/android-9172442-2-1.html,http://bbs.gfan.com/android-9161132-1-1.html],i only get 0.358913813459268,what do you think?i think it should be similar

ImportError: No module named html_similarity

How to reproduce the problem?

from html_similarity import style_similarity, structural_similarity, similarity

Versions

html-similarity version: 3.7.1
Python Version: 3
Operating System: Raspberry Pi (Raspbian)

Feature request. TED algorithm

First of all, thanks for the amazing tool!

In the research of Thamme Gowda and Chris Mattmann they use ZhangShasha’s tree edit distance (TED) algorithm for comparing HTML's DOM trees. I've found the python library implements this algorithm: https://zhang-shasha.readthedocs.io/en/latest/#tree-format-and-usage

I think it could be more accurate than SequenceMatcher in difflib. But I ran into the problem of creating a node from HTML.

Do you have any thoughts or ideas about how to create a node from HTML? Do you think it could be helpful for compare?

matiskay / html-similarity Goto Github PK