michaelharms / comcrawl Goto Github PK

search_single_index doesn't handle pagination. As mentioned on the Common Crawl blog (Ctrl + f Pagination), this means it will max out at the first 15000 results. For my use case I was getting only half my dataset.

Hi,

I've been using this library to fetch the html files from quite sometime but today I'm facing error which says " Failed to establish the connection" I've tried with 2 different internet connection and it is not working on both.

Is it an issue from common crawl server?

Is there any way to search over a locally downloaded common crawl index with this library instead of hitting commoncrawl.org's index? It keeps timing out for me and having a copy locally would be more stable.

EDIT: Or maybe pointing it at a self-hosted version of https://github.com/commoncrawl/cc-index-server instead?

I have instlled comcrawl via pip.

client = IndexClient(["2019-51", "2019-47"])
client.search("reddit.com/r/MachineLearning/*")
client.download()

trying to download html pages but not working. Gives error (Not a gzipped file (b'<?')).

Hello, I'm trying to run the following code and getting this error (It's in a virtual environment on a Mac, with Python v3.7.0. I tried on Google Colab, and got the same error as well.

Any idea what I might be doing wrong?

Thanks!

>>> from comcrawl import IndexClient
>>> client = IndexClient(verbose=True)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): index.commoncrawl.org:443
DEBUG:urllib3.connectionpool:https://index.commoncrawl.org:443 "GET /collinfo.json HTTP/1.1" 502 182
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Elias/Desktop/Temp/commoncrawl/venv/lib/python3.7/site-packages/comcrawl/core/index_client.py", line 49, in __init__
    self.indexes = fetch_available_indexes()
  File "/Users/Elias/Desktop/Temp/commoncrawl/venv/lib/python3.7/site-packages/comcrawl/utils/initialization.py", line 20, in fetch_available_indexes
    .get("https://index.commoncrawl.org/collinfo.json")
  File "/Users/Elias/Desktop/Temp/commoncrawl/venv/lib/python3.7/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

$ pip freeze 

pip freeze
certifi==2020.6.20
chardet==3.0.4
comcrawl==1.0.2
idna==2.10
requests==2.24.0
urllib3==1.25.10

I am trying to get the data using comcrawl for year 2019 - 2020 using client = IndexClient(['2019-04', '2020-45']). Looks like it is giving the data only for 2019-04 & 2020-45. Is there a way to filter data year wise? Do I have to write all the index as parameter for all the years needed?

From what I understand, the index offsets are different for WET vs WARC. So is there a way to search for the WET files using index.commoncrawl.org or would we need to download all the indexes and re-write this script to read from there instead of querying index.commoncrawl.org?

Hi @michaelharms
Thanks for releasing such a nice toolkit. I would like to know is it possible to extract specific language with your tool?