skillachie / news-corpus-builder Goto Github PK

Is there a way to export the tables as CSV files instead of database? Are you planning to add more news sites like yahoo news, bing news, reuters etc

The news corpus builder pip install does not get completed . I think the issue for this is using beautifulsoup , it has been termed OBSOLETE and they have migrated to beautifulsoup4 if this problem could be solved asap it would be good , or if there were a turn around this problem that would be appreciated as well

Collecting` beautifulsoup (from goose-extractor)
  Using cached BeautifulSoup-3.2.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\VENKAT~1\AppData\Local\Temp\pip-build-i083otnm\beautifulsoup\setup.py", line 22
        print "Unit tests have failed!"
                                      ^
    SyntaxError: Missing parentheses in call to 'print'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\VENKAT~1\AppData\Local\Temp\pip-build-i083otnm\beautifulsoup\

Although i understand that this is because of my python running 3+ , will the software be migrated anytime soon .

Hi

I am trying the example.py, however got the error below. Is this a lib issue?

Traceback (most recent call last): File "example.py", line 94, in <module> ex.generate_corpus(article_links) File "/home/work/.jumbo/lib/python2.7/site-packages/news_corpus_builder/news_corpus_generator.py", line 101, in generate_corpus 'category':category}) File "/home/work/.jumbo/lib/python2.7/site-packages/news_corpus_builder/news_corpus_generator.py", line 105, in _save_article print "Saving article %s..." %(clean_article['title']) UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2018' in position 39: ordinal not in range(256)

Hey,
Trying to crawl the data from google news but it is giving 30 links per term. How can i increase no of links per term atleast 100.

pip3 install news-corpus-builder generates an error

_Collecting beautifulsoup (from goose-extractor>=1.0.25->news-corpus-builder)
Using cached BeautifulSoup-3.2.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-o2uyq317/beautifulsoup/setup.py", line 22
print "Unit tests have failed!"
^
SyntaxError: Missing parentheses in call to 'print'

I believe this can be solved by updating to beautfiulsoup4_

environment: Linux
#5 Is there a solution or workaround?