Comments (6)
Ok bois, I had the same errors while crawling comments from a specific post.
How I got around this was to use "-a post=" instead of "-a page=".
I learned about the new feature from this: #27
It is working for me now :)
from fbcrawl.
I have the same problem
from fbcrawl.
I'm stucked with the same problem too. Can't figure out the solution yet.
from fbcrawl.
How I got around this was to use "-a post=" instead of "-a page=".
It was still throwing uncaught error, I had to do like this -a page="" -a post="FULL_POST_PATH"
Note there will still be KeyError: 'flag'
after that but that error is atleast caught.
from fbcrawl.
I was getting the same error as the OP. I tried ngbtri's suggestion of using "post" instead of "page" but now I'm getting a different error(see below). I tried ishandutta2007's suggestion but I get another error:
*** With -a page="" -a post="https://mbasic..."
2019-10-05 08:56:47 [twisted] CRITICAL:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python3.7/site-packages/scrapy/crawler.py", line 85, in crawl
self.spider = self._create_spider(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/scrapy/crawler.py", line 108, in _create_spider
return self.spidercls.from_crawler(self, *args, **kwargs)
File "/usr/lib/python3.7/site-packages/scrapy/spiders/init.py", line 50, in from_crawler
spider = cls(*args, **kwargs)
File "/home/redacted/github/fbcrawl/fbcrawl/spiders/comments.py", line 24, in init
raise AttributeError('You need to specifiy only one between post and page')
AttributeError: You need to specifiy only one between
*** With -a post instead of -a page
2019-10-05 08:51:42 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://mbasic.facebook.com/login/%22https:/mbasic.facebook.com/story.php?story_fbid=redacted&id=redacted%22>: HTTP status code is not handled or not allowed
2019-10-05 08:51:42 [scrapy.core.engine] INFO: Closing spider (finished)
from fbcrawl.
Ok bois, I had the same errors while crawling comments from a specific post.
How I got around this was to use "-a post=" instead of "-a page=".
I learned about the new feature from this: #27
It is working for me now :)
Thanks this worked for me too
from fbcrawl.
Related Issues (20)
- no module named urllib2
- . HOT 1
- [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://mbasic.facebook.com/gettingstarted/groups/2296031677301785>: HTTP status code is not handled or not allowed
- INFO: Ignoring response <404
- Error from VMWare Windows Instance HOT 1
- Spider error processing HOT 11
- Page scraped, clicking on "more"! ERROR HOT 3
- How do I run this
- "IndentationError: Unexpected Indent" when running scrapy, blank csv file HOT 2
- after finishing successfully, it results an empty csv file fbcrawl.py HOT 1
- scraping multiple pages by adding text file with page links to be extracted
- Empty page comments file HOT 2
- DNSLookupError: DNS lookup failed: no results for hostname lookup: mbasic.facebook.com.
- IndexError: list index out of range in comments.py line63 HOT 1
- "Show More" apparently not working. HOT 6
- The crawler has stopped working! HOT 2
- Ignoring response 404 - getting empty csv HOT 1
- Comments crawler not parsing pages correctly HOT 1
- Cannot login
- No <form> element found with HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fbcrawl.