Comments (9)
Is it possible and I didnt find the option?
There is no option for this indeed.
If it's just not implemented but possible maybe I can try to contribute if maybe you can show me where to look.
It's possibly not implemented indeed. Worst case scenario might be this info is collected through another API call by the website and we don't have it readily available in the parsed payloads.
You need to log the payload in this function here: https://github.com/medialab/minet/blob/master/minet/instagram/api_scraper.py#L381
Then the dataclass representing a post is there: https://github.com/medialab/minet/blob/master/minet/instagram/types.py#L84 and this is the method used to convert the payload into the beforementioned dataclass: https://github.com/medialab/minet/blob/master/minet/instagram/types.py#L104
I'll try to give it a look later if you don't find the time to look on your end.
from minet.
Okey thanks! 'm currently debugging but there does not seem to be a video url indeed, only images. So it's probably another API call, I will try to dig a bit.
from minet.
Okey so I found how to retrieve the url, so a simple reproducing step would be like that. But I don't know if it's the way you want it integrated, and how you want to integrate this new field in the output.
I guess the datamodel of InstagramHashtagPost should be modified ton include video_url, and then the csv output too, I can look it up if you give me some guidelines for proper contributing. Or I can let you do it.
But I need to mention that making the request seems to be super slow so maybe it's better if it's optionnal. And maybe the graphql can accept multiple shortcodes at once ? It would speed things up a lot.
for edge in edges:
post = InstagramHashtagPost.from_payload(edge["node"])
if post.media_type == "GraphVideo":
url = forge_post_url_from_shortcode(post.shortcode)
data = self.request_json(url)
post.video_url = data['graphql']['shortcode_media']['video_url']
yield post
from minet.
In this case, since we are using the same routine as the post-infos
command I think it would be better to split the workflow in two using the existing commands no? 1. you collect the posts using the hashtag
command 2. you use the post-infos
command on the relevant column of the output from step 1.
And maybe the graphql can accept multiple shortcodes at once ?
Maybe, I have never tested this. If you find it to be the case, tell me and we will improve the related command.
from minet.
Hmmm so.
I tried
minet instagram post-infos shortcode -i res.csv
but i'm getting a
File "d:\alie\documents\projets\minet\minet\instagram\types.py", line 171, in from_payload media_type = INSTAGRAM_MEDIA_TYPE.get(payload["media_type"])
error, I tried debugging it and indeed minet tries to access InstagramPost.from_payload(getpath(data, ["items", 0]))
and "items" does not exist in the data dict, I guess it should be ["graphql", "shortcode_media"] from what I'm seeing. But even like this some required fields like media_type
don't exist.
So maybe the result from the request has changed since the code was written.
I also tried
minet instagram post-infos id -i res.csv > output.csv
but there i'm only getting raw html in stdout and no error nor output in output.csv (except for the header).
If the requests have changed, maybe it could be intersting to defer the calls to a dedicated lib like https://github.com/subzeroid/instagrapi ? Don't know if it's feasible or not, but it seems to be up to date.
But again, maybe i'm just mistaken and using minet the wrong way.
from minet.
Hmm.
On my end minet instagram post-infos shortcode -i res.csv
works. Can you give me a shortcode that fails? Best case scenario it is a simple fix.
If the requests have changed, maybe it could be intersting to defer the calls to a dedicated lib like https://github.com/subzeroid/instagrapi ? Don't know if it's feasible or not, but it seems to be up to date.
Looks like a good inspiration indeed.
from minet.
I just tried again and I get the same error with minet instagram post-infos shortcode -i res.csv > output.csv
Here is my res.csv (it's only 5 rows)
res.csv
from minet.
@Tyrannas I failed to reproduce using your file. Can you double check your minet version?
from minet.
I'm on 2.0.3, using dev version (cloned the repo and pip installed -e), I tried to pull but apparently everything is up to date.
I'll try on another computer though.
from minet.
Related Issues (20)
- os.makedirs is already threadsafe
- There should be a Crawler side global callback for each job
- Method of CrawlJob to get an identical CrawlTarget to retry
- Upgrade to min version py3.8
- Add a playwright version of the crawler
- Spider process error should lead to errorred crawl? HOT 1
- Upgrade trafilatura and deal with lxml_html_clean
- Upgrade rich and other deps
- Refactor Crawler request_args as inheritance
- instagram post-infos should have line parity in the output and increase a stat rather than log
- ThreadsafeBrowser enhancements
- Add LoadingBar.track
- Improve ThreadsafeBrowser.request stability by retrying content acquisition if needed
- Draw edges kwarg
- Scrapping 1000's of comments on Instagram HOT 3
- Error on wikipedia pageviews HOT 1
- Add FORWARD_SPIDER option
- Spider process exceptions should at least be raised with some context around them
- tiktok search-videos error HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from minet.