king-millez / snapmap-archiver Goto Github PK
View Code? Open in Web Editor NEWDownload all Snap Map content from a specific location.
License: GNU General Public License v3.0
Download all Snap Map content from a specific location.
License: GNU General Public License v3.0
SnapmapArchiver.py
file and move many of its functions elsewhere. There is also some ugly state management where there are multiple snap caches in use at one time. This can be cleaned.print()
is handling all the logging at the moment, which should change in favour of a proper logging library.I notice you're using Aria for downloading the videos. Is there a significant enough time difference using the external library over say a piece of code like
import requests
...
with open(filename + '.mp4', 'w+') as f:
# req_headers from get_data.py
f.write(requests.get(snap['media']['raw_url'], headers=req_headers))
now that above code most likely won't run out of the box but I feel removing the Aria requirement would make it more attractive to people as they could get it running straight from pip
And sorry for opening an issue to discuss this, I don't know where the best place to talk about the code is.
Error message:
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='ms.sc-jpl.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Reason:
Certificate has been renewed on 30.7.2024
Quick workaround - VERIFIED:
Potential permanent solution:
Add certificate check to requests and implement certificate renewal in case of expiration.
Snaps with text and stickers don't include said graphics in the video file, instead they're stored in an image called overlay.png
, and displayed over the top by the browser/app.
We could have an option like --merge-overlay
which would use something like ffmpeg
or avconv
to put the image over the top of the media.mp4
file and export it as a new file.
A better way to name the downloaded Snaps would be nice, rather than [SNAP_ID].mp4
, which isn't really searchable. The data returned by utils.organise_data()
could be used, which includes the Snap location and the timestamp from when it was posted.
The actual API also contains some extra data for certain snaps, like the raw text used for a video.
At the moment I'm thinking a good naming convention could be [MINOR_LOCATION] - [TIMESTAMP] - [SNAP_ID].mp4
...
This should be changed:
epoch_endpoint = "https://ms.sc-jpl.com/web/getLatestTileSet"
->
epoch_endpoint = f"{self.api_host}/web/getLatestTileSet"
The ability to constrain archived Snaps not only by location but also by time would be useful. Just to illustrate the idea:
snapmap-archiver -o . -l='LAT,LONG' -r 5000 -t 5d --write-json
snapmap-archiver -o . -l='LAT,LONG' -r 5000 -t 36h --write-json
The new option being -t
which accepts some concept of time in either hours or days (minutes seems unneccesasry, but ๐คทโโ๏ธ)
When querying the API, you can pass the parameter radiusMeters
, which will give you more/less specific content the lower/higher the number you provide is. Starting from 95000
and working down to 0
, we could iterate through the maximum amount of content in a specific area. I'm thinking iterating by -2500
for values > 10000
would be good, then -1000
until 1000
, then -100
to 100
, then -10
to 0
, arbitrarily ending with 1
.
Nvm
Hello, I can't get the new version 2.0.1 to work.
my command :
snapmap-archiver -o ~/Desktop/projet/ -l='123.123,123.123' -l '445.445,445.445'
-r 1000
my error :
'''
Traceback (most recent call last):
File "/home/user/.local/bin/snapmap-archiver", line 8, in
sys.exit(main())
File "/home/user/.local/lib/python3.10/site-packages/snapmap_archiver/init.py", line 34, in main
sm_archiver.main()
File "/home/user/.local/lib/python3.10/site-packages/snapmap_archiver/SnapmapArchiver.py", line 128, in main
self.download_snaps(self.query_coords(coords))
File "/home/user/.local/lib/python3.10/site-packages/snapmap_archiver/SnapmapArchiver.py", line 122, in query_coords
return self._transform_index(to_download.values())
TypeError: SnapmapArchiver._transform_index() takes 1 positional argument but 2 were given
'''
I've tried this with a few different locales, and it seems not to be working:
โ snapmap-archiver -o . -l='decimal.lat,decimal.long' -r 10460 --write-json -t 7d
2024-05-20 10:30:45.564 | INFO | snapmap_archiver.SnapmapArchiver:__init__:52 - Skipping Snaps older than [1715610645].
Location: (decimal.lat,decimal.long) |โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ | โโโ
96% [10000/10460] in 9s (~0s, 1091.9/s) 2024-05-20 10:30:55.006 | WARNING | snapmap_archiver.SnapmapArchiver:_coordinate_query_failure:223 - You have been rate limited. Sleeping for [60] seconds...
Location: (decimal.lat,decimal.long) |โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ | โโโ 96% [10000/10460] in 1:24 (~4s, 119.9/s) 2024-05-20 10:32:09.328 | WARNING | snapmap_archiver.SnapmapArchiver:_coordinate_query_failure:223 - You have been rate limited. Sleeping for [60] seconds...
Location: (decimal.lat,decimal.long) |โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ | โโโ 96% [10000/10460] in 1:46 (~5s, 94.1/s)
The average rate slowly approaches 0.0/s
over the course of many more retries and eventually just hangs. Perhaps a bug was introduced in the refactor?
There is a type of video that isn't handled by the tool, currently throws an error when queried.
I think git messed up the coordinates.py
file with a capital C which VS Code didn't catch.
I'll fix this some time but it's time for bed rn
I have the impression that when you export the json with the command --write-json the file generates this kind of output:
"create_time": "1673031690121"
If we do :
date -d @1673031690121
Thu April 06, 54986 19:02:01 CEST
but if we remove the last three characters :
date -d @1673031690
Fri Jan 06 2023 20:01:30 CET
which seems to correspond more to the real date.
I assume the date is taken by snapmap explorer, but there seems to be a format problem.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.