luux / arbsmapdo Goto Github PK

It seems there have been major changes in both https://github.com/andruzzzhka/BeatSaberScrappedData/ and the beatsaver API.
As there are some major changes needed in order to get it running again and I'll be quite busy the next days, ARBSMapDo currently does not work.

I'll try to fix these issues as soon as I have some spare time, however this could take a while as we really need some deeper changes here. Sorry :/

Filtering by additional info from beastsaber. For example, allows filtering by genre (looking for non-Anime songs? Here you go!)
This will of course be limited by the information that is currently present on bsaber. Many songs don't have genre tags for example. Could be useful nevertheless. However, missing tags must be taken into consideration.

For uploading maps for the Quest, files are uploaded as *.zip. Implement an option for not extracting downloaded files.

7B9DAB9DDAD1BD3234C1E12D83E6BC2D4A48EC5B_Skeelie--AaltopahWi_Splatter-Party.zip cannot be unzipped by the zipfile package and returns BadZipFile("File is not a zip file"). This results in an error message, the song doesn't get extracted and the tmp dir is not cleared automatically.

The windows explorer isn't able to extract the zip as well. 7zip seems to do the job. However, checking the archive reveals that the included "SplatterVocal.egg" is erroneous.

Oof. Idk if I can fix this one easily. But at least a workaround should be implemented in order to proceed normally.

Optional, of course.

https://stackoverflow.com/questions/389204/how-do-i-create-my-own-url-protocol-e-g-so

For example, input URI https://bsaber.com/songs/new/?uploaded_by=ARBSMapDoDummy or https://scoresaber.com/?search=ARBSMapDoDummy instead of a specific map would result in downloading all maps uploaded by the specified user (just as it already works with playlists)

The Error:
Found Levels: 52/200 | | # | 52 Elapsed Time: 0:00:15 Traceback (most recent call last): File "arbsmapdo.py", line 208, in <module> File "downloader.py", line 47, in start levels_to_download = self.fetch_and_filter() File "downloader.py", line 150, in fetch_and_filter if self._filter_level_with_beatsaver_info(level) is True: File "downloader.py", line 206, in _filter_level_with_beatsaver_info bs_metadata = bs_info["metadata"] KeyError: 'metadata' [11716] Failed to execute script arbsmapdo
This is checking for all ranked songs between 6 and 7 stars. The same thing also happened while filtering between 7 and 10 stars.

At least direct upload to BMBF could be supported in the future. I'll have to check if there is any way to see the already present maps there.

Since BeatDrop2 doesn't work at all since months and doesn't seem to get updated and this is already a Beat Saber download tool - why not support downloading songs given by .bplist files directly?

Specify a playlist in parameters/config and add downloaded filtered songs into that. Seems like a natural addition. Goes hand in hand with #4

Due to ARBSMapDo getting more complex, some automatic tests would be nice to prevent breaking bugs. This includes configuration stuff as well as filtering and downloading functions.

For example:

• Unit Tests such as testing if ARBSMapDo correctly downloads a given list of levels, correctly loads a given configuration, ...
• Integration Tests such as filtering a handcrafted list of map information given a specific set of filtering rules

When using multiple presets, download_dir currently has to be saved in each one seperately. A hierarchical preset architecture would solve that. (Note for myself: do not allow cycles)

Here's the error:

Scanning already existing maps...
Traceback (most recent call last):
  File "D:\Documents\bs\ARBSMapDo\arbsmapdo.py", line 331, in <module>
    downloader = advanced_downloader(config_handler.config)
  File "D:\Documents\bs\ARBSMapDo\downloader.py", line 61, in __init__
    self.cache = cache.Cache(config)
  File "D:\Documents\bs\ARBSMapDo\cache.py", line 35, in __init__
    self.update_levelhash_cache()
  File "D:\Documents\bs\ARBSMapDo\cache.py", line 143, in update_levelhash_cache
    levelhash = utils.calculate_Level_hash_from_dir(self.download_dir.joinpath(entry.name))
  File "D:\Documents\bs\ARBSMapDo\utils.py", line 62, in calculate_Level_hash_from_dir
    info_data = json.loads(info_binary, encoding="utf-8")
  File "C:\Python39\lib\json\__init__.py", line 359, in loads
    return cls(**kw).decode(s)
TypeError: __init__() got an unexpected keyword argument 'encoding'

I'll leave that here. andruzzzhka/BeatSaberScrappedData#6

Handling multiple requests at the same time, starting downloads in parallel to searching songs that meet the criteria.

The executable seems to be picked up as trojan by virustotal. No idea why, just thought I'd let you know. Malwarebytes also seems to immediately remove the file upon download, and windows smartscreen doesn't like it either

https://www.virustotal.com/gui/file/81ab76c818e322e3b55305fef5fdc00614ca7cd9137ccd436010a45acf25492b/detection

Making it pip-ready. :)

89219da contains an experimental first approach of filtering maps by game mode.
This allows searching specifically for OneSaber or 360° maps for example. However, this first implementation has flaws which is why it won't go into master right now: a) it's slow, b) it spams the BeatSaver API with hundreds of requests to find a few songs, since OneSaber maps are quite rare. This is not optimal and cannot be changed directly right now because of how the BeatSaver API works.

However, I could download a regularly updated scrape of the BeatSaver data regularly and perform the search locally on that copy. Daily updated scrapes are already available at https://github.com/andruzzzhka/BeatSaberScrappedData
The problem is that at the moment, the scraped data from BeatSaver doesn't contain the much needed characteristic infos which is where the game mode information lies. If andruzzzhka includes this information in the future, that would be the solution as long as the BeatSaver API is as it is.
andruzzzhka/BeatSaberScrappedData#3

Check if the user input is valid when asking for options, for example at vote_ratio_min.

Currently, detecting existing songs works on directory name level, which works just fine for maps downloaded with ARBSMapDo. However, maps from other locations don't follow ARBSMapDo's naming structure, so they won't be detected.

This can be solved by generating hashes of other maps and comparing them against a maintained list.