mrepoapp / magisk-modules-repo-util Goto Github PK

Would be great to have a per-module override for max_num (could go to track.json). Two use cases:

• a (small) module with frequent updates (avoid 404 when the user tries to download a version already purged)
• a large module with rare updates (to not waste space with "dead dinosaurs")

(idea again "borrowed" from how F-Droid deals with its ArchivePolicy, having a default one for the entire repo which can be overridden per app)

Hello

I have installed your (beta) MRepo app and seen your new Repo for Magisk modules

I'd like to test and to submit my module ToyBox-Ext

Hopefully, the config.json and hosts.json are filled correctly:

config.json:
{
"repo_name": "ToyBox-Ext",
"repo_url": "https://github.com/zgfg/ToyBox-Ext/",
"repo_branch": "main",
"sync_mode": "git",
"max_num_module": "",
"show_log": "",
"log_dir": ""
}

hosts.json:
[
{
"id": "ToyBox-Ext",
"update_to": "https://raw.githubusercontent.com/zgfg/ToyBox-Ext/main/update.json",
"license": "OBSD",
"changelog": ""
}
]

If one downloads the ZIP or TAR from releases/, all that cli.py will ever do is showing a stack trace:

Traceback (most recent call last):
  File "./cli.py", line 26, in <module>
    sys.exit(Main.exec())
  File "/mnt/data/web/ftp/repo/magisk/util/sync/cli/Main.py", line 50, in exec
    parser = Parameters.generate_parser()
  File "/mnt/data/web/ftp/repo/magisk/util/sync/cli/Parameters.py", line 76, in generate_parser
    version=str(get_version_code()),
  File "/mnt/data/web/ftp/repo/magisk/util/sync/__version__.py", line 36, in get_version_code
    count = int(GitUtils.commit_count())
  File "/mnt/data/web/ftp/repo/magisk/util/sync/utils/GitUtils.py", line 58, in commit_count
    return int(cls.exec("git rev-list --count HEAD"))
ValueError: invalid literal for int() with base 10: ''

It seems cli.py depends on the presence of the .git/ directory just to look up how many commits there were (git rev-list --count HEAD) – which is not really needed to run the utility. Maybe this should be eliminated, or at least a fallback implemented. Not everyone wants to run from the latest HEAD (which might not be stable) – some would prefer to stick to a stable release and download it from there.

Currently, the timestamp added to the corresponding version in update.json as well as the last_update in track.json take the timestamp of the modules ZIP file – which usually is the time the ZIP was built (or attached at the place update_to's update.json points to). There's no way to tell what time this version reached the Magisk repo – last_update is rather last_build.

Would you consider an added field for the corresponding version entries in update.json to point out the date this specific version was added to the repo? Not sure how that should be reflected in track.json, as there a last_update is already present (maybe something like last_update_repo or last_repo_update).

Thanks for considering!

I've never checked before but always thought you'd just extended the format used by the "official repos". But comparing the modules.json of the Magisk-Alt-Repo with mine, I see the fields are named entirely different. This is what the Alt-Repo's looks like:

{
    "last_update": 1598596016000,
    "modules": [
        {
            "id": "acc",
            "last_update": 1680282963000,
            "notes_url": "https://raw.githubusercontent.com/Magisk-Modules-Alt-Repo/acc/master/README.md",
            "prop_url": "https://raw.githubusercontent.com/Magisk-Modules-Alt-Repo/acc/master/module.prop",
            "stars": 7,
            "zip_url": "https://github.com/Magisk-Modules-Alt-Repo/acc/archive/master.zip"
        },

and this is from mine:

{
  "name": "IzzyOnDroid Magisk Repo",
  "timestamp": 1681150305.993857,
  "modules": [
    {
      "id": "AOSPMods",
      "license": "GPL-3.0-only",
      "version": "2.7.2",
      "versionCode": 233,
      "name": "AOSP Mods (Xposed version)",
      "author": "Siavash79 + ElTifo",
      "description": "Xposed based module for customizations on SystemUI on AOSP based roms. Supports Android 12+",
      "states": {
        "zipUrl": "https://apt.izzysoft.de/magisk/modules/AOSPMods/2.7.2_233.zip",
        "changelog": "https://apt.izzysoft.de/magisk/modules/AOSPMods/2.7.2_233.md"
      }
    },

Now trying to add your or mine to a client like FoxMMM won't work, as the only matching field name is the module's id. If the fields that are matching would be named alike, other clients (hopefully) would simply ignore the additional ones. What do you think?

(I just stumbled on this as some folks asked me why they couldn't add my repo to FoxMMM. Which also raised the question if they could e.g. add the Alt-Repo to MRepo directly, or only via your mirror?)

When running cli.py sync, despite of the log output stating for each module already the latest version, update.json, Readme and ZIP of each module are touched/changed – and thus transferred again to the remote server when running rsync. If a module is already up-to-date, those files IMHO should NOT be touched/modified. Not only does that cause unnecessary network traffic (on syncing to the server later), it also makes it harder to spot what time a module has been really last updated (especially the timestamp of the ZIP) when looking at it in the file system.

That would probably be this line:

if versions_item.versionCode <= latest_version.versionCode:

As this is inside if local_update_json.exists(), shouldn't that rather be if versions_item.versionCode < latest_version.versionCode: (instead of <=)?

git clone sets the timestamps of files to now() (i.e. the time of the clone), which does not reflect their creation or commit times. That way, building a module ZIP from a repo that has not seen any commits for 2 years would result in a module being announced as "brand new". To avoid that, timestamps should be adjusted after cloning and before zipping.

To speed up implementation for that, you can find matching Python code here. Quoting for your convenience:

#!/usr/bin/env python
# Bare-bones version. Current directory must be top-level of work tree.
# Usage: git-restore-mtime-bare [pathspecs...]
# By default update all files
# Example: to only update only the README and files in ./doc:
# git-restore-mtime-bare README doc

import subprocess, shlex
import sys, os.path

filelist = set()
for path in (sys.argv[1:] or [os.path.curdir]):
    if os.path.isfile(path) or os.path.islink(path):
        filelist.add(os.path.relpath(path))
    elif os.path.isdir(path):
        for root, subdirs, files in os.walk(path):
            if '.git' in subdirs:
                subdirs.remove('.git')
            for file in files:
                filelist.add(os.path.relpath(os.path.join(root, file)))

mtime = 0
gitobj = subprocess.Popen(shlex.split('git whatchanged --pretty=%at'),
                          stdout=subprocess.PIPE)
for line in gitobj.stdout:
    line = line.strip()
    if not line: continue

    if line.startswith(':'):
        file = line.split('\t')[-1]
        if file in filelist:
            filelist.remove(file)
            #print mtime, file
            os.utime(file, (mtime, mtime))
    else:
        mtime = long(line)

    # All files done?
    if not filelist:
        break

No worries about performance impact, if the author is to be believed. They state it took 0.27s for the entire Bash repo, and still less than a minute for the entire Linux repo even. So it should not even be noticable for a single Magisk module 🙈

I couldn't find the point in the code, but recently I had it quite frequently that cli.py sync returned with code "1" without specifying an error. Sometimes there's an indicator, as I had just now:

[2024/01/06 20:16:23] Pull ERROR: from_json: [adrianmmiller.KSUdebloat] -> ConnectionError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
'cli.py sync' returned '1', aborting.

(the second line is from my wrapper), but sometimes not even that like right now on a second run:

[2024/01/06 20:19:59] Pull INFO: _check_version_code: [SuperuserListBackup] -> already the latest version
'cli.py sync' returned '1', aborting.

Could you please include a corresponding error message when the tool exits with anything other than a "0"? It's hard to tell if an error-exit means I shall not push the changes to my server, and when it means I can ignore it.

Is there a list of defined exit codes? If so, where can I find it? If not, where in the code do I find what threw the error?

PS: This time, the 3rd run succeeded. Other days I had 5+ runs without success. Would be nice knowing the culprit so it can be addressed – whether in the code, or at my end, whichever it is. Having clear hints would help with that. Thanks in advance!

With the new release, running cli.py sync crashes when pulling an update:

[2023/08/30 00:58:01] Pull DEBUG: _get_changelog_common: [jdtoolbox] -> Add support for using bootable kernel flashers as input to kernel installer
Add kernel+dtb install menu is not in /sanitized/repo/magisk/local
[2023/08/30 00:58:01] Pull DEBUG: from_track: [jefferderp.keyboardremaps] -> type: ONLINE_JSON
Traceback (most recent call last):
  File "util/cli.py", line 26, in <module>
    sys.exit(Main.exec())
  File "/sanitized/repo/magisk/util/sync/cli/Main.py", line 53, in exec
    code = cls._check_args()
  File "/sanitized/repo/magisk/util/sync/cli/Main.py", line 73, in _check_args
    return cls.sync()
  File "/sanitized/repo/magisk/util/sync/cli/Main.py", line 265, in sync
    sync.update(
  File "/sanitized/repo/magisk/util/sync/core/Sync.py", line 141, in update
    online_module = self._update_jsons(track=track, force=force)
  File "/sanitized//repo/magisk/util/sync/core/Sync.py", line 38, in _update_jsons
    online_module, timestamp = self._pull.from_track(track)
  File "/sanitized/repo/magisk/util/sync/core/Pull.py", line 221, in from_track
    return self.from_json(track, local=False)
  File "/sanitized/repo/magisk/util/sync/core/Pull.py", line 167, in from_json
    online_module = self._from_zip_common(track.id, zip_file, changelog, delete_tmp=True)
  File "/sanitized/repo/magisk/util/sync/core/Pull.py", line 124, in _from_zip_common
    changelog_url = self._get_file_url(module_id, target_changelog_file)
  File "/sanitized/repo/magisk/util/sync/core/Pull.py", line 49, in _get_file_url
    if not (file.is_relative_to(module_folder) and file.exists()):
AttributeError: 'PosixPath' object has no attribute 'is_relative_to'

This did not happen with the repo-util version from April.

Here's the corresponding track.json for this module:

{
  "id": "jefferderp.keyboardremaps",
  "update_to": "https://github.com/Jefferderp/Magisk-KeyboardRemaps/raw/master/update.json",
  "license": "GPL-2.0-only",
  "homepage": "",
  "source": "https://github.com/Jefferderp/Magisk-KeyboardRemaps",
  "support": "https://github.com/Jefferderp/Magisk-KeyboardRemaps/issues",
  "donation": "",
  "changelog": "https://github.com/Jefferderp/Magisk-KeyboardRemaps/raw/master/changelog.md",
  "added": 1679706725.0,
  "last_update": 1647987772.0,
  "versions": 1
}

and the corresponding update.json:

{
  "id": "jefferderp.keyboardremaps",
  "timestamp": 1647987772.0,
  "versions": [
    {
      "timestamp": 1647987772.0,
      "version": "1",
      "versionCode": 10,
      "zipUrl": "https://apt.izzysoft.de/magisk/modules/jefferderp.keyboardremaps/1_1.zip",
      "changelog": "https://apt.izzysoft.de/magisk/modules/jefferderp.keyboardremaps/1_1.md"
    }
  ]
}

Looking into the corresponding module directory: the new ZIP as well as the changelog have been pulled. No other files seem to have been updated (especially neither update.json nor track.json). There are 2 ZIPs now in the directory. Funnily, both (1_1.zip and v1.0_10.zip) seem to use versionCode: 10. And there was no commit or new release in that repo since April, so repo-util should not even have pulled anything. No idea why the two ZIP files even differ in size (the newer one is smaller).

Removing the original ZIP and MD and running cli.py sync again seems to solve it for this module, but then fails for another with a stack trace:

[2023/08/30 01:12:45] Pull DEBUG: from_track: [unlimited-hotspot] -> type: LOCAL_ZIP
Traceback (most recent call last):
  File "util/cli.py", line 26, in <module>
    sys.exit(Main.exec())
  File "/sanitized/repo/magisk/util/sync/cli/Main.py", line 53, in exec
    code = cls._check_args()
  File "/sanitized/repo/magisk/util/sync/cli/Main.py", line 73, in _check_args
    return cls.sync()
  File "/sanitized/repo/magisk/util/sync/cli/Main.py", line 265, in sync
    sync.update(
  File "/sanitized/repo/magisk/util/sync/core/Sync.py", line 141, in update
    online_module = self._update_jsons(track=track, force=force)
  File "/sanitized/repo/magisk/util/sync/core/Sync.py", line 38, in _update_jsons
    online_module, timestamp = self._pull.from_track(track)
  File "/sanitized/repo/magisk/util/sync/core/Pull.py", line 229, in from_track
    return self.from_zip(track)
  File "/sanitized/repo/magisk/util/sync/core/Pull.py", line 206, in from_zip
    last_modified = zip_file.stat().st_mtime
  File "/usr/lib/python3.8/pathlib.py", line 1198, in stat
    return self._accessor.stat(self)
FileNotFoundError: [Errno 2] No such file or directory: '/sanitized/repo/magisk/local/unlimited_hotspot_v5_5.zip'

Again, this is a module available in my repo for quite a while, and sync never complained before the update to the new v2.0 release of repo-util. Why does it suddenly look for the ZIP in the local/ directory? It is properly located in modules/unlimited_hotspot/v5_5.zip (a relative path to local/ would be needed for LOCAL_JSON – I see that's now also needed for LOCAL_ZIP). With the module's source repo gone, I simply disabled updates for now (cli.py trrack -i unlimited_hostspot --disable-updates), which seems to have worked fine. sync runs through now – but after index I notice one module seems to be gone (count decreased). Digging into it I see this log entry:

[2023/08/30 01:26:00] LocalTracks INFO: get_tracks: size = 81

But there are only 80 modules in the index. Luckily I kept the previous one: it's our first candidate, jefferderp.keyboardremaps, which is missing now; it's update.json still had the original file names for zipUrl and changelog (that wasn't updated when sync crashed). Fixing those manually, too, brought the module back.

TL;DR: while the second case (LOCAL_ZIP) could be declared "works as designed", something is definitely wrong with the first one. Not sure what changed in the handling here between the versions. Further, sync should not crash in such a case but rather leave the affected module as.is, just throwing an appropriate warning/error – else automated updates via cron would simply get stuck for the entire repo if a single module fails to sync/update.

I also miss any hints in the log output indicating which modules got updates (there were such entries with the older repo-util).

One more strange message:

[2023/08/30 01:25:51] Pull DEBUG: _get_changelog_common: [jdtoolbox] -> Add support for using bootable kernel flashers as input to kernel installer
Add kernel+dtb install menu is not in /sanitized/repo/magisk/local

Oof. That's an error in the module's repo, see https://raw.githubusercontent.com/JoshuaDoes/jdtoolbox/master/update.json – maybe ignore URLs which do not start with http (or any other supported protocol)?

Why are these inverted?

impossible to get the current version when the index is 0

      "versions": [
        {
          "timestamp": 1667163698.0,
          "version": "3.1.0",
          "versionCode": 118,
          "zipUrl": "https://ya0211.github.io/magisk-modules-alt-repo/modules/open_fonts/3.1.0_118.zip",
          "changelog": "https://ya0211.github.io/magisk-modules-alt-repo/modules/open_fonts/3.1.0_118.md"
        },
        {
          "timestamp": 1682263480.0,
          "version": "3.1.1",
          "versionCode": 119,
          "zipUrl": "https://ya0211.github.io/magisk-modules-alt-repo/modules/open_fonts/3.1.1_119.zip",
          "changelog": "https://ya0211.github.io/magisk-modules-alt-repo/modules/open_fonts/3.1.1_119.md"
        }
      ]

Can we have a website property in track.json, to link to the module's website for further information? A property to point to the module's issue tracker would also be nice. Both properties should be optional, or at least allow for empty values when a corresponding URL is not available.

In this context: how would the util deal with

• additional properties I'd add to track.json which it doesn't know itself (e.g. a comment property)
• additional files I'd place in the module's directory

Would they simply be ignored – or would they be removed? I might need some"control data" for automation/maintenance tasks not covered by repo-util and think about where to place them.

EDIT: details of the requested properties in this comment below.

Two things I encountered with sync that might need some improvement:

Check module id: once a ZIP has been downloaded, sync extracts details like the module name from module.prop to add them to the index. While on it, it should also check if the id from that very module.prop matches the one specified in track.json, update.json and by the given directory name – and log an ERROR when it doesn't match in the JSON files, maybe a WARNING when it matches there but not the directory name (the latter maybe only if it technically matters, or if it doesn't matter there only if specified thus in the config). I had a typo in some of them when setting them up, which was only discovered days later "by accident".

Syncing only specified module(s): This would be helpful when adding a new module to a growing collection, e.g. running cli.py sync <module_id> would only sync the specified module (not all; this then might be a reason why to ensure the directory name should match the id). Optionally, indexing (building up modules.json) could be separated to a new index command.

When using an update.json provided by the module's own repo, things are clear: I point update_to to that, and updates are handled upstream. But when upstream does not provide an update.json, what options do I have to handle updates?

local update.json

This is not supported by repo-util as it requires the address to start with http (isWith('http','json')). Would be great if a local update.json could be supported; while the name is already reserved in this space, one could use e.g. upstream.json (or any other name as long as it ends with .json) which would be looked for in the module's folder then. What do you think?

from URL

This is described here. What is not 100% clear is how that deals with multiple versions. Say I initially set update_to to .../version_v1.0.zip, that would be added to our Magisk.repo. Now v1.1 is released, and I update the URL in update_to to point to .../version_v1.1.zip. What would happen on the next sync? Would the existing version_v1.0.zip be replaced – or would the new ZIP be added (until max_num is reached, which then would purge the oldest ZIP)? I assume the latter, but just want to make sure.

upload from local

As I understand the description, I'd need to have a directory local/ at the same level where modules, json/ etc are located (maybe you add that in the for developer section?). There I'd place the ZIP file, then specify it with update_to. Will the ZIP then be duplicated to the modules/<id> directory? Or will it be moved there? If I update to the next version, will both ZIPs have to stay in local/?

from .git

Here I assume it's a set-and-forget like with update.json. Out of curiosity, what would be needed for that to work? How does it pull updates? What's needed in the referenced git repo for it to work? I guess it's using upstream's module.prop then; but ho does it identify the matching ZIP? Is it building the module itself?

Thanks in advance for your explanations, @ya0211!

If an update.json has a trailing comma (like this one), it causes sync to fail, throwing a

JSONDecodeError(Expecting property name enclosed in double quotes: line 6 column 1 (char 243))

A possible solution to this is e.g. outlined here: JSON5 allows trailing commas. So maybe that could be used for parsing? Quoting:

>>> import json5
>>> json5.loads('{"key1": "{my special value,}",}')
{u'key1': u'{my special value,}'}

From a quick glance it could be a drop-in replacement.

I know this is rather the fault of the module author, but maybe if versionCode is defined as string but isNumeric, it could simply be converted to int? Example from here:

{
    "version": "2.7f",
    "versionCode": "27",
    "zipUrl": "https://github.com/Magisk-Modules-Alt-Repo/AdGuardDNS4Magisk/releases/download/2.7f/AdGuardDNS4Magisk-v2.7f.zip",
    "changelog": "Update to Magisk v24"
}

on sync gives

ERROR: AdGuardDNS4Magisk: update module failed: ValueError(invalid literal for int() with base 10: '27f')

I just tried with the latest code from git, and have another crash with cli.py sync:

[2023/08/30 20:46:24] Pull DEBUG: from_track: [NoProcStatRestriction] -> type: ONLINE_JSON
Traceback (most recent call last):
  File "/sanitized/magisk/util/cli.py", line 26, in <module>
    sys.exit(Main.exec())
  File "/sanitized/magisk/util/sync/cli/Main.py", line 53, in exec
    code = cls._check_args()
  File "/sanitized/magisk/util/sync/cli/Main.py", line 73, in _check_args
    return cls.sync()
  File "/sanitized/magisk/util/sync/cli/Main.py", line 265, in sync
    sync.update(
  File "/sanitized/magisk/util/sync/core/Sync.py", line 134, in update
    online_module = self._update_jsons(track=track, force=force)
  File "/sanitized/magisk/util/sync/core/Sync.py", line 38, in _update_jsons
    online_module, timestamp = self._pull.from_track(track)
  File "/sanitized/magisk/util/sync/core/Pull.py", line 242, in from_track
    return self.from_json(track, local=False)
  File "/sanitized/magisk/util/sync/core/Pull.py", line 174, in from_json
    if not self._check_version_code(track.id, update_json.versionCode):
  File "/sanitized/magisk/util/sync/core/Pull.py", line 55, in _check_version_code
    if len(update_json.versions) != 0 and version_code > update_json.versions[-1].versionCode:
TypeError: '>' not supported between instances of 'str' and 'int'

This seems to be a regression from the last commit(s) as it didn't happen with the code from the tag. The culprit probably lies in upstream's update.json I guess:

{
    "version": "v0.0.2",
    "versionCode": "2",
    "zipUrl": "https://github.com/Magisk-Modules-Alt-Repo/NoProcStatRestriction/releases/download/v0.0.2/NoProcStatRestriction_v0.0.2.zip",
    "changelog": "https://github.com/Magisk-Modules-Alt-Repo/NoProcStatRestriction/releases/tag/v0.0.2"
}

As you can see, it defines versionCode as a string. Changing the indicated line 55 in Pull.py to

        if len(update_json.versions) != 0 and int(version_code) > update_json.versions[-1].versionCode:

solves this. But then there's the next crash with another module, unrelated to the above:

[2023/08/30 20:58:26] Pull DEBUG: from_track: [zygisk_lsposed] -> type: ONLINE_JSON
Traceback (most recent call last):
  File "/sanitized/magisk/util/cli.py", line 26, in <module>
    sys.exit(Main.exec())
  File "/sanitized/magisk/util/sync/cli/Main.py", line 53, in exec
    code = cls._check_args()
  File "/sanitized/magisk/util/sync/cli/Main.py", line 73, in _check_args
    return cls.sync()
  File "/sanitized/magisk/util/sync/cli/Main.py", line 265, in sync
    sync.update(
  File "/sanitized/magisk/util/sync/core/Sync.py", line 134, in update
    online_module = self._update_jsons(track=track, force=force)
  File "/sanitized/magisk/util/sync/core/Sync.py", line 38, in _update_jsons
    online_module, timestamp = self._pull.from_track(track)
  File "/sanitized/magisk/util/sync/core/Pull.py", line 242, in from_track
    return self.from_json(track, local=False)
  File "/sanitized/magisk/util/sync/core/Pull.py", line 188, in from_json
    online_module = self._from_zip_common(track.id, zip_file, changelog, delete_tmp=True)
  File "/sanitized/magisk/util/sync/core/Pull.py", line 148, in _from_zip_common            
    changelog_url = self._get_file_url(module_id, target_changelog_file)
  File "/sanitized/magisk/util/sync/core/Pull.py", line 65, in _get_file_url            
    if not (file.is_relative_to(module_folder) and file.exists()):
AttributeError: 'PosixPath' object has no attribute 'is_relative_to'

Here it seems you've changed the naming of the files, but not in all places. The new ZIP is v1.9.1_6990.zip, but going by its module.prop should rather be v1.9.1_(6990)_6990.zip (corresponding to the previous v1.9.0_(6986)_6986.zip). I've tried renaming the files accordingly and running sync again (so it would store the files once more using the wrong name but find the correct ones), but that didn't work out. As the error is

AttributeError: 'PosixPath' object has no attribute 'is_relative_to'

I've added some debug (print(file)) to see which one it is: magisk/modules/zygisk_lsposed/v1.9.1_6990.md. That file does exist. But indeed PosixPath has no is_relative_to – only PurePosixPath has that. So I try file = PurePosixPath(file), which only changes the error message to AttributeError: 'PurePosixPath' object has no attribute 'is_relative_to'. I'm clueless with this one, and why it only happens with this module – hope you have an idea there.

Some modules are updated on each scan despite not having been changed for many month, and I have no idea why. Here are 2 examples using remove update.json:

• https://github.com/CrazyBytesVE/QS-Modded (last updated 8 months (!) ago)
• https://github.com/Jefferderp/Magisk-KeyboardRemaps (last updated 3 months ago)

Both still have versionCode=1. For both, the last_update stays the same after the ZIP has been fetched again. But for both the timestamps of the files inside modules/ are touched on each run, and the log indicates they have been updated.

With a growing repo it might be easier to keep track of modules if one could configure them in separate JSON files, one per repo, and named e.g. by their ID. Those files could just contain the basic info which is then "compiled" into json/modules.json. It would be great to also have some timestamps there, too, for maintenance purposes. Picking an example from the Readme here:

modules/zygisk_lsposed.json:

{
  "id": "zygisk_lsposed",
  "update_to": "https://lsposed.github.io/LSPosed/release/zygisk.json",
  "license": "GPL-3.0-only",
  "added": "2022-10-21",
  "last_update": "2023-01-23",
  "versions": 2
}

The timestamps would allow to spot modules having received no updates "for ages" – and the versions (to be increased whenever a new version was fetched successfully) help identify very actively maintained ones. What "time format" to use (just the date as above, or datetime, or a Unix timestamp) might be worth thinking about, but IMHO date-as-above would suffice.

As indicated, I'd suggest to place those per-module JSON into the modules/ directory. Not sure if you'd see that as a conflict to the name of the json/ directory name, but mixing them in there could become a bit confusing (and make parsing harder as you had to skip those two JSON files when scanning the per-module JSONs).

PS: Please do not delete solved issues, just close them – they can be helpful in the future for references. I just wanted to look up something in our initial issue here (#1, "Refactor magisk-modules-repo-util") and found it was gone 😢 I wanted to look/ask there about the "current final state", especially concerning support for the format of "established repos" – to know if it makes sense to start on my repo already, or if I had to wait for MRepo being ready for the refactored format.

Currently, there's no -q parameter to make scan "quiet" e.g. when running from Cron. Redirecting STDOUT to /dev/null redirects ALL output, including errors – so one would not notice if automated runs had errors.

Errors should hence be directed to STDERR, not STDOUT.

A -q parameter would be welcome, too 😉

{
  "id": "microg_installer",
  "update_to": "https://raw.githubusercontent.com/nift4/microg_installer_revived/master/update.json",
  "license": "GPL-3.0-only",
  "homepage": "",
  "source": "https://github.com/nift4/microg_installer_revived",
  "support": "https://github.com/nift4/microg_installer_revived/issues",
  "donation": "",
  "changelog": "https://raw.githubusercontent.com/nift4/microg_installer_revived/master/CHANGELOG",
  "added": 1679623730.41043,
  "last_update": 1685780590.0,
  "versions": 3,
  "max_num": 1
}

After having run cli.py sync multiple times now, there are still 3 zips in this module's directory. Maybe older versions only get removed once an update was fetched (I will have to wait for that), but I'd expect sync to take care for that when the value has been changed (especially when versions > max_num). If you consider that "works as designed", what would be the proper way to clean up (apart from manually changing the files with the risk of "messing up")? Sometimes "waiting for a new update" might be in vain, or take half a year or more 😉

I have a module in my collection where the Github repo was (maybe temporarily) set to private (or was removed, I do not know) – see #13). So I would like to (temporarily) disable that module to avoid errors on cli.py sync. Setting upload_to to an empty string leads to

Sync ERROR: unlimited-hotspot: update module failed: AttributeError('NoneType' object has no attribute 'replace')

Maybe some disabled property (defaulting to 0 or false) would be needed? Simply accepting the URL can be empty (for this reason) might work as well, but one might just want to temporarily disable a module for some reason, so a better approach would probably be

  "disable": "<reason why this module should be excluded from modules.json>",
  "disable_updates": "<reason why update checks should be skipped for this module>"

This way

• the URL can be kept
• updates can easily be re-enabled
• one keeps a pointer on the reason to follow up
• one can easily list disabled modules

The Github repo for one of the modules I have configured was removed today.

{
  "id": "unlimited-hotspot",
  "update_to": "https://github.com/felikcat/unlimited-hotspot/raw/master/update.json",
  "license": "WTFPL",
  "changelog": "",
  "last_update": 1679527941.189307,
  "versions": 1
}

So on sync, cli.py spits out the entire content of the Github error page source:

[2023/03/30 19:53:06] Sync ERROR: unlimited-hotspot: update module failed: HTTPError(





<!DOCTYPE html>
<html lang="en" data-color-mode="auto" data-light-theme="light" data-dark-theme="dark" data-a11y-animated-images="system">
  <head>

(much more following that – like 20 screens or so). This might be fine when debug was turned on – but without that I'd rather expect something like

[2023/03/30 19:53:06] Sync ERROR: unlimited-hotspot: update module failed, got a 404 (not found) for <failed_url_here>

While most of the fields requested in #7 and became available with v2.0.0 (thanks a lot!), 3 fields have been left out: antifeatures (array), category (str) or categories (array/CSV) and translation. The former two are essential for my repo:

• antifeatures: as my repo is focused on FOSS and privacy, "borderline-cases" (e.g. a module having proprietary components or depending on such) need to be clearly marked.
• category/categories: especially with larger repos (mine currently holds 82 modules) this helps narrowing down the list to the area you need a module for. With my repo e.g. filtering on "Multimedia" narrows down results to 7, or "Theming" to 6 out of the 82 modules.

Both of these are already used with my repo, which you can see here in its web interface. It's a "custom hack" I'd prefer to have supported "officially". So if you could just integrate these 2 fields that would be great; translation (linking to where one can support localization of a module e.g. via Crowdin, Transifex, Weblate etc) seems o be rarely offered by module developers (only 2 out of the 82 modules in my repo have this) – so while it would be nice-to-have, it's not as essential as the other two and I'd simply drop it if you'd at least integrate the former 2. For categories I'd suggest the "plural" variant, either using a string of comma-separated values or an array, whatever suits you best; antifeatures should be an array as sometimes multiple of them must be applied.

Thanks again for all your great work – and for considering this addition!

I've noticed this already for a while:

Pull ERROR: from_json: [Hc_memory] -> ConnectionError(HTTPSConnectionPool(host='ghproxy.com', port=443): Max retries exceeded with url: /https://github.com/OneB1ank/A1Memory/releases/download/Richard8/A1Memory-2023-11-26.zip (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f558fbf2430>: Failed to establish a new connection: [Errno 110] Connection timed out')))

Here's the corresponding track.json:

{
  "id": "Hc_memory",
  "enable": true,
  "update_to": "https://github.com/OneB1ank/A1Memory/raw/main/version.json",
  "license": "GPL-3.0-only",
  "source": "https://github.com/OneB1ank/A1Memory",
  "support": "https://github.com/OneB1ank/A1Memory/issues",
  "added": 1688711365.002626,
  "last_update": 1699429588.0,
  "versions": 3
}

The problem lies in that project's update.json (called version.json there):

{
	"version": "v4 (2023.11.26)",
	"versionCode": 439,
	"zipUrl": "https://ghproxy.com/https://github.com/OneB1ank/A1Memory/releases/download/Richard8/A1Memory-2023-11-26.zip",
	"changelog": "https://ghproxy.com/https://raw.githubusercontent.com/OneB1ank/A1Memory/main/changelog.md"
}

Maybe repo-util could simply s!https://ghproxy.com/https:!https:! on the zipUrl (as there might be other proxies, though I haven't encountered any yet: I basically mean "extracting the real URL")? Currently, according to the logs it somehow adds a "leading slash" to the full URL, which then of course does not work.

Another crash now that an update is available for a module:

Traceback (most recent call last):
  File "/web/ftp/repo/magisk/util/cli.py", line 26, in <module>
    sys.exit(Main.exec())
  File "/mnt/data/web/ftp/repo/magisk/util/sync/cli/Main.py", line 53, in exec
    code = cls._check_args()
  File "/mnt/data/web/ftp/repo/magisk/util/sync/cli/Main.py", line 73, in _check_args
    return cls.sync()
  File "/mnt/data/web/ftp/repo/magisk/util/sync/cli/Main.py", line 265, in sync
    sync.update(
  File "/mnt/data/web/ftp/repo/magisk/util/sync/core/Sync.py", line 134, in update
    online_module = self._update_jsons(track=track, force=force)
  File "/mnt/data/web/ftp/repo/magisk/util/sync/core/Sync.py", line 84, in _update_jsons
    track.write(track_json_file)
  File "/mnt/data/web/ftp/repo/magisk/util/sync/model/TrackJson.py", line 55, in write
    JsonIO.write(new, file)
  File "/mnt/data/web/ftp/repo/magisk/util/sync/model/JsonIO.py", line 12, in write
    json.dump(self, f, indent=2)
  File "/usr/lib/python3.8/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib/python3.8/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.8/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable

Module in question would be isodrive. And the worst thing about this: the crash also destroyed the corresponding track.json, which now looks like this:

{
  "id": "isodrive",
  "update_to":

(yes, that's the entire file) Luckily I still have not set up update via cron or things would have been broken online as well. And I kept a backup from before the repo-util upgrade, so I was able to restore the track.json (with manual corrections). To help you debug, here's the full track.json:

{
  "id": "isodrive",
  "update_to": "isodrive.json",
  "license": "GPL-3.0-only",
  "source": "https://github.com/nitanmarcel/isodrive-magisk",
  "support": "https://github.com/nitanmarcel/isodrive-magisk/issues",
  "donation": "https://github.com/sponsors/nitanmarcel",
  "changelog": "",
  "added": 1690407638.719967,
  "last_update": 1690292969.0,
  "versions": 2
}

and the corresponding update.json:

{
  "id": "isodrive",
  "timestamp": 1690292969.0,
  "versions": [
    {
      "timestamp": 1690292969.0,
      "version": "1.2",
      "versionCode": 12,
      "zipUrl": "https://apt.izzysoft.de/magisk/modules/isodrive/1.2_12.zip",
      "changelog": ""
    },
    {
      "timestamp": 1693645491.0,
      "version": "2.1",
      "versionCode": 21,
      "zipUrl": "https://apt.izzysoft.de/magisk/modules/isodrive/2.1_21.zip",
      "changelog": ""
    }
  ]
}

Looks like that was updated properly, as v2.1 was the update being pulled (released 2h ago) – so for tests, just set "versions": 1 in track.json and remove v2.1 from update.json (but whom do I tell that 🙈).

Oh, the corresponding local/isodrive.json (looks like my local updater needs to be adjusted to not escape slashes – no idea why it suddenly does that):

{
    "version": "2.1",
    "versionCode": "21",
    "moduleProp": "https:\/\/github.com\/nitanmarcel\/isodrive-magisk\/raw\/master\/magisk-module\/module.prop",
    "urlPattern": "https:\/\/github.com\/nitanmarcel\/isodrive-magisk\/releases\/download\/v%v\/isodrive-magisk-v%v.zip",
    "zipUrl": "https:\/\/github.com\/nitanmarcel\/isodrive-magisk\/releases\/download\/v2.1\/isodrive-magisk-v2.1.zip",
    "changelog": ""
}

I've just tried to obtain a module via its git URL. The resulting ZIP does not match the one provided by upstream; it seems some directory-separator is missing:

The left screenshot shows the structure of the ZIP as provided by the module's author. The right one is what repo-util created. Note the files:

• commonfunctions.sh in the original module is common/functions.sh
• systemplaceholder originally is system/placeholder

No files are missing. Is there some bug with _get_module_from_git? Or rather with git_clone?

    with ZipFile(out, "w", ZIP_DEFLATED) as f:
        for dir_path, dir_names, file_names in os.walk(repo_dir):
            file_path = dir_path.replace(repo_dir.as_posix(), "")
            for file_name in file_names:
                f.write(os.path.join(dir_path, file_name), file_path + file_name)

The last line looks wrong. Shouldn't that be

               f.write(os.path.join(dir_path, file_name), os.path.join(file_path,file_name))

Adjusting it that way makes the archive looking fine. Further, the two lines before above block:

    shutil.rmtree(repo_dir.joinpath(".git"), ignore_errors=True)
    shutil.rmtree(repo_dir.joinpath(".github"), ignore_errors=True)

miss some more removals: .gitlab tree, and files like .gitattributes and .gitignore.