repology / repology-updater Goto Github PK

Need a way to make specific repositories not produce lonely packages. This will allow incompatible repositories which have too many packages which do not match with other repos to still take part in comparison. Useful for OpenSUSE as long as it's based on binary package lists and non-unix repositories like F-Droid and Chocolatey which have too many non-portable projects.

Some repos include patches, it would be nice to report them upstream.

Code is there, need an easy way to download all .spec files. Todo: concact sisyphus guys

Keep transformed name in the process, pass it from rule to rule; pass terminates (rename to stop)

Since repositories are independent, this should be easy to fetch and parse them in parallel. And it will really be useful for slow repositories such as pkgsrc (index generation) and Fedora (slow fetching)

E.g. darker red color

wget -> python requests; gzip/bzip?

Currently we use list of packages instead of parsing pkgsrc, because the latter process (make index) is too slow. Not sure what we can do here now though.

Failing to fetch single repository shouldn't interrupt the whole fetch, it should only stop single repository from updating. Also, failure to fetch most repositories should remove their state, so incomplete state is not parsed.

Merge maintainers case-insensitively, such as:

Debian JavaScript Maintainers <[email protected]>
Debian Javascript Maintainers <[email protected]>

ver, verpat and setver

The list is too long otherwise

E.g. debian uses fake versions for perl modules: these should be ignored not to clobber other distros.

Make versions link to packages in specific repos (or VCS pages)

Currenly, each metapackage only allow single package per repo (e.g. only one package for FreeBSD). This leads to shadowing and information loss (when e.g. php55, php56, php70 are merged into a single metapackage). Allow multiple packages per repo, always take highest version, but leave all information there.

This depends on some refactoring though

E.g. will draw repository names in table header with gray and display a tooltip, to show users package data for specific repository is incomplete.

Before proper dynamic backend is developed, let's just make a static site generator. Required features:

• Pagefied package lists
  • Plain (just packages)
  • Outdated for each repository
  • Absent for each repostitory
  • Per maintainer
  • Per category
• Summary page for each package

• Browse pagefied package lists A...Z
• Browse by category
• Browse by maintainer
• Browse outdated packages per repository
• Browse by packages (packages with best support along distros)
• Compare repositories
• Compare maintainers
• Suggest maintainer with same interests

--tag foo,bar --tag baz == (foo OR bar) AND baz

Packages are named differently across repos. Need rules to merge differently named packages into single entity. Need single package rules (extreme-tuxracer + extremetuxracer) as well as generic rules (FreeBSD: p5-Foo-Bar, Debian libfoo-bar-perl).

E.g. 20[01][0-9]\.?[01][0-9]\.?[0123][0-9]

What to do with this:

• Convert with single format (see abcmidi package problem) to fix comparison
• If all items contain date, compare these instead (other parts are likely non-unformative, e.g. 0.0.20160916 vs. git20160916 vs. 2016.09.16

And right-align package names?

To make pagination more usable, it needs to work with package names (e.g. aa..ak, ak..bc instead of 1, 2)

Useful for testing

Fetchable via web: https://admin.fedoraproject.org/pkgdb/packages/

Available for testing in newrepos branch. Still TODO:

• Improve fetching. Sequential fetching takes ~6 hours. Maybe do a parallel fetch.
• Improve parsing. Need more clever .spec parser which will be useful for other RPM repos.
  • Add substitution support (Package: %foovar%)
  • Fix parsing of multiline fields (%description)

autossh: 1.4c > 1.4e ???
lsof: 4.89 > 4.90.f ???

Need to be split: btf (dev-java and sci-libs)

Actually, a lot more:

find gentoo.git -type d -maxdepth 2 -mindepth 2 |
   egrep -v 'dev-(perl|python|haskell)' |
   awk -F/ '{print $NF}' |
   sort | uniq -d

ace acl ada amap analog apel atlas attica auctex baloo balsa barcode bbdb bfm binclock bluedevil bson btf build c-support calc calendar cdcover cdrtools charm checkpassword coffee-script color crystal csv daemontools dash dictionary dirdiff docker dolphin ebuild-mode ecb eject elib emacs ess exo fam fcgi ffmpeg fuse gambit gdl git glade glu gnupg gnuplot gom gpgme grip gsasl haskell-mode highline icecream igrep info jack jal jama jde jpeg json kactivities kde-gtk-config kdeplasma-addons kdesu kfilemetadata kglobalaccel khotkeys kinfocenter kmenuedit knewstuff krunner kscreen kstart ksysguard kwin kwrited languagetool launchy lemon libelf libffi libgudev libiconv libintl libkscreen libnet libusb locale lookup lzma magic mailcrypt mailx man mars mash mavros mc mediawiki mew milou mime-types mldonkey mmix mmm-mode modutils mongo mpack mpc msgpack muse mysql nagios nemesis ninja nitrogen notification-daemon nut ocaml openmsx otter pam par pcl pdv picard pkgconfig planner plasma-mediacenter plasma-nm plasma-workspace pmake pms polkit-kde-agent polyglot powerdevil psgml psi python-mode rails re2 redis reduce riece ruby rubygems screen session shadow signify silo ski skkserv slim slurm smack sml-mode snappy spice spin splat splice sqlite3 ssh surf systemsettings szip teco texinfo tf time tokyocabinet tornado tree uclibc udev vc vm w3m xclip xslide xsp yacc zenburn zenirc

How do I download a single-file AUR index (the one pacman uses?) or whole AUR as a single repository? No idea for now. As last resort, AUR website may be scrapped.

Please share any ideas on what additional repositories we can support. A description on how to fetch all package data from specific repository is preferred. Approved repositories with determined fetching algorithm are split to separate bugs and eventually implemented.

Classic *nix package repositories

• Any RPM repos
  • ✅ Fedora (see #36)
  • ✅ OpenSUSE (see #44)
  • ✅ AltLinux Sisyphus (see #24)
  • ✅ Fedora EPEL
• ✅ slackware (#331)
• ✅ homebrew (see #198)
• ✅ DragonFlyBSD's dports (pretty much the same as FreeBSD ports minus some packages which don't build on DragonFly)
• 💡 VectorLinux

From Fedora release-monitoring

• ✅ Alpine linux
• ✅ CRUX whatever that is

Unsorted

• ✅ SliTaz (#664)
• 💡 NuTyX (only shell based pkgbuilds are available)
• ❌ Sabotage (do not use versions)
• ✅ salix
• ✅ solus (https://git.solus-project.com/, https://packages.solus-project.com/unstable/) (#341)

Other platforms

• ✅ NixOS
• ✅ YACP (Yet another cygwin ports) (though project is somewhat inactive)
• ✅ Rosa
• ✅ GuixSD
• 💡 OpenPandora
• ✅ buckaroo 👍 json recipes, 👎 custom naming (fixed by some rules). Also looks dead already (true, 75% outdatedness).
• ✅ MSYS2 (see #262 )

Since these will contain too many unrelevant unique packages, doable as shadow repos:

• ✅ Chocolatey (see #43)
• ✅ F-Droid (only a handful of packages manually whitelisted, too much android garbage)

Upstream repos

Doable as shadow repos as well

• ✅ CPAN (perl packages)
• ✅ PyPi (python packages)
• ✅ RubyGems
• 💡 Others? (node, etc).
• 💡 GitHub, for projects which fetch from there. Need feedback loop here (parse normal repos -> get github urls -> parse github urls for latest tags)

New version detectors

• 💡 FreeBSD's portscout
• 💡 pkgsrc's thingy
• 💡 AllMyChanges.com just for completeness, usefullness is questionable: it's focused on iOS apps and stuff which intersects with *nix software is fairly outdated

...more ideas?

This is of course planned with proper backend.

Mark vulnerable package versions

The plan:

• Implement harvesting CPE data from upstream repositories
  • ❌ GUIX contains cpe_name (is useless without vendor)
  • ❌ FreeBSD ports define CPE_VENDOR and CPE_PRODUCT, but these are not exposed in INDEX
  • Gentoo contains usable CPE metadata
• Implement database storage for project → CPE relations
• Implement fetching and parsing CPE data (https://nvd.nist.gov/vuln/data-feeds#JSON_FEED)
• Implement setting vulnerable flag for affected packages
  • Match incoming packages against vulnerable version ranges in the database
  • Force project update on new CPE for it (by resetting its hash)
  • It turned out to be viable to bulk update vulnerable status on all packages no it didn't, as we can't update binding tables properly this way to be able to do filtering based on vulnerable property
• Implement stub for handling patched vulnerabilty information from repositories (discussed in #1045
• Add vulnerable flag to binding tables to allow using it in project filtering
• Integrate vulnerability updates into delta update process properly
  • During update, only run on incoming packages
  • When vulnerabilities are updated, queue affected projects in dedicated table, reset their hashes before pushing new packages
• Add per-maintainer and per-repository vulnerable packages/projects counters
• When all code is in place, force update for all projects with defined cpe_vendor/cpe_product
• Add per-project vulnerability flag in order to be able to show "Vulns" project page conditionally
• Add/ensure that vulnerability counts are saved in repository history, so we can plot graphs
• Implement vulnerability based history events

If package was removed, it's probably problematic and there's little reason bringing it back. For FreeBSD, may use MOVED file

At least for rules processing and version comparison done, now complete unit test is needed, which includes parsing and processing fake repository data.

Instead of polluting stderr

Available for testing in newrepos branch. Uses binary package lists, so partially unsuitable for comparison, will be implemented as a shadow repo. Also contains too little info. Investigate a possibility of fetching complete data.

• apmod:perl
• apmod:python
• apmod:wsgi
• apr (ignore on freebsd)
• argus, argus-clients (-sasl on freebsd)
• asterisk (merge 11, 13 on freebsd)
• autodia (gentoo wtf)
• bonnie (pkgsrc wtf)

• Ignore only newer (single version bad)
• Ignore always (versioning schema totally broken)

Non-matching rule may indicate a bug or a need for update

These are not tied together. Fetching is more generic and may be common to multiple parsing techniques. Currently, there are just 2 types of fetchers:

• plain file [with options to gunzip or bunzip it after downloading]
• git repository

Track package state changes (such as version updates, new packages). Provide RSS streams of such events, display icons for recently updated/added packages.

For named package, generate a badge with repositories it's present in with version info

Fetcher is rather trivial: get https://chocolatey.org/api/v2/Packages()?$filter=IsLatestVersion, parse XML, get next page from <link rel="next" href="">. All package info is available in XML, including name, version, tags, comment, author, www.