jsdelivr / libgrabber Goto Github PK

I found a project that seemed to fail to upload the new files after bumping a new release. 2 clues:

• file dates are 2 weeks old after the new 0.3.1
• top-line comment still reads the old 0.3.0 version.

I'd think it would be nice to at least warn of old files, so our admins won't push a non-update update when they're multitasking.

I noticed that project.js looks at SemVer to flag updates, which is great.

Should it be in scope to have a new update be pushed without a SemVer increase? Sometimes a dev might forget a minor change, then add it & push out an update with out feeling they need to increase the SemVer.

The bot should have an option to disable publishing of alpha-, beta- or rc-* releases

Some projects may not push releases on the "master" branch but rather do the dev work on the master branch. It would be great if repos with releases on other branches are supported.

Fetch files off release

Many smaller projects, particularly those of CSS and JavaScript such as hosted on jsDelivr, utilize a major.minor structure. Despite not being SemVer compliant per-se, libgrabber should probably support them.

The way I would do this is to detect the versions with only one dot separating two integers early on, and appending a second minor version to fix the detection of new versions, but use the original for the folder names in the projects.

Example

1.2 => 1.2.0
1.4.3 => 1.4.3

I need libgrabber to grab the following url for handsontable to be updated by jsdelivr

https://api.github.com/repos/handsontable/jquery-handsontable/tags?per_page=200

There are a lot of tags and they are not being returned in an order that the default pagination will list the latest ones.

• dist/*.js - copies javascript files from dist dir. Directory structure will not be retained (e.g. dist dir will be stripped when copied). To retain directory structure prepend glob with ./ e.g. ./dist/*.js

What happens if all files for consumption are inside dist, but they have a dist/vendors folder, & they want only the /vendors folder to be a directory in jsDelivr?

For example this fails

{
  "packageManager": "github",
  "name": "matreshka",
  "repo": "finom/matreshka",
  "files": {
    "include": ["matreshka.min.js", "matreshka.js", "matreshka.min.map"]
  }
}

because finom/matreshka is now matreshkajs/matreshka

The bot should be able to follow the redirects and use the new repo names.

Eg: for "include": [ "thing.css", "thing..min.css", "thing.js", "thing.min.js"], if jquery.thing.js's SemVer is changed, then all files are uploaded to a new version on jsDelivr, even if the other files' versions aren't bumped? How about if a non-mainfile version is bumped like a CSS?

Could be a race condition if a PR is accepted before the min files are updated.

As discussed on Gitter, I think it'd be useful to add support for ftp and file indexs. For example, see angular's index.

Could make managing pull requests like this jsdelivr/jsdelivr#5335 significantly easier via auto-update.

/cc @jimaek @UnbounDev

I'm interested in hearing your opinions about how libgrabber should approach with pushing project updates to git. My idea is that libgrabber doesn't work directly with jsdelivr/jsdelivr repo, but uses its own fork (with a separate github account). For example, let say that librabber found a new version in one of the projects, then it does following things:

1. Creates local branch <project name>/<version>
2. Adds project files to this branch
3. Commits to branch
4. Pushes branch to origin
5. Removes local branch
6. Creates pull request to merge remote branch <project name>/<version> into jsdelivr/jsdelivr

Every time libgrabber get scheduled to run, it merges changes from upstream (jsdelivr/jsdelivr). On the next execution it will not update same version again because the pull request is accepted by @jimaek and new version is found in the file system.

If the pull request is still not accepted, then libgrabber checks if there is a remote branch (<project name>/<version>) in its own fork and skips updating project again. This works rather well in cases when there is an error in update.json or a bug in libgrabber. You could just fix update.json and delete remote branch directly from github, libgrabber will then try to update the project again.

This is implemented as of now, with the exception of doing pull requests. There are couple of branches created in test jsdelivr fork https://github.com/aleksandarabot/jsdelivr/branches.

Please tell me your suggestions and improvements to this workflow.

1. Can you verify this file is correct? https://github.com/jsdelivr/jsdelivr/blob/5262636dd7b8d9f438077e39b5964bdf1adb90ca/files/semantic-ui/update.json
2. Any ideas on how we can validate an update.json before pushing to github?

https://github.com/zmmbreeze/lining.js/releases/tag/0.3.1 is up for a day, but https://github.com/zmmbreeze/lining.js/blob/master/build/lining.min.js etc still have v0.3.0. I assume you either didn't bump package.json until after you ran Grunt, or forgot to upload the new files?

https://github.com/wavded/humane-js
How do we include the themes folder?

{
  "packageManager": "github",
  "name": "humane.js",
  "repo": "wavded/humane-js",
  "files": {
    "basePath": "/",
    "include": ["humane.min.js", "humane.js", "themes"]
  }
}

Doesnt work.

Updating using metadata/home/jsdelivr/files/humane.js/update.json
 Read metadata
 { metadata:
 { packageManager: 'github',
 name: 'humane.js',
 repo: 'wavded/humane-js',
 files:
 { basePath: '/',
 include:
 [ 'humane.min.js',
 'humane.js',
 'themes' ] },
 path: '/home/jsdelivr/files/humane.js' },
 versions:
 { localVersions: [ '3.0.6', '3.1.0' ],
 remoteVersions:
 [ '1.1.0',
 '1.2.0',
 '1.2.1',
 '1.3.0',
 '1.4.0',
 '2.0.0',
 '2.1.0',
 '2.1.1',
 '2.2.0',
 '2.2.5',
 '2.2.6',
 '2.2.7',
 '2.2.8',
 '2.5.0',
 '2.6.0',
 '2.7.0',
 '2.7.1',
 '2.7.2',
 '2.8.0',
 '2.8.1',
 '3.0.0',
 '3.0.1',
 '3.0.2',
 '3.0.3',
 '3.0.4',
 '3.0.5',
 '3.0.6',
 '3.1.0',
 '3.2.0' ],
 branchVersions: [] } }
Updating humane.js to 3.2.0
Copied 3 files
{ name: 'humane.js',
version: '3.2.0',
files:
[ '/home/jsdelivr/files/humane.js/3.2.0/humane.min.js',
'/home/jsdelivr/files/humane.js/3.2.0/humane.js',
'/home/jsdelivr/files/humane.js/3.2.0/themes' ] } 

And themes/* would simply add the contents of the folder.

Hi there,
We've recently pushed the latest version of our javascript client: v2.6.6 and we'd like to make it available to our users through jsdelivr to always provide them our latest version, so they won't have to worry about minor updates.
We just discovered that aliases point to different versions, and not to the last one.

Latest version: 2.6.6 (added 13 days ago)
http://cdn.jsdelivr.net/algoliasearch/latest/algoliasearch.min.js -> 2.6.2
http://cdn.jsdelivr.net/algoliasearch/2/algoliasearch.min.js -> 2.6.5

Sounds like a caching issue. Could you please have a look into it.
Otherwise, can you tell me how often does the libgrabber bot crawle github repos?

Thanks,

I'm trying to understand; so people have to install libgrabber on their local machine?
For some reason I thought it was a jsDelivr bot...

So far the bot already has 3 branches + master.
On every new run can we delete old branches and leave only master?

Github is not the place to store so many images

Not sure what the file cap should be?

Idea source

Likely a v2 thing, but could work something like this:

1. Repo managers adds the Update.json
2. When libbrabber hits their repo, it also grabs the README.*
3. Looks for "CDN" or "jsDelivr" keywords
4. If doesn't exist, then a PR is prepped to add instructions on how to use jsDelivr. Best as a subsection under "Usage" or "Download" if any. If not, before the "License" at the end.
5. PR is pushed after the repo is active on the CDN.

Unfortunately, badges can't be used anymore for notifications.

Several projects maintain multiple major version (notably jQuery). It'd be useful to support syncing latest tags on different major versions

When libgrabber checks if project is updated, it reads metadata from update.json. Metadata contains information like name of package, type of package manager (npm or bower), repository url (if project is not using package manager) and a list of files to be included or excluded.

For example, for a project that uses npm or bower:

{
  "packageManager": "npm" // "bower"
  "name": "lodash",
  "files": {
    "basePath": "dist/",
    "include": ["*.js"],
    "exclude": ["lodash.underscore.js", "lodash.underscore.min.js"]
  }
}

If the project is not using package manager, then the repository value identifies project location. For example:

{
  "repository": "https://github.com/lodash/lodash"
  "files": {
    "basePath": "dist/",
    "include": ["*.js"],
    "exclude": ["lodash.underscore.js", "lodash.underscore.min.js"]
  }
}

packageManager - type of package manager (npm or bower)
name - package name
repository - required if packageManager or name are not defined, refers to the project's git repository
files.basePath (optional) - base path to use when looking for resources
files.include (optional, multiple) - files or paths to include
files.exclude (optional, multiple) - files or paths to exclude

Includes/excludes might be specified using glob (see https://github.com/isaacs/node-glob).

Besides getting project updates from github, libgrabber supports popular package managers npm and bower.

I assume now that this means NPM or Bower may be used only to flag new releases, instead of Releases?
And the files are still uploaded directly from GitHub?

name (required) - refers to package name on npm or Bower, or repo name when GitHub is used

Seems that name really reflects jsDelvr's folder name if you look at this example? If true, then I'll update.

Lately the bot started closing update PRs immediately after opening them and deleting the branch. This had never happened before. Any ideas?

jsdelivr/jsdelivr#1047
jsdelivr/jsdelivr#1043

Is there any way to make the bot rename some files before pushing them?
This way we can make sure that all projects will follow our structure.

Discussion jsdelivr/jsdelivr#915

We need a detailed readme for this project describing how to install and use the tool for people that want to contribute code and for the authors how to keep their projects auto-updated.

For example chartist changed their structure and libgrabber stopped updating it.
The only way to see the problem is to read the logs

info: Updating chartist.js to v0.4.3
info: Copied 0 files name=chartist.js, version=v0.4.3, files=[]
warn: Something might be wrong with update.json name=chartist.js, version=v0.4.3

an other example

error: Error in reading metadata file /home/jsdelivr/files/underscore.string/update.json
warn: Failed to read metadata metadataPath=/home/jsdelivr/files/underscore.string/update.json,
error: Failed to update project /home/jsdelivr/files/underscore.string/update.json
error: Failed to update project

There needs to be a way to automatically notify the admin via email about a library that fails to update

If a project doesn't have any local versions then libgrabber should simply download the latest available.
Example:

files/jquery.cookie/
info.ini
update.json

This is going to be awesome as it doesnt require the author to fork jsDelivr and add an initial version. He can simply use Web GUI to add update.json and libgrabber will take care of the rest.

Lots of big projects want instant updates on our CDN.
Any ideas how to use Github's webhooks functionality to ping libgrabber every-time there is a new release and quickly update that project?

We accidentally added an update.json to a project that doesnt have any tagged releases. So libgrabber crashed. Can we catch this kind of errors and simply skip these projects?

If its possible it would help a lot to add a parameter to update.json.
Something like "auto-merge": true.
It can be enabled per project and if the bot sees it then it will automatically merge the PR for that project without waiting for a mod.