It would be nice if you could see at a glance in the draft folder what dates the revision numbers correspond to.

Rename the PACKAGE_TEMPLATE_NAME into APPLICATION_PACKAGE_TEMPLATE_NAME
this will avoid confusion between the revision of the application package (app_revision) with the revision on the extension

For now the logic is built on the name (which is choose by the application revision) but can be changed. This could bring errors

Looking at:

• http://192.168.113.208:8080/api/v1/app/5aa3da37f3eb080001497b3d/extension?app_revision=29173&os=macosx&arch=amd64&sort=created&sortdir=-1
• http://192.168.113.208:8080/#item/5ef328794db7a400019d60f4

The category metadata is not listed.

Proposed solution

• Edit metadata of extension package already uploaded
• Update client invocation to associated category with newly uploaded extension. See Slicer/Slicer#5010

http://slicer-extension-manager.readthedocs.io/en/latest/ -> http://slicer-package-manager.readthedocs.io/en/latest/

The front end for presenting Slicer packages is https://download.slicer.org/

The associated code can be found here: https://github.com/mhalle/slicer4-download

Step 1: API update

Update slicer4-download so that it can work with the API of the slicer_package_manager server.

cc: @Pierre-Assemat

@mhalle: Would it be possible to setup a test-download.slicer.org that we could update our-self to test ?

Step 2: Stats

Investigate how we could also keep track of more sophisticated stats for extension packages

The download.slicer.org currently takes care of generating the download stats: https://download.slicer.org/download-stats/

The download stats are first gathered by parsing the Apache log and keeping track of the results in a local sqlite database. It also look the stats can be exported to json.

@mhalle We would like to also support similar stats for the new extensions infrastructure. We were thinking to generalize your work and keep track of stats for both application packages and extension packages.

This issue was created to document recent improvements related to the continuous integration infrastructure

• #111
• #112
• #113

See KitwareMedical/slicer-extensions-webapp#2 (comment)

By using utilities functions instead of duplicate the code for application and extension packages

This will allow us to address this issue: https://issues.slicer.org/view.php?id=3696

The following error is reported:

$ git clone --branch 2.x-maintenance https://github.com/girder/girder.git girder-2x
$ cd girder-2x
$ docker-compose up
[...]
ERROR: Could not find a version that satisfies the requirement hachoir-metadata (from girder==2.5.0) (from versions: none)
ERROR: No matching distribution found for hachoir-metadata (from girder==2.5.0)

We want to be able to sort extensions by date from the more recent one to the last.
Using the Girder API give us the sorted list of extension, but using the shell commands don't...

Is your feature request related to a problem? Please describe.

Downloading a specific package without using the python client currently requires multiple calls. This lead to complex implementation.

For example:

# Get app_id
app_id=$(curl -X GET --header "Accept: application/json" "https://slicer-packages.kitware.com/api/v1/app?name=Slicer" | jq ".[0]._id" -r)
https://slicer-packages.kitware.com/api/v1/app?name=Slicer

app_id=5f4474d0e1d8c75dfc705482
echo "app_id: ${app_id}"

# Get item_id
revision=29917
os=macosx
arch=amd64
item_id=$(curl -X GET --header "Accept: application/json" "https://slicer-packages.kitware.com/api/v1/app/${app_id}/package?os=${os}&arch=${arch}&revision=${revision}" -s | jq ".[0]._id" -r)
echo "item_id: ${item_id}"

# Get file_id
curl -X GET "https://slicer-packages.kitware.com/api/v1/item/${item_id}/download" -O

Describe the solution you'd like

Add new endpoints.

There are few possible approaches to define the routes

Explicitly include release_id_or_name, os, arch, revision

For example:

GET /app/:app_id_or_name/:release_id_or_name/package/:os/:arch/:revision/download
GET /app/:app_id_or_name/:release_id_or_name/extension/:os/:arch/:revision/download

where:

• app_id_or_name: 5f4474d0e1d8c75dfc705482 or Slicer
• release_id_or_name: draft, release name like 4.10, or id of the release
• os: linux, macosx or `win
• arch: amd64
• revision: 29917

Examples:

/app/Slicer/4.15/package/macosx/amd64/29917/download
/app/Slicer/draft/package/macosx/amd64/29917/download

Only with os, arch, revision

GET /app/:app_id_or_name/package/:os/:arch/:revision/download
GET /app/:app_id_or_name/extension/:os/:arch/:revision/download

and then additional parameter would be passed using &param1=value1.

All filters as parameters

GET /app/:app_id_or_name/package/download
GET /app/:app_id_or_name/extension/download

If more than packages is found, report an error requesting user to be more specific with specific filters.

• Do not display API key
• Display text before listing list of revision to delete
• Instead of --force use -y. I think it convey better the idea of answering yes to the question

Get extensions older/newer than a specific date or revision.

First: by date

The basic use of the client should approaching the following command:

slicer_extension_manager_client extension get {app_name} --older {date} --newer {date}

Then: Link the revision with date of upload of the extension

We should be able to get only the extensions older/newer than a given revision.

slicer_extension_manager_client extension get {app_name} --older {app_revision} --newer {app_revision}

We should only use:

• getExtensions
• getReleases

To get both extension and release by name/id or a list of them. So we should remove:

• get /app/{app_id}/extension/{extension_name}
• get /app/{app_id}/release/{release_id_or_name}

And remove or rename with maybe /revisions instead of /draftrelease

• get /app/{app_id}/release/draftrelease

Because it's function is to literally get all the revisions (as sub-folder of the 'draft' release) that contain the nightly extensions...

instead of

The Slicer Package Manager is a Girder plugin that allows you to manage Slicer package
 and extension within Girder. It provides a simple API to Upload and Download extensions 
or packages, create new applications, and manage releases of your applications.

consider something like this where you would link to the different part of the documentation you already wrote:

The Slicer Package Manager includes a Girder plugin and a CLI allowing you
to manage Slicer packages for Slicer-based applications and associated extensions.

In a nutshell:

• Data model specific to this project is implemented by organizing data using standard
  Girder constructs (collection, folder and item) and by associating metadata.

• By default, a top-level collection named Applications is created with a packages folder
  organizing the different application.

• Each application folder contain a draft folder where unreleased packages are uploaded and one or multiple release folders (e.g 1.0, 2.0, ...).

• Each release folder contain application packages (installers for the different platforms), and an extensions folder containing a flat list of extension packages.

• Each extension packages is associated with metadata like application revision, extension revision, operating system and architecture,

The diagram below represents the organization:

Applications
   |--- packages
   |        |----- Slicer
   |        |         |----- 1.0
   |        |         |        |---- Slicer-linux.tar.gz
   |        |         |        |---- Slicer-macos.dmg
   |        |         |        |---- Slicer-win.exe
   |        |         |        |---- extensions
   |        |         |        |             |---- Plugin1
   |        |         |        |             |---- Plugin1
   |        |         |        |             |---- Plugin1
   |        |         |        |             |---- Plugin2
   .        .         .        .             .
   .        .         .        . 
   |        |         |----- 2.0
   .        .         .        .
   .        .         .        
   |        |         |----- draft
   |        |         |        |--- r100
   |        |         |        |        |---- Slicer-linux.tar.gz
   |        |         |        |        |---- Slicer-macos.dmg
   |        |         |        |        |---- Slicer-win.exe
   |        |         |        |        |----- extensions
   |        |         |        |        |             |---- Plugin1
   .        .         .        .        .             .
   .        .         .        . 
   |        |         |        |--- r101
   .        .         .        .       |
   .        .         .        .
   |        |            
   |        |------SlicerCustom

See https://issues.slicer.org/view.php?id=2171

The field will be a free form text field.

For reference, here are some example of identifiers: https://spdx.org/licenses/

Adapt from https://github.com/scikit-build/scikit-build/blob/master/docs/make_a_release.rst

The slicer_package_manager_client_test.sh required an python utility function to extract the ID from the output of the command. So it has to be run within its on directory, which is under:
slicer_package_manager/plugin_tests/python_client_tests

• Share fixtures between test_python_client and test_client_cli
• Add tests for regression fixed in #85

This function return two different results when we try to find a specific release by name:

• listRelease('nightly') will return the list of all revision containing in the default release folder (named 'nightly')

• listRelease('Nightly') will return the 'nightly' folder...
  Or in the source code both of this command request the same endpoint :
  GET: app/{app_id}/release/{release_id_or_name}

But using the web interface of the server API, the two call with both Nightly & nightly parameters return the same response : the 'nightly' folder

It just tells you to 'cd' into the cloned folder. And then?

This change should spread on the Documentation, code source comments and also constants.

Following the integration of #78, docker-compose has been disabled.

It should be re-enabled.

See https://github.com/girder/slicer_package_manager/pull/78/files#diff-1d37e48f9ceff6d8030570cd36286a61

This should help address https://issues.slicer.org/view.php?id=4296

I am not sure if we need both. To move the discussion forward, I will try to explain:

On windows, after generating stable release, the package is automatically uploaded to the server.

• but it is not yet ready to be distributed to the user.
• the package needs to be signed. This is currently a manual process, first download from the server, sign and .. then re-upload.

In the current Midas based system, the build system first upload the stable release as an "experimental" package. Then, we manually download, sign... and re-upload. See https://www.slicer.org/wiki/Documentation/Nightly/Developers/ReleaseProcess#Manually_sign_packages

If needed, we can also use some SQL to explicit tag a package as "release". See https://www.slicer.org/wiki/Documentation/Nightly/Developers/ReleaseProcess#Explicit_tagging_of_release_packages

Ideally, the API and the client should update the different "state" of the stable release. Before implementing, we need to identify these states.

get /app/{app_id}/extension/downloadStats
->
get /app/{app_id}/downloadStats

Need to have a What is a Slicer application package? section.

#101

Data Structure

Ideally the structure would look like this:

<Name of a collection>
  |
  |--------- packages
  |             |---- Slicer
  |             |       |
  |             |      4.8.1
  |             |         |----- Slicer-4.8.1-linux-amd64.tar.gz
  |             |         |----- Slicer-4.8.1-macos-amd64.dmg
  |             |         |----- Slicer-4.8.1-win-amd64.exe
  |             |         |----- extensions
  |             |         |             |--------- ext1
  |             |         |             [...]
  |             |      nightly
  |             |         |---- r1000
  |             |         |          |--------- [...]
  |             |         |---- r1042
  |             |                    |----- Slicer-4.9.1-2018-02-29-linux-amd64.tar.gz
  |             |                    |----- Slicer-4.9.1-2018-02-29-macos-amd64.dmg
  |             |                    |----- Slicer-4.9.1-2018-02-29-win-amd64.exe
  |             |                    |----- extensions
  |             |                            |--------- ext1
  |             |                            [...]
  |             | 
  |             |---- SlicerCustom1
  |
  |
  |--------- other unrelated folder

Download stats

Download stats currently stored as metadata under each "releases" (4.8.1, nightly, ...) and should be updated to also contain the download stats of the application package:

{
    "26953": {
        "applications": {
            "linux": {
                "amd64": 1
            },
            "macosx": {
                "amd64": 1
            }
        },
        "extensions": {
            "CardiacAgatstonMeasures": {
                "macosx": {
                    "amd64": 1
                }
            },
            "DebuggingTools": {
                "macosx": {
                    "amd64": 1
                }
            }
        }
    },
    "26962": {
        "applications": {
        },
        "extensions": {
            "Autoscroll": {
                "macosx": {
                    "amd64": 1
                }
            },
            "EasyClip": {
                "macosx": {
                    "amd64": 1
                }
            }
        }
    }
}

API

get /app/{app_id}/package 
delete /app/{app_id}/package/${package_id} 
post /app/{app_id}/package 

Features

For a given set of app_revision, os and arch, user should be able to easily search extension by text.

Text should be looked up within the following metadata fields:

baseName
description

Questions

(a) How to implement autocomplete in web app backed by Girder 3 API endpoint ?

(b) Is there any existing implementation that could re-use or adapt from ? (ansible, config, end-point implementation, ....)

(c) Current plan is to experiment with option (5) and (3) described below.

Known MongoDB Limitations

• Using the built-in $text: {$search: 'sometext'} does not allow to implement search using arbitrary text as it is word based.

Options

In the context of the Slicer Extensions Manager migration where we are transitioning from my Midas to Girder for the backend, we have a Girder3 plugin called ^{1 2 3}.

The autocomplete capabilities associated with the Midas instance is implemented leveraging LIKE SQL operator. It allowed us to quickly display relevant extensions while the user was typing text in a search box.

See https://github.com/midasplatform/slicerpackages/blob/5c4c6cabd391609c57b4d2c4d06fc11658c8ebd5/models/pdo/ExtensionModel.php#L36-L73

This feature is important, and we have been exploring how to implement this using Girder 3.

Few options:
(1) Allocate funding for a MongoDB Atlas instance. See https://docs.atlas.mongodb.com/reference/atlas-search/autocomplete/ (btw $500 to $1000 / year)
(2) On premise install of Solr with use of mongodb connector ⁴.
(3) On premise install of Elasaticsearch with use of the mongdb connector "monstache" implemented in go ^{5 6}.
(4) On premise custom map/reduce job for extracting and updating collection to maintain map to original object⁷
(5) Simulate use of LIKE using regular expression⁸

References

For example:

EXTENSION ID             	NAME                          		REVISION    RELEASE NAME    APP REVISION  
-------------------------	------------------------------		--------    ------------    ------------
5aa008503b94df000180b499 	macosx_amd64_mpReview_1bf2be5 		 1bf2be5    draft           r20050
5aa0084e3b94df000180b496 	macosx_amd64_iGyne_a57c534    		 a57c534    draft           r20000
5aa0084c3b94df000180b493 	macosx_amd64_exStone_c71cd38  		 c71cd38    4.8.1           r19000   
5aa0084a3b94df000180b490 	macosx_amd64_ZFrameReg_28d43dd	         28d43dd    draft           r18500

[...]

May a "Tutorials" section could be added to the user guide...

Also review this issue: https://issues.slicer.org/view.php?id=4296 and assess if the new infrastructure effectively address the problem.

Read https://issues.slicer.org/view.php?id=2334 and discuss possible approaches.