Spec: https://git.cs.upb.de/5gtango/generalInfo/wikis/user-story-t41-1-packaging

Lets annotate our package descriptors uploaded to the catalogue so that we do not need the mapping endpoint anymore.

Makes the design mich clearer. (We annotate UUIDs to the files anyhow. )

give transaction UUID as response since the unpackaing should be async

• implement REST endpint
• document REST endpoint
• synchronous vs. asynchronous request handling
• add a callback mock
• perform callback after unpackaging
• improve unittests (for all modules!!!)

You can get the information in the wiki:
https://git.cs.upb.de/5gtango/deliverables/wikis/repository-that-contains-the-api-documentation

I know we're all still building this, but I needed one already... :-)

• call unpacking using CLI
• call unpacking using REST
• unpackaging procedure (TODO more details)
  • create wiki page to document TangoPackager and EtsiPackager
  • extract package file to temp dir.
  • collect all infos about package (define a clear order in which manifests are checked, document it)
    • read TOSCA.meta, ETSI MF, NAPD.yaml
    • validate NAPD against its schema
    • unify metadata from different files (fill empty NAPD skeleton if no NAPD is there)
    • assert that a minimum set of meta data is available or abort packaging process (generic function that should work on all package spec. levels)
  • validate checksums
    • compute and compare checksums
    • error handling CLI mode
    • error handling REST mode
    • --ignore flag
  • add example commands from comments to README.md
  • call storage backend for each extracted artifact
  • generate and return result (w. links if service)
  • REST: send full callback payload (id, metadata, ...) -> return record as result of _do_unpack...
• ~~~store to local file backend (replaces catalogue for first version)~~~
• failure management: produce logs in service mode, but do not stop service if something fails
• IMPORTANT implement specific unittests for 5GTANGO packager (use example package)

Additional features

We might create independent issues for these.

• CLI mode: nicer output and result overview (cli.py dispatch method)
• helper to guess mime types (e.g. packager: _etsi_to_napd_package_content)

We skip label and reference support in v0.

If validator is installed, use it.

• install validator in Docker image (have it always there in service mode)
• have a command line flag to manually skip validation
• implement validation as such
  • validate during packaging
    • good case tng-pkg -p misc/5gtango_ns_project_example1/
    • bad case tng-pkg -p misc/5gtango_ns_project_example1_bad/
      • nice error handling
    • add unittest
  • validate during unpackaging (inside Docker!!!!) (problem: we do not always unpackage to a tng-project! solution: run a second project sotrage backend always that unpacks to tmp and validates there)
    • good case tng-pkg -u misc/5gtango-ns-package-example.tgo -o /tmp
    • bad case tng-pkg -u misc/5gtango-ns-package-example-bad.tgo -o /tmp
    • add unittest
• check that old validation is not called anymore (or keep it as fallback)

Implement/wrap in validation.py

POST to /project

Input: ZIP file with project structure/files

Processing: Extract ZIP to local FS (temp), run packaging process on it.

Return: Link to *.tgo file for download.

Document: Endpoint documentation using Swagger and Wiki

2019-05-13:

Also have a GET /project that lists all created packages (JSON list with package names, uuids, links)

https://git.cs.upb.de/5gtango/UserStories/issues/291

endpoint: /api/v1/ping

Goal of this issue is to capture the developments needed to enable basic package creation using the 5GTANGO SDK.

Scenario: A NS/VNF/test developer has created a local project using the tng-sdk-project tool and has added files/descriptors to it:

tng-workspace
tng-project -p test_project1

Now she wants to use tng-sdk-package to package this project:

# minimal
tng-pkg -p misc/5gtango_ns_project_example1
# w. more options
tng-pkg -p 5gtango_ns_project_example1 \
    --workspace path/to/workspace \
    --format eu.5gtango
    --output my_package.tgo

The resulting package can be uploaded to the SP/V&V.

Steps to be performed by the packager:

• find project.yml or project.yaml in path given by -p
• open, read, validate project.yaml
• setup unittest
• create NAPD data structure (use existing NAPDR structure) based on project.yml
• re-write file paths: descriptors are always put to Definitions (TOSCA)
• update timestamps
• compute checksums
• create a temp. working directory (WD) to build package in
• (--- from here things depend on output format ---)
• create empty package skeleton in WD
• copy all files from project to WD
• write NAPD (and clean it if needed)
  • add package type (MIME depending on output format)
• gen. and write ETSI MF (ETSI compatibility)
• gen. and write TOSCA.meta (TOSCA compatibility)
• zip WD and write to --output or vender.name.version.tgo in PWD
• evolve existing unittest
• add a packager ent-to-end untittest (first package then unpackage)
• examples to README.md

ETSI is about to release SOL007 at the end of 2018:

• SOL007_NSD_file_structure_spec

Will allow us to stop early if artifacts vendor.name.version are there already.

curl -H "Content-type:application/json" http://pre-int-sp-ath.5gtango.eu:4011/api/v2/packages?name=asdas&vendor=asdasd&version=0.1

No temp. names when uploading the package files to tng-cat.

see sonata-nfv/tng-cat#35

ONAP uses ETSI SOL004 aligned CSAR packages (a ZIP file with *.csar as ending).

• Examples:

  • Docu with examples
  • Example Package 1
  • Example Package 2
  • Example Package (correct?)

• Documentation:

  • Doc 1 (only upper part relevant)
  • Doc see Dublin Release
  • Doc more reqs.

• Tosca CSAR Spec

  • CSAR1.1 (Chapter6)

• ETSI SOL004

Have a tng-pkg alias for tng-sdk-package

File gets huge. Split it into multiple files. Move helper functions to packager.__init__.py

Have different modules for different package types etc.

packager.tango.py
packager.osm.py

Maybe: packager.commons.py etc.

• configure catalogue endpoint with ENV variables using the service's docker file (use localhost as fallback, e.g., local dev env)
• upload NAPD.yaml (Attention: ETS-only packages would not work with this backend!)
• upload VNFDs
• upload NSDs
• upload TESTDs
• upload original package file
• upload other arbitrary files to catalogue (e.g. test binaries)
• upload additional catalogue support mapping (endpoint in catalog?, implement in _build_catalog_metadata)
  • fix integration bug (400) File with {name => upb_logo.png, uuid => 2dd10810-d608-45cb-8712-4491df20fbca} not found in the Catalogue
• return package UUID and send it in callback
• unified upload error handling
• unit tests for storage.py (currently skip_store=True in test_unit_rest.py, re-enable!)
• expose skip_store as CLI and ENV config
• test with all three example packages from tng-schema and misc/

Skip all other files in the first prototype.

ENV variable naming:

 CATALOGUE_URL: http://tng-cat:4011/catalogues/api/v2

• success
• failed

Flow:

• fetch existing descriptors
• if descriptor found, do a PUT instead of a POST to upload it to tng-cat

give infos about the unpackaging status

• add to REST
• document using swagger
• unit tests

JSON can then easily opened with Swagger UI.

Users that do not want install the tng-sdk-package can still view the API specs.

This one is required for tng-sdk-profile: sonata-nfv/tng-sdk-benchmark#6

The output format should be a TANGO project folder that can be packaged again.

We should implement this as a new storage backend: TangoProjectFileSystemBackend

Inputs: destination folder

• implement as storage backend
  • translate NAPD to a valid project.yaml
  • create output project folder
  • create folder structure of packe in project folder
  • copy all files to new folders
  • write project.yaml
• write tests
• update readme

If you do: tng-pkg -p project -o ../some/folder/ you will hit an error (21)!

The tool should notice that -o points to a directory (and not a full file name) and use the automated name generation, combined with the dir given in -o.

on unpackaging:

1. extract OSM descriptors
2. re-pack OSM descriptors into OSM packages
3. push to OSM's NBI

Required feature for UPB's planned NFVSDN demo.

Unpackaging of example packages from tng-schema repo and upload to tng-cat.

End-to-end test across:

• tng-schema
• tng-sdk-package
• tng-cat

Start implementing this test once #14 is closed.

If someone want's to unpackage a package without checking it's checksums he/she should be able to do it.

• CLI --ignore-fingerprints
• REST parameter: 'ignore-fingerprints'

Start packager in service mode: Running in a Docker container, offering a REST API

see PkgSepc.

See ETSI:

• SOL004 (VNF)
• SOL007 (NS)
• SEC021_VNF_Package_Security_Spec

https://docbox.etsi.org/isg/nfv/open/drafts/

We want to be able to create also native OSM packages from an 5GTANGO SDK project.

OSM packages are a bit different, e.g., there are single packages for NSD, and every single VNFD. Packages are simple tar.gz files.

tng-pkg -p myproject --format osm

Documentation:

• (not very useful): https://osm.etsi.org/wikipub/index.php/Creating_your_own_VNF_package

Examples:

• SDK Project with OSM descriptors: https://github.com/sonata-nfv/tng-tests/tree/master/packages/NSID1V_OSM
• NSDs: https://osm.etsi.org/gitweb/?p=osm/vim-emu.git;a=tree;f=examples/services;h=bb80f8b63648f29c105ae5e5021fb2d1684eedab;hb=HEAD
• VNFDs: https://osm.etsi.org/gitweb/?p=osm/vim-emu.git;a=tree;f=examples/vnfs;h=cfaac02fffe07dcd83728f37177901ccd0fb832b;hb=HEAD
• Many other, larger examples: http://osm-download.etsi.org/ftp/osm-5.0-five/6th-hackfest/packages/

Testing:

OSM release 5.0 instance at: http://fgcn-osm1.cs.upb.de/

2019-06-03:

• Checksums: MD5 algorithm
• Tagging to control destination package

Unify data models of POST endpoint and status endpoint.

POST should also return the following structure

{
"error_msg": null,
"package_process_uuid": "e6eec600-2990-470a-87c0-dbd3d0dd11f4",
"status": "success"
}

Automatically increase version numbers of a project when a new package is created. Makes use much more convenient (we noticed this during SONATA pilot integration).

However, feature is optional and can be disabledif needed e.g. --autoversion (remove the old --no-autoversion flag)

Attention: Versions come as strings "0.1" or floats 0.1 both needs to work.

We add/increment the third part of the version field.

Examples:

0.1 -> "0.1.1"
"0.1.1" -> "0.1.2" etc.

The first version always uses a fixed name for the ETSI manifest when a package is created.

This is still aligned to the ETSI spec but requires Entry-Manifest: etsi_manifest.mf to be present in the TOSCA.meta (which is the case in the current implementation)

We can improve this by also dynamically naming the *.mf like the main descriptor to be even more aligned.

This is not super important.

Do not output packaging infos on stdout. Helps if called from another tool.

see sonata-nfv/tng-sdk-benchmark#6

Having the UUIDs of the uploaded artifacts in the package descriptor (NAPD) will make re-use in the rest of the system much easier, as discussed with @jbonnet

TODO in tng-sdk-package code

• upload NAPDR instead of NAPD
• remove ugly additional metadata from NAPDR
• re-order uploads to upload NAPDR after uploading the other artifacts
• annotate NAPDR with UUID received from catalogue when the other artifacts are uploaded

Not idempotent.

 tng-pkg -p misc/5gtango_ns_project_example1/
 tng-pkg -u eu.5gtango.5gtango-project-sample.0.1.tgo
 tng-pkg -p eu.5gtango.5gtango-project-sample.0.1
 tng-pkg -u eu.5gtango.5gtango-project-sample.0.1.tgo
 tng-pkg -p eu.5gtango.5gtango-project-sample.0.1
 tng-pkg -u eu.5gtango.5gtango-project-sample.0.1.tgo
 tree eu.5gtango.5gtango-project-sample.0.1

TOSCA and ETSI packages have no MIME types in their descriptors. We should have a module that tries to guess them based on file ending or descriptor content.

This will improve compatibility to legacy package formats.

Two fold:

• normal artifact
• other packages

If other packages, we might think about an automated recursive unpackaging procedure.

Automatically create packages from using the integrated 5GTANGO SDK GUI: sonata-nfv/tng-sdk-descriptorgen#45

#Example:

tng-package -h
usage: tng-package [-h] [-p PACKAGE] [-u UNPACKAGE] [--format FORMAT] [-v]
                   [-s] [--address SERVICE_PORT] [--port SERVICE_PORT]

5GTANGO SDK packager

optional arguments:
  -h, --help            show this help message and exit
  -p PACKAGE, --package PACKAGE
                        Create package from given project.
  -u UNPACKAGE, --unpackage UNPACKAGE
                        Unpackage given package.
  --format FORMAT       Package format [5GTANGO|OSM]. Default: 5GTANGO
  -v, --verbose         Output debug messages.
  -s, --service         Run packager in service mode with REST API.
  --address SERVICE_PORT
                        Listen address of REST API when in service mode.
                        Default: 0.0.0.0
  --port SERVICE_PORT   TCP port of REST API when in service mode. Default:
                        5099

Skips the validation against online schemas (e.g. NAPD.yaml).
Skips resolving of online resources.