swissdatasciencecenter / renku-python Goto Github PK

• Implement following helper attributes builders: jsonld.container.(set|list|index)
  • example usage: jsonld.container.list(Author)

Options

a) build a single @context from all nested objects
b) include @context in each nested object + implement custom loader?

include contexts with:

• project
  CWL steps/workflows
  .renga metadata (is included in project)
• datasets

nested serialization to be handled via #119

want something like:

client = renga.from_env()

allow e.g. the following:

$ renga runner notebook --notebook-path notebooks/my_notebook.ipynb

add renga-python to PyPI

renku init has some side effects if run in an existing git repo (inits lfs, for example). The user should be made aware of these changes and also instructed on how to bring the existing repo in-line with what we expect (e.g. command to run to add existing files to git-lfs).

Need a CLI that allows for this workflow at a minimum:

$ renga login

• guides the user through obtaining an offline token
• set up the platform access points
• create a ~/.renga.conf file that stores user settings and tokens

$ renga init <project>

• initialize a project, including adding a node to the KG
• creates a .renga metadata file for project-specific configuration

$ renga add

• add code and/or data from KG
• from git repo
• from URL

renga notebook

• launch a notebook, mounting . in the notebook container and setting it up with the proper environment for interacting with the platform

make several general notebook images with renga pre-installed based on https://hub.docker.com/r/jupyter/

renga workflow create <FILE> generates invalid CWL due to wrong YAML serialisation.

In other words, the default bucket is not created by the cli at project creation time

If you run renga init in an already existing git repo, the error is confusing, and hard to know about the -f flag.
Also, it does not rollback or cleanup the .renga folder it created in the process, which throws error when you try a renga init -f later.

If you run renga init in an already existing renga repo, the error is confusing as you get a FileExistsError.

Basic python API that can be used inside e.g. a jupyter notebook to interact with the platform.

• renga notebook fails
• renga notebook show doesn't show the running notebooks
• implement renga notebook stop
• implement numbering of running notebooks for easier navigation?
• implement automatic opening of chosen notebook in browser

will be done in #31

• all CWL outputs should be produced from latest version of inputs

--
addresses SwissDataScienceCenter/renku#117

Sequence to reproduce:

mkdir foo
mkdir bar
cd foo
renga init
echo woop > ../woop
renga datasets add dataset ../woop
cd ../bar
renga init
renga datasets create dataset
renga datasets add dataset ../foo/data/dataset/woop
renga run wc data/dataset/foo/data/dataset/woop > woop.wc
cd ../foo
echo woop2 > data/dataset/woop
git commit -am 'commiting changes to woop'
cd ../bar
git submodule update --rebase --remote
git commit -am 'update submodule'
renga status

On branch master
Files generated from outdated inputs:
  (use "renga log <file>..." to see the full lineage)

	woop.wc:

Normally it should display: <name_of_submodule>@<commit_sha1>.

Scenario:

renga init --endpoint https://example.com --autosync

Then the following is not working:

renga contexts list

I can fix by changing .renga/config.yml from:

core:
  autosync: true
  generated: '2017-10-25T11:36:10.689296'
  name: MyProject
endpoints:
  https://example.com:
    vertex_id: '20688'

to

core:
  autosync: true
  generated: '2017-10-25T11:36:10.689296'
  name: MyProject
  default: https://example.com
endpoints:
  https://example.com:
    vertex_id: '20688'

but it would be nice to have it inferred from the init command.

Importing from a git repository that contains a .renku directory should automatically reuse the included metadata about authors/creators of various entities.

• remove the local filesystem path (privacy issues)
• reference the original dataset metadata file: $ref: ...
• use submodule index to iterate over files when importing from a Git repo

addresses SwissDataScienceCenter/renku#135

if no cwl is found, we should not raise an error because that breaks the pipeline on repos that have not used renga from the outset. Instead, fail gracefully to allow other stages of the pipeline to queue.

addresses SwissDataScienceCenter/renku#117

• use the CI pipeline functionality of gitlab to rerun steps
• create custom images for rerunning renga-generated steps
• create a .gitlab-ci.yml automatically in each renga repo

- [ ] renga init should also create a .gitattributes file that will sync the data/ directory.

• cli functions like renga run should add appropriate paths to git lfs for tracking
  • renga run invocations should add the outputs to .gitattributes by default -- provide a flag to override the behavior (done in #116)
  • renga datasets add should add all files except for metadata.json

addresses SwissDataScienceCenter/renku#117

addresses SwissDataScienceCenter/renku#117

Define inputs and outputs and labels of a context:

labels:
  - renga.contexts.inputs.<name>(=[<bucket_id>,<file_id>])

relates to SwissDataScienceCenter/renga-deployer#46

• authors are removed after calling add (related to #119)
• files/<NAME>/path is not serialized as str but pathlib.PosixPath
• check how the target is joined to origin path (//)
• warn when importing local git repository
• adding a specific file without using -t doesn't work

currently the project vertex_id is passed as a label -- move it to headers to have the same behavior as storage

done in #31

• fix DAG display when the same file is used as input in multiple steps

--
addresses SwissDataScienceCenter/renku#117

like for example (it can be different):

renga io buckets upload BUCKET_ID FILENAME

renga io download FILE_ID > FILENAME

Currently we pass an access token to the deployed container, which will expire. On solution could be to simply send a refresh token instead of the access token, but this has security implications. We need to solve this problem either by having a flow with authentication via the python client or some other means.

We need a way to link notebook code in the graph

link together several steps to form a workflow

• construct a CWL workflow from a file's provenance graph
• resolve dependency paths and save workflow to disk for reuse

addresses #62

Image name cannot be renga-deployer:
https://github.com/SwissDataScienceCenter/renga-python/blob/2f1aa1246e86063355116bdcc4fbaf4afaca45b6/.travis/deploy-docker.sh#L4-L5

Travis cannot build docker images with sudo: false:
https://github.com/SwissDataScienceCenter/renga-python/blob/2f1aa1246e86063355116bdcc4fbaf4afaca45b6/.travis.yml#L22

inside a renga-deployed notebook:

import renga
client = renga.from_env()
client.contexts[0]

--> HTTP 500

from the logs:

sqlalchemy.exc.StatementError: (builtins.ValueError) bytes is not a 16-char string 
[SQL: 'SELECT contexts.created AS contexts_created, contexts.updated AS contexts_updated, contexts.id AS contexts_id, contexts.spec AS contexts_spec, contexts.jwt AS contexts_jwt 
FROM contexts 
WHERE contexts.id = %(param_1)s'] [parameters: [{'%(140547263475216 param)s': '0'}]]

enable this:

$ renga runner rerun --job job.yml .renga/workflow/step12345.cwl 

we should recognize /tree/<branch-name> and checkout the branch

Now: <renga.models.deployer.Context at 0x7f54d419eb00>
Expected: Context(id=foobar, ...) on all models

• include autodoc for clients and cli
• enable hook for Read the Docs

renga init tries to post to /api/projects/ but the URL should be /api/projects