zenotech / epic-cli Goto Github PK

For now we should not expose the clusters interface via the cli. Focus is on jobs and data.

Move away from just printing response text on error, give more generic and user friendly error code

It would be nice to be able to submit a job using the application name and version rather than the ID.

When downloading a single file you get at 403 from AWS:

(venv) Mikes-MacBook-Pro:epic-cli mike$ python epiccli.py data download /data/file.py ./data/

| | _ _ / | | | _ _ | / _ __ ___ | | ___ | |
| | | |) | | | | ' | | | | / // _ \ '_ \ / _ | / _ / | '
| || /| | |_ | |) | || | / /| / | | | (_) | || / (| | | |
||| |_| |./ _, | /____|| ||_/ ___|_|| ||
|__/

Loading config from /Users/mike/.epic/config
Downloading /data/file.py to ./data/
Download failed, An error occurred (403) when calling the HeadObject operation: Forbidden

Looks like the key prefix isn't being assembled correctly.

A number of the methods in pyepic core are just placeholders and need completing.

Should the API use an epic specific url to signify a remote location? Something like:

epiccli cp epic://folder/file ./local_file
epiccli ls epic://folder/

We could then drop the download/upload options and do it both with cp. I suggest this because I was using cp expecting it to copy to a local folder but was actually making copies in the S3 bucket...

Current data functions do not handle paths on windows and so the folder name is included in the s3 object name.

When using relative paths to upload data base keys on the location specified rather than the location relative to the cli execution.
For example running:
python epiccli.py data upload --dryrun ~/Downloads/Zenotech/example/ /example/

Produces the following keys in S3:

Uploading directory /Users/mike/Downloads/Zenotech/example/ to /example/
Uploading /PIV_E/../../Downloads/Zenotech/example/.DS_Store (dryrun)

Currently the s3 sync command requires a trailing slash to be present on paths.
We should drop this requirement if the folder already exists.

Should this line be

arn = get_request('/data/aws/get', get_request_headers())

We need to have some public facing documentation for the API that also describes the json inputs required for submitting jobs etc. Ideally this would be autogenerated but might require manual work.
Examples of API usage are also very useful.

The job submission interface is fairly minimal at the moment, for a real case we would also need to pass the application_options. The contents of this depend a lot on which application is being submitted and so a json input of the application options probably makes most sense.
In addition to the application options we should expose the other options available on job submission - basically anything in "BaseBatchJobCreateMixin".

All failure paths should return a non-zero error code - e.g. not having a trailing / on a source / dest local directory currently returns 0.

If AWS credentials haven't been setup in cli then upload / download fails with s3 error:

Upload failed, Failed to upload /Users/jamessharpe/myfile.txt to epic-data-ireland//: An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied

Add functions to allow moving of files and dirs (S3 to S3 copy without downloading).

It would be nice to pass the input data and any excluded folders by path rather than ID when submitting a job.

As well as move would be good to gave a copy function to create a copy of a directory in the epic store. Same as move but leave original in place.

Would be good to be able to switch team on the cli, something like:
python epiccli.py accounts set_team <team_id/team name>

If team_id or team_name not specified then present the list of teams.
Once switched updated team should be stored in the config file.

Readme should be updated to show usage of new epic-cli