encapsia / encapsia-cli Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 7.0 1.19 MB

Command line tools to talk to an Encapsia system.

License: MIT License

Python 39.57% Shell 5.02% HTML 53.72% PLpgSQL 1.69%

encapsia-cli's People

Contributors

Stargazers

Watchers

Forkers

petrem adiniscmed rosselliott atofei pocroianu mpopetcmed trellixvulnteam

encapsia-cli's Issues

support filtering by tag

At least encapsia plugins status, but other sub-commands too, would benefit from the ability to filter by tag(s).

"running task ..." message only displays namespace

encapsia run --host core1 task pluginsmanager taskapi.initialise
Running task pluginsmanager.....Done

Expected e.g.

encapsia run --host core1 task pluginsmanager taskapi.initialise
Running task pluginsmanager.taskapi.initialise .....Done

Setting --include-prereleases flag in config file

When working on developing Insight plugins my team and I will more regularly need to install the development versions of plugins, which now requires the additional --pre or --include-prereleases flag

To improve our workflow it would be great if we could set a default value in the ~/.encapsia/config.toml file allowing us to always include pre-releases on our machines

`config get` raises KeyError instead of signaling unauthrised access

encapsia  config get trial
...
KeyError: 'trial'

while show works properly

encapsia config show
...
encapsia_api.EncapsiaApiError: 401 UNAUTHORIZED
Full response:
b'{"message":"Invalid token","status":"error","status_code":401,"status_long_message":"No permission -- see authorization schemes","status_short_message":"Unauthorized"}'

Create "plugins reactor" subcommand

E.g. forward local calls to remote for even easier dev without installing anything except the SQL.

Please add encapsia-plugins/cmedtech/sysops/production/sysadmin to the s3_buckets for plugins

Cannot add to local store from versions toml

(.venv) cmed.peter ~/work/inf-ice-maker$ encapsia  plugins add --versions foo.toml 
Found: /Users/peter/.encapsia/plugins/plugin-analytics_permissions-2.0.0.tar.gz (Skipping)
Found: /Users/peter/.encapsia/plugins/plugin-archiver-1.5.3.tar.gz (Skipping)
Found: /Users/peter/.encapsia/plugins/plugin-esourcetraining-1.1.3.tar.gz (Skipping)
Found: /Users/peter/.encapsia/plugins/plugin-insights-2.0.1.2.tar.gz (Skipping)
Traceback (most recent call last):
  File "/Users/peter/work/inf-ice-maker/.venv/bin/encapsia", line 8, in <module>
    sys.exit(encapsia())
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/encapsia_cli/encapsia.py", line 135, in encapsia
    main(auto_envvar_prefix="ENCAPSIA")
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/encapsia_cli/plugins.py", line 763, in add
    _add_to_local_store_from_s3(pi, plugins_local_dir, force=plugins_force)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/encapsia_cli/plugins.py", line 86, in _add_to_local_store_from_s3
    s3.download_file(pi.get_s3_bucket(), pi.get_s3_name(), str(filename))
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 170, in download_file
    return transfer.download_file(
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/boto3/s3/transfer.py", line 307, in download_file
    future.result()
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/s3transfer/download.py", line 340, in _submit
    response = client.head_object(
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/client.py", line 337, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/client.py", line 628, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/client.py", line 674, in _convert_to_request_dict
    api_params = self._emit_api_params(
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/client.py", line 703, in _emit_api_params
    self.meta.events.emit(
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.8/site-packages/botocore/handlers.py", line 200, in validate_bucket_name
    if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
TypeError: expected string or bytes-like object

... because bucket is None.

Error when trying to install a new (version of a) plugin from file path

$ encapsia --host hostname plugins install path/to/plugin-foobar-1.0.0.tar.gz
Added to local store: path/to/plugin-foobar-1.0.0.tar.gz
Calling: api.run_view('pluginsmanager', 'installed_plugins_with_tags') (attempt 1/10)
Traceback (most recent call last):
  File "/home/kpp/.virtualenvs/encapsia/bin/encapsia", line 8, in <module>
    sys.exit(encapsia())
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/encapsia_cli/encapsia.py", line 141, in encapsia
    main(auto_envvar_prefix="ENCAPSIA")
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/encapsia_cli/plugins.py", line 333, in install
    plan = _create_install_plan(
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/encapsia_cli/plugins.py", line 100, in _create_install_plan
    candidate = local_store.latest_version_matching_spec(spec)
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/encapsia_cli/plugininfo.py", line 281, in latest_version_matching_spec
    return PluginSpec.make_from_spec_or_string(spec).filter(self).latest()
  File "/home/kpp/.virtualenvs/encapsia/lib/python3.8/site-packages/encapsia_cli/plugininfo.py", line 361, in make_from_spec_or_string
    raise TypeError(f"Unknown spec type {type(spec_or_string)}")
TypeError: Unknown spec type <class 'encapsia_cli.plugininfo.PluginInfo'>

$ encapsia version
Encapsia CLI version: 0.5.0
Encapsia API version: 0.2.9

click-completion is not being used correctly

Use click-web to create an encapsia webserve command?? Put in a plugin?

make encapsia plugins status output less wide

encapsia plugins status output width does not fit a normal terminal's columns. Most of the time we care about 'name' and 'version' only, maybe 'available' too.
Consider adding a --verbose (or --full) flag and limit the number of columns by default.

Fix mismanaged plugins local_dir setting

Expanding user ~ and making default directory not working due to out of date code.

E.g. results in plugins_local_dir directory being made in the cwd upon running things like:

encapsia plugins --force dev-update .

installed plugins with no manifest are causing tracebacks

e.g. running encapsia plugins install, freeze, or status:

Traceback (most recent call last):
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/bin/encapsia", line 8, in <module>
    sys.exit(encapsia())
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/encapsia_cli/encapsia.py", line 142, in encapsia
    main(auto_envvar_prefix="ENCAPSIA")
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/encapsia_cli/plugins.py", line 377, in install
    installed = PluginInfos.make_from_encapsia(host)
  File "/home/pnedoiu/.local/share/virtualenvs/encapsia-api/lib/python3.10/site-packages/encapsia_cli/plugininfo.py", line 247, in make_from_encapsia
    tags = i.get("manifest").get("tags")
AttributeError: 'NoneType' object has no attribute 'get'

Depending on command, the correct behaviour can be:

install: report an error and immediately fail
status, freeze: report an error for the "offending" plugin but keep going for the others, then exit with error code (for scripts to notice the error).

Add package command for running standard tasks

DRAFT: enhance versions.toml format to support tiers/dependencies

Enhance versions.toml to support the layers/tiers idea so that we can define them in a single file, and/or manage dependencies between plugins (e.g. order of installation).

`encapsia plugins install` should return error when install fails

The errors returned as part of the install log are displayed but the command returns 0.

When scripted, there is no easy way to determine that a plugin installation failed.

Support exporting and importing users (and roles)

Use `pluginsmanager.plugins()` instead of `pluginsmanager.installed_plugins_with_tags()`

See in plugininfo.py: make_from_encapsia()

'encapsia plugins install' exitcode on error is 0

adinis@inf2:~/adi-test$ encapsia --host adi-test-01.encapsia.com plugins --local-dir /etc/cmed/encapsia/plugins/ install --versions system-plugins.toml
name existing version new version action

sync 0.0.2 install
test 0.0.1 0.0.1 skip
tmp_maintenance 0.0.1 0.0.1 skip

(Equivalent semver versions are shown in brackets when non-semver version is used)
Do you wish to proceed with the above plan? [y/N]: y

Uploaded /etc/cmed/encapsia/plugins/plugin-sync-0.0.2.tar.gz to blob: 213e2bb19b39438595b85ac29861fc35
Installing..................................................Done
Status: error
Downloaded blob 213e2bb19b39438595b85ac29861fc35 to /tmp/tmps8gyprxr
Extracted /tmp/tmps8gyprxr into /tmp/tmp8r4u7ujk
Registered plugin sync (0.0.2) from blob 213e2bb19b39438595b85ac29861fc35 with id 399184cf695644e0ae741401e8eb2d84
/var/lib/pluginsmanager-virtualenv/lib/python3.8/site-packages/icepluginsmanager/plugins.py:221: FutureWarning: reset_on_install is deprecated and will be treated as always true in the future
warnings.warn(
Destroying plugin namespace: sync
Stopped all task workers for namespace: sync
Destroyed virtualenv: sync
Removed tasks directory: /home/ice/sync
Removed webfiles directory: /var/share/nginx/iceplugins/sync
Views namespace does not exist: sync
Remove published permissions, if any
Remove published tags, if any
Removing all/any schedules from plugin namespace: sync
Installing plugin sync (0.0.2)
Made webfiles directory: /var/share/nginx/iceplugins/sync
Installing web files in to namespace sync
Installed manifest /var/share/nginx/iceplugins/sync/plugin.toml
pew new -d sync
created virtual environment CPython3.8.5.final.0-64 in 173ms
creator CPython3Posix(dest=/home/ice/.local/share/virtualenvs/sync, clear=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ice/.local/share/virtualenv)
added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
pew in sync pip install -c /var/lib/pluginsmanager-virtualenv/config/pluginsmanager/requirements.txt redis rq
Collecting redis
Using cached redis-3.5.3-py2.py3-none-any.whl (72 kB)
Collecting rq
Using cached rq-1.5.1-py2.py3-none-any.whl (62 kB)
Collecting click>=5.0.0
Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Installing collected packages: redis, click, rq
Successfully installed click-7.1.2 redis-3.5.3 rq-1.5.1
WARNING: You are using pip version 20.2.4; however, version 21.0 is available.
You should consider upgrading via the '/home/ice/.local/share/virtualenvs/sync/bin/python -m pip install --upgrade pip' command.
Made virtualenv: sync
Installing wheels in to namespace sync
pew in sync pip install --find-links /tmp/tmp8r4u7ujk/plugin-sync-0.0.2/wheels --requirement /tmp/tmp8r4u7ujk/plugin-sync-0.0.2/wheels/requirements.txt
Looking in links: /tmp/tmp8r4u7ujk/plugin-sync-0.0.2/wheels
Collecting encapsia-api
Using cached encapsia_api-0.2.7-py3-none-any.whl (19 kB)
Collecting python-pidfile==2.0.0
Using cached python_pidfile-2.0.0-py3-none-any.whl (4.9 kB)
Collecting requests[security]<3.0,>=2.24
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting toml<0.11,>=0.10
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting arrow<0.17,>=0.16
Using cached arrow-0.16.0-py2.py3-none-any.whl (50 kB)
Processing ./.cache/pip/wheels/a9/92/74/7694cd5d3f10ea335ef507c10e877b2150de20975a986782b9/psutil-5.8.0-cp38-cp38-linux_x86_64.whl
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting pyOpenSSL>=0.14; extra == "security"
Using cached pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Processing ./.cache/pip/wheels/9b/bd/12/c040f2df6b28138b66b0361cd218180a278b95763fc2466951/cryptography-3.3.1-cp38-cp38-linux_x86_64.whl
Collecting python-dateutil>=2.7.0
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.5.2
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Processing ./.cache/pip/wheels/74/c4/b2/301f50054a0b5635fc25567992701a66ccb924f38a85e1133c/cffi-1.14.4-cp38-cp38-linux_x86_64.whl
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Installing collected packages: idna, urllib3, certifi, chardet, six, pycparser, cffi, cryptography, pyOpenSSL, requests, toml, python-dateutil, arrow, encapsia-api, psutil, python-pidfile
Successfully installed arrow-0.16.0 certifi-2020.12.5 cffi-1.14.4 chardet-4.0.0 cryptography-3.3.1 encapsia-api-0.2.7 idna-2.10 psutil-5.8.0 pyOpenSSL-20.0.1 pycparser-2.20 python-dateutil-2.8.1 python-pidfile-2.0.0 requests-2.25.1 six-1.15.0 toml-0.10.2 urllib3-1.26.3
WARNING: You are using pip version 20.2.4; however, version 21.0 is available.
You should consider upgrading via the '/home/ice/.local/share/virtualenvs/sync/bin/python -m pip install --upgrade pip' command.
Installing tasks in to namespace sync
Installed task file /home/ice/sync/sync.py
Configuring 1 task workers for namespace sync
Created/updated system user sync with capabilities: ['login.transfer.from', 'login.transfer.to', 'login.extend', 'config.read.sync', 'login.federate.sync.from', 'login.federate.sync.to', 'synchronise.in', 'synchronise.out', 'blob.read', 'blob.write']
Not publishing empty tags list.
Installing schedules in to namespace sync
Unknown error. Check system log with id d98492e8f1cf44f4beb9be0f31b7ec94 for details.
adinis@inf2:/adi-test$ echo $?
0
adinis@inf2:/adi-test$ encapsia version
Encapsia CLI version: 0.4.0
Encapsia API version: 0.2.6

`plugins add` from versions toml does not report error for plugins that aren't found

See code around line 652 in add().

encapsia database backup fails when tmp and destination are on different devices

encapsia database --host adi-test.encapsia.com backup /backups/daily/adi-test/Friday/adi-test.encapsia.com-201219.sql
Backing up database....Done
Traceback (most recent call last):
File "/usr/local/bin/encapsia", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/encapsia_cli/database.py", line 28, in backup
temp_filename.rename(filename)
File "/usr/lib/python3.6/pathlib.py", line 1309, in rename
self._accessor.rename(self, target)
File "/usr/lib/python3.6/pathlib.py", line 393, in wrapped
return strfunc(str(pathobjA), str(pathobjB), *args)
OSError: [Errno 18] Invalid cross-device link: '/tmp/tmpncvu23yb' -> '/backups/daily/adi-test/Friday/adi-test.encapsia.com-201219.sql'

'encapsia config load' does not print anything on success

Should retry on connection errors and timeouts

Print messages on successful operations

gracefully handle key not found error in `config get`

encapsia --host foo config get deploy
Traceback (most recent call last):
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 655, in get_config
    return self.get(("config", key))["result"][key]
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 115, in get
    return self.call_api("get", *args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 97, in call_api
    raise encapsia_api.EncapsiaApiError(
encapsia_api.EncapsiaApiError: 404 NOT FOUND
Full response:
b'{"message":"No configuration with this key","status":"error","status_code":404,"status_long_message":"Nothing matches the given URI","status_short_message":"Not Found"}'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/peter/work/inf-ice-maker/.venv/bin/encapsia", line 8, in <module>
    sys.exit(encapsia())
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_cli/encapsia.py", line 141, in encapsia
    main(auto_envvar_prefix="ENCAPSIA")
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_cli/config.py", line 44, in get
    lib.pretty_print(api.get_config(key), "json")
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 657, in get_config
    raise KeyError(key)
KeyError: 'deploy'
(.venv) cmed.peter ~/work/tcp$ encapsia --host launch2 config get deploy
Traceback (most recent call last):
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 655, in get_config
    return self.get(("config", key))["result"][key]
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 115, in get
    return self.call_api("get", *args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 97, in call_api
    raise encapsia_api.EncapsiaApiError(
encapsia_api.EncapsiaApiError: 404 NOT FOUND
Full response:
b'{"message":"No configuration with this key","status":"error","status_code":404,"status_long_message":"Nothing matches the given URI","status_short_message":"Not Found"}'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/peter/work/inf-ice-maker/.venv/bin/encapsia", line 8, in <module>
    sys.exit(encapsia())
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_cli/encapsia.py", line 141, in encapsia
    main(auto_envvar_prefix="ENCAPSIA")
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_cli/config.py", line 44, in get
    lib.pretty_print(api.get_config(key), "json")
  File "/Users/peter/work/inf-ice-maker/.venv/lib/python3.9/site-packages/encapsia_api/rest.py", line 657, in get_config
    raise KeyError(key)
KeyError: 'deploy'

'encapsia database backup' should allow cleanup after download

At the moment a backup download leaves the generated file in tmp and cannot be cleaned up unless covered by tmp_cleanup_plugin.
I propose to extend dbctl_download_data from encapsia_api to allow passing of cleanup variable, thus allowing to delete the file after it is downloaded

plugins dev-uninstall --all also removes the pluginsmanager itself

add plugin variants

Installing groups of plugins with one command

There doesn't seem to be a neat way to add/update plugins on a server based on an S3-bucket using the encapsia_cli

e.g. we have the latest releases in the "production" S3 Insight folder(s) which I want to install onto a server.

Something like
$ encapsia --host encval-v15-uat plugins --s3-bucket encapsia-plugins/cmedcro/standards/production/insights install

To use 'encapsia plugins install' without confirmation you need to use --force; Maybe create a -y for click.confirm?

Allow for system-wide configuration and/or command line override of user-level configuration file.

For being able to use encapsia-cli by several users on a central system, it would be useful to have a common configuration file. This could be achieved with one of the following

Read a system-wide configuration file from a few well-known locations (e.g. /etc/encapsia-cli/config.toml, /usr/local/etc/encapsia-cli-config.toml or similar).
Add a command line option to override user-level config files: encapsia --config path/to/config.toml ...

semver bug

The full semver syntax is not supported, only the major.minor.patch

The pre-release / build metadata is not supported

I created a plugin using the semver 1.0.0-dev.1 and it failed, however removing the pre-release section of the file name succeeded

semver library should be used instead of the regex:

m = re.match(r"plugin-([^-]*)-([^-]*).tar.gz", full_name)

DRAFT: re-think plugin version "spec"ing, and variants, to be more consistent and user friendly

(this would not be backwards compatible)

encapsia plugins install should add to local store if not available

> encapsia --host t2208-1086 plugins install entry-2.0.2 serverlib-1.1.1
Could not find plugin matching entry-2.0.2 in local store!
Aborted!

✖1 > encapsia --host t2208-1086 plugins add entry-2.0.2 serverlib-1.1.1
Downloaded encapsia-plugins/cmedtech/pd/production/apps/entry/plugin-entry-2.0.2.tar.gz and saved to /home/pmierlutiu/.encapsia/plugins/plugin-entry-2.0.2.tar.gz
Downloaded encapsia-plugins/cmedtech/pd/production/middleware/plugin-serverlib-1.1.1.tar.gz and saved to /home/pmierlutiu/.encapsia/plugins/plugin-serverlib-1.1.1.tar.gz

It should add it in the first place, as I think it does when used with --all-from-s3 or --versions.

plugins add / install should consider only release versions by default

When using encapsia plugins add or encapsia plugins install, development versions/builds should not be considered "latest", by default. This would also mean encapsia plugins install entry-2.1 should select latest 2.1.x release version.

To be considered if we fix this or jump to a more comprehensive change of how we allow versions to be specified (see #93 but it doesn't say much at this time).

upgrade to lastest Click and adapt to changes to shell completions support

Add a `schedule replace` sub-command that removes a task by id and adds it back with some changed fields (those present on the CLI)

Add plugins command for linting e.g. consistency of capabilities

Error on installing plugin directly from filename

According to the documentation for encapsia plugins install, it should be possible to pass in a filename directly. But you obtain a traceback. For example:

[13:01] Raymond Ponoran
    encapsia --host rponoran3 plugins install plugin-missingdatalib-1.0.0-pre.1.tar.gz
[13:01] Raymond Ponoran
    but I do get the following error
[13:02] Raymond Ponoran
    Traceback (most recent call last):
  File "/home/rponoran/.local/share/virtualenvs/cli/bin/encapsia", line 8, in <module>
    sys.exit(encapsia())
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/encapsia_cli/encapsia.py", line 141, in encapsia
    main(auto_envvar_prefix="ENCAPSIA")
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/encapsia_cli/plugins.py", line 327, in install
    plan = _create_install_plan(
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/encapsia_cli/plugins.py", line 100, in _create_install_plan
    candidate = local_store.latest_version_matching_spec(spec)
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/encapsia_cli/plugininfo.py", line 278, in latest_version_matching_spec
    return PluginSpec.make_from_spec_or_string(spec).filter(self).latest()
  File "/home/rponoran/.local/share/virtualenvs/cli/lib/python3.10/site-packages/encapsia_cli/plugininfo.py", line 358, in make_from_spec_or_string
    raise TypeError(f"Unknown spec type {type(spec_or_string)}")
TypeError: Unknown spec type <class 'encapsia_cli.plugininfo.PluginInfo'>

A workaround is to add to the local cache first, and then install. For example:

encapsia plugins add plugin-missingdatalib-1.0.0-pre.1.tar.gz
encapsia plugins install missingdatalib

Error if "variant" is specified in encapsia plugin uninstall

dconstantinescu@inf1:~/inf-systems/production/Sponsor/trial-name$ encapsia --host trial-name-01.encapsia.com plugins uninstall "shared_trial_settings-variant-trial-name"
Preparing to uninstall: shared_trial_settings-variant-trial-name
Are you sure? [y/N]: y
Calling: api.run_task(pluginsmanager, icepluginsmanager.uninstall_plugin) (attempt 1/10)
Uninstalling shared_trial_settings-variant-trial-name...Done
Status: error
Manifest for shared_trial_settings-variant-trial-name not found. Not installed?
Could not determine installed plugin_id for shared_trial_settings-variant-trial-name. Giving up.
Aborting...

dconstantinescu@inf1:~/inf-systems/production/Sponsor/trial-name$ encapsia --host trial-name-01.encapsia.com plugins uninstall "shared_trial_settings"
Preparing to uninstall: shared_trial_settings
Are you sure? [y/N]: y
Calling: api.run_task(pluginsmanager, icepluginsmanager.uninstall_plugin) (attempt 1/10)
Uninstalling shared_trial_settings...............Done

uninstall plugin causes traceback

Steps to reproduce:
uninstall a plugin using encapsia plugins --host inscro1 uninstall tg001tokengenerator

Expected Behaviour:
Plugin is uninstalled successfully

Actual Behaviour
Output displays "Uninstalling tg001tokengenerator...Done" but then displays traceback KeyError:'id'

ENCAPSIA_URL and ENCAPSIA_TOKEN are not usable

Code that reads environment variables if host is not supplied is not reachable, so the "Unable to determine host" message is returned to the user.

def get_api(**obj):
    host = obj.get("host")
    if host is None:
        log_error("Unable to determine host.")
        log_error(
            "Please specify via the command line, env variable, or ~/.encapsia/config.toml file.",
            abort=True,
        )
    try:
        url, token = encapsia_api.discover_credentials(obj["host"])
    except encapsia_api.EncapsiaApiError as e:

Message wrongly acknowledges adding a plugin after error

...
Unable to download: analytics_permissions/plugin-analytics_permissions-1.0.0.tar.gz
Added to local store: /Users/peter/.encapsia/plugins/plugin-analytics_permissions-1.0.0.tar.gz

`plugins install plugin_name-a.b.c-d.tar.gz` cannot be completed (pre-release version specifier)

When passing a path to an existing plugin archive on the file system to the plugins install command, the following error is returned

Could not find plugin matching name-a.b.c.d [exact] in local store nor S3

Because a pre-release version is used, it needs the --pre option passed as well to continue with the installation (that is also the workaround)

The fix would be to automatically use the --pre flag when a plugin file is used.

allow adding name to scheduled task and allow removing scheduled task by namespace and name

Invalid names are allowed when building a plugin

Steps to Reproduce:

In the plugin.toml file set an invalid name (e.g. containing underscores)
Build the plugin

Expected Behavior:
Invalid plugin names are prevented and the plugin is not built

Current Behavior:
The tar.gz file is created (the plugin is built)

Add UTF BOM for run view

Please can the UTF BOM be added to the encapsia run view output so UTF chars open correctly by default in Excel.

https://stackoverflow.com/questions/17879198/adding-utf-8-bom-to-string-blob

STR:

$ encapsia --host encval-v15-uat run view extract_dataviews dataview_ae --save-as encval-v15-uat-extract_dataviews-ae.csv
open in Excel unicode characters aren't shown correctly (screenshot1637606135744.png)

Running the below and then opening in Excel and the characters open fine (screenshot1637606329500.png)
$ sed -i 's/^/\xef\xbb\xbf/' encval-v15-uat-extract_dataviews-ae.csv

`--all-available` option for `plugins add` and `install` commands should be improved

When trying to download plugins from S3 to local store (the add subcommand) or install plugins to an encapsia server (the install subcommand), the user can specify the -all-available flag in the command line.
It will try to download all the plugins from the configured S3 buckets; in case of install, that's before figuring out what to install on the target encapsia server.

I haven't tried how long it takes to download all the plugins, but, at the time I'm writing this, there are 1315 encapsia plugin archives on S3. There is no warning to the user before commencing, it will just start to download (it can be interrupted, though).

As part of patch for #109, the prerelease versions are discarded, unless --include-dev-builds is specified. About 2/3 of the files are dev builds, so it saves quite a bit. However, I think alerting the user and waiting for confirmation would be better (overridden by --yes, for headless runs)

Additionally, the behaviour could be refined by adding a --only-latest-versions (or similar) suboption, in case the user does not want to host the entire release archive.

As a note, the help for --all-available option does not clearly say it's going to download all the versions, only all of the plugins, so that should be corrected, as well.

Don't use datetime.datetime.fromisoformat because not present in Python 3.6

Traceback (most recent call last):                                                                                                                                                                                                                                                              
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/bin/encapsia", line 8, in <module>                                                                                                                                                                                                          
    sys.exit(main())                                                                                                                                                                                                                                                                            
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/click/core.py", line 829, in __call__                                                                                                                                                                           
    return self.main(*args, **kwargs)                                                                                                                                                                                                                                                           
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/click/core.py", line 782, in main                                                                                                                                                                               
    rv = self.invoke(ctx)                                                                                                                                                                                                                                                                       
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/click/core.py", line 1259, in invoke                                                                                                                                                                            
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                                     
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/click/core.py", line 1259, in invoke                                                                                                                                                                            
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                                     
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/click/core.py", line 1066, in invoke                                                                                                                                                                            
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                                                              
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/click/core.py", line 610, in invoke                                                                                                                                                                             
    return callback(*args, **kwargs)                                                                                                                                                                                                                                                            
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/click/decorators.py", line 33, in new_func                                                                                                                                                                      
    return f(get_current_context().obj, *args, **kwargs)                                                                                                                                                                                                                                        
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/encapsia_cli/plugins.py", line 290, in status                                                                                                                                                                   
    "installed": _format_datetime(i["when"]),                                                                                                                                                                                                                                                   
  File "/Users/tcorbettclark/.pyenv/versions/3.6.11/lib/python3.6/site-packages/encapsia_cli/plugins.py", line 26, in _format_datetime                                                                                                                                                          
    return datetime.datetime.fromisoformat(dt).strftime("%a %d %b %Y %H:%M:%S")                                                                                                                                                                                                                 
AttributeError: type object 'datetime.datetime' has no attribute 'fromisoformat'