Bug Report

Describe the bug

Getting the error 'SEC_ERROR_INVALID_TIME' when trying to switch to dab. This is because dab uses a shorter TTL for each certificate than previously used, requiring the user to manually accept the certificate

To Reproduce

If using a different service, switch to DAB and send a request to the api or view a web page

Expected behavior

Certificates should already be accepted

dab version output

benjaminm@benjaminm-pc:~$ dab version
Dab wrapper script appears to have an update available!
Dab Version: 27b4e5083265950bcdad9cc5840793456bf5056a
WARNING: No swap limit support
Server Version: 18.01.0-ce
Runtimes: runc
Kernel Version: 4.4.0-98-generic
Operating System: Ubuntu 16.04.3 LTS
Architecture: x86_64

Is your feature request related to a problem? Please describe

Single adding only a service each time can be time consuming and can lead to problem for networking in the future

Describe the solution you'd like

A new feature should be engineered, allowing groups of microservices depend on other groups. This
way the start/stop procedure can be standardised resulting in lower times for network and debugging.

Describe alternatives you've considered

Additional context

Is your feature request related to a problem? Please describe

#83 could have been caught earlier if the update command was rigorously tested.

Describe the solution you'd like

When using tests.sh a different tag other than latest should be used such as test which should allow pulling of latest which will be different to the test image allowing a test of a full update process.

Describe alternatives you've considered

Well not testing it, but that clearly did not work out.

Additional context

None.

Bug Report

Describe the bug

When executing dab changelog or dab shell cat /LOG there is only one line.

To Reproduce

Steps to reproduce the behavior:

1. Run dab changelog

Expected behavior

Show show each commit on a different line, much more than one.

Example

None.

dab version output

Dab Version: 2a45970cb77d62f555d676f7de1589c852a5069d
Server Version: 18.06.1-ce
Runtimes: runc
Kernel Version: 4.14.76
Operating System: NixOS 18.09.884.d96c7a35638 (Jellyfish)
Architecture: x86_64

Additional context

None.

Is your feature request related to a problem? Please describe.
Cannot easily determine what username ran the dab container.

Describe the solution you'd like
DAB_USER should be set to the running user.

Describe alternatives you've considered
Could do some gross hacks I guess.

Additional context
N/A.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Cover more cases, check more things.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
Containers created by dab pki may be left behind causing clutter.

Describe the solution you'd like
run commands should remove the container when it is finished.

Describe alternatives you've considered
None.

Additional context
None.

Is your feature request related to a problem? Please describe.
Currently subcommands are scripts in a single directory even though there are now many nested subcommands, this has made the scripts bloated and hard to grok/read/refactor.

Describe the solution you'd like
Allow directories under app/subcommands to become subcommand namespaces like dab repo is that does nothing but provide info and access to other subcommands represented by scripts in that directory.

Continuing with the repo example, app/subcommands/repo.sh would become something like app/subcommands/repo/{entrypoint,group,manage}.sh which would require the current repo add top level repo sub subcommands to be moved to something like manage.

Describe alternatives you've considered
Could rewrite in a different language that allows better composition, too big of an increment though.

Additional context
N/A.

When the lab network config changes it cannot be directly rebuilt because the ensure network hook fails before reaching the recreate subcommand.

The hook should be skipped when doing a network recreate or move the hook to subcommands that require it.

Is your feature request related to a problem? Please describe.
N/A.

Describe the solution you'd like
Some sub commands do allow --help some do not, some commands are suffixed with subcommands some or suffixed with targets.

Describe alternatives you've considered
N/A.

Additional context
N/A.

Bug Report

Describe the bug

Since #97 CI has taken 2-3 times as long and failed about 50% of the time.

To Reproduce

Steps to reproduce the behavior:

1. Run Circle CI workflow

Expected behavior

Should take less than 20 mins and not need any retries.

Example

https://circleci.com/gh/Nekroze/dab/298

dab version output

N/A.

Additional context

N/A.

Is your feature request related to a problem? Please describe.
Sometimes the repos my repo depends on are updated without me knowing it leading to services not working without manually updating each dependent repository.

Describe the solution you'd like
A CLI or web based tool to indicate which repositories have updates and perhaps to do the updates for you on selected repo's.

Describe alternatives you've considered
git-run might be able to do something like this however smarts would be limited and requires git tool knowledge and configuration making it less than a turnkey solution.

Additional context
N/A

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Whenever an entrypoint script is ran, display all shellcheck issues as warnings on that script.

Describe alternatives you've considered
I have not.

Additional context
Should help entrypoints stay sane and resilient.

Describe the bug
dab repo entrypoint should not delete existing script unless forced

To Reproduce
Run dab repo entrypoint create with existing start and stop scripts

Expected behavior
The existing script should not be overridden by the default scripts

Screenshots

dab version output:

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Some way to select specific versions of dab tools. Likely environment variables like DAB_TOOL_CONSUL that the host/user can define.

Describe alternatives you've considered
Doing nothing would probably be fine but would be somewhat inflexible.

Additional context
N/A

Feature Request

Is your feature request related to a problem? Please describe

Nope.

Describe the solution you'd like

After updating or on demand, display the changelog.

Describe alternatives you've considered

It would be nice to automatically generate a changelog of sorts like vim-plug does but appears much more involved.

Additional context

The idea is to surface important changes to the users attention before they cause an issue.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Some way to group multiple repo's under a single name for bulk actions.

Describe alternatives you've considered
Different subcommand, seems messy.

Additional context
The idea is for a dev to be able to group logically clustered repositories or commonly used ones together. Provides a way to easily manage the group and execute entrypoints in a specific sequence.

Is your feature request related to a problem? Please describe.
If the wrapper is updated, a user will not know.

Describe the solution you'd like
Include the wrapper in the container, mount the wrapper used to run the container, compare hashes and provide update info.

Describe alternatives you've considered
None.

Additional context
Should always recheck and keep bugging the user.

Describe the bug
If you run dab update, after 24 hours, dab will try to update twice.

To Reproduce
Steps to reproduce the behavior:

1. Run dab update
2. Wait 24 hours
3. Run `dab update

Expected behavior
Should only update once.

Screenshots
N/A.

Desktop (please complete the following information):

• OS/Distro: NixOS
• Docker Version: 18.06.1-ce

Additional context
N/A.

Is your feature request related to a problem? Please describe.
Tools are not changed often but the ones using custom Dockerfile's must be built by each user before they can be used.

Describe the solution you'd like
The tools using a custom Dockerfile should tag as a member of the dab image for pushing to the docker hub. This should allow a user to download a tool without having to build it.

Describe alternatives you've considered
Could use another docker hub repository instead of per tool tags on the hub, would be more complex to admin though.

Additional context
Might need to split tool upgrading into pulling and building, or make building a part of scripts/build.sh instead.

Is your feature request related to a problem? Please describe.
When sharing config directories some environment variables must be distributed too, this should be easier.

Describe the solution you'd like
The ability to at least define default environment variable values such as DAB_LAB_INTERNET to be loaded for dab runs. This might also be a good way to provide additional tool configuration such as kafka topics or database names/users.

Describe alternatives you've considered
Could all be done manually but that is error prone and time consuming.

Additional context
None.

Is your feature request related to a problem? Please describe.
Might accidentally add a dependency to a group twice!

Describe the solution you'd like
Config set and add operations should be idempotent so repeat operations do not alter state.

Describe alternatives you've considered
None.

Additional context
N/A.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
The repo group subcommand should not be under repo as it handles tools and services as well..

Describe alternatives you've considered
Leaving it where it is would be confusing.

Additional context
Will have to change group config keys to match.

Is your feature request related to a problem? Please describe.
As a rando, I want to know how dab works and how to contribute.

Describe the solution you'd like
Expand markdown documents for developers and users alike.

Describe alternatives you've considered
N/A.

Additional context
N/A.

Is your feature request related to a problem? Please describe.
Completion was inaccurate

Describe the solution you'd like
Some way to test tab completion binary.

Describe alternatives you've considered
Nope.

Additional context
Nope.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
The tools subcommand still follows the old subcommand scheme and is as such is large and unwieldy while having an unusual UX compared to the rest of dab.

Describe alternatives you've considered
N/A.

Additional context
N/A.

Describe the bug
When running dab tools list there should only be two columns however pgadmin username and password are in additional columns without headers.

To Reproduce
Steps to reproduce the behavior:

1. Run 'dab tools list'

Expected behavior
All columns should either have headers or not exist.

Screenshots

      TOOL | DESCRIPTION                                                         |             | 
 cyberchef | The Cyber Swiss Army Knife                                          |             | 
 portainer | Docker Management Web UI                                            |             | 
chronograf | Chronograf metrics explorer from the TICK stack                     |             | 
 kapacitor | Real-Time metrics processing from the TICK stack                    |             | 
   traefik | The Cloud Native Edge Router, docker reverse proxy                  |             | 
     ngrok | Secure introspectable tunnels, points to traefik                    |             | 
watchtower | Watches labelled container images for updates and applies them      |             | 
    sysdig | Universal system visibility tool with native support for containers |             | 
   grafana | The open platform for analytics and monitoring                      |             | 
    serveo | Expose local servers to the internet                                |             | 
    ntopng | Monitor network interfaces, requires redis service                  |             | 
  vaultbot | Automate interaction with Hashicorp Vault                           |             | 
   pgadmin | Postgres administration console, usr                                | pw=user@dab | admin

dab version output:

Dab Version: c2ec12ab605d220ba78dfd7ef58759bacdd03808
Server Version: 18.06.1-ce
Runtimes: runc
Kernel Version: 4.14.74
Operating System: NixOS 18.09.800.21293d2576a (Jellyfish)
Architecture: x86_64

Additional context
Caused by a colon in the description.

Is your feature request related to a problem? Please describe.
N/A.

Describe the solution you'd like
Provide at least bash shell completions.

Describe alternatives you've considered
Too many sub commands too not.

Additional context
N/A.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
entrypoint scripts should have access to dab.

Describe alternatives you've considered
N/A.

Additional context
Should not launch an additional container, nor should it re-run main hooks.

Feature Request

Is your feature request related to a problem? Please describe

No.

Describe the solution you'd like

Use subcommander to replace app/subcommander.sh

Describe alternatives you've considered

N/A.

Additional context

None.

Is your feature request related to a problem? Please describe.
Promotes more stable changes to the code.

Also pull requests to master require checks to be passed that use the code of the merge request to check, allowing all checks to pass if disabled inadvertently or not.

Describe the solution you'd like
Move as much test code as possible to another repository or branch.

Describe alternatives you've considered
Leaving as is, almost every project seems fine with the status quo.

Additional context
Needs a mechanism to still allow breaking changes, with alarm bells if possible.

Is your feature request related to a problem? Please describe.
Untrusted certificate errors in browsers.

Describe the solution you'd like
Use certutil to add the Vault PKI CA to browser cert8.db and cert9.db files.

Describe alternatives you've considered
Installing the in OS is also an option but does not often propogate to browsers and cli tools often have switches for untrusted certs.

Additional context
N/A.

Is your feature request related to a problem? Please describe.
N/A.

Describe the solution you'd like
lib.sh or a subset should be available from entrypoint scripts.

Describe alternatives you've considered
I have not.

Additional context
Could also be used to provide useful entrypoint features like wait for port, wait for docker health, silently, etc.

Is your feature request related to a problem? Please describe.
inconsistent and confusing

Describe the solution you'd like
DAB_REPO_PATH should be available in the container, mounted and as an environment variable, so that containers that use the hosts DAB_REPO_PATH are easily to configure. Do the same for DAB_CONF_PATH.

Describe alternatives you've considered
N/A.

Additional context
N/A.

Describe the bug
Completion still has dab repo entrypoint set which was replaced by dab repo entrypoint create.

Bug Report

Describe the bug

Tools like watchtower do not have a network interface with which you can interact but the start script assumes it does giving an error. This prevents using headless tools in groups as well.

To Reproduce

Steps to reproduce the behavior:

1. Run dab group tools misc watchtower
2. Run dab group tools misc cyberchef
3. Run dab group start misc
4. See only watchtower start

Expected behavior

Exit code should be non zero.

Example

Nope.

dab version output

Dab Version: c28bd799a0d96baeeff7d5dc152257dd792ac54a
Server Version: 18.06.1-ce
Runtimes: runc
Kernel Version: 4.14.76
Operating System: NixOS 18.09.884.d96c7a35638 (Jellyfish)
Architecture: x86_64

Additional context

Nope.

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
Similar to --help and its variations displaying usage info, --version and its variants should display git hash the image was built from.

Describe alternatives you've considered
Could use some other versioning token however I believe Dab should stick with the rolling release model and a git commit aligns better with that, in my opinion.

Additional context
None.

Docker compose files already define all services and with the addition of a description label the configuration could be used to dynamically generate the tools and services ´list´ subcommands.

This will prevent list and compose being out of sync.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Make x509 PKI management easy.

Describe alternatives you've considered
either subcommands or lib functions.

Additional context
N/A.

Describe the bug

When the dab image has been updated dab update should pull the latest image but it does not appear to do that.

To Reproduce

Steps to reproduce the behavior:

1. Run dab update
2. See no update even if docker pull nekroze/dab:latest does update.

Expected behavior

Should pull the nekroze/dab:latest image.

Example

None.

dab version output

Dab Version: feb5648123bab3eed9e02763f99ef2f987fc5280
Server Version: 18.06.1-ce
Runtimes: runc
Kernel Version: 4.14.74
Operating System: NixOS 18.09.800.21293d2576a (Jellyfish)
Architecture: x86_64

Additional context

Must be an update available to test, that is why this is not a part of the tests at present.

Feature Request

Is your feature request related to a problem? Please describe

Updated dab but completion was not auto updated at the same time.

Describe the solution you'd like

Should do it at the same time as an image update or first usage of the new image.

Describe alternatives you've considered

None.

Additional context

None.

Feature Request

Is your feature request related to a problem? Please describe

No.

Describe the solution you'd like

Replace travis and circle ci with github actions.

Describe alternatives you've considered

Leaving as is.

Additional context

Still in limited beta at time of issue creation.

Feature Request

Is your feature request related to a problem? Please describe

When starting a group with many services and tools the update output can make it hard to find the url output.

Describe the solution you'd like

When you start a group, one of the earliest actions should be to update all services and tools that group depends on.

Describe alternatives you've considered

Could suppress update information on start but that would just hide problems.

Additional context

None.

Is your feature request related to a problem? Please describe.
When dab tools all destroy is executed the lab network will be destroyed as well.

Describe the solution you'd like
A new subcommand to handle the lab network in its own docker-compose file making lab external to the tools docker-compose.yml config.

Describe alternatives you've considered
Leaving it where it is will cause issues in the future.

Additional context
N/A.

Is your feature request related to a problem? Please describe.
N/A.

Describe the solution you'd like
Add services, like tools, provide standard service instances like postgresql, consul, etc.

Describe alternatives you've considered
N/A.

Additional context
Services should be a class of group deps started before repos.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Use Nix for building docker images to keep images small and deterministic.

Describe alternatives you've considered
Leaving as is, based on alpine is probably fine.

Additional context
https://grahamc.com/blog/nix-and-layered-docker-images

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Add ntopng as a tool. This should allow monitoring of all docker networks.

Describe alternatives you've considered
Nope.

Additional context
N/A

Feature Request

Is your feature request related to a problem? Please describe

Cannot run dab network recreate with any services running.

Describe the solution you'd like

Should disconnect all containers before the recreation of the network and reconnect them afterwards.

Describe alternatives you've considered

None.

Additional context

None.

Is your feature request related to a problem? Please describe.
N/A

Describe the solution you'd like
Add a docker registry configured for caching pass through proxy, then setup tools and provide facilities for easily using it similar to the lab network.

Describe alternatives you've considered
N/A.

Additional context
N/A.

Describe the bug
Docker hub image is not using .dockerignore it seems, causing image build failures.

To Reproduce
https://hub.docker.com/r/nekroze/dab/builds/

Expected behavior
should pass

Screenshots
N/A.

Desktop (please complete the following information):
N/A

Additional context
N/A.