to not make so big jumps between version. big jumps always come with big changes in the code. and with a fast update cycle we can prevent this.

@tuxmea are you ok with this?

• HW/SW requirements
• Installation
• Configuration

get all environments from puppetdb even if no node is running against it

• assets precompile: bundle exec rails assets:precompile
• ENV: RAILS_ENV=production
• Puppet-HDM Module:
  • database.yml
  • production.sqlite3

When building on a tag: the tag is available as latest
When building on main, this build should not be tagged as latest, but as development (maybe also main).

• we want to got to Rails 7.0 (?)
• we want to update to ruby 2.7.x at least

when i bundle without development, the app does not start.
but if i understand this right, from concept, development group shouldn't be necessary for a production run?

hdm  | rails aborted!
hdm  | LoadError: cannot load such file -- listen
hdm  | /hdm/config/environments/development.rb:75:in `block in <top (required)>'
hdm  | /hdm/config/environments/development.rb:3:in `<top (required)>'
hdm  | /hdm/config/environment.rb:5:in `<top (required)>'
hdm  | Tasks: TOP => db:create => db:load_config => environment
hdm  | (See full trace by running task with --trace)

to fix this, i moved the listen gem out of the development group.

Usually we have short living feature branch based environments.

Consider that someone creates the feature branch, runs a single node against the feature branch, and merges his change.
In this case the code deployment will remove the feature branch environment from file system, PuppetDB will get updated, when Puppet agent is running the next time.

Within HDM:

1. we get a list of environments from PuppetDB
2. we select an environment and query PuppetDB for nodes

Desired solution:

1. HDM queries PuppetDB for environments
2. prior showing the list HDM checks if any of the returned environments is no longer available on file system and marks the unavailable environments as non-selectable in the drop-down menu

switch to ruby-slim to be debian based and so solve the problem with alpine and dart-lang/sdk#40906

so we then should be able to run the precompile of our assets

bundle exec rake assets:precompile

When running browser in fullscreen, the content is centered and uses maybe 50% of window width.

HDM should use 80% of window width if possible for the data content.
It is ok to keep the hiera key width as is.

Screen recording to show installation, configuration, usage.

Should HDM show that access to the key is denied?
Should HDM show nothing?

container build is atm on ruby 3.1.3, we want to update this to 3.2.1 or newer.

We first wait for the logo, then we can adopt the colour scheme.

Some thoughts about a CI/CD strategy that could be implemented. This is based on my experience with vox-pupuli-tasks (also a Rails application) and the different gems Vox Pupuli hosts:

Enable dependabot alerts

• GitHub will scan the Gemfile.lock for potential updates and CVEs in dependencies daily

Define a GitHub action that runs on pull requests:

• Define the lowest Rubyversion we support
• Run rubocop with that Rubyversion as target
• Define a list of Ruby versions we want to support + latest ruby
• Use that as a matrix where we run the following for each ruby version
• Run the existing rspec tests in a matrix on each Ruby version
  • On one Ruby version, run codecov for coverage reporting
  • Run brakeman for Rails specific reporting
• build the application (generating assets...)
• Start the application and validate some http endpoints

Define a GitHub action that runs on merges

• Apply the action from above
• Build a docker image
  • publish it at least to github container registry
  • Tag the image with the commit sha
  • Tag it also with latest or testing
  • Start the application via docker-compose with the new Image+dependencies
• build at least rpms+debs and upload them

Define a GitHub action that runs on tags

• same as above, but tag packages/images with the name of the pushed tag, not a commit sha

Additional Resources

• https://github.com/voxpupuli/vox-pupuli-tasks/blob/master/.github/workflows/test-build.yml
• https://github.com/voxpupuli/vox-pupuli-tasks/blob/master/.github/workflows/test-build-release.yml
• https://github.com/voxpupuli/beaker/blob/master/.github/workflows/test.yml
• voxpupuli/vox-pupuli-tasks@32e8c7d

• final color scheme
• logo
• license

something like: bundle exec rake db:admin:reset:<username> should be possible.
Return value should be a new random password

We currently have a bunch of rubocop violations listed in https://github.com/betadots/hdm/blob/main/.rubocop_todo.yml. This needs to be fixed or we need to disable cops we do not like.

The following data produce no error in hiera:

---
# comment

HMD YAML parser complains about NIL.

When adding the curly braces, HDM will not complain:

--- {}
# comment

Accept NIL data and continue to follow Hiera behavior

is datadir is not set in hiera.yaml, one gets an error in the intrface about datadir being empty/nil.

datadir is not mandatory in hiera.yaml, if not set, it is assumed to be "data".

https://github.com/betadots/hdm/security/dependabot/77

 Dependabot cannot update activerecord to a non-vulnerable version

The latest possible version that can be installed is 7.0.2.4 because of the following conflicting dependencies:

rails (7.0.2.4) requires activerecord (= 7.0.2.4) via actionmailbox (7.0.2.4)
rails (7.0.2.4) requires activerecord (= 7.0.2.4) via actiontext (7.0.2.4)
rails (7.0.2.4) requires activerecord (= 7.0.2.4) via activestorage (7.0.2.4)
rails (7.0.2.4) requires activerecord (= 7.0.2.4)

The earliest fixed version is 7.0.3.1.

Static public error pages.
No need to be 100% identical to HDM web UI.
More important: where should a user look for data errors.

HTTP error codes:

• 422 - rails default for API results (unprocessible entity)
• 404 - not found - e.g. typo in URL
  • short message: check the URL
  • home button
• 500 - internal server error
  • NIL yaml data
  • link to issue on github
  • hint to check logfile
  • home button
• 403 - no auth, not privileged/authorized
  • hint: not logged in, user gets admin URL, general permissions
  • home button

Open Source. Check dependencies.

SAML implementation?
Check possible options.

A hierarchy may make use of an absolute path.
We should verify if HDM can deal with this.

We are considering the idea to let HDM run in a container without authentication for local (dev workstation) usage.

General question:

is it possible to easily disable authentication?

Was following old links from the Puppet blog and it may be a good idea just to update the readme on https://github.com/example42/hdm to direct here and archive it to avoid HDM looking like its not maintained.

Idea is to have an API endpoint where we can query for results.

e.g.

POST http://<server>:<port>/api/v1/data?env=production&node=node.domain.tld&key=classes or
http://<server>:3000/environments/production/nodes/puppet.matest.betadots.training/keys/classes?search=&format=json

Result (needs review and optimization)

{
  'yaml hierarchy': [
    {
      'nodes/puppet.yaml': 'RESULT'
    }
  ],
  'defaults': [
    {
      'defaults/auditbeat.yaml': 'RESULT'
    },
    {
      'defaults/filebeat.yaml': 'RESULT'
    },
  ]
  'common': [
    {
      'common.yaml': 'RESULT'
    }
  ]
}

RAILS is capable to provide different data format based on URL.
Do we want users to have a specific role to access API.
Basic Authentication is possible.

divide objects and export hiera parsing into an own library.

• setup pe main server
• setup pe puppetdb
• add sample data/control_repo
• use vagrant (?)
• use docker(?)

when installing a new hdm instance and having no users, one gets redirected to http to create a new user. but connection was initialized from https. we schould redirect to the same protocol as the initial reuqest

This is OK for read/write mode.
In read-only mode, HDM should only show hierarchy files with content.

At the moment we can enter data in the LDAP Login UI and receive an error message.

• webpack
  • @rails/weppacker needs 4.x of webpacker
• bootstrap
  • needs version of ^4 something

In hiera.yaml file one can specify a fact in multiple ways.

1. Top Scope Variable (Using :: syntax like %{::customer})
2. Facts Hash (Using facts. syntax like %{facts.customer})

At the moment HDM only recognizes option 1.
HDM should be able to also understand and parse the option 2.

Some people might use %{::environment} in Hiera.yaml.
This is not a fact but should have the content from the selected environment.

Provide an example hiera.yaml file which uses multiple hierarchies, where single path elements might be take from relative and absolute paths.
This example should provide information about a hierarchy structure which uses data from hiera data (maybe control-repo data) and data from another git repo which overwrites data from control-repo.

Good starting point: https://github.com/tuxmea/hdm/pull/25#issuecomment-920749909

Some customers may only use web-interfaces which follow a "barrier free" concept.

Possible steps:

• black/white UI - better: contrast.
• ALT Naming to any field/button?
• Red/Green should not be paired.

hdm  | rails aborted!
hdm  | ExecJS::RuntimeUnavailable: Could not find a JavaScript runtime. See https://github.com/rails/execjs for a list of available runtimes.
hdm  | /usr/local/bundle/gems/execjs-2.8.1/lib/execjs/runtimes.rb:58:in `autodetect'
hdm  | /usr/local/bundle/gems/execjs-2.8.1/lib/execjs.rb:5:in `<module:ExecJS>'
hdm  | /usr/local/bundle/gems/execjs-2.8.1/lib/execjs.rb:4:in `<top (required)>'
hdm  | /usr/local/bundle/gems/autoprefixer-rails-10.4.7.0/lib/autoprefixer-rails/processor.rb:4:in `require'
hdm  | /usr/local/bundle/gems/autoprefixer-rails-10.4.7.0/lib/autoprefixer-rails/processor.rb:4:in `<top (required)>'
hdm  | /usr/local/bundle/gems/autoprefixer-rails-10.4.7.0/lib/autoprefixer-rails.rb:39:in `require_relative'
hdm  | /usr/local/bundle/gems/autoprefixer-rails-10.4.7.0/lib/autoprefixer-rails.rb:39:in `<top (required)>'
hdm  | /usr/local/bundle/gems/bootstrap-4.6.1/lib/bootstrap/engine.rb:3:in `require'
hdm  | /usr/local/bundle/gems/bootstrap-4.6.1/lib/bootstrap/engine.rb:3:in `<top (required)>'
hdm  | /usr/local/bundle/gems/bootstrap-4.6.1/lib/bootstrap.rb:61:in `require'
hdm  | /usr/local/bundle/gems/bootstrap-4.6.1/lib/bootstrap.rb:61:in `register_rails_engine'
hdm  | /usr/local/bundle/gems/bootstrap-4.6.1/lib/bootstrap.rb:11:in `load!'
hdm  | /usr/local/bundle/gems/bootstrap-4.6.1/lib/bootstrap.rb:75:in `<top (required)>'
hdm  | /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/runtime.rb:60:in `require'
hdm  | /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
hdm  | /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/runtime.rb:55:in `each'
hdm  | /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/runtime.rb:55:in `block in require'
hdm  | /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/runtime.rb:44:in `each'
hdm  | /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/runtime.rb:44:in `require'
hdm  | /usr/local/bundle/gems/bundler-2.3.11/lib/bundler.rb:176:in `require'
hdm  | /hdm/config/application.rb:7:in `<top (required)>'
hdm  | /hdm/Rakefile:14:in `require_relative'
hdm  | /hdm/Rakefile:14:in `<top (required)>'
hdm  | /usr/local/bundle/gems/railties-7.0.3/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'
hdm  | /usr/local/bundle/gems/railties-7.0.3/lib/rails/commands/rake/rake_command.rb:18:in `perform'
hdm  | /usr/local/bundle/gems/railties-7.0.3/lib/rails/command.rb:51:in `invoke'
hdm  | /usr/local/bundle/gems/railties-7.0.3/lib/rails/commands.rb:18:in `<top (required)>'
hdm  | bin/rails:4:in `require'
hdm  | bin/rails:4:in `<main>'
hdm  | (See full trace by running task with --trace)
…

i fixed this with adding

gem 'mini_racer' # minimal Google V8 JS engine for execjs

or is it better to install nodejs into the container?

Proposal:

• If LDAP is enabled, only LDAP logins shoud be used.
• Only Admin will remain local user

Maybe remove the possibility to switch (local login, ldap login)?

We decided to have the Open Source Version only.
Any code which is in hdm-pro can be merged into this repo.

This should only be the RBAC part.

Check possibilities for a HDM-Foreman integration.
Design pattern insist that Foreman is talking to a Smart-Proxy only.

We might need:

• HDM API
• HDM Smart-Proxy
• HDM Foreman Web UI

Foreman HDM WebUI queries information from HDM Smart-Proxy.
HDM Smart-Proxy queries information from HDM API.

Foreman HDM Web UI should display in the same way as we do it in HDM.
User authentication is done by Foreman.

We have a basic systemd unit file at https://github.com/betadots/hdm/pull/40/files#diff-6a4ba7e2b78ee8953da5086899d9ba08d3cdb26164e9b4ecf7d5aa87fe665438

while this seems to work, we should implement some hardening. systemd provides many options for that.

Following the default and taking the latest HDM module and latest Docker module and including HDM on primary. When it tries to start the container it gets

Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: sethostname: invalid argument: unknown
Error: failed to start containers: hdm

As we now have removed NodeJS and Yarn, we must update the documentation and the Docker container build.
We also must update the documentatoin in terms of development vs production.

The production env must use precompiled assets.
On development assets are compiled on demand.

At the moment we can search for environments -> nodes -> hiera keys.

Users want to see in which environment, hierarchies and files a hiera key is made available.

This feature should allow users to select an environment and then search for a key.

The webinterface should show the hiera.yaml hierarchies and mark them if the key is available in an hierarchy (bold, like with the node data).

When klicking on a hierachy, a list with all files should be listed which contain the key.
Files without the key should not be shown.
The value should be shown for each file when klicking on the file name.

View only mode!