hpc / hpc-collab Goto Github PK

in its most basic form, an empty OverlayFS would only include patches

ip command needs to be loaded vi brew on MacOS X, for example

Alternatives:

• push & pull from external source via one/all of: git, nfs, sshfs
• NFS or other external storage for initial /vagrant (/) rather than vboxsf (also beneficial for I/O speed)
  NFS may require external host privileges. :-(

Templates of interest:

1. minimal cluster, similar to a stripped down "vc"
2. use-case: (re)create use problem found on a production cluster, so that the packaged hpc-collab cluster may be used to a) demonstrate the problem, b) (in)validate the fix for the problem, c) provide archival artifacts, potentially serving as points of integration into larger integration & validation test suites
3. minimal cluster,

• illustrating slurm configuration comparison scenarios, ex. configless slurm, [alternate backend data bases]
  (https://bugs.schedmd.com/show_bug.cgi?id=8537)
• comparing alternative configuration options (ex. FairShare vs Fairtree, with a queue loading generator)

4. clusters with login front-ends in different security zones

This issue arose from a discussion regarding the USRC Resilience team's requirements for fault detection and signature generation.

It may be a justification for a lighterweight underlying provider, such as libvirt/ovirt or docker/container.

This would allow node configuration changes to be tested in a CI/CD pipeline, self-validating an hpc-collab node recipe & configuration change.

Another alternative: , returning node back to an earlier full state
Perhaps an underlying vagrant snapshot/suspend mechanism could be utilized, provided:
there are sufficient hooks to trigger a mini-provision, consisting mostly of verification, and
there is a mechanism to do multi-machine snapshot/resume in the preserving dependencies.

Alternatives could be:

• separate external 2nd disk, possibly shared between vcdb and vcaltdb
• storage provided in-cluster from vcfs
• external iscsi mount

• Issue #1/AddNodeCfg
• Issue #2/Refactor ... common
• Issue #22/External source of truth cfg mgmt
• Makefile/Autoconf/CMake/Ansible-glue to generate Vagrantfile for a meta-make cluster recipe (using templates)
  • constructs other shared views into common data such as:
    /etc/ethers, /etc/networks, /etc/hosts and Vagrantfile IP and MAC addresses dynamically
  • Use this instantiation of a file system import into the cluster as a consistent set checkout to interface with an external configuration management system
    populate user data, slurmdb, full node config? from external source of truth, ex. HESIOD, or vice versao, populate external directory entry form local recipe (alternate option) file system configuration

• execution scripts move into per-/
• data files move into common hierarchy, per-cluster where necessary, per-global-common wherever possible (ex. /etc/hosts)

Issue #2, Issue #22, Issue #52

• Makefile target rules (build, install, configure, verify) rather than in provision.sh driver
• precise ordering
  • precisely specify order of operations of service hierarchy in sub-directories of {build,config,install,verify}
  • including the {inc,lib}/*.sh headers which should not be loaded in alphabetic order (ex. dynamic before cfgfs)
    each directory contains a "cursor", "next" and a "requires"

This issue arose from the shared system (muon) deployment.

This request came about in preparation for a student joining the project next week (18 May)

Current process is as follows:

1. Add an entry into clusters/<cluster-id>/common/etc/{hosts,ethers} for the new host picking a unique
   IP and MAC address
2. Put in a corresponding entry into clusters/<cluster-id>/Vagrantfile.<cluster-id>.nodes
3. Add an entry into clusters/<cluster-id>/common/etc/slurm.conf, for the node itself and whichever partition(s) are appropriate.
4. Update clusters/<cluster-id>/cfg/<new-node>/requires/<pre-requisite-nodes>/<pre-requisite-svcs> up

see also issue #20
provisioning extension that just instantiates this hierarchy for the invoking user dynamically ("make clean" removes)

This issue arose during the muon (USRC) deployment.

nodename aliases vary depending upon whether the node has been provisioned correctly. If the node is not fully provisioned, the alias maps to 'make '. If the node is fully provisioned, the alias maps to 'ssh -o UserKnownHostsFile=/dev/null '

The ideal provider would be able to be run within an existing HPC job allocation, on an existing, unmodified HPC cluster.

Use case: on-going in-pipeline validation of upcoming, scheduled HPC changes

• node state diagram
• the annotated cfg directory structure
• node dependency diagram (using sdedit.jar?)
  make -Bnd | make2graph | dot -Tsvg -o out.svg
  show dependencies: https://unix.stackexchange.com/questions/283478/how-to-display-dependencies-given-in-a-makefi
  le-as-a-tree
  https://plindenbaum.blogspot.com/2012/11/visualizing-dependencies-of-makefile.html
  https://github.com/lindenb/makefile2graph
• self-documenting Makefile rules

complexity when this is (mostly) preformed in the requires/* stage, before most services DNS, NFS

This issue arose out of the muon (USRC) deployment.

record machine's GUID into the state file
If the GUID doesn't exist, invalidate that state file
If a machine with that name and GUID exists, report EX_ALREADY or EX_OK (context defines)
If a machine with that name and a different GUID exists, EX_CANTCREAT, EX_TEMPFAIL, EX_ALREADY

implies deployment of an in-cluster configuration management system, such as ansible
interim step until node (re)construction is truly lightweight

• RHEL variant: TOSS, mostly as a proof of concept
• SuSE Linux
• SuSE/Cray Linux Environment -- shasta variant

interesting but below the line:

• minmal linux, such as Alpine
• *BSD such as FreeBSD OpenBSD NetBSD

See also issue #52

This should improve rpm loading timing, therefore reduce node rebuild time.
Needs to leave a usable hierarchy whether or not local repos, remote repos are flagged.

possibly combine cfg//rpms/early/... into a common (cluster-external) early-localrepo.repo

non-relocatable paths in Vagrantfile, generating errors such as the following if 'vagrant' is not invoked from the directory where Vagrantfile is located:

      % vagrant ssh vcfs                                                                   
      nodes_file_path:/home/sts/hpc-collab/clusters/vc/cfg/vcfs/verify/firewall/Vagrantfile.firewall.nodes does not exist