Hello again nextPYP Developers,

I just wanted to leave another quick suggestion for how nextPYP is set up. From an operational perspective, it would likely be a lot easier to deal with the web app portion of nextPYP if it were containerized using Docker instead of Singularity. My reasoning on this point is that Docker is much better suited for web applications due to its support for network namespacing and its ability to manipulate iptables rules. Singularity, in contrast, looks to be much more oriented towards batch computing and non-interactive applications.

Additionally, Docker has a deployment/testing mechanism (docker-compose) that is a bit more intuitive and reproducible when deploying web applications. docker-compose codifies port mappings, storage volumes, etc using a YAML file and allows you to reliably bring up an application without needing to deal with an installer script, systemd units or the underlying container runtime.

I've seen docker-compose used with some success in SmartScope and I believe that it would work well here too.

--John

Hi nextPYP Team,

I'm a systems administrator at an institution (NYSBC/SEMC) that is interested in using nextPYP with some regularity. We recently performed a test installation of nextPYP and came across a few architectural decisions that make it particularly unwieldy from an operational perspective.

One particular issue that we have with the software design is the fact that SLURM jobs run as a service account instead of the user who actually submitted the job. The main problem with this setup is that SLURM comes with an accounting system that we can use to gather statistics about usage. This accounting system isn't just used for reporting purposes, however; the database where job metrics is kept also can influence scheduling decisions (via fairshare) and resource limits. When all users run jobs as the same service account, this effectively blurs usage patterns from all users together, and makes it impossible to use SLURM's accounting functionality in any meaningful way.

At a small institution such as NYSBC, not being able to fully leverage the accounting functionality would not be too big of a deal, but at larger institutions (universities, the NIH, etc) this would likely be a bit of a showstopper. For some individual labs within a larger university, a potential workaround could be to use an individual grad student/postdoc's account as the service account, but this would come with the caveat that that individual's fairshare factor would be affected and their jobs would be unjustly deprioritized by the activities of their colleagues.

Ideally, instead of using a service account researchers would be able to log in as themselves using the same identity provider as the SLURM cluster. In this way, the identity provider and authentication functionality (which currently seems to be handled by MongoDB entries) would be decoupled from authorization. One could, for instance use LDAP as an identity provider and store authorization information (i.e., which identities can access which projects) in the MongoDB database.

A concrete example of this can be seen in how JupyterHub currently handles authentication (see here), where there are multiple authenticators that can be used with different identity providers. Multiple authenticator classes is probably overkill for nextPYP, but at the same time I feel like it would be useful to at least be offload authentication onto PAM.

I don't know how difficult what I described would be to implement on your end, but would be happy to chat some more if you find this feedback useful (or alternatively, if you would like some clarification about what I'm describing / asking for).

--John

Hi again,

Just wanted to leave a quick bug report (that might be more of a documentation issue than a code issue). I noticed that by default the Caddyfile generated here uses Let's Encrypt / ACME to fetch an SSL certificate. However, ACME requires that Caddy perform a challenge to verify domain ownership. If the reverse proxy is not on a publicly accessible server both of the main challenge types out there will fail (i.e., HTTP-01 fails because Let's Encrypt can't reach the server; DNS-01 fails because the DNS record is internal-only).

One workaround would be to set the value of the tls environment variable to tls internal (either by adding export tls="tls internal" to /usr/bin/nextpyp-startrprox or by using a systemd unit file drop-in to alter the environment variables using systemd) to use a local CA. One could also modify the Caddyfile to use tls [<cert_file> <key_file> although this only would work if you had already had a certificate issued by a recognized CA (e.g., InCommon).

With tls internal, however, pyp jobs will generally fail with errors like this:

    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/envs/pyp/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/envs/pyp/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/usr/local/envs/pyp/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nextpyp.semc.nysbc.org', port=443): Max retries exceeded with url: /pyp (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

In this case, they'll only work if you use an insecure connection by updating config.toml to bypass the reverse proxy:

# URL of the web server, from a SLURM compute node's point of view
webhost = 'http://nextpyp.myinternaldomain.com:8080'

--John