AWS Batch runner does not support runtime-configurable Docker images about cli HOT 4 CLOSED

If I understand correctly:

• The job definition nextstrain-job would be the main definition for every job using the latest image.
• A new job definition should be created before the job submission if a new image is specified.

This raise a few questions:

• The naming convention of the job definition. Something like nextstrain-job_<image_tag> would probably work fine. For example nextstrain-job_branch-python-base.
• How should the accumulation of job definition be handled? Since it'd be only linked to an image, it's not clear how/when a job definition could/should be erased. After some time, you could have a lot of definitions created, but only a few used.

from cli.

Thanks for thinking about this, @jstoja! Here are my thoughts, would appreciate feedback.

The job definition nextstrain-job would be the main definition for every job using the latest image.

Yes, assuming the requested job/image are default as shipped. Any given build job should use the requested job description as-is it uses the requested image.

Note that the name of the job definition can be controlled by config, env, or option and may not be nextstrain-job.

Also note that the requested image name will be nextstrain/base (implicitly :latest) by the shipped default, but it can be set by config, env, or option to something else, like nextstrain/base:branch-python-base or even nextstrain/seasonal-flu:latest (hypothetical, doesn't exist).

A new job definition should be created before the job submission if a new image is specified.

Correct.

This raise a few questions:

The naming convention of the job definition. Something like nextstrain-job_<image_tag> would probably work fine. For example nextstrain-job_branch-python-base.

The image part will have to be the fully-specified name, not just the tag of nextstrain/base since that may not always be the image name. I think some suffixing scheme will be good enough, though. I'm not sure what characters AWS Batch allows in job definition names, but I'd prefer a less common delimiter char than underscore. Perhaps nextstrain-job|nextstrain/base:branch-python-base or nextstrain-job:nextstrain/seasonal-flu:latest or nextstrain-job~nextstrain/base:branch-python-base.

How should the accumulation of job definition be handled? Since it'd be only linked to an image, it's not clear how/when a job definition could/should be erased. After some time, you could have a lot of definitions created, but only a few used.

This is a reasonable concern, but I think for now this issue can be punted. Unless you have some clever ideas here?

from cli.

Happy to contribute, awesome project guys, the amount of work and the quality of the software is incredible!

I'm not sure what characters AWS Batch allows in job definition names, but I'd prefer a less common delimiter char than underscore.

Documentation lists explicitly hyphens and underscores:

jobDefinitionName
    The name of the job definition to register. Up to 128 letters (uppercase and lowercase), numbers, hyphens, and underscores are allowed.

I've tested with / and ~ in the AWS interface: /, :, | and ~. None of them work with the following message: Error executing request, Exception : Job definition name should match valid pattern.

Therefore, I would suggest:

• We use nexstrain-job_xxx
• xxx would be the docker image name and tag, but with the characters /, and _ replaced to -, and : with _.

With this proposal a job definition with image nextstrain/seasonal-flu:latest would have a job like: nextrain-job_nextstrain-seasonal-flu_latest. This allows to list jobs and split by _ to have the name of the job definition, the image and the tag (if we agree that there shouldn't be _ in the job definition name).

This is a reasonable concern, but I think for now this issue can be punted. Unless you have some clever ideas here?

No I don't.

from cli.

Documentation lists explicitly hyphens and underscores:

Ah, well, underscores it is! I think your proposal for substitutions makes sense.

from cli.