It seems that when commands such as rsync error, the output is captured in the logs as a blob of data. It would be useful to be able to see this output in the logs.

dacd: [754B blob data]

Job 556 references a file that does not exist for stage in, and 557 references a file that does not exist for stage out.

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               556     debug use-mult    test1 SO       0:10      1 (burst_buffer/datawarp: dws_data_out: exit status 23
)
               557     debug use-mult    test1 SO       0:09      1 (burst_buffer/datawarp: dws_data_out: exit status 23
)

scancel 556 and 557 does nothing. The buffers also hang around but that is probably expected:

JobID=557 CreateTime=2019-10-02T10:10:38 Pool=default Size=3200GiB State=staged-in UserID=test1(1001)
JobID=556 CreateTime=2019-10-02T10:07:44 Pool=default Size=3200GiB State=staged-in UserID=test1(1001)

Its possible two buffers try to claim the same bricks, we correctly go into the error state for the looser, and the transaction to claim the bricks fails.

However, when we try to clean up this buffer, we wait for ever stuck in the DeleteRequested state, because we never reached the bricks assigned state.

Possible fix ideas: we should track if bricks were ever assigned, and don't wait for dacd when deleting a volume that never got its bricks ever assigned.

When a node needs to be remove from service there is currently no formal mechanism for removing keys and state from etcd.

bricks get deleted, but volumes are still present

repeating teardown fails, because dacd is no longer watching that volume any more

idea: now we stop watching things when we hit Error state, does this break the dacd event watching when we hit errors??

It looks like I can hit a 5 minute timeout after about 1 minute (assuming it is waiting for the mount to complete and not from the start of the session; in that case about 2 minutes). Here is some relevant information from the logs:

Jan 10 09:50:31 dac-e-16 dacd[8349]: starting action {Uuid:2fd7d0cb-94ff-4794-b8ce-dc1d3aa72365 Session:{Name:6153 [snip]
Jan 10 09:51:25 dac-e-16 dacd[8349]: Mount for: 6153
Jan 10 09:51:25 dac-e-16 dacd[8349]: Mounting 6153 on host: dac-e-16 for session: 6153
Jan 10 09:51:25 dac-e-16 dacd[8349]: SSH to: dac-e-16 with command: mkdir -p /mnt/dac/6153_job
Jan 10 09:51:25 dac-e-16 dacd[8349]: Completed remote ssh run: mkdir -p /mnt/dac/6153_job
Jan 10 09:51:25 dac-e-16 dacd[8349]: SSH to: dac-e-16 with command: mount -t lustre -o flock,nodev,nosuid dac-e-16-opa@o2ib1:/xpgSIqqB /mnt/dac/6153_job
Jan 10 09:52:25 dac-e-16 dacd[8349]: Time up, waited more than 5 mins to complete.
Jan 10 09:52:25 dac-e-16 dacd[8349]: Error in remote ssh run: 'mount -t lustre -o flock,nodev,nosuid dac-e-16-opa@o2ib1:/xpgSIqqB /mnt/dac/6153_job' error: signal: killed

Balance load between the NVMe drives, rather than just picking at random.

Likely should update drive health on restart of dacd and after delete of buffers. Need to look into secure erase / reset / discard at delete of buffer, to make sure Lustre does the correct thing.

when we recompiled dacd to test all names on the DAC, previous nvme0n1 was not updated or removed. it should also be noted that dacd currently treats nvme0n1 as the mgt and the mdt in lustre

Using access_mode=private results in no buffer being created.

The $DW_JOB_PRIVATE environment variable is not set at least.

Is this expected or should one be able to request a private buffer without stripped?

We occasionally hit an error "burst_buffer/datawarp: setup: error during ansible create: exit status 2".

Looking at the dacd journal, I see: mkfs.lustre FATAL: loop device requires a --device-size= param

TASK [lustre : Reformat MDTs] **************************************************
failed: [dac-e-12] (item={'key': u'nvme7n1', 'value': 3}) => 
{
  "ansible_loop_var": "item",
  "changed": true,
  "cmd": [
    "/usr/sbin/mkfs.lustre",
    "--mdt",
    "--reformat",
    "--fsname=hOvicamu",
    "--index=3",
    "--mgsnode=dac-e-20-opa@o2ib1",
    "/dev/nvme7n1p1"
  ],
  "delta": "0:00:00.004677",
  "end": "2020-01-09 13:37:15.215476",
  "item": {
    "key": "nvme7n1",
    "value": 3
  },
  "msg": "non-zero return code",
  "rc": 22,
  "start": "2020-01-09 13:37:15.210799",
  "stderr": "\nmkfs.lustre FATAL: loop device requires a --device-size= param\nmkfs.lustre: exiting with 22 (Invalid argument)",
  "stderr_lines": [
    "",
    "mkfs.lustre FATAL: loop device requires a --device-size= param",
    "mkfs.lustre: exiting with 22 (Invalid argument)"
  ],
  "stdout": "\n   Permanent disk data:\nTarget:     hOvicamu:MDT0003\nIndex:      3\nLustre FS:  hOvicamu\nMount type: ldiskfs\nFlags:      0x61\n              (MDT first_time update )\nPersistent mount opts: user_xattr,errors=remount-ro\nParameters: mgsnode=10.47.18.20@o2ib1",
  "stdout_lines": [
    "",
    "   Permanent disk data:",
    "Target:     hOvicamu:MDT0003",
    "Index:      3",
    "Lustre FS:  hOvicamu",
    "Mount type: ldiskfs",
    "Flags:      0x61",
    "              (MDT first_time update )",
    "Persistent mount opts: user_xattr,errors=remount-ro",
    "Parameters: mgsnode=10.47.18.20@o2ib1"
  ]
}

we need to be careful when detaching multi-job volumes, when attached by two jobs.

Currently slurm picks up the #DW Swap and add that as additional storage request, and also rounds it up to the nearest granularity, and multiples by the max number of nodes.

Not sure we actually want it to work like how Slurm has it right now.

ansible has a drama with loop: {{ foo.keys()}} this can be changed to with_dict: {{ foo }} and {{ item.key}}

further investigation is required

Ignore error 17 on format existing msg

There is a hardcoded timeout of 5 minutes:

We hit this timeout occasionally so it would be nice to give it a bit more time.

Thanks

If you name a persistent burst buffer with an underscore it causes creation to fail, but not only that, running an squeue for example prints out a huge error, the top of which I've pasted here but this is only a snippet:

(burst_buffer/datawarp: _create_persistent: panic: invalid session name: 'test_1' [recovered]
        panic: invalid session name: 'test_1'

goroutine 1 [running]:
main.main.func1()
        /home/circleci/data-acc/cmd/dacctl/main.go:187 +0xb9
panic(0xaccca0, 0xc0002c4060)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
log.Panicf(0xbf1b17, 0x1a, 0xc0001546c0, 0x1, 0x1)
        /usr/local/go/src/log/log.go:340 +0xc0
github.com/RSE-Cambridge/data-acc/internal/pkg/registry_impl.getSessionKey(0x7ffccbdd0e74, 0x7, 0x42c64f, 0x8)
[snip]

Not allowing underscores is fine (unless this is an easy fix) however maybe the DAC should handle this issue a little better?

Needs investigation and log files from dacd, but @sjr20 has repored issues creating swap files greater than 7GB

The existing bug was creating problems when tidying up after the above failure:
#15

as this retry is unlimited it can mean that /tmp will be filed with fs_{job} directories and files. if /tmp was to small this will fill up the disk.

Currently we do the mount in a serial loop of ssh requests. This gets very slow with large numbers of compute nodes.

We should move to ansible to do the mount on both clients and servers. That way we have a single way to manage concurrent ssh connections.

Firstly need to understand the info / find some docs, then work out the equivelent in our system, then implement it.

I am seeing this issue on the compute nodes.

Also after unmounting we should issue an "rm -df" on the mountpoint to tidy up that directory without recursing into it (which should not be necessary). This also goes for persistent buffers; they are currently unmounted but not removed.

I know you are working on this but here's a placeholder to track the issue.

Buffers are currently being mounted under /mnt/lustre/ on the DAC nodes but /dac/ on the compute nodes. Let's consolidate them to be /mnt/dac/. This will help with staging data in and out.

When adding hosts to new pools, the pool resources are not separated. For example the TotalSpace for the pools are the same, and requesting one pool spills over and uses hosts from another pool.

We can use 99% (technically 100% minus one unit of granularity) of available space however when we create a buffer to consume the remaining space, it looks to be created successfully but then squeue reports:

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                20     debug create-p    test2 PD       0:00      1 (burst_buffer/datawarp: setup: panic: runtime error: index out of range [recovered]
        panic: runtime error: index out of range

goroutine 1 [running]:
main.main.func1()
        /home/circleci/data-acc/cmd/dacctl/main.go:187 +0xb9
panic(0xb0d7c0, 0x12394c0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/RSE-Cambridge/data-acc/internal/pkg/dacctl/workflow_impl.sessionFacade.doAllocationAndWriteSession(0xce3440, 0xc000176d00, 0xcdf780, 0xc000179560, 0xce0e80, 0xc00017c750, 0xcccf
60, 0x126c0c8, 0x7ffff8268d30, 0x2, ...)
        /home/circleci/data-acc/internal/pkg/dacctl/workflow_impl/session.go:166 +0x6ee
github.com/RSE-Cambridge/data-acc/internal/pkg/dacctl/workflow_impl.sessionFacade.CreateSession.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/circleci/data-acc/internal/pkg/dacctl/workflow_impl/session.go:110 +0xfb
github.com/RSE-Cambridge/data-acc/internal/pkg/dacctl/workflow_impl.sessionFacade.submitJob(0xce3440, 0xc000176d00, 0xcdf780, 0xc000179560, 0xce0e80, 0xc00017c750, 0xcccf60, 0x126c0c8, 0x7
ffff8268d30, 0x2, ...)
        /home/circleci/data-acc/internal/pkg/dacctl/workflow_impl/session.go:53 +0x3e8
github.com/RSE-Cambridge/data-acc/internal/pkg/dacctl/workflow_impl.sessionFacade.CreateSession(0xce3440, 0xc000176d00, 0xcdf780, 0xc000179560, 0xce0e80, 0xc00017c750, 0xcccf60, 0x126c0c8,
 0x7ffff8268d30, 0x2, ...)
        /home/circleci/data-acc/internal/pkg/dacctl/workflow_impl/session.go:107 +0x1b4
github.com/RSE-Cambridge/data-acc/internal/pkg/dacctl/actions_impl.(*dacctlActions).CreatePerJobBuffer(0xc000179580, 0xcdcac0, 0xc0000e71e0, 0xc000179580, 0x0)
        /home/circleci/data-acc/internal/pkg/dacctl/actions_impl/job.go:93 +0x6f5
main.setup(0xc0000e71e0, 0x0, 0x0)
        /home/circleci/data-acc/cmd/dacctl/actions.go:92 +0xa3
github.com/urfave/cli.HandleAction(0xade2a0, 0xc15140, 0xc0000e71e0, 0xc0000e71e0, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:514 +0xbe
github.com/urfave/cli.Command.Run(0xbe38a2, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0a8db, 0x47, 0x0, ...)
        /go/pkg/mod/github.com/urfave/[email protected]/command.go:171 +0x4d2
github.com/urfave/cli.(*App).Run(0xc0000dc540, 0xc0000b4000, 0xe, 0xf, 0x0, 0x0)
        /go/pkg/mod/github.com/urfave/[email protected]/app.go:265 +0x733
main.runCli(0xc0000b4000, 0xf, 0xf, 0xbe21f8, 0x1)
        /home/circleci/data-acc/cmd/dacctl/main.go:172 +0x1255
main.main()
        /home/circleci/data-acc/cmd/dacctl/main.go:194 +0x1f1
)

At this point everything looks normal with the buffers with now 0 FreeSpace as expected:

Name=datawarp DefaultPool=default Granularity=1600GiB TotalSpace=24000GiB FreeSpace=0 UsedSpace=24000GiB
  Flags=EnablePersistent,PrivateData
  StageInTimeout=3600 StageOutTimeout=3600 ValidateTimeout=5 OtherTimeout=3600
  GetSysState=/usr/local/bin/dacctl
  GetSysStatus=/usr/local/bin/dacctl
  Allocated Buffers:
    Name=small CreateTime=2019-10-02T14:24:55 Pool=default Size=3200GiB State=allocated UserID=test2(1002)
    Name=full CreateTime=2019-10-02T14:23:14 Pool=default Size=20800GiB State=allocated UserID=test2(1002)
  Per User Buffer Use:
    UserID=test2(1002) Used=24000GiB

$DW_JOB_STRIPED includes the "global" directory in the path.
$DW_PERSISTENT_jobname does not, so users must specify $DW_PERSISTENT_jobname/global when using persistent buffers.

Can we make the behaviour of these consistent?

It would be useful to provide a prefix to stage in and out requests so that we can have storage mounted in a configurable location.

Paths that the user requests would be resolved first (with realpath or similar) to strip ../ and expand ~ and $HOME, and then prefixed with the configurable path.

probably should be a force

Sometimes dacd never transitions to the BricksAllocated state, i.e. it doesn't notice its new volume appear.

This has happened twice. We do at least now delete things correctly if you do an teardown via scancel.

It's useful to have the option to retain all ansible playbooks created for later debugging/review. Currently the method captures it along with the .venv and binaries which can fill the filesystem.
Using ansible tempdir: /tmp/fsPlQCMRnH_830051621

Perhaps call it DAC_RETAIN_INVENTORY

It looks like a gid is assumed to be the same as a uid and so my persistent buffers have group rwx given to the wrong group. This could also lead to the chown failing if the gid doesn't exist.

Per-job buffers are unaffected. You can see below the gid is, and should be different to the uid but it incorrectly matches on the persistent buffer.

Example user:

$ id
uid=17121 gid=17124

Buffers:

/dac/2307_job:
total 4
drwxrwx--- 2 17121 17124 4096 Aug  7 12:19 global

/dac/2307_persistent_jsteel:
total 4
drwxrwx--- 2 17121 17121 4096 Aug  7 10:17 global

Running stat $DW_JOB_PRIVATE inside a job that requests a private buffer, shows that the permissions are set to 777. It seems that shortly after these are changed to 700 but I think we should use "install" to create the directory with 700 permissions from the start, as they are not fixed before the user could start writing data there (eg my stat command runs and notices the issue).

• This repo is currently unmaintained.
• Dependabot is flagging dependancies with security updates

The proposal is:

• Update README with an obvious notice about the status
• Archive the repository in GitHub

On a per-job burst buffer, requesting for example 2.8TB (or any fractional value) causes the per-job buffer not to be created and therefore not mounted for the running job.

On a persistent burst buffer, requesting for example 2.8TB (or any fractional value) always gives 1.4TB.

Need to have a list of devices that may be broken, so we just advertise the working ones.

We could do automated checks, but the admin may want to swap out a disk using this system.

Specifying something like:

#DW stage_out source=$DW_PERSISTENT_STRIPED_mybuffer/myfile destination=[...snip...]

Attempts to copy /myfile so it looks like that environment variable is not set. That environment variable can be used in the job successfully, just not in stage in or out definitions.

Note that I can successfully use the $DW_JOB_STRIPED environment variable for per-job buffers in the stage in and out definitions so I was expecting the persistent one to also work in the same way.

@sjr20 has reported issues where request just swap and no per job buffer means the swap isn't added.

Slighly unsure on the expected behaviour, but it seems likely the swap should work as expected.

On the DAC nodes, The /mnt/dac/$JOBID_job_private symlink does not get created, I guess this is because it links to a compute hostname.

I guess one would want to copy to private buffers to give every compute node the same data but its own copy. I would expect it to copy to /mnt/dac/$JOBID_job/private/*/ and copy out would create a directory tree eg copying out $DW_JOB_PRIVATE/file would copy out for eg:

private/node1/file
private/node2/file

But how feasible is this or do we not support copying into and out of private buffers?

5567c09 and 8c02d2f were reverted to fix some issues with failed clean ups. Needs more work.

We should be able to create a working ansible directory to do ad hoc mantainance / debugging on any buffer.

Once added we can remove the /tmp/fs_* directories on failures. This stops /tmp filling up the local disk.

We are seeing an issue occasionally where a job will hit an error "Burst buffer pre_run error". I see this in the dacd journal:

Jan 09 13:38:07 dac-e-19 dacd[57485]: SSH to: cpu-e-335 with command: mount -t lustre -o flock,nodev,nosuid dac-e-19-opa@o2ib1:/RTsjlttY /mnt/dac/6038_job
Jan 09 13:38:08 dac-e-19 dacd[57485]: Error in remote ssh run: 'mount -t lustre -o flock,nodev,nosuid dac-e-19-opa@o2ib1:/RTsjlttY /mnt/dac/6038_job' error: exit status 16
Jan 09 13:38:08 dac-e-19 dacd[57485]: Warning: Permanently added 'cpu-e-335,10.43.1.79' (ECDSA) to the list of known hosts.
                                      mount.lustre: mount dac-e-19-opa@o2ib1:/RTsjlttY at /mnt/dac/6038_job failed: Device or resource busy
                                      Is the backend filesystem mounted?
                                      Check /etc/mtab and /proc/mounts

This might be due to the compute node mounting other file systems at the same time (at least that is how I can replicate the issue). In any case, should we add a retry, or put in a locking mechanism here to avoid this bombing out?