tazend / me Goto Github PK

@tazend, many thanks for this and the ingenious bug-finding....

First of all, my test and CORE clusters are set up completely identically. The only difference between the two is that the test cluster consists of real machines and the CORE cluster of cloud instances.

Whether one can really ignore this is really the question here, which I am still trying to answer through error analysis. I see the error message under SLURM on the CORE system:

SlurmCTLD:
2023-02-13T14:28:34.802] JobId=370421 nhosts:1 ncpus:1 node_req:64000 nodes=CompNode01
[2023-02-13T14:28:34.802] Node[0]:
[2023-02-13T14:28:34.802] Mem(MB):15998:0 Sockets:1 Cores:6 CPUs:6:0
[2023-02-13T14:28:34.802] Socket[0] Core[0] is allocated
[2023-02-13T14:28:34.802] Socket[0] Core[1] is allocated
[2023-02-13T14:28:34.802] Socket[0] Core[2] is allocated
[2023-02-13T14:28:34.802] Socket[0] Core[3] is allocated
[2023-02-13T14:28:34.802] Socket[0] Core[4] is allocated
[2023-02-13T14:28:34.802] Socket[0] Core[5] is allocated
[2023-02-13T14:28:34.802] --------------------
[2023-02-13T14:28:34.802] cpu_array_value[0]:6 reps:1
[2023-02-13T14:28:34.802] ====================
[2023-02-13T14:28:34.803] sched/backfill: _start_job: Started JobId=370421 in Artificial on CompNode01
[2023-02-13T14:28:34.910] _slurm_rpc_requeue: Requeue of JobId=370421 returned an error: Only batch jobs are accepted or processed
[2023-02-13T14:28:34.914] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=370421 uid 0
[2023-02-13T14:28:34.915] job_signal: 9 of running JobId=370421 successful 0x8004
[2023-02-13T14:28:35.917] _slurm_rpc_complete_job_allocation: JobId=370421 error Job/step already completing or completed

SlurmD:
[370420.extern] fatal: Could not create domain socket: Operation not permitted
[2023-02-13T14:13:12.412] error: _forkexec_slurmstepd: slurmstepd failed to send return code got 0: Resource temporarily unavailable
[2023-02-13T14:13:12.417] Could not launch job 370420 and not able to requeue it, cancelling job

And with this, the SlurmD process aborts the processing and reports back to the CTLD that the JOB cannot be executed. And I find absolutely no explanation for this. I only see on both sides CTLD and SlurmD, the "unauthorised credential for client .....". - How did you solve the problem in the end? With this FLAG under MUNGE or rather under SLURM?
Best regards from Berlin.....

Z. Matthias

Originally posted by @ZXRobotum in dun/munge#130 (comment)