son-of-gridengine / sge Goto Github PK

I was able to fix it, although I suspect that my fix may have been disruptive to the jobs.

Firstly, I  believe the problem was that gridengine does not handle a deleted job that is on a host that has been deleted, and it dies when it sees it.   Presumably the bug is in allowing it to be deleted in the first place.

Anyway, my fix (after backing up the directory /var/spool/gridengine) was to move the file /var/spool/gridengine/spooldb/sge_job to a temporary location, restart the qmaster, add the host back with qconf -ah, stop the qmaster, restore the old database  /var/spool/gridengine/spooldb/sge_job, and restart the qmaster.

Before doing that whole procedure, to stop the hosts getting confused I stopped all the gridengine-exec services.  That probably wasn't optimal because clients like qsub and qstat would still have been able to access the queue in the interim, and it definitely would have confused them and killed some processes.  Unfortunately I had to do this on short notice and wasn't sure how to use iptables to close off those ports from outside the qmaster while I did the maintenance-- that would have been a better solution. 

Also I encountered a hiccup that `systemctl stop gridengine-qmaster` didn't actually work the second time, the process was still running, with the old database, so I had to manually kill it and retry.

Anyway this whole episode is making me think more seriously about moving to Univa GridEngine.  I've known for a long time that the free version has a lot of bugs, and I just don't have time to deal with this type of thing.



On Sat, Nov 10, 2018 at 4:49 PM Marshall2, John (SSC/SPC) <[email protected]> wrote:
Hi,

I've never seen this but I would start with:
1) strace qmaster during restart to try to see at which point it is dying (e.g.,
loading a config file)
2) look for any reference to the name of the host you deleted in the spool
area and do some cleanup
3) clean out the jobs spool area

HTH,
John

On Sat, 2018-11-10 at 16:23 -0500, Daniel Povey wrote:
Has anyone found this error, and managed to fix it?
I am in a very difficult situation.
I deleted a host (qconf -de hostname) thinking that the machine no longer existed, but it did exist, and there was a job in 'dr' state there.
After I attempted to force-delete that job (qdel -f job-id), the queue master died with out-of-memory, and now I can't restart qmaster.

So now I don't know hw to fix it.  Am I just completely lost now?

Dan
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users

cc -DSGE_ARCH_STRING=\"lx-amd64\" -O2 -Wstrict-prototypes -DLINUXAMD64 -DLINUXAMD64 -D_GNU_SOURCE -DGETHOSTBYNAME_R6 -DGETHOSTBYADDR_R8  -DTARGET_64BIT  -DSGE_PQS_API -DSPOOLING_classic  -DHAVE_HWLOC=1 -DNO_JNI -DCOMPILE_DC -D__SGE_COMPILE_WITH_GETTEXT__  -D__SGE_NO_USERMAPPING__ -I../common -I../libs -I../libs/uti -I../libs/juti -I../libs/gdi -I../libs/japi -I../libs/sgeobj -I../libs/cull -I../libs/comm -I../libs/comm/lists -I../libs/sched -I../libs/evc -I../libs/evm -I../libs/mir -I../daemons/common -I../daemons/qmaster -I../daemons/execd -I../clients/common -I. -fPIC -c ../libs/sgeobj/sge_binding.c
In file included from ../libs/uti/sge_binding_hlp.h:45:0,
                from ../libs/sgeobj/sge_binding.h:43,
                from ../libs/sgeobj/sge_binding.c:39:
../libs/uti/hwloc.h:49:10: fatal error: hwloc/autogen/config.h: No file or folder of this kind
#include <hwloc/autogen/config.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
../libs/sgeobj/Makefile:184: recipe for target 'sge_binding.o' failed
make: *** [sge_binding.o] Error 1
not done

as written on Github, here is the link to the Git repo I created from the DARCS repo of SGE.

https://gitlab.bfabric.org/schmidt/sge

As far as I can see this is equal to the DARCS repo (diff). 

 README                                                               |  7 -------
 README.md                                                            | 15 +++++++++++++++
 debian/rules                                                         |  0
 ... all other diffs are empty.