Funnel responds with 'Closed explicitly' about funnel HOT 9 OPEN

Funnel wasn't handling mongo sessions properly. I'm guessing the session was becoming stale and was being closed by the mongo server.

Try the fix referenced in the PR above.

from funnel.

The new version does not seem to work properly.
The jobs are submitted and immediately go to state 'SYSTEM_ERROR' without being sent to AWS batch.
In the log files (with the debug enabled) there is no information.

Using the previous version (v0.9.0 from DockerHub) which eventually gets to the 'Closed explicitly' state, the jobs are submitted as expected and go to states QUEUED and RUNNING etc

from funnel.

The changes I made shouldn't have affected jobs being submitted to batch. Nonetheless I found a spot where I wasn't logging a potential error in the CreateTask method. I pushed up a new commit to that branch that should provide this missing error logging.

from funnel.

Thanks, there is still the same problem
This version is still not able to submit jobs to AWS Batch. (the previous version v0.9.0 can on teh same node, with the same config files)
The only relevant error I can see is

{"error":"RequestError: send request failed\ncaused by: Post https://batch.us-east-1.amazonaws.com/v1/submitjob: x509: failed to load system roots and no roots provided","level":"error","msg":"error submitting task to compute backend","ns":"server","taskID":"bk3ut6qi76gg008vei1g","time":"2019-06-17T19:48:44Z"}

Thanks
K

from funnel.

Are you running your funnel server in a docker container? If so, I think the problem may have been with the base image I was using. Try rebuilding the container (make docker) using the branch in #613.

from funnel.

I am using it in an image, but I compiled it myself.

FROM ubuntu:18.04



RUN apt-get update &&\
    apt-get -y install git golang go-dep
    
ENV GOROOT=/usr/lib/go-1.10/
ENV GOPATH=/opt/
ENV PATH=$PATH:$GOROOT/bin

WORKDIR /opt

 
WORKDIR /opt/src/github.com/ohsu-comp-bio/
RUN git clone https://github.com/ohsu-comp-bio/funnel.git
WORKDIR /opt/src/github.com/ohsu-comp-bio/funnel
#RUN git checkout mongo-session && git pull origin mongo-session
RUN make && go build


FROM ubuntu:18.04

RUN mkdir -p /opt/funnel/funnel-work-dir
COPY --from=0 /opt/src/github.com/ohsu-comp-bio/funnel/funnel /opt/funnel/funnel

ENTRYPOINT ["/opt/funnel/funnel"]


CMD []

Nevertheless, I will try your suggestion and let you know

from funnel.

Ah ok, I think the source of the error is that the ubuntu:18.04 image doesn't com with ca-certificates installed. See http://vinceyuan.github.io/solving-ssl-root-certificates/ and https://stackoverflow.com/questions/45388934/how-do-i-make-an-https-call-in-a-busybox-docker-container-running-go

from funnel.

Perfect, thanks for the pointer.
This solved the problem...

from funnel.

For completion the dockerfile is now:

FROM ubuntu:18.04



RUN apt-get update &&\
    apt-get -y install git golang go-dep

ENV GOROOT=/usr/lib/go-1.10/
ENV GOPATH=/opt/
ENV PATH=$PATH:$GOROOT/bin

WORKDIR /opt


WORKDIR /opt/src/github.com/ohsu-comp-bio/
RUN git clone https://github.com/ohsu-comp-bio/funnel.git
WORKDIR /opt/src/github.com/ohsu-comp-bio/funnel
RUN make && go build


FROM ubuntu:18.04

RUN mkdir -p /opt/funnel/funnel-work-dir
COPY --from=0 /opt/src/github.com/ohsu-comp-bio/funnel/funnel /opt/funnel/funnel

RUN apt-get update
RUN apt-get install -y ca-certificates

ENTRYPOINT ["/opt/funnel/funnel"]


CMD []

from funnel.