Issues accessing presigned s3 URLs about funnel HOT 9 CLOSED

What is the error message being returned and what version of Funnel are you using?

from funnel.

This is the version I am using (I am running this on a Mac, I used the Homebrew instructions from here https://ohsu-comp-bio.github.io/funnel/download/):

git commit: 3eaec933
git branch: master
git upstream: 
build date: 2018-06-20T16:17:18Z
version: 0.8.0

This is an example of the error I get (I shortened the URL a bit, but the relevant parts are there):

funnel task get bpnlopovlv353e62s680
{
  "id": "bpnlopovlv353e62s680",
  "state": "SYSTEM_ERROR",
  "name": "Hello world",
  "inputs": [
    {
      "url": "\"https://[...].s3.amazonaws.com/ga4gh_demo/README?AWSAccessKeyId=[...]5DT\u0026Signature=cUa[...]g%3D\u0026Expires=158[...]02\"",
      "path": "/inputs/README"
    }

If I look at the task in the Web UI (http://localhost:8000/v1/tasks/bpnlopovlv353e62s680) I get the correct URL:
"https://[...].s3.amazonaws.com/ga4gh_demo/README?AWSAccessKeyId=[...]5DT&Signature=cUa[...]g%3D&Expires=158[...]02"

from funnel.

What is the error message that gets logged by the storage backend when the task is run?

The encoding stuff you are seeing is probably related to golang/protobuf#407, but I am not sure that it is the underlying issue.

from funnel.

The storage back (s3) end simply responds with an authorization error:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2F[...]87</RequestId><HostId>ss[...]0K/w=</HostId></Error>

But I think this is due to the access credentials not being formatted correctly.

from funnel.

I don't think it is an issue with the access credential formatting - Funnel tries to execute a HEAD request on URL prior to downloading which is going to fail for a pre-signed URL that authorizes a GET request.

I will see what I can do to resolve this.

from funnel.

Thank you!

from funnel.

Are you using presigned URLs just for the inputs or for the outputs as well?

I was able to add download support, but upload support would be a bit more work.

from funnel.

I would like to do both, eventually. But being able to access data first would be OK.

(Related to this: if I could specify s3 URLs for input/output formatted like s3://[...]/ga4gh_demo/ via cwl-tes that would also be helpful; but that is a separate issue).

from funnel.

I think it works! I just tested the commit locally, and I do no longer get an error. Thank you.

from funnel.