Hello,
I've run into this issue:

1. I've created a fresh bucket
2. gsutil cp data to the bucket
3. ran GCSFS.fix_storage(), root_path is default ("")
4. GCSFS created directory markers, also created a "/" directory marker for the bucket's root
5. issue -> I want to create a new directory, so I run GCSFS.fix_storage() again, but it is stuck

gcsfs.fix_storage()
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): www.googleapis.com:443
DEBUG:urllib3.connectionpool:https://www.googleapis.com:443 "GET /storage/v1/b/redacted/o?projection=noAcl&prefix= HTTP/1.1" 200 249836
<nothing happens>

6. If I delete "/" directory marker in bucket root, GCSFS.fix_storage() runs correctly and marks the new directory properly

Btw thanks for this lib.

Since fs 2.4.15 getbasic() method has been deprecated. gscfs calls getbasic() at least in opendir() (gcsfs 1.4.5).

The code of GCSFS below results in a deprecation warning issued by urllib3

if retry:
    adapter = HTTPAdapter(max_retries=Retry(total=retry,
                                             status_forcelist=[429, 502, 503, 504],
                                             method_whitelist=False,  # retry on any HTTP method
                                             backoff_factor=0.5))
    self.client._http.mount("https://", adapter)

The warning:
DeprecationWarning: Using 'method_whitelist' with Retry is deprecated and will be removed in v2.0. Use 'allowed_methods' instead

Versions:
fs-gcsfs 1.4.1
urllib3 1.26.2

P.S. retry is not described in the GCSFS.init() docstring.

s3fs has a handy class called S3Map that wraps the file system in a MutableMapping. This is a necessary feature for use with some third party libraries (in my case, xarray and zarr).

The implementation is pretty simple and can be ported directly from s3fs. I have submitted a pull request doing so.

Hi!
I have some problems accessing certain files using fs_gcsfs. Im struggling with figuring out what exactly the problem is, as it does not affect all files. Accessing the files directly through gcsfs works though.

>>> import gcsfs
>>> t = gcsfs.GCSFileSystem(project='some-company')
>>> p = t.open('gs://my-bucket/my-folder/README.md')
>>> p
<File-like object GCSFileSystem, my-bucket/my-folder/README.md>
>>>
>>> from fs import open_fs
>>> t = open_fs('gs://my-bucket/')
>>> t.open('my-folder/README.md')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/fs/base.py", line 1166, in open
    bin_file = self.openbin(path, mode=bin_mode, buffering=buffering)
  File "/usr/local/lib/python3.7/site-packages/fs_gcsfs/_gcsfs.py", line 348, in openbin
    info = self.getinfo(path)
  File "/usr/local/lib/python3.7/site-packages/fs_gcsfs/_gcsfs.py", line 127, in getinfo
    raise errors.ResourceNotFound(path)
fs.errors.ResourceNotFound: resource 'my-folder/README.md' not found
>>> t.exists('my-folder/README.md')
True
>>> t.open('other-folder/some-file.csv')
    <_io.TextIOWrapper name='other-folder/some-file.csv' encoding='utf-8'>

So for some reason i can access other-folder/some-file.csv but not my-folder/README.md when using fs_gcsfs. From what i see, the acl and iam for both files look pretty much the same:

% gsutil acl get gs://my-bucket/my-folder/README.md
[
  {
    "entity": "project-owners-605693490522",
    "projectTeam": {
      "projectNumber": "605693490522",
      "team": "owners"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-editors-605693490522",
    "projectTeam": {
      "projectNumber": "605693490522",
      "team": "editors"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-viewers-605693490522",
    "projectTeam": {
      "projectNumber": "605693490522",
      "team": "viewers"
    },
    "role": "READER"
  },
  {
    "email": "[email protected]",
    "entity": "[email protected]",
    "role": "OWNER"
  }
]
% gsutil iam get gs://my-bucket/my-folder/README.md
{
  "bindings": [
    {
      "members": [
        "projectViewer:some-company"
      ], 
      "role": "roles/storage.legacyObjectReader"
    }, 
    {
      "members": [
        "projectOwner:some-company", 
        "projectEditor:some-company", 
        "serviceAccount:[email protected]"
      ], 
      "role": "roles/storage.legacyObjectOwner"
    }
  ], 
  "etag": "CAE="
}

% gsutil acl get gs://my-bucket/other-folder/some-file.csv
[
  {
    "email": "[email protected]",
    "entity": "[email protected]",
    "role": "OWNER"
  },
  {
    "entity": "project-owners-605693490522",
    "projectTeam": {
      "projectNumber": "605693490522",
      "team": "owners"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-editors-605693490522",
    "projectTeam": {
      "projectNumber": "605693490522",
      "team": "editors"
    },
    "role": "OWNER"
  },
  {
    "entity": "project-viewers-605693490522",
    "projectTeam": {
      "projectNumber": "605693490522",
      "team": "viewers"
    },
    "role": "READER"
  }
]
% gsutil iam get gs://my-bucket/other-folder/some-file.csv
{
  "bindings": [
    {
      "members": [
        "serviceAccount:[email protected]",
        "projectOwner:some-company",
        "projectEditor:some-company"
      ],
      "role": "roles/storage.legacyObjectOwner"
    },
    {
      "members": [
        "projectViewer:some-company"
      ],
      "role": "roles/storage.legacyObjectReader"
    }
  ],
  "etag": "CAE="
}

The GOOGLE_APPLICATION_CREDENTIALS env-variable is set to a service-account file for some-service-account.
Any idea how i can debug/solve this? Using

fs-gcsfs                   1.0.0    
gcsfs                      0.3.0 

>>> gcsfs.__version__
'0.2.3'

The glob() function without star (*) returns wrong value. This is an inconsistent behavior different than glob.glob or tensorflow.io.gfile.glob.

>>> fs.glob('gs://mybucket/folder/*.csv')
['mybucket/folder/a.csv']          # why without gs:// ???

>>> fs.glob('gs://mybucket/folder/a.csv')
[]

>>> fs.exists('gs://mybucket/folder/a.csv')
True

Im wondering: im creating files on the bucket from another application which doesnt create markers, so i have to write some additional code somewhere to handle this. But i think if i would do a small "does the file exists but the markers along the path are missing? if so, create the markers" snippet somewhere on getinfo under the block

if check_parent_dir:
            [...]
            if parent_dir != "/" and not self._get_blob(parent_dir_key):
                raise errors.ResourceNotFound(path)

would fix this problem. Is there any reason why this is a bad idea?

Hi,
When trying to open

gcsfs = GCSFS(bucket_name=bucket)
I get the following error
OSError: Project was not passed and could not be determined from the environment.

The problem is that when creating the Client there is no option to pass Google Project ID

Currently, all Travis builds fail as the encryption keys securing the service-account-key.json seem to be missing/broken.

Hi,

When doing:

fs = open_fs("gs://my_bucket/path/to/blob")

I'm getting:

  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/fs/opener/registry.py", line 228, in open_fs
    default_protocol=default_protocol,
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/fs/opener/registry.py", line 189, in open
    open_fs = opener.open_fs(fs_url, parse_result, writeable, create, cwd)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/fs_gcsfs/opener.py", line 25, in open_fs
    return GCSFS(bucket_name, root_path=root_path, create=create)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/fs_gcsfs/_gcsfs.py", line 88, in __init__
    raise errors.CreateFailed("Root path \"{}\" does not exist".format(root_path))
fs.errors.CreateFailed: Root path "path/to/blob" does not exist

But I'm sure gs://my_bucket/path/to/blob does exist (running gsutil ls gs://my_bucket/path/to/blob returns files)

Wondering if #9 broke something cc @birnbaum

from fs import open_fs
open_fs("gs://valid_bucket_name/directory_that_does_not_exist")

completes without error, even though the directory does not exist

Hi, I get the following error using either of the two options (obviously I replaced this with the real name of my bucket):

from fs_gcsfs import GCSFS
gcsfs = GCSFS("mybucket")

or

from fs import open_fs
gcsfs = open_fs("gs://mybucket/mydir") 

Error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-da26c33d3a55> in <module>
----> 1 gcsfs = open_fs("gs://mybucket/mydir")

/opt/conda/lib/python3.7/site-packages/fs/opener/registry.py in open_fs(self, fs_url, writeable, create, cwd, default_protocol)
    224                 create=create,
    225                 cwd=cwd,
--> 226                 default_protocol=default_protocol,
    227             )
    228         return _fs

/opt/conda/lib/python3.7/site-packages/fs/opener/registry.py in open(self, fs_url, writeable, create, cwd, default_protocol)
    185         opener = self.get_opener(protocol)
    186 
--> 187         open_fs = opener.open_fs(fs_url, parse_result, writeable, create, cwd)
    188         return open_fs, open_path
    189 

/opt/conda/lib/python3.7/site-packages/fs_gcsfs/opener.py in open_fs(self, fs_url, parse_result, writeable, create, cwd)
     37             client.client_options = {"api_endpoint": api_endpoint}
     38 
---> 39         return GCSFS(bucket_name, root_path=root_path, create=create, client=client, strict=strict)

/opt/conda/lib/python3.7/site-packages/fs_gcsfs/_gcsfs.py in __init__(self, bucket_name, root_path, create, client, retry, strict)
     80                                                     status_forcelist=[429, 502, 503, 504],
     81                                                     allowed_methods=False,  # retry on any HTTP method
---> 82                                                     backoff_factor=0.5))
     83             self.client._http.mount("https://", adapter)
     84 

TypeError: __init__() got an unexpected keyword argument 'allowed_methods'

import fs_gcsfs
fs_gcsfs.__version__
# 1.4.2

The used version of packaging is old and conflicts with updating in my other dependency.

  Because poetryup (0.7.1) depends on packaging (>=21.3,<22.0)
   and fs-gcsfs (1.4.5) depends on packaging (>=20.0,<21.0), poetryup (0.7.1) is incompatible with fs-gcsfs (1.4.5).
  And because no versions of fs-gcsfs match >1.4.5,<2.0.0, poetryup (0.7.1) is incompatible with fs-gcsfs (>=1.4.5,<2.0.0).
  So, because *** depends on both fs-gcsfs (^1.4.5) and poetryup (0.7.1), version solving failed.

GCP can compute the md5 hash server-side: https://googleapis.dev/python/storage/latest/blobs.html#google.cloud.storage.blob.Blob.md5_hash

hash() is not typically implemented in pyfilesystem according to: https://docs.pyfilesystem.org/en/latest/implementers.html#helper-methods however, in this case it could be argued that there is a performance benefit to getting the hash from the server rather than downloading+computing it client-side.

Note that this only works for md5 and crc32c. Therefore the implementation could just fallback to the parent class otherwise. An implementation might look like this (complelely untested):

def hash(self, path, name):
    if name.lower() == 'md5':
        _path = self.validatepath(path)
        _key = self._path_to_key(_path)
        blob = self._get_blob(_key)
        if not blob:
            raise errors.ResourceNotFound(path)
        return blob.md5_hash
    else:
        return super().hash(self, path, name)

I'm not completely familiar with Travis for CI, but it seems like a credential to deploy to pypi might be exposed here?

Within _gcsfs.py in the fix_storage process if the root_path ends with a slash, there is an infinite loop. This is because the dirname function will return each parent directory without the trailing slash.

I think this can be resolved easily, by a safety check for a trailing slash on the root path when the file system is created.

I'm working on a large codebase using fs-gcsfs and we'd like to test our code without having to: 1) create development versions of buckets and 2) go through thousands of lines of code to find every bucket, change the name during testing, and remember to change every single one back before deployment.

It'd be really great if there were a way to supply the api_endpoint argument to the Client object via the URL. I haven't thought too long about this but I was thinking adding client_options as an argument and the value would be a JSON object that we'd then pass directly to the client_options argument of Client:

fs.open_fs("gs://my-bucket?client_options=%7B%22api_endpoint%22%3A%20%22http%3A//localhost%3A1234%22%7D")

You'd then modify the opener and add:

options_string = parse_result.params.get("client_options")
if options_string is not None:
    # You may need to run options_string through urllib.parse.unquote() first
    client_options = json.loads(options_string)
else:
    client_options = None

return GCSFS(..., client_options=client_options)

Then in the GCSFS constructor you'd add an optional client_options argument and make a small change to how the client is created:

if self.client is None:
    self.client = Client(client_options=client_options)

I know JSON is quite ugly but it'll get the job done and is extensible. Another possibility is using dot notation along with dpath to get something much more readable:

fs.open_fs("gs://my-bucket?client_options.api_endpoint=http%3A//localhost%3A1234")

It's a tad bit more work but I think it'll be fine? Google's API client only defines one client option so far and it's a string. If they add another option (say a timeout length) and it's an integer then we'd have to handle that some other way.

Currently all files created in GCS are created with application/octet-stream MIME type, that makes it impossible to upload images for hotlinking from GCS.

It should be relatively easy to add support for mimetypes.guess_type, but it will increase usability of the module a lot.

When using upload() we get an error if the "directory" doesn't exist. With S3 this isn't the case.

Thanks for the otherwise brilliant package 👍