Giter Club home page Giter Club logo

pysssix's Introduction

pysssix

Mount S3 as a POSIX like file system. Access is read only and assumes data on S3 is static.

Set up AWS credentials as per http://boto3.readthedocs.io/en/latest/guide/configuration.html using a method that doesn't require accessing the boto Session or boto.client object/methods.

pysssix <mount_point> [--debug]

<mount_point> should be a existing empty directory.

Requires libfuse to be installed on the system.

Ubuntu:

sudo apt-get install libfuse-dev

Work in progress use at own risk :)

Developer notes, set up and running in container.

  • build:
docker build  -t pysssix .
  • run:
docker run -i -t --cap-add SYS_ADMIN --device /dev/fuse  -v ~/.aws:/root/.aws -v $(pwd):/root/pysssix pysssix

Using

cd /root
mkdir s3
pip install --upgrade --no-deps --force-reinstall  ./pysssix/
pysssix s3 --debug

In another process

cat /root/s3/<my bucket>/<my key>

pysssix's People

Contributors

tam203 avatar jacobtomlinson avatar

Stargazers

HalfBuddhist avatar Theo Barber-Bany avatar avatar

Watchers

avatar avatar Niall Robinson avatar avatar

Forkers

uk-gov-mirror lemaral

pysssix's Issues

Error when no AWS credentials provided

An error is thrown if AWS credentials cannot be found. There are two problems with this:

  • If the bucket is public it should not require credentials
  • If credentials are required the error should still be caught and a more helpful error message should be printed.
$ ls /mnt/pysssix/mybucket/
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/local/lib/python3.4/site-packages/fuse.py", line 693, in readdir
    fip.contents.fh):
  File "/usr/local/lib/python3.4/site-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "/usr/local/lib/python3.4/site-packages/pysssix/s3_fuse_ops.py", line 56, in readdir
    return list_bucket(path)
  File "/usr/local/lib/python3.4/site-packages/pysssix/s3.py", line 93, in list_bucket
    items = boto3.client('s3').list_objects_v2(Bucket=bucket,Prefix=key)['Contents']
  File "/usr/local/lib/python3.4/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.4/site-packages/botocore/client.py", line 602, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python3.4/site-packages/botocore/endpoint.py", line 143, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.4/site-packages/botocore/endpoint.py", line 168, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.4/site-packages/botocore/endpoint.py", line 152, in create_request
    operation_name=operation_model.name)
  File "/usr/local/lib/python3.4/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.4/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.4/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.4/site-packages/botocore/signers.py", line 154, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.4/site-packages/botocore/auth.py", line 420, in add_auth
    super(S3SigV4Auth, self).add_auth(request)
  File "/usr/local/lib/python3.4/site-packages/botocore/auth.py", line 352, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
ls: reading directory /mnt/pysssix/mybucket/: Bad address

Unable to read files when creds are stores in ~/.aws

When running pysssix on a server which has no IAM role I am unable to read files despite having my credentials configured with aws configure and stored in ~/.aws/credentials.

$ cat /mnt/pysssix/mybucket/myfile.txt
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/local/lib/python3.4/site-packages/fuse.py", line 511, in getattr
    return self.fgetattr(path, buf, None)
  File "/usr/local/lib/python3.4/site-packages/fuse.py", line 759, in fgetattr
    attrs = self.operations('getattr', self._decode_optional_path(path), fh)
  File "/usr/local/lib/python3.4/site-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "/usr/local/lib/python3.4/site-packages/pysssix/s3_fuse_ops.py", line 34, in getattr
    attrs =  s3_getattr(path)
  File "/usr/lib64/python3.4/functools.py", line 472, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/local/lib/python3.4/site-packages/pysssix/s3.py", line 71, in getattr
    return {'st_mode': 33188, 'st_size': get_size(path)} if obj_type(path) == 2 else {'st_mode': 16877}
  File "/usr/local/lib/python3.4/site-packages/pysssix/s3.py", line 57, in obj_type
    if get_size(path) <= 0:
  File "/usr/lib64/python3.4/functools.py", line 472, in wrapper
    result = user_function(*args, **kwds)
  File "/usr/local/lib/python3.4/site-packages/pysssix/s3.py", line 20, in get_size
    response = s3.head_object(Bucket=bucket, Key=key)
  File "/usr/local/lib/python3.4/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.4/site-packages/botocore/client.py", line 602, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python3.4/site-packages/botocore/endpoint.py", line 143, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.4/site-packages/botocore/endpoint.py", line 168, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.4/site-packages/botocore/endpoint.py", line 152, in create_request
    operation_name=operation_model.name)
  File "/usr/local/lib/python3.4/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.4/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.4/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.4/site-packages/botocore/signers.py", line 154, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.4/site-packages/botocore/auth.py", line 420, in add_auth
    super(S3SigV4Auth, self).add_auth(request)
  File "/usr/local/lib/python3.4/site-packages/botocore/auth.py", line 352, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
cat: /mnt/pysssix/mybucket/myfile.txt: Bad address

When doing an ls on the bucket it successfully gets the file list but gives the error for every file in the bucket before showing the list of files. I assume this is trying to look up some info on each file like file size and failing each time.

Maintain python 3.4 support with asyncio

We are currently running this in Python 3.4 on our Kubernetes cluster. The cluster is built on Debian Jessie which comes with Python 3.4 by default. I feel this is a good minimum version to maintain.

Therefore the addition of asyncio in the block-cache branch should ensure to maintain 3.4 support and use the "classic" asyncio syntax.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.