Giter Club home page Giter Club logo

s3-cf-service-broker's Introduction

Cloud Foundry Service Broker for Amazon S3

A Cloud Foundry Service Broker for Amazon S3 built using the spring-boot-cf-service-broker.

The broker currently publishes a single service and plan for provisioning S3 buckets.

Design

The broker uses meta data in S3 and naming conventions to maintain the state of the services it is brokering. It does not maintain an internal database so it has no dependencies besides S3.

Releases

Stable versions have been tagged as releases.

Running

Simply run the JAR file and provide AWS credentials via the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables.

Locally

mvn package && AWS_ACCESS_KEY=secret AWS_SECRET_KEY=secret java -jar target/s3-cf-service-broker-2.0.0-SNAPSHOT.jar

In Cloud Foundry

Build s3-cf-service-broker and push it to Cloud Foundry:

mvn package
cf push s3-cf-service-broker -p target/s3-cf-service-broker-2.0.0-SNAPSHOT.jar --no-start
cf set-env s3-cf-service-broker AWS_ACCESS_KEY "MYAWSKEY"
cf set-env s3-cf-service-broker AWS_SECRET_KEY "MYAWSSECRET"
cf set-env s3-cf-service-broker SECURITY_USER_PASSWORD "mysecret"

Start the service broker:

cf start s3-cf-service-broker

Create Cloud Foundry service broker:

cf create-service-broker s3-cf-service-broker user mysecret http://s3-cf-service-broker.cfapps.io

Add service broker to Cloud Foundry Marketplace:

cf enable-service-access amazon-s3 -o ORG

Using the services in your application

Format of Credentials

The credentials provided in a bind call have the following format:

"credentials":{
	"username":"cloud-foundry-s3-c5271ba4-6d2f-4163-843c-6a5fdceb7a1a",
	"access_key_id":"secret",
	"bucket":"cloud-foundry-2eac2d52-bfc9-4d0f-af28-c02187689d72",
	"secret_access_key":"secret",
	"host":"s3.amazonaws.com",
	"uri":"s3://secret:[email protected]/cloud-foundry-2eac2d52-bfc9-4d0f-af28-c02187689d72"
}

Java Applications - Spring Cloud

For Java applications, you may consider using Spring Cloud and the spring-cloud-s3-service-connector.

Configuration

General Configuration

The following general configuration options are available.

Environment Variable Required Default
AWS_ACCESS_KEY x
AWS_SECRET_KEY x
AWS_REGION US
PROXY_HOST none
PROXY_PORT none
PROXY_USERNAME none
PROXY_PASSWORD none
PREEMPTIVE_PROXY_BASE_AUTH false

Broker Security

spring-boot-starter-security is used. See the documentation here for configuration: Spring boot security

The default password configured is "password" (see application.properties).

You may also configure security via environment variables as noted in the Spring Boot documentation.

Environment Variable Required Default
SECURITY_USER_NAME user
SECURITY_USER_PASSWORD password

User for Broker

An AWS user must be created for the broker. The user's accessKey and secretKey must be provided using the environments variables AWS_ACCESS_KEY and AWS_SECRET_KEY as noted above.

An example user policy for the broker user is provided in broker-user-iam-policy.json. If desired, you can further limit user and group resources in this policy based on prefixes defined above.

Note: The S3 policies could be more limited based on what is actually used.

Basic Plan

A service provisioning call will create an S3 bucket, an IAM group, and an IAM Policy to provide access controls on the bucket. A binding call will create an IAM user, generate access keys, and add it to the bucket's group. Unbinding and deprovisioning calls will delete all resources created.

The following names are used and can be customized with a prefix:

Resource Name is based on Custom Prefix Environment Variable Default Prefix Example Name
S3 Buckets service instance ID BUCKET_NAME_PREFIX cloud-foundry- cloud-foundry-2eac2d52-bfc9-4d0f-af28-c02187689d72
IAM Group Names service instance ID GROUP_NAME_PREFIX cloud-foundry-s3- cloud-foundry-s3-2eac2d52-bfc9-4d0f-af28-c02187689d72
IAM Policy Names service instance ID POLICY_NAME_PREFIX cloud-foundry-s3- cloud-foundry-s3-2eac2d52-bfc9-4d0f-af28-c02187689d72
IAM User Names binding ID USER_NAME_PREFIX cloud-foundry-s3- cloud-foundry-s3-e9bea699-aa68-4464-bb8f-0c8622884b43

Also the following paths are used for IAM resources and can be customized with a prefix:

Resource Custom Path Environment Variable Default Path
IAM User USER_PATH /cloud-foundry/s3/
IAM Group GROUP_PATH /cloud-foundry/s3/

Bucket Policy

The group policy applied to all buckets created is provided in default-bucket-policy.json.

Bucket Tagging

All buckets are tagged with the following values:

  • serviceInstanceId
  • serviceDefinitionId
  • planId
  • organizationGuid
  • spaceGuid

The ability to apply additional custom tags is in the works.

Registering a Broker with the Cloud Controller

See Managing Service Brokers.

Testing

Export AWS credentials environment variables:

export AWS_ACCESS_KEY="YOUR_AWS_ACCESS_KEY"
export AWS_SECRET_KEY="YOUR_AWS_SECRET_KEY"

and execute tests with maven:

mvn test

Contributing

In the spirit of free software, everyone is encouraged to help improve this project. All contributions should be done through pull requests.

Here are some ways you can contribute:

  • by using alpha, beta, and prerelease versions
  • by reporting bugs
  • by suggesting new features
  • by writing or editing documentation
  • by writing specifications
  • by writing code (no patch is too small: fix typos, add comments, clean up inconsistent whitespace)
  • by refactoring code
  • by closing issues
  • by reviewing patches

Submitting an Issue

We use the GitHub issue tracker to track bugs and features. Before submitting a bug report or feature request, check to make sure it hasn't already been submitted. You can indicate support for an existing issue by voting it up. When submitting a bug report, please include a Gist that includes a stack trace and any details that may be necessary to reproduce the bug, including your Golang version and operating system. Ideally, a bug report should include a pull request with failing specs.

Submitting a Pull Request

  1. Fork the project.
  2. Create a topic branch.
  3. Implement your feature or bug fix.
  4. Commit and push your changes.
  5. Submit a pull request.

s3-cf-service-broker's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

s3-cf-service-broker's Issues

Optional KMS Key ID in credential payload

An admin should be able to specify an option kms_key_id that is added to the service credentials payload to indicate what KMS key a service consumer can use for server side encryption.

An admin cloud enforce the usage of KMS by added something like the following to their bucket policy:
{ "Version":"2012-10-17", "Id":"PutObjPolicy", "Statement":[{ "Sid":"DenyUnEncryptedObjectUploads", "Effect":"Deny", "Principal":"*", "Action":"s3:PutObject", "Resource":"arn:aws:s3:::YourBucket/*", "Condition":{ "StringNotEquals":{ "s3:x-amz-server-side-encryption":"AES256" } } } ] }

support for ceph radosgw (onsite s3)

is there a possibility to set the host?
because ceph(radosgw) have almost full s3 support
see http://docs.ceph.com/docs/master/radosgw/s3/

we already use ceph for cf droplets

    droplets:
      droplet_directory_key: <%= root_domain %>-cc-droplets-new
      fog_connection:
        provider: "AWS"
        host: "s3.eden.klm.com"
        scheme: "http"
        port: 80
        aws_signature_version: "2"
        aws_access_key_id: "XXXXXXXXXXXXXXXXXX
        aws_secret_access_key: "XXXXXXXXXXXXXXXXXXXXXXX"

so we only need to be able to set ( host, scheme, port, aws_signature_version )

A way to bind an app to multiple S3 buckets?

Is there a way or support to bind an app to multiple S3 buckets?

My app is currently binded to an S3 bucket and has access to the following environments:

    "credentials": {
     "access_key_id": "MXHUA",
     "bucket": "cloud-foundry-82-59cf655a61ef",
     "host": "s3.amazonaws.com",
     "secret_access_key": "RBiB0Or1LaukH",
     "uri": "s3://AKJMXHUA:[email protected]/cloud-foundry-c882-59cf655a61ef",
     "username": "cloud-foundry-s3-79031277d"
    }

I'm wondering if the app can be binded to multiple buckets?

Can't delete service instance

Hi,

I am having issues deleting an s3-cf-service-broker instance can anyone help? Error message is:

cf delete-service brooklyn-s3Test -f
Deleting service brooklyn-s3Test in org CloudSoft-01 / space CS-01 as [email protected]...
FAILED
Server error, status code: 502, error code: 10001, message: Service instance brooklyn-s3Test: Service broker error: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 5E202360E87EBB71)

also tried without the -f flag and same message occurs

Many thanks

Adam

create-service-key does not create IAM user

It creates the IAM group, but not a user.

Later, when deleting the service key, an error is in the logs for its inability to delete the missing user.

Binding to an application generates IAM user & credentials correctly.

issue with mvn package

Hello,
I installed the package from github and wanted to build the package using mvn.but i revcieved the below error with the pom.xml file:

[INFO] Scanning for projects...
Downloading: http://repo.spring.io/libs-snapshot/org/springframework/boot/spring-boot-starter-parent/1.1.4.RELEASE/spring-boot-starter-parent-1.1.4.RELEASE.pom
Downloading: https://repo.maven.apache.org/maven2/org/springframework/boot/spring-boot-starter-parent/1.1.4.RELEASE/spring-boot-starter-parent-1.1.4.RELEASE.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.cloudfoundry.broker:s3-cf-service-broker:2.3.0-SNAPSHOT: Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:1.1.4.RELEASE from/to spring-io (http://repo.spring.io/libs-snapshot): repo.spring.io: unknown error and 'parent.relativePath' points at wrong local POM @ line 4, column 10
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.cloudfoundry.broker:s3-cf-service-broker:2.3.0-SNAPSHOT (/home/js37gfy/s3-cf-service-broker/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.cloudfoundry.broker:s3-cf-service-broker:2.3.0-SNAPSHOT: Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:1.1.4.RELEASE from/to spring-io (http://repo.spring.io/libs-snapshot): repo.spring.io: unknown error and 'parent.relativePath' points at wrong local POM @ line 4, column 10: Unknown host repo.spring.io: unknown error -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
bash-4.1$

clould you please help sort out this.

Custom tags on buckets

In addition to the standard tags added to buckets, an admin should be able to provide additional custom tags.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.