osaddon / cdmi Goto Github PK

View Code? Open in Web Editor NEW

22.0 7.0 9.0 326 KB

CDMI on OpenStack Swift implementation

License: Apache License 2.0

Python 99.77% Shell 0.23%

cdmi's Introduction

CDMI for OpenStack's Swift

A Python egg that adds support for the CDMI protocol to OpenStack Swift.

Setup

Install Openstack with Swift
Grab the code from github: git clone http://github.com/osaddon/cdmi
Install this python egg: sudo python setup.py install
Configure Swift:

In /etc/swift/proxy-server.conf, add cdmi filter before proxy-server

[pipeline:main]
pipeline = healthcheck cache tempauth *cdmi* proxy-server

And add the following section to the file:

[filter:cdmi]
use = egg:cdmiapi#cdmiapp

To use CDMI ReSTful API to access containers and data objects, a CDMI client normally need to know the root path plus the host name and port number. This implementation uses "cdmi/" as its root and the port number is 8080. An example url to access a container may look like this:

http://hostname:8080/cdmi/root_container/container1/data1

To access the capability, a CDMI client should simply use the host name, port and the root path "cdmi/" and the returned capability path. For example, to access the data1 capability, the url may look like this:

http://hostname:8080/cdmi/cdmi_capabilities/root_container/container1/data1

Running tests

First copy a test config file to /etc/swift:

cp /opt/stack/swift/test/sample.conf /etc/swift/test.conf

if you are using keystone configuration for authentication, you will have to make changes in /etc/swift/test.conf and to make sure that you have all the information correct as follows:

auth_host = 127.0.0.1
auth_port = 5000
access_port = 8080
auth_ssl = no
auth_prefix = /v2.0/tokens

account = your_tenant_name
username = your_user_name
password = your_password

if you have swift all-in-one environment, then make sure the information in /etc/swift/test.conf is as follows:

auth_host = 127.0.0.1
auth_port = 8080
access_port = 8080
auth_ssl = no
auth_prefix = /auth/v1.0

account = test
username = tester
password = testing

Now the test cases in the test directory can be run using python <name_of_test.py> in the tests directory.

Development with devstack

Please make sure swift is enabled in your devstack environment file localrc.

Some sample curl commands

Authenticate and get an auth token (Swift All-in-one environment):

curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0

Authenticate and get an auth token (DevStack or Keystone 2.0):

curl -d '{"auth": {"passwordCredentials": {"username": "<<your name>>",
"password": "<<your password>>"}, "tenantName":"<<your tenant name>>"}}'
-H "Content-Type: application/json" http://127.0.0.1:5000/v2.0/tokens

Use the Authentication token and URL which is in the response from the last curl command to perform an GET operation:

curl -v -X GET -H 'X-Auth-Token: AUTH_tk56b01c82710b41328db7c9f953d3933d'
http://127.0.0.1:8080/cdmi/<user_account>/

Create a Container: (replace the auth token with the token obtained above, also replace the user_account with your user account)

curl -v -X PUT -H 'X-Auth-Token: <<your token>>'
-H 'Content-tType: application/directory'-H 'Content-Length: 0'
http://127.0.0.1:8080/cdmi/<user_account>/<root container>

Query the capabilites of a Container:

curl -v -X GET -H 'X-Auth-Token: <<your token>>'
-H 'X-CDMI-Specification-Version: 1.0.1'
http://127.0.0.1:8080/cdmi/<user_id>/cdmi_capabilities/container/

Add an Object to a Container:

curl -v -X PUT -H 'X-Auth-Token: <<your token>>'
-H 'X-CDMI-Specification-Version: 1.0.1'
-H 'Accept: application/cdmi-object'
-H 'Content-Type: application/cdmi-object'
http://127.0.0.1:8080/cdmi/<user_account>/<root container>/<container_name>/<object_name> -d '<Some JSON>'

cdmi's People

Contributors

Stargazers

Watchers

Forkers

tmetsch gwdg csleex ctbrizhp zhangkaixuan songguang-2010 macressler intellabseurope yangkf1985

cdmi's Issues

tests use auth_host for testing and such expect auth_host == swift proxy host

This might not always be the case - e.g. when keystone and swift proxy are on different host. Maybe we should asume swift proxy == localhost & auth_host == keystone ip?

See:

 conn = httplib.HTTPConnection(self.conf.get('auth_host'), self.conf.get('access_port'))

If you're happy with this I can make the changes accordingly.

run run_tests.py end with errors

Hi,
I installed cdmi to use with swift and keystone. But curl command is always get errors. So I run run_tests.py in cdmi install dir, and end with errors. Please help!
I followed the readme file, and installed the cdmi by python, then configured proxy-server.conf at /etc/swift. Restart the proxy-server. Here is my proxy-server.conf
///////////////////////////////////////////proxy-server.conf/////////////////////////////////////
[DEFAULT]
bind_ip = 0.0.0.0
bind_port = 8080
swift_dir = /etc/swift
workers = 8
user = swift

cert_file = /etc/swift/syncsort.cer

key_file = /etc/swift/private.pem

log_name = swift
log_facility = LOG_LOCAL0
log_level = DEBUG

[pipeline:main]
pipeline = catch_errors healthcheck cache swift3 s3token authtoken keystone cdmi proxy-server

[app:proxy-server]
use = egg:swift#proxy
account_autocreate = true
log_level = DEBUG

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory

the host must point to your keystone server

auth_host = 10.200.33.30
auth_port = 35357
auth_protocol=http
service_host = 10.200.33.30
service_port = 5000
admin_token = ADMIN
admin_user = admin
admin_password = openstack
admin_tenant_name = adminTenant
signing_dir = /etc/swift

[filter:keystone]
paste.filter_factory = keystone.middleware.swift_auth:filter_factory
operator_roles = adminRole, swiftoperator
is_admin = true
reseller_prefix=AUTH

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 127.0.0.1:11211

[filter:domain_remap]
use = egg:swift#domain_remap

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:swift3]
use = egg:swift3#swift3

[filter:cdmi]
use = egg:cdmiapi#cdmiapp

[filter:s3token]
paste.filter_factory = keystone.middleware.s3_token:filter_factory
auth_port = 35357
auth_host = 10.200.33.30
auth_protocol = http
reseller_prefix=AUTH
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////
And here is my test.conf at /etc/swift/
//////////////////////////////////////////test.conf//////////////////////////////////////////////////////////
[func_test]

sample config

auth_host = 10.200.33.30
auth_port = 5000
access_port = 8080
auth_ssl = no
auth_prefix = /v2.0/tokens

account = adminTenant
username = admin
password = openstack

Primary functional test account (needs admin access to the account)

account = test

username = tester

password = testing

User on a second account (needs admin access to the account)

account2 = test2

username2 = tester2

password2 = testing2

User on same account as first, but without admin access

username3 = tester3

password3 = testing3

Default constraints if not defined here, the test runner will try

to set them from /etc/swift/swift.conf. If that file isn't found,

the test runner will skip tests that depend on these values.

Note that the cluster must have "sane" values for the test suite to pass.

max_file_size = 5368709122

max_meta_name_length = 128

max_meta_value_length = 256

max_meta_count = 90

max_meta_overall_size = 4096

max_object_name_length = 1024

container_listing_limit = 10000

account_listing_limit = 10000

max_account_name_length = 256

max_container_name_length = 256

collate = C

[unit_test]
fake_syslog = False

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////
And here is my log when run run_tests.py
/////////////////////////////////////////runtest.log////////////////////////////////////////////////////////
************* Module cdmi
C0111: 1,0: Missing docstring
W0403: 17,0: Relative import 'cdmi', should be 'cdmi.cdmi'
C0111: 37,4:filter_factory.cdmi_filter: Missing docstring
************* Module cdmi.cdmi
W0403: 31,0: Relative import 'cdmiapp.cdmibase', should be 'cdmi.cdmiapp.cdmibase'
W0403: 33,0: Relative import 'cdmiapp.cdmibase', should be 'cdmi.cdmiapp.cdmibase'
W0403: 34,0: Relative import 'cdmiapp.cdmicontrollers', should be 'cdmi.cdmiapp.cdmicontrollers'
W0403: 36,0: Relative import 'cdmiapp.cdmicommoncontroller', should be 'cdmi.cdmiapp.cdmicommoncontroller'
W0403: 38,0: Relative import 'cdmiapp.noncdmicontrollers', should be 'cdmi.cdmiapp.noncdmicontrollers'
W0403: 40,0: Relative import 'cdmiapp.cdmiutils', should be 'cdmi.cdmiapp.cdmiutils'
W0613: 50,0:CdmiMiddleware.init: Unused argument 'args'
W0613: 50,0:CdmiMiddleware.init: Unused argument 'kwargs'
C0103: 59,4:CdmiMiddleware.get_container_controller_by_version: Invalid name "get_container_controller_by_version" (should match [a-z_][a-z0-9_]{2,30}$)
C0111: 59,4:CdmiMiddleware.get_container_controller_by_version: Missing docstring
R0201: 59,4:CdmiMiddleware.get_container_controller_by_version: Method could be a function
C0103: 66,4:CdmiMiddleware.get_object_controller_by_version: Invalid name "get_object_controller_by_version" (should match [a-z_][a-z0-9_]{2,30}$)
C0111: 66,4:CdmiMiddleware.get_object_controller_by_version: Missing docstring
R0201: 66,4:CdmiMiddleware.get_object_controller_by_version: Method could be a function
C0111: 73,4:CdmiMiddleware.get_controller: Missing docstring
R0914: 73,4:CdmiMiddleware.get_controller: Too many local variables (21/15)
C0103:132,16:CdmiMiddleware.get_controller: Invalid name "d" (should match [a-z_][a-z0-9_]{2,30}$)
C0103:157,16:CdmiMiddleware.get_controller: Invalid name "d" (should match [a-z_][a-z0-9_]{2,30}$)
C0103:166,16:CdmiMiddleware.get_controller: Invalid name "d" (should match [a-z_][a-z0-9_]{2,30}$)
C0103:172,16:CdmiMiddleware.get_controller: Invalid name "d" (should match [a-z_][a-z0-9_]{2,30}$)
C0103:177,12:CdmiMiddleware.get_controller: Invalid name "d" (should match [a-z_][a-z0-9_]{2,30}$)
W0612: 90,12:CdmiMiddleware.get_controller: Unused variable 'content_length'
W0612: 81,12:CdmiMiddleware.get_controller: Unused variable 'content_is_object'
R0912: 73,4:CdmiMiddleware.get_controller: Too many branches (26/12)
R0915: 73,4:CdmiMiddleware.get_controller: Too many statements (68/50)
W0142:200,25:CdmiMiddleware.call: Used * or ** magic
C0111:226,4:filter_factory.cdmi_filter: Missing docstring
W0611: 44,0: Unused import split_path
W0611: 41,0: Unused import Request
W0611: 31,0: Unused import Consts
W0611: 41,0: Unused import Response
R0801: 1,0: Similar lines in 2 files
==cdmi.init:29
==cdmi.cdmi:218
cdmi_root = conf.setdefault('cdmi_root', 'cdmi')
cdmi_root = cdmi_root.lstrip('/ ').rstrip('/ ')
conf['cdmi_root'] = cdmi_root
conf['cdmi_root_length'] = len(cdmi_root.split('/'))
conf.setdefault('cdmi_version_supported', '1.0.1')
conf.setdefault('cdmi_capability_id', 'cdmi_capabilities')

def cdmi_filter(app):
    return CdmiMiddleware(app, conf)

return cdmi_filter

cdmi/cdmiapp/cdmicommoncontroller.py:384:57: W291 trailing whitespace
1 W291 trailing whitespace
cdmi/cdmi.py:31: 'Consts' imported but unused
cdmi/cdmi.py:41: 'Request' imported but unused
cdmi/cdmi.py:41: 'Response' imported but unused
cdmi/cdmi.py:44: 'split_path' imported but unused
cdmi/cdmi.py:81: local variable 'content_is_object' is assigned to but never used
cdmi/cdmi.py:90: local variable 'content_length' is assigned to but never used
cdmi/cdmiapp/cdmicontrollers.py:16: 'Controller' imported but unused
cdmi/cdmiapp/cdmicontrollers.py:22: 'unquote' imported but unused
cdmi/cdmiapp/cdmicontrollers.py:22: 'quote' imported but unused
cdmi/cdmiapp/cdmicontrollers.py:23: 'Response' imported but unused
cdmi/cdmiapp/cdmicontrollers.py:24: 'get_logger' imported but unused
cdmi/cdmiapp/cdmicontrollers.py:25: 'split_path' imported but unused
cdmi/cdmiapp/cdmicontrollers.py:26: 'http_connect_raw' imported but unused
cdmi/cdmiapp/cdmicommoncontroller.py:21: 'get_logger' imported but unused
cdmi/cdmiapp/cdmicommoncontroller.py:26: 'mimetypes' imported but unused
cdmi/cdmiapp/cdmicommoncontroller.py:291: undefined name 'Excetpion'
cdmi/cdmiapp/cdmiutils.py:23: 'HTTPConnection' imported but unused
cdmi/cdmiapp/cdmiutils.py:173: undefined name 'HTTPSConnection'
cdmi/cdmiapp/noncdmicontrollers.py:16: 'Controller' imported but unused
cdmi/cdmiapp/noncdmicontrollers.py:16: 'Consts' imported but unused
cdmi/cdmiapp/noncdmicontrollers.py:22: 'unquote' imported but unused
cdmi/cdmiapp/noncdmicontrollers.py:23: 'split_path' imported but unused
cdmi/cdmiapp/noncdmicontrollers.py:24: 'Response' imported but unused

Pychecker report

Warnings...

[system path]/httplib.py:807: INTERNAL ERROR -- STOPPED PROCESSING FUNCTION --
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pychecker/warn.py", line 242, in _checkFunction
_checkCode(code, codeSource)
File "/usr/lib/python2.7/dist-packages/pychecker/warn.py", line 155, in _checkCode
raise NotImplementedError('No DISPATCH member for op %r' % op)
NotImplementedError: No DISPATCH member for op 50

[system path]/urlparse.py:352: (dict) shadows builtin

[system path]/swift/common/bufferedhttp.py:101: Overridden method (getresponse) doesn't match signature in class (httplib.HTTPConnection)

[system path]/swift/common/utils.py:716: Function (import) doesn't support **kwArgs

[system path]/webob/request.py:154: No class attribute (environ) found
[system path]/webob/request.py:161: No class attribute (environ) found
[system path]/webob/request.py:164: No class attribute (environ) found
[system path]/webob/request.py:166: No class attribute (environ) found
[system path]/webob/request.py:167: No class attribute (environ) found
[system path]/webob/request.py:190: No class attribute (environ) found
[system path]/webob/request.py:206: No class attribute (environ) found
[system path]/webob/request.py:207: No class attribute (environ) found
[system path]/webob/request.py:216: No class attribute (environ) found
[system path]/webob/request.py:218: No class attribute (environ) found
[system path]/webob/request.py:220: No class attribute (environ) found
[system path]/webob/request.py:233: No class attribute (environ) found
[system path]/webob/request.py:378: No class attribute (environ) found
[system path]/webob/request.py:379: No class attribute (environ) found
[system path]/webob/request.py:380: No class attribute (environ) found
[system path]/webob/request.py:381: No class attribute (environ) found
[system path]/webob/request.py:384: No class attribute (environ) found
[system path]/webob/request.py:388: No class attribute (environ) found
[system path]/webob/request.py:400: No class attribute (environ) found
[system path]/webob/request.py:401: No class attribute (environ) found
[system path]/webob/request.py:402: No class attribute (environ) found
[system path]/webob/request.py:403: No class attribute (environ) found
[system path]/webob/request.py:404: No class attribute (environ) found
[system path]/webob/request.py:407: No class attribute (environ) found
[system path]/webob/request.py:421: No class attribute (environ) found
[system path]/webob/request.py:422: No class attribute (environ) found
[system path]/webob/request.py:429: No class attribute (environ) found
[system path]/webob/request.py:441: No class attribute (environ) found
[system path]/webob/request.py:442: No class attribute (environ) found
[system path]/webob/request.py:443: No class attribute (environ) found
[system path]/webob/request.py:446: No class attribute (environ) found
[system path]/webob/request.py:466: No class attribute (environ) found
[system path]/webob/request.py:467: No class attribute (environ) found
[system path]/webob/request.py:469: No class attribute (environ) found
[system path]/webob/request.py:471: No class attribute (environ) found
[system path]/webob/request.py:473: No class attribute (environ) found
[system path]/webob/request.py:474: No class attribute (environ) found
[system path]/webob/request.py:588: No class attribute (environ) found
[system path]/webob/request.py:673: No class attribute (environ) found
[system path]/webob/request.py:684: No class attribute (environ) found
[system path]/webob/request.py:711: No class attribute (environ) found
[system path]/webob/request.py:715: No class attribute (environ) found
[system path]/webob/request.py:774: (input) shadows builtin
[system path]/webob/request.py:817: No class attribute (environ) found
[system path]/webob/request.py:818: No class attribute (environ) found
[system path]/webob/request.py:837: No class attribute (environ) found
[system path]/webob/request.py:849: No class attribute (environ) found
[system path]/webob/request.py:862: No class attribute (environ) found
[system path]/webob/request.py:869: No class attribute (environ) found
[system path]/webob/request.py:1022: No class attribute (environ) found
[system path]/webob/request.py:1033: Function return types are inconsistent
[system path]/webob/request.py:1170: Invalid arguments to (setattr), got 4, expected 3
[system path]/webob/request.py:1175: No class attribute (environ) found
[system path]/webob/request.py:1177: Invalid arguments to (getattr), got 3, expected 2
[system path]/webob/request.py:1179: No class attribute (environ) found
[system path]/webob/request.py:1183: Invalid arguments to (delattr), got 3, expected 2
[system path]/webob/request.py:1187: No class attribute (environ) found

[system path]/webob/response.py:162: Invalid arguments to (str), got 2, expected 1
[system path]/webob/response.py:166: Statement appears to have no effect
[system path]/webob/response.py:980: Function return types are inconsistent

cdmi/cdmi.py:50: Parameter (args) not used
cdmi/cdmi.py:50: Parameter (kwargs) not used
cdmi/cdmi.py:82: Local variable (content_is_object) not used
cdmi/cdmi.py:90: Local variable (content_length) not used

cdmi/cdmiapp/cdmibase.py:65: Parameter (account_name) not used
cdmi/cdmiapp/cdmibase.py:65: Parameter (env) not used
cdmi/cdmiapp/cdmibase.py:65: Parameter (kwargs) not used
cdmi/cdmiapp/cdmibase.py:73: Parameter (env) not used
cdmi/cdmiapp/cdmibase.py:73: Parameter (kwargs) not used
cdmi/cdmiapp/cdmibase.py:82: Parameter (env) not used
cdmi/cdmiapp/cdmibase.py:82: Parameter (start_response) not used
cdmi/cdmiapp/cdmibase.py:181: Parameter (account_name) not used
cdmi/cdmiapp/cdmibase.py:181: Parameter (kwargs) not used
cdmi/cdmiapp/cdmibase.py:186: Parameter (start_response) not used
cdmi/cdmiapp/cdmibase.py:217: Parameter (kwargs) not used
cdmi/cdmiapp/cdmibase.py:222: Parameter (start_response) not used

cdmi/cdmiapp/cdmicommoncontroller.py:26: Imported module (mimetypes) not used
cdmi/cdmiapp/cdmicommoncontroller.py:39: Parameter (kwargs) not used
cdmi/cdmiapp/cdmicommoncontroller.py:67: Parameter (start_response) not used
cdmi/cdmiapp/cdmicommoncontroller.py:111: Parameter (start_response) not used
cdmi/cdmiapp/cdmicommoncontroller.py:278: Parameter (valid_units) not used
cdmi/cdmiapp/cdmicommoncontroller.py:291: No global (Excetpion) found
cdmi/cdmiapp/cdmicommoncontroller.py:308: Parameter (start_response) not used
cdmi/cdmiapp/cdmicommoncontroller.py:397: Parameter (env) not used
cdmi/cdmiapp/cdmicommoncontroller.py:397: Parameter (start_response) not used
cdmi/cdmiapp/cdmicommoncontroller.py:493: Parameter (start_response) not used

cdmi/cdmiapp/cdmiutils.py:95: Parameter (logger) not used
cdmi/cdmiapp/cdmiutils.py:117: Setting method to itself has no effect
cdmi/cdmiapp/cdmiutils.py:118: Setting path to itself has no effect
cdmi/cdmiapp/cdmiutils.py:152: Parameter (get_body) not used
cdmi/cdmiapp/cdmiutils.py:152: Parameter (logger) not used
cdmi/cdmiapp/cdmiutils.py:152: Parameter (method) not used
cdmi/cdmiapp/cdmiutils.py:152: Parameter (query_string) not used
cdmi/cdmiapp/cdmiutils.py:173: No global (HTTPSConnection) found
cdmi/cdmiapp/cdmiutils.py:174: Object (req) has no attribute (server_portport)

7 errors suppressed, use -#/--limit to increase the number of errors displayed
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////
So what is the problem? Please help! Thank you in advance!

Sincerely,
Peng.Shi

Listing large numbers of objects

Hi,

it seems that the Swift API, which is ultimately called for requests to the CDMI API, has an upper page size limit of 10000 elements. We have a container that has more than 10000 elements under a single (pseudo) path.

Containers (pseudo-directories) with more than 10000 elements must be listed through pagination (last element of a page given as marker to start the next page).

The implementation of cdmiutils.check_resource() does not do this at the moment. It merely lists the first 10000 elements and returns them.

Cheers,
Björn

Running test within the test directory will fail

Test should be update so the can be run in working directory. Currently when running 'python test_cdmi_object.py' the test will fail:

ERROR: test_create_object_cdmi_multipart (main.TestCDMIObject)

Traceback (most recent call last):
File "test_cdmi_object.py", line 172, in test_create_object_cdmi_multipart
fp = open(path + os.sep + 'desert.jpg', 'rb')
IOError: [Errno 2] No such file or directory: '/desert.jpg'

ERROR: test_create_object_non_cdmi_multipart (main.TestCDMIObject)

Traceback (most recent call last):
File "test_cdmi_object.py", line 216, in test_create_object_non_cdmi_multipart
fp = open(os.sep.join([path, 'desert.jpg']), 'rb')
IOError: [Errno 2] No such file or directory: '/desert.jpg'

Running test from 'python cdmi/test_cdmi_object.py' works.

Also tests should make use of os.sep instead of '/'

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.