Whenever files are transferred from a directory, an archive file is created in the directory (if it did not exist before)

The file is named .ditto_archived

Files with this name are excluded from the file transfer processes

Hi Devs,

Is there an obvious licence file that repo should have? If so it would be good to get that in place.

Cheers,
James

Bucket size for each

Ensure that when a bucket name that breaks the S3 convention is entered an appropriate error catching mechanism is in place and a warning returned

To avoid summer-time/winter-time clashes it would be best to store object-last-updated timestamps in Unix time (or at least in UTC). We need to clarify what is returned by the S3 interface and may need to update DITTO's code accordingly.

Reviewer: Ray
Pull request: #15

Require access token to run any commands

Service to connect with LDAP

Reviewer: Ray, Dan

Python distribution options: Wheel, Egg

Running an POST listpresent/ request with a bucket name that doesn't exist causes the server to crash rather than returning a 404.

When copying using copydir with directory argument as None, directory is treated as None so assumed doesn't exist in S3 storage and thus all files will be copied again even though they can already exist in the S3 end.

I think we will need to use a "user" field in the POST JSON data to pass the user from SimDB (the CCFE simulation management tool) to Ditto, so Ditto can query for accessible buckets.

Reviewer: Rob

A POST request to the Web API (route "copydir") copies all files in a directory, so long as that directory didn't previously exist at the S3 end

Acceptance criteria:
POST request to: http://172.28.129.160:8888/copydir/ with correctly formatted JSON body (e.g. http://172.28.129.160:8888/copydir/ {"bucket": "test-bucket-name", "directory": "testdir/testsubdir"})

Should return a warning message if there are no files in the local directory or the bucket does not exist.

Should return a warning message if the (relative) directory already exists at the S3 end, and not transfer any files. (Assumption: empty directories do not exist in S3; directories only exist when they contain at least one file)

Otherwise, copies all files in the directory (and sub-directories) to the S3 storage, preserving the directory structure. Returns a message summarizing the transfer (number of files, amount of data).

Story points: 6

A POST request to the Web API (route "copyupdate") copies or update all files

Acceptance criteria:
Both the web app and the S3 interface need to be up and running.

POST request to http://172.28.129.160:8888/copyupdate/ with correctly formatted JSON body (e.g. http://172.28.129.160:8888/copyupdate/ {"bucket": "test-bucket-name", "directory": "testdir/testsubdir"}) returns a JSON response

The relative path to directory is an optional argument (i.e. http://172.28.129.160:8888/copyupdate/testbucket/ should also work). When it is absent all new files should be copied/updated.

Should return a warning message if the bucket does not exist or there are no files in the local directory.

Otherwise, copies all new files in the directory (and subdirectories) to the S3 storage, preserving the directory structure. Updates all existing files that have changed (local file has more recent last-modified tag). Returns a message summarizing the new files transferred (number of files, amount of data), the updated files transferred, and also the number of files not transferred because they have not changed.

Story points: 2

A DELETE request to the Web API (route "deletefile") deletes a single file at the S3 end

Acceptance criteria:
GET request to: http://172.28.129.160:8888/deletefile/{bucket-name}/{relative-path-to-file} (e.g. http://172.28.129.160:8888/deletefile/test-bucket/testdir/testfile.txt)

Should return a warning message if the bucket or the file does not exist at the S3 end

Otherwise, deletes the file and returns a message confirming the deletion

Only users in admin group authorised to create buckets

New buckets included in bucket settings service and added to bucket_settings.ini

Attempting to upload a directory with broken links or unreadable files currently causes an Internal Server Error. These kind of generic errors (most seemingly coming from boto) should be caught and returned as an error message.

The content header of the ditto web api handlers returns the server type and version by default. This is a security risk and so needs to be disabled.

Acceptance criteria

None of the ditto web api handlers returns the server in its response header.

Note that tasks related to createbucket are in separate issues:

• (#67) only admin users can create buckets
• (#82) when creating a bucket, the bucket settings file is updated

Therefore, these tasks are not covered in this PR.

Acceptance criteria:
A bucket_settings.ini file must be written in in ditto_web_api/DittoWebApi, using example_bucket_settings.ini as a template. Note that the group tags for each bucket will be those used to compare against the groups tags in security_configuration.ini

Confirm that a user can only interact with a bucket that they have authorisation for: i.e. one or more of the groups they belong to is on the list of groups for that bucket. Any un-authorised requests should return a 403 error.

When creating a new bucket need to specify where the bucket location is. e.g. us-east-1. Need to look at why this matters, what we need and if we need to configure. Currently used value is eu-west-1

Reviewer: Ray

Following on from #7 whenever a file is transferred from a directory this event is recorded in the archive file located in that directory.

The following information is captured

• File name
• Operation (copy new, update, download, etc)
• Version of DITTO used
• Last updated timestamp
• Name of bucket where it is replicated
• File size (bytes)

Story points: 4

A POST request to the Web API (route "copynew") copies all files that didn't previously exist at the S3 end

Acceptance criteria:
Both the web app and the S3 interface need to be up and running.

POST request to http://172.28.129.160:8888/copynew/ with correctly formatted JSON body (e.g. http://172.28.129.160:8888/copynew/ {"bucket": "test-bucket-name", "directory": "testdir/testsubdir"}) returns a JSON response

The relative path to directory is an optional argument ({"bucket": "test-bucket-name"} should also work). When it is absent all files should be copied.

Should return a warning message if there are no files in the local directory.

Otherwise, copies all files in the directory (and subdirectories) to the S3 storage, preserving the directory structure. Returns a message summarizing the transfer (number of files, amount of data) and also the number of files not transferred because they already exist at the S3 end.

Reviewer: Rob, Dan

A POST request to the Web API (route "listpresent", JSON body) returns a list of the files already present at the server end.
#16

Acceptance criteria:
Both the web app and the S3 interface need to be up and running.

POST request to http://172.28.129.160:8888/listpresent/ with correctly formatted JSON body (e.g. http://172.28.129.160:8888/listpresent/ {"bucket": "test-bucket-name", "directory": "testdir/testsubdir"}) returns a JSON response

The relative path to directory is an optional argument ({"bucket": "test-bucket-name"} should also work). When it is absent all files should be returned.

• When the bucket does not exist a warning message should be returned
• When there are no files in the directory at the S3 end this should return an empty array
• When there are files in the directory at the S3 end this should return information for each one: name (including relative path), which bucket it is in, size in bytes, last modified (in UNIX time)

Acceptance criteria

Failed requests should return a status: fail and status code: 400 or 404, where appropriate, when unsuccessful requests are made. This includes:

• listpresent to a bucket that doesn't exist
• copydir, copynew, copyupdate when a directory doesn't exist or bucket doesn't exist
• deletefile when file doesn't exist or bucket doesn't exist
• createbucket when bucket name is rejected or already exists

Story points: 2
Reviewer: Rob

A POST request to the Web API (route "createbucket") creates a bucket at the server end.

Acceptance criteria:
Both the web app and the S3 interface need to be up and running.

Bucket names are standardised: they must either start with test- or with ditto-. This standardisation should be specified in the configuration file.

POST request to http://172.28.129.160:8888/createbucket/{bucket-name} (e.g. http://172.28.129.160:8888/createbucket/ditto-example/) returns a JSON response confirming that the bucket was created

Returns a warning message (and makes no changes) if

• no bucket name is given in the route
• the bucket name does not fit the pattern specified in the configuration file
• a bucket of that name already exists

#36

Continuous integration using Travis

Reviewer: Oliver

Acceptance criteria:
The LoggingLevel setting in the configuration.ini file can be set to the options listed in the Deployment section of the wiki.
The app should still run and log when the level is set to any of these options, and refuse to run when it is set to anything else.

For requests, "http://url/listpresent/" is found but "http://url/listpresent" returns a 404. Does the API need to be this fussy? Would it cause issues to allow both formats?