eliotfowler / pigion-core Goto Github PK

We need a way to have a list of users that are whitelisted to sign up. That way, with a public web app, people can't sign up at will and rack up huge charges on S3.

There are going to be numerous clients - iOS, OS X, web, Windows, Android, Linux - uploading files that need to be managed by the Pigion server. For all intents and purposes, "storage" in this issue can be replaced with "S3 Bucket". There are two current approaches that are appealing to me at the moment:

Solution 1 - Upload from Client to Pigion, then from Pigion to Storage

This has the benefit of being able to process all files completely as they are being uploaded. It will be easy to monitor size restrictions as well as failed transfers. However, there will be a loss in efficiency as each chunk of each file will need to take 2 hops to get to its destination.

As the file is being streamed to the server, the chunks will be immediately passed on to the storage, so memory usage on the server will not be an issue.

Solution 2 - Send before and after requests to Pigion, then upload directly form Client to Storage

This solution is advantageous in terms of upload efficiency and implementation easiness, but it doesn't allow for as much control, at least it doesn't appear to.

In reality an approach like this would be a three step process:

First, a request will be made to Pigion with the file/user details to verify that uploading to the server shall be permitted.

For example, if the user had a filesize restriction and the requested file to be uploaded was too large, the request would be denied and the file would not be uploaded.

Assuming the request was granted, the file would be uploaded to the storage with a client-specific implementation.

Note: This is a disadvantage as each client becomes responsible for handling the direct interaction with the storage. With solution 1, Pigion could expose simple APIs such as:

POST: /files/upload/dropbox
POST: /files/upload/s3
POST: /files/upload/drive

The final step in solution 2 is notifying Pigion that the request finished. At this point, Pigion can move the file from a temporary directory to the users/directory and do any post-processing it requires.

Which approach makes more sense?