Tried to change AWS_DEFAULT_REGION and got err: {status: InvalidBucketName}

This is what I tried:

"use strict";

process.env.AWS_DEFAULT_REGION = 'eu-west-1';

const DB = require('s3-db');
const db = new DB();
// console.log('db:', db);

db.createCollection('del_man_orders_s3_db')
  .then(() => console.log('success'))
  .catch(err => console.log('err:', err));

Added a console.log() to print defaults variable from src/index.js

Defaults: { db: 
   { name: 's3-db',
     environment: 'dev',
     namePattern: '${db.name}.${db.environment}-${name}' },
  provider: { name: 'aws-s3', region: 'eu-west-1' },
  collections: { default: {} },
  serializer: {} }
err: { status: 'InvalidBucketName' }

I think its possible to have a policy apply when a request parameter is supplied. This is completely untested but it should be possible to disable deleteObject deleteObjectVersion if a header or value is provided in the request. If thats true this policy could be applied conditionally to essentially allow creation only for specific calls, we could have create and overwrite options in the API.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ExamplePolicies_EC2.html

https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html

https://aws.amazon.com/about-aws/whats-new/2018/09/amazon-s3-announces-new-features-for-s3-select/

https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html

Possibly for hydrate type functionality to load part of an object and not the entire thing? Maybe as an index?

Hi there,

when I install this module I'm not able to import it in my code.

I noticed that your main property in the package.json says /dist/index but after the command npm i I don't have the dist folder.

Why?

Ability to execute code during a collection event. Ideally, its able to be configured to call a lambda event directly. This way a primitive method of a webhook can be fairly easily implemented.

const users: Collection<User> = new Collection(User);
const newUsers: Collection<NewUser> = new Collection(NewUser);
const transformer: DuplexStream = ....;

//This should read from users, transform using the transformer and then write to newUsers.
users
   .pipe(transformer)
   .pipe(newUsers);

• .pipe() should produce a ReadStream that makes use of .find with no prefix.
• Collection itself should implement WritableStream so it can be used as a value into .pipe() as a write destination.
• ReferenceList should have a .pipe() method to produce a ReadStream which will produce documents starting with the prefix defined in .find() provided to get the ReferenceList.

Is that possible already with validator? if not that would be a good feature.

When updating a document check for version/rev to keep data consistent, on a successful update increment the version/rev number.

ex: 2 updates on the same document, one goes through the other will fail because of version mismatch.

The version/rev data would have to be stored in the metadata, and checked kind of like you do in the collection.

1. An example where the method saveDocument is used would be helpful.

2. Maybe a quick explanation or link to es6 to provide an understanding of the arrow function and promises.

const Database = require('s3-db');
const database = new Database();
const users    = database.getCollection('users');
users.get('my-user')
  .then( user => {user.age = 32; return user} )
  .then( user => user.save() );

This doesn't work because getCollection returns a promise.

Typo in collection section: database.getColletion('x',{id:{propertyName:'name'}})

I'd like to see how to create a collection, create a new document, then get document and update document.

database.createCollection('test')
      .catch(console.error)
  } )

database.getCollection('test')
  .then( collection => {
    collection.saveDocument({ "id":"321", "name":"test2", "value":123 })
      .catch(console.error)
  } )

How to (upsert) create a document if one doesn't exist, or update a document.

Some config examples would be nice.

const Database = require('s3-db');
const configuration = {
  db: {
    name: 's3-db',
    environment: 'dev',
    pageSize : 10,
    allowDrop: true
  },
  provider: {
    name: 'azure-drive',
    region: 'us-east-1'
  },
  collections: {
    default: {
      onlyUpdateOnMD5Change: false,
      collideOnMissmatch: false,
      pageSize: 1000,
      encryption: false,
      id: {
        propertyName: 'id',
        generator: collectionName => `${configuration.db}-${collectionName}-${new Date().getTime()}`
      }
    }
  }
};

The above creates an a file with: %5Bobject%20Object%5D-%5Bobject%20Object%5D-1497313127381 for some reason.

I did find some nice documentation in your repo, after I wrote this.

This example is giving me an error: TypeError: users.save is not a function, the first one is users.saveDocument because its a collection.

const Database = require('s3-db');
const database = new Database();
const user     = {name : 'Richard Cranium'};

database.getCollection('users')
  .then( users => users.save(user ))
	.then( user => {
		user.size = 1234;
		user.sex  = 'male';
		return user;
	})
	.then( user => user.save() )
	.then( user => {
		user.size = 122345;
		user.sex  = 'female';
		return user;
	})
	.then( user => user.refresh() )
	.then( user => user.delete() )
	.catch( error => console.error(error.stack) )

Because the ID is added 'to' the JSON object the current save behavior would muck up any buffer that was attempted to be saved. We also would attempt to serialize an object when it should probably be left as a Buffer.

• TypeScript
• Golang

Many services these days, such as Digital Ocean, Backblaze B2, Google Cloud Storage, and others are S3 compatible. By adding an endpoint option, it will be possible to use these competitive services too.

If I invoke locally

sls invoke local -s produccion -f Some

I get

{ db: 
   { name: 's3-db',
     environment: '$LATEST',
     namePattern: '${db.name}.${db.environment}-${name}' },
  provider: { name: 'aws-s3', region: 'us-west-2' },
  collections: { default: {} },
  serializer: {} }

Which later gives me an InvalidBucketName exception.

My workaround was setting STAGE=mystage on command line but I'm curious on how it will behave on deploy.

calling .find() for the subcollection returns the objects of the parent collection:

const companyTable = await database.getCollection('companies');
const table = companyTable.subCollection(`${company.id}-${TABLE_NAME}`);
const documentReferences = await table.find();

documentReferences would contain the files from 'companies'

Applies only to an existing object, fails if none found.
Argument takes in a function to manipulate the loaded object.
eTag matching used to reject the change if the object is modified 'underneath' the change.
If rejected, re-attempts the operation 3 times.
If fails 3 times then fails completely.
Object is re-loaded for each attempt with the current eTag.

Hi @matt-filion

👋

You missed a letter for database. getColletion instead database.getCollection ! (the "c" letter)

3 times at the end of readme.md

Have a nice day, thx for s3-db 😃

It would be helpful to have a method that compares documents and returns a true or false.

Also helpful, a method that compares key,value pairs of documents and returns a true or false.

Use case: check if a phone number already exists

Hi,
It possible to use it as is on AWS elastic beanstalk app ?

Thanks

Hello, how does s3-db deal with S3 key duplication limitation? For example you want to save a User document, meaning the username needs to be unique, S3 from what I know, does not have such facility to prevent duplication and it's eventual consistency nature will make it hard for any application to prevent such.

For example, two client creates a document with the same username (but different passwords) at about the same time which client will be able to actually make it?

http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html

Im not exactly sure what is going on here but when I call new Database() I get

SyntaxError: Unexpected token =
 at exports.runInThisContext (vm.js:53:16
    at Module._compile (module.js:414:25)
    at Module._extensions..js (module.js:442:10)
    at Object.require.extensions.(anonymous function) [as .js] (S:\\repos\\api\\api-media\\lambda\\node_modules\\babel-register\\lib\\node.js:152:7)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Module.require (module.js:366:17)
    at require (module.js:385:17)
    at new module.exports (S:\\repos\\api\\api-media\\lambda\\node_modules\\s3-db\\src\\index.js:38:27
   at Object.create (S:/repos/api/api-media/lambda/src/db/s3/index.js:4:3)
    at S:/repos/api/api-media/lambda/src/sdk/asset/index.js:36:7
    at process._tickDomainCallback (node.js:411:9)

'use strict';
var Database = require('s3-db');

//breaks right away if i comment below
//const Database = new Database();

var breaks = (accountId,type) => {
		return new Database();
}

module.exports = {
        breaks 
}

This is a really cool project. I'm trying to determine the limit of what it could/should be used for when it comes to situations where multiple users/sessions could be trying to write to the DB at the same time. I see the checkIsModified option, which seems to leave only a small window (between the checking if its been modified and the actual write) where another user/session could have written to the DB undetected. Would you say that one would need to implement there own lockfile solution to close that window completely?

For atomic transactions involving multiple models, I'm thinking a custom solution would also be needed. Do you agree?

Thanks again for the cool library.

http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#copyObject-property

• Rename: Make sure that if collideonmissmatch flag is set, that the CopySourceIfMatch attribute is used with the eTag of the source. Add configuration, that causes a renaming of the source instead of delete.
• Move: convenience method for moving a document to another collection. Then deletes the source document when confirmed successfully.
• Add a duplicate method as well, which falls back on a configuration to determine the name of the new object. Will duplicate the object into the same collection.

When listing multiple buckets to deal with under the resource, they don't end up the policy so I get permission denied. How can multiple buckets be used like below?

  iamRoleStatements:
    -  Effect: "Allow"
       Action:
         - "s3:*"
       Resource:
         Fn::Join:
           - ""
           - - "arn:aws:s3:::bucket1*"
           - - "arn:aws:s3:::bucket2*"
           - - "arn:aws:s3:::bucket2*"

Files under 4k could be stored under only the file name (might be lower, need to double check AWS s3 limits). Would need to change the lookup/find mechanism to locate files that are only a file name. In these cases the updates would be fast since they would bypass write operations and only modify metadata.