ca98am79 / connect-dynamodb Goto Github PK
View Code? Open in Web Editor NEWDynamoDB session store for Connect
Home Page: http://ca98am79.github.com/connect-dynamodb/
License: MIT License
DynamoDB session store for Connect
Home Page: http://ca98am79.github.com/connect-dynamodb/
License: MIT License
DynamoDB doesn't support encryption at rest, which means if you have sensitive data that you want to encrypt, this must be done at the application level.
Session data contains this kind of sensitive data, so one blocker for us using connect-dynamodb
is the ability encrypt session data before saving it to dynamodb. One path we're pursuing is creating a connect-session-encrypted
session store that wraps an existing session store and adds this kind of encryption. Another approach would be adding it to connect-dynamodb
directly, since it might be a common-enough use case. I'm curious what you think.
In DynamoDBStore.prototype.get function,
now >= result.Item.expires always returns false, while debugging figure out that result.Item.expires is not a number so its unable to compare with 'now' and always returns false.
else block gets executed always :
else if (result.Item.expires && now >= result.Item.expires) {
fn(null, null);
} else {
var sess = result.Item.sess.S.toString();
sess = JSON.parse(sess);
fn(null, sess);
}
Did anyone face similar issue?
The library currently creates the DynamoDB table if it doesn't exists. In that process it doesn't seem to be properly setting the table TTL to the expires
attribute of the sessions. Is there any reason for this?
2 questions:
Hello,
Is it possible to use ES6 import like this (import DynamoDBStore from 'connect-dynamodb';) instead of require?
When I try the standard import I get errors relating to:
TypeError: Cannot read properties of undefined (reading 'Store')
Thanks, Alex
Hi Team,
I have inherited a project that uses this package. Do you have any details of why this wrapper around the native aws-sdk for dynamo exists? What does this package offer over using the native?
Hi guys,
I've just tried the module and it works well. I found that the session items are stringified stored in DynamoDB, and I think there should be another way to store the session if it is an object: flattern the nested object and store the session as an item in db. Is it feasible/make sense?
On the master
branch, using the commands README:
docker run -it --rm \
--name=dynamodb-test \
-p 127.0.0.1:8000:8000 \
deangiberson/aws-dynamodb-local
export AWS_CONFIG_JSON='{"endpoint": "http://127.0.0.1:8000", "region": "us-east-1", "accessKeyId": "accesskey", "secretAccessKey": "secretaccesskey"}'
npm test
I run into this error:
TypeError [ERR_INVALID_CALLBACK]: Callback must be a function. Received undefined
at maybeCallback (node:fs:177:3)
at Object.exists (node:fs:260:3)
at Object.<anonymous> (/Users/lancegliser/Projects/lancegliser/connect-dynamodb-v2-dependency/test/test.js:9:17)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:190:29)
at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
at async Promise.all (index 0)
at async ESMLoader.import (node:internal/modules/esm/loader:281:24)
at async importModuleDynamicallyWrapper (node:internal/vm/module:437:15)
at async formattedImport (/Users/lancegliser/Projects/lancegliser/connect-dynamodb-v2-dependency/node_modules/mocha/lib/nodejs/esm-utils.js:7:14)
at async Object.exports.requireOrImport (/Users/lancegliser/Projects/lancegliser/connect-dynamodb-v2-dependency/node_modules/mocha/lib/nodejs/esm-utils.js:38:28)
at async Object.exports.loadFilesAsync (/Users/lancegliser/Projects/lancegliser/connect-dynamodb-v2-dependency/node_modules/mocha/lib/nodejs/esm-utils.js:91:20)
at async singleRun (/Users/lancegliser/Projects/lancegliser/connect-dynamodb-v2-dependency/node_modules/mocha/lib/cli/run-helpers.js:125:3)
at async Object.exports.handler (/Users/lancegliser/Projects/lancegliser/connect-dynamodb-v2-dependency/node_modules/mocha/lib/cli/run.js:370:5)
test/test.js:9:17 is:
var fs = require('fs'),
// Other stuff
var config = fs.exists('./aws-config.json') && fs.readFileSync('./aws-config.json');
The definition of fs.exists
reads:
deprecated export function exists( path: PathLike, callback: (exists: boolean) => void): void
Test whether or not the given path exists by checking with the file system. Then call the callback argument with either true or false:
import { exists } from 'fs';exists('/etc/passwd', (e) => {
console.log(e ? 'it exists' : 'no passwd!');
});
The parameters for this callback are not consistent with other Node.js callbacks. Normally, the first parameter to a Node.js callback is an errparameter, optionally followed by other parameters. The fs.exists() callback has only one boolean parameter. This is one reason fs.access() is recommended instead of fs.exists().
Using fs.exists() to check for the existence of a file before callingfs.open(), fs.readFile() or fs.writeFile() is not recommended. > Doing so introduces a race condition, since other processes may change the file's state between the two calls. Instead, user code should open/read/write the file directly and handle the error raised if the file does not exist.
If I update the test config file read to the follow it works:
let config;
try {
config = fs.readFileSync('./aws-config.json')
} catch {}
if (config) {
client = new AWS.DynamoDB(JSON.parse(config));
}
else if (process.env.AWS_CONFIG_JSON) {
var AWS = require('aws-sdk');
config = JSON.parse(process.env.AWS_CONFIG_JSON);
client = new AWS.DynamoDB(config);
}
Not sure I care to fix it, as I'm looking a different configuration in #76 but thought I'd document the issue and solution.
I created a hash key called sessionID
and got:
ValidationException: One or more parameter values were invalid: Missing the key sessionID in the item
Found out my table had to use the hash id
to work properly with this module.
Maybe add the option to choose your own hash key? Or explicitly note it within the Readme/wiki?
Thanks!
Hi, I'd rather not create a new session table every time I Deploy. I have a large cloud formation yaml file that spins up all the AWS resources I need in my stack. I was hoping to use this package to connect to the existing table. But the documentation looks like it only creates tables.
Is there a way to use this package to connect to my existing table?
I have gotten it to work on express 4 by doing
var connect = require('connect');
var DynamoDBStore = require('connect-dynamodb')(connect);
is the the right way of doing it?
express-session now supports a .touch
method on session stores.
This method should just update the expiry. It allows you to set resave: false
and avoid race conditions that cause session data trampling.
Hello,
I have tried to implement your solution and have managed to store sessions in dynamodb and retrieve them when my client make requests.
However, I have ran into an issue with some sessions that only contains an id, an expiration number but no other field (sess and type). When my client make a request with this id, I get a "Cannot read property 'S' of undefined" error in connect-dynamodb.js line 146
var sess = result.Item.sess.S.toString();
It makes sense, because there is no valid sess object.
I can try to detail my configuration more if needed, but do you please have any idea about what may cause this error ?
Thank you very much, best regards
With the advent of DynamoDB TTL, it is now possible to automatically delete rows (sessions) from DynamoDB automatically based on a field (e.g. expires
). Unfortunately, it only currently supports seconds since epoch (not milliseconds, which is what is stored in expires
).
Is there a way to leverage this new DynamoDB feature?
Related: https://forums.aws.amazon.com/thread.jspa?threadID=251306, as it could also be a feature request on AWS's end.
pls
When connect-dynamodb creates the table for the first time, it logs the result of the request out to console.log. Is this correct behavior? Doesn't it need to block somehow until the creation of the table is complete? I understand this would require some work on the API, but it also seems like a race condition on whether the table gets created before connect-dynamodb uses it.
https://github.com/ca98am79/connect-dynamodb/blob/master/lib/connect-dynamodb.js#L69-L89
Is there a way to disable always checking if the table exists?
The reason for this is that we plan to use express-session with dynamoDb and serverless (using aws lambda), so checking if the table exists each time a lambda function gets fired up seems redundant.
I'm having trouble figuring out how you set the session timeout. I see the reap interval, but that looks like simple clean up of dead session records. How can you set the lifespan of the session?
Hi,
We have noticed that the sessions table is not being cleared down by the reap function.
We think the problem is that the type of the "expires" field used in the reap scan filter is an "N", whereas the "expires" field is initially saved as an "S". This results in the "LT" scan not matching any rows.
Regards,
@roylines and @jamesbloomer
tsc -v
Version 5.0.4
tsc
node_modules/connect-dynamodb/index.d.ts:10:3 - error TS2666: Exports and export assignments are not permitted in module augmentations.
10 export = ConnectDynamoDB;
~~~~~~
We use claudiajs to package our functions with:
claudia update --no-optional-dependencies ... (Idem: npm install --no-optional)
to reduce the size of our package.
As recommended by AWS to not include AWS-SDK in node_modules when deploying to lambda ...
and we notice that you install this in dependencies:
"dependencies": {
"aws-sdk": "",
"connect": ""
},
it should be in optionalDependencies:
"optionalDependencies": {
"aws-sdk": "*"
},
It will not change anything, except that those that wish to not include this package will be able to do it...
by using npm install -no-optionnal
So the node_modules/connect-dynamodb directory will be a few Ko instead of ~30.4Mo
Small change for a huge difference, especially when sending to aws lamda
Thanks
The size of the package is the only buttleneck for us now...
Love your package (use it in Api-Gateway->Lambda->Dynamo)
Yves Laroche
PS.: I did a pull request...
Not sure why, but reaping does not appear to be working for me, and the number of sessions stored in dynamo is growing seemingly without bound. I'm not using a custom reapInterval so it should be using the 10 min default, but there are sessions that are weeks old in my table. Any help would be appreciated!
would you accept a PR to allow ProvisionedThroughput to be configurable?
e.g.
var options = {
// Optional DynamoDB table name, defaults to 'sessions'
table: 'myapp-sessions',
// Optional path to AWS credentials and configuration file
// AWSConfigPath: './path/to/credentials.json',
// Optional JSON object of AWS credentials and configuration
AWSConfigJSON: {
accessKeyId: <YOUR_ACCESS_KEY_ID>,
secretAccessKey: <YOUR_SECRET_ACCESS_KEY>,
region: 'us-east-1'
},
// Optional client for alternate endpoint, such as DynamoDB Local
client: new AWS.DynamoDB({ endpoint: new AWS.Endpoint('http://localhost:8000')}),
// Optional clean up interval, defaults to 600000 (10 minutes)
reapInterval: 86400000, // 1 day
readCapacityUnits: 20,
writeCapacityUnits: 20
};
The documentation for the AWS SDK mentions being able to set the AWS region via an environment variable here. connect-dynamodb doesn't yet respect this environment variable. If the developer is unaware of this it can lead to the session table being hosted in a different region than the rest of the database.
By Default Dynamodb issues non-consistent reads. It would be nice to have an option to use consistent reads.
express-session deprecated undefined resave option; provide resave option at src/config/session.ts:15:42
express-session deprecated undefined saveUninitialized option; provide saveUninitialized option at src/config/session.ts:15:42
Seeing the above in my API as I use it. Not sure if this is related to the #76 as I'm using that or a constant. Should confirm what it affects.
Is it possible to create an ondemand Table instead of a provisioned one.
I looked over your code and did not seem to think this was possible?
Is there a reason why?
Thanks so much for the package.
I omitted the table attribute to the config options and got this error.
Running "watch" task
Waiting...Express server listening on port 3000
{ [MissingRequiredParameter: Missing required key 'TableName' in params]
code: 'MissingRequiredParameter',
name: 'MissingRequiredParameter' } null
OK
It worked after adding the table attribute.
Might be more efficient to use DynamoDB's new parallel scan feature for the reap method.
Hi,
As the title, is there @types/connect-dynamodb or index.d.ts for Typescript? Thank you :)
When sess.cookie.maxAge is missing from the session options the expiry is defaulting to 86400 milliseconds. It should default to 86400000 milliseconds
Updated to 1.0.8 to get the fix for reaping but now encountering a different issue during testing. It appears to come from the line that references self.prefix.length
in the reaping logic. Has anyone else seen/resolved this problem?
TypeError: Cannot read property 'length' of undefined
at destroyDataAt (/.../node_modules/connect-dynamodb/lib/connect-dynamodb.js:211:48)
at Response.destroy (/.../node_modules/connect-dynamodb/lib/connect-dynamodb.js:219:9)
at Response.onScan (/.../node_modules/connect-dynamodb/lib/connect-dynamodb.js:196:21)
at Request.<anonymous> (/.../node_modules/aws-sdk/lib/request.js:355:18)
at Request.callListeners (/.../node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit /.../node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/.../node_modules/aws-sdk/lib/request.js:615:14)
at Request.transition (/.../node_modules/aws-sdk/lib/request.js:22:10)
(paths are abbreviated)
Can you update the npm package with a new version so that we have the new package.json ( with optionalDepencencies)
Thx...
I did not do a pull since i don't know what version you want to put ...
Yves Laroche
Key: {
id: {
'S': sid
}
}
Uses id
as a hard-coded value. Should use value of hashKey
option.
var KeyValue = {}
KeyValue[this.hashKey] = { 'S' : sid };
...
Key : KeyValue
Results in ValidationException
connect-dynamodb/lib/connect-dynamodb.js
Line 345 in 2e73c15
I think this should be ? now + sess.cookie.maxAge/1000
It doe snot work without a credentials.json file
{ accessKeyId: AWS_ACCESS_KEY_ID, secretAccessKey: AWS_SECRET_KEY, region: 'us-east-1' } or smth like that would enable us not to have credential in the repo. I would improve it but I am sure someone understands these things much better than me.
The date math introduced in version 1.0.12, moving from millisecond unix epoch timestamps to second unix epoch timestamps, breaks session reaping when using sessions across multiple servers that serve sites to the same domain.
One of our servers, running connect-dynamodb v1.0.11 was reaping sessions and comparing against millisecond epoch timestamps, thus deleting every session made by our other server, running connect-dynamodb v1.0.12, since these timestamps are always shorter than their millisecond-length counterparts.
This results in sessions lasting no longer than 10 minutes (default reapInterval), and users being logged out automatically.
I'd consider this a breaking change, which definitely shouldn't happen when updating the patch version.
Allow the flexibility to pass the express-session directly as a parameter to the adapter constructor.
Today, then way connect-dynamodb seems to allow initialization is like this:
var session = require('express-session');
var DynamoDBStore = require('connect-dynamodb')({session: session});
Where as frameworks such as sails, initialize the store like this. Consequently, connect-dynamodb is not directly compatible with sails, which is gaining popularity quickly.
The issue is to request support initialisation by passing the express-session directly as a param. Like this:
var DynamoDBStore = require('connect-dynamodb')(require('express-session'));
This is today also supported by connect-mongo and will bring us feature-parity.
HI,
Is there anyway i can ignore aws-sdk from dependencies. I tried with
npm install --no-optional still its packed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.