2do2go / rate-limit-mongo Goto Github PK

Can this be optionally turned off?

The implementor could create the indexes manually if they were detailed in the readme.

This is currently failing for me with Atlas even when the writer has admin permissions, and the index exists. Looking into this more on my end.

Thanks

Hello,

Before trying to implement rate-limit-mongo, to successfully connect to MongoDB I use a connection string in these formats:

Local - there is no username, password or database defined:

connection_string = mongodb://localhost:27017"

Production Server (MongoDB Atlas)
The related env property value is defined like this - a username, password and database is defined:

connection_string = "mongodb+srv://<user-name>:<user-password>@cluster0.vewnz.azure.mongodb.net/Cluster0?retryWrites=true&w=majority&useNewUrlParser=true&useUnifiedTopology=true"

Note that Cluster0 (with a capital 'C') is the database name defined.

The connection is made with this code:

const MongoClient = require('mongodb').MongoClient;  

MongoClient.connect(connection_string, function(err, client) {
    if (err) {
        throw err;
    } else {
        mongo_client = client;
        console.log("made a connection to server");
    }
});

And when I want to make requests to a database, I use something like this (database and collection is defined):

var collection = mongo_client.db("some-database-name").collection("some-collection-name");

However, now I am trying to implement rate-limit-mongo.

I can see that the default configuration options for MongoStore are:

store: new MongoStore({
  uri: "mongodb://127.0.0.1:27017/test_db",
  collectionName: "expressRateRecords",
  user: "",
  password: "",
  authSource: "",
  collection: "",
  expireTimeMs: 60 * 1000,
  resetExpireDateOnChange: false,
  errorHandler: "_.noop",
  createTtlIndex: true
});

I am wondering whether I can define MongoStore with these parameters only:

uri: connection_string,
collectionName: "expressRateRecords"

Because the user and password are already defined in my connection string (in production environment) and locally they are not required?

Thank You.

Edit:

The suggestion above works in local environment with the connection string mongodb://localhost:27017, but not in production environment with MongoDB Atlas and the connection string:

mongodb+srv://<user-name>:<user-password>@cluster0.vewnz.azure.mongodb.net/Cluster0?retryWrites=true&w=majority&useNewUrlParser=true&useUnifiedTopology=true

There is no error message displayed on the server, but the entry is not added to the expressRateRecords (database) > expressRateRecords (collection).

It also looks like it has created a database called Cluster0 containing a collection called expressRateRecords.

And I had already created a database called expressRateRecords containing a collection called expressRateRecords.

Entries are not going into either database or collection.

Edit 2:

I have also tried the following:

var apiLimiter = rateLimit({
    windowMs: 5 * 60 * 1000, // 5 minutes
    max: 3,
    headers: false,
    handler: function(req, res) {
        res.status("429");
    },
    store: new MongoStore({
        uri: process.env.RATE_LIMIT_MONGO_CONFIG_CONNECTION_STRING,
        collectionName: "expressRateRecords",
        user: process.env.RATE_LIMIT_MONGO_CONFIG_USER,
        password: process.env.RATE_LIMIT_MONGO_CONFIG_PASSWORD
    })
});

When the connection string is:

mongodb+srv://cluster0.vewnz.azure.mongodb.net/expressRateRecords?retryWrites=true&w=majority&useNewUrlParser=true&useUnifiedTopology=true

I get 500 error:

MongoError: Database name expressRateRecords?retryWrites=true&w=majority&useNewUrlParser=true&useUnifiedTopology=true is too long.  Max database name length is 38 bytes.

When the connection string is:

mongodb+srv://cluster0.vewnz.azure.mongodb.net/expressRateRecords

I get 500 error:

"MongoError: Authentication failed"

When the connection string is:

mongodb+srv://user-name:[email protected]/expressRateRecords

I get 500 error:

"MongoError: Authentication failed"

After updating to 2.3.2 version of rate-limit-mongo, getting an error whenever the application starts up. The exception stack trace:

[TypeError]: Cannot read property 'counter' of null
      at Group.<anonymous> (/fusion/node_modules/rate-limit-mongo/lib/mongoStore.js:158:32)
      at Group.<anonymous> (/fusion/node_modules/twostep/lib/twoStep.js:171:15)
      at next (/fusion/node_modules/twostep/lib/twoStep.js:159:9)
      at Group.done (/fusion/node_modules/twostep/lib/twoStep.js:30:16)
      at Group._fillSlot (/fusion/node_modules/twostep/lib/twoStep.js:52:8)
      at /fusion/node_modules/twostep/lib/twoStep.js:77:9
      at processTicksAndRejections (internal/process/task_queues.js:79:11) {

Noticed in 2.3.2, there was a commit (41e917b) that changed the options passed to be returnDocument or returnNewDocument instead of returnOriginal.

The error occurs whenever the upserts needs to create a new record. It returns an object which doesn't have the counter property, which seems to be in line with returning the pre-upsert record rather than the post-upsert record.

Great work!

Would it possible to add a CHANGELOG? Thanks!

There is no LICENSE file in the project, but there is a license field in package.json indicating that it is licensed by MIT. I have a fork of this repository and i want to redistribute it legally.

What is the current license status?

Hello,

I am referring to:

https://github.com/2do2go/rate-limit-mongo#configuration

I am confused as to why there are two parameters that seem to refer to the same thing?

collectionName: string -- name of collection for storing records. Defaults to expressRateRecords

collection: object -- mongodb collection instance. Required if uri hasn't been set.

Sorry if I am not understanding this correctly.

Thank You.

(node:26384) [MONGODB DRIVER] DeprecationWarning: collection.findOneAndUpdate option [returnOriginal] is deprecated and will be removed in a later version.

Seem to just use the 1 minute default no matter what I set for the options. Tried:

var RateLimit = require('express-rate-limit')
var MongoStore = require('rate-limit-mongo')

var limiter = new RateLimit({
    store: new MongoStore({
        uri: databaseUrl,
        expireTimeMs: 1,
    }),
    skip: shouldSkipRateLimiter,
    windowMs: 1,
    max: 5,
})
app.use(limiter)

Hi @okv,

I recently helped out with a Typescript rewrite for the express-rate-limit package, which also introduced a couple of changes to the Store interface:

• Added the init method for stores to set themselves up using options passed to the middleware.
• Rename the incr method to increment.
• Allow the increment, decrement, resetKey and resetAll methods to return a promise.
• Old stores will automatically be promisified and used.

As the external tests show, the store still works (without any changes) with v6 of express-rate-limit. However, implementing the newer Store interface allows for the following:

• The users do not need to pass expireTimeMs/windowMs twice (once to the middleware and once to the store) with the introduction of the init method.
• Since all functions can now return Promises, it removes the need for the numerous callbacks used in the code.

I would be happy to open/help out with a PR that implements the new Store interface.

Also, I see that #27 is still open and there is no @types/rate-limit-mongo package. Would you like this PR to also:

1. Rewrite the library in Typescript, OR
2. Add a type definition file (index.d.ts like what @gtmsingh has suggested in #27 (comment)) so that Typescript users don't encounter errors while using this library.

Please let me know your thoughts on the above.

Thanks,
Vedant

// cc @nfriedly

The Readme currently states:

"MongoStore class provides public methods (incr, decrement, resetKey) required by express-rate-limit."

It is not immediately clear that these methods can (or should) take an optional callback.

When using mongoose to provide a backing store collection, its deleteOne() method requires a callback for proper operation: https://mongoosejs.com/docs/queries.html#executing

This could easily lead to an improper (synchronous) implementation of resetKey(), causing a race condition between mongo and express.

Thank you for your great work.

I am having trouble making a test exit because the mongo connection is not closed. It would be nice if mongostore can provide a method for disconnecting. Alternative it could expose the mongoClient so user can disconnect with it.

In my app, this is the code I use to get the client's real ip address, it works even if behind heroku

    let ip = req.headers["x-forwarded-for"];
    if (ip){
      var list = ip.split(",");
      ip = list[list.length-1];
    } else {
      ip = req.connection.remoteAddress;
    }

But heroku has different IP than what the above code will give you.

Here is an example heroku IP, it's different everytime, so the blocking of IP isn't so effective in this case, as they all have different counter values for the different IPs, I think it would be better if the real client IP was used, this way there is one IP with one counter value, making the blocking of IP more effective.

Is there a configuration that allows to set the counter to 0 vs it deleting the record? It would be a great addition to have an additional field for total and when the resetKey is called it resets the counter to 0 and increments the total. This could allow for easily being able to see who has been hitting your server. Currently you could have someone hitting your sever but you not know it b/c the key is deleted after the expiration. I get why the in memory store or redis store wouldn't want this but for a persistent store like mongo it would be a big bonus imo.

Is there any effort to develop the @types/rate-limit-mongo module?

As soon as I add some connectionOptions to the mongo store, the application will throw an error:

[ERROR] RangeError: Maximum call stack size exceeded
    at Object.hasOwnProperty (<anonymous>)
    at C:\Users\mfr\Git\app\node_modules\rate-limit-mongo\node_modules\mongodb\lib\cmap\connection_pool.js:99:41
    at Array.reduce (<anonymous>)
    at resolveOptions (C:\Users\mfr\Git\app\node_modules\rate-limit-mongo\node_modules\mongodb\lib\cmap\connection_pool.js:98:53)
    at new ConnectionPool (C:\Users\mfr\Git\app\node_modules\rate-limit-mongo\node_modules\mongodb\lib\cmap\connection_pool.js:153:20)       
    at new Server (C:\Users\mfr\Git\app\node_modules\rate-limit-mongo\node_modules\mongodb\lib\core\sdam\server.js:126:19)
    at createAndConnectServer (C:\Users\mfr\Git\app\node_modules\rate-limit-mongo\node_modules\mongodb\lib\core\sdam\topology.js:866:18)     
    at updateServers (C:\Users\mfr\Git\app\node_modules\rate-limit-mongo\node_modules\mongodb\lib\core\sdam\topology.js:910:22)
    at NativeTopology.serverUpdateHandler (C:\Users\mfr\Git\app\node_modules\rate-limit-mongo\node_modules\mongodb\lib\core\sdam\topology.js:567:5)
    at Server.emit (node:events:513:28) uncaughtException

const path = require("path");
const os = require("os");
const rateLimit = require("express-rate-limit");
const MongoStore = require("rate-limit-mongo");

const config = require("../config.json");

const mongoUri = "mongodb+srv://" + config.mongodb.host + "/" + process.env.DB + "?authSource=%24external&retryWrites=true&w=majority";

module.exports = (count = 30, time = 60) => rateLimit({
    store: new MongoStore({
        uri: mongoUri,
        collectionName: "rateLimits",
        connectionOptions: {
            ssl: true,
            sslValidate: false,
            sslCert: path.join(os.homedir(), "X509-cert.pem"),
            sslKey: path.join(os.homedir(), "X509-cert.pem"),
            authMechanism: "MONGODB-X509"
        }
    }),
    windowMs: time * 1000,
    max: count
});

Hello,

Can I just clarify what these configuration parameters mean:

expireTimeMs - is this how often the background task is run that removes expired entries?

resetExpireDateOnChange - i don't understand the description:

indicates whether expireDate should be reseted when changed or not

I can see that expirationDate is a property in a MongoDB entry for a related hit from an IP address.

But I don't understand what "when changed" means?

When would an entry "be changed", other than when the counter property in the MongoDB entry is incremented after another hit from the same IP address?

And why would it be helpful for resetExpireDateOnChange to be reset every time an IP address makes another request? Wouldn't that just give them the ability to make unlimited hits if the expirationDate keeps being reset on each request?

Thank You.