rahulpsd18 / cognito-backup-restore Goto Github PK

I am doing some migrations. All was good until I hit a google_xxx account. I haven't been able to locate the error. Is it possible? Links to documentation would be great.

Note: I am raising it here because these value came out of a backup and then I tried to restore and can imagine other people having this problem. In practice what happens is that the non external accounts (ie cognito) restore but it is very hard to notice this.

My assumption is that either we need to strip values of external accounts on restore or setup the pool differently. The former seems more probable because I don't/can't change my pool without deleting it.

UPDATE: Looking at linking cognito user to federated accounts it would seem that it is not surprising we can't out of the box.

General Settings > Attributes

• has email as required.

Federation > Google > Attribute mapping

• email
• email_verified
• name
• given_name
• family_name
• sub (--> username)

Here's my script, sample json from a backup and the error message.

➜  cognito-migrate node ./node_modules/.bin/cbr restore \
>      --file to_add.json \
>     -r xxxx \
>     -p default \
>     --pool XXXXXXX
✔ Users imported successfully to XXXXXXX

[
  {
    "Username": "google_100051227669222736914",
    "Attributes": [
      {
        "Name": "sub",
        "Value": "93085109-16ac-4997-9680-0b61dbd4771d"
      },
      {
        "Name": "identities",
        "Value": "[{\"userId\":\"100051227669222736914\",\"providerName\":\"Google\",\"providerType\":\"Google\",\"issuer\":null,\"primary\":true,\"dateCreated\":1609832494628}]"
      },
      {
        "Name": "email_verified",
        "Value": "true"
      },
      {
        "Name": "name",
        "Value": "J T"
      },
      {
        "Name": "given_name",
        "Value": "J"
      },
      {
        "Name": "family_name",
        "Value": "T"
      },
      {
        "Name": "email",
        "Value": "[email protected]"
      }
    ],
    "UserCreateDate": "2021-01-05T07:41:34.635Z",
    "UserLastModifiedDate": "2021-01-05T07:41:34.635Z",
    "Enabled": true,
    "UserStatus": "EXTERNAL_PROVIDER"
  }
]

cognito-migrate node ./node_modules/.bin/cbr restore \
>      --file to_add.json \
>     -r xxxx \
>     -p default \
>     --pool xxxxxx
✔ Users imported successfully to xxxxxx

cognito-migrate/node_modules/aws-sdk/lib/protocol/json.js:52
  resp.error = util.error(new Error(), error);
                          ^

InvalidParameterException: Cannot modify the non-mutable attribute identities
    at Request.extractError (cognito-migrate/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.callListeners (cognito-migrate/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (cognito-migrate/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (cognito-migrate/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (cognito-migrate/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (cognito-migrate/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at cognito-migrate/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (cognito-migrate/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (cognito-migrate/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (cognito-migrate/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'InvalidParameterException',
  time: 2021-06-18T06:40:27.805Z,
  requestId: '90f09f62-7b7e-412f-a3c4-eb6296e2f727',
  statusCode: 400,
  retryable: false,
  retryDelay: 64.47731523692524

Hi guys!

I am unable to look out and maintain this project anymore (barely doing it anyway since 2020😞)
I am looking for contributors and maintainers to keep this project alive.
Please let me know if you want to be added as a collaborator here and on the npm package.

Thanks!

cc: @jstarmx @jfstephe @mholger

Hi there!

I installed [email protected] using yarn global add

success Installed "[email protected]" with binaries:
      - cbr
      - cognito-backup-restore
✨  Done in 14.36s.

However, when running the executable cbr, I was greeted with this error:

kanerogers@Kanes-MacBook-Pro ~/.c/y/g/n/c/n/.bin> cbr
env: node\r: No such file or directory

To fix this, I simply ran dos2unix on the executable:

kanerogers@Kanes-MacBook-Pro ~/.c/y/g/n/.bin> dos2unix ../cognito-backup-restore/build/cli/cli.js
dos2unix: converting file ../cognito-backup-restore/build/cli/cli.js to Unix format...

And now it runs fine!

kanerogers@Kanes-MacBook-Pro ~/.c/y/g/n/.bin> cbr
? Choose the mode (Use arrow keys)
❯ Backup
  Restore

While taking the backup from Aws cognito user pool, if the users are more than 10,000. it's giving rate limit exedeed error.
can you look into that and let use know when can you resolve that?
I Can also help if you need any help

I don't know about the others, but I would like to use this backup solution with a scheduled lambda function. It would be sweet to have the possibility to export to a S3 bucket. And then it would also make sense to allow restoring from an export in a S3 bucket.

Hi there,

It looks like npm audit is flagging inquirer-file-path as having a high-severity vulnerability in its own dependency (lodash).

Within my backup process, I'm calling the backupUsers process and expect that the file is complete when the promise returned from that function resolves, however the data I can immediately pull from disk is not complete (the stream has not finished writing all of the data to the file).

I see here (src/index.ts#L61) that there is no promise wrapping the callback pattern used for the stream, so there's no way to wrap this function call to ensure that the stream has finished writing all of the data to disk.

The following example:

const cbr = require('cognito-backup-restore');
const aws = require('aws-sdk');
const fs = require('fs');
const userPoolId = 'us-east-1_abc123';
const cognitoIsp = new aws.CognitoIdentityServiceProvider();
cbr.backupUsers(cognitoIsp, userPoolId, '/tmp').then(() => {
  JSON.parse(fs.readFileSync(`/tmp/${userPoolId}.json`).toString())
});

will yield the error:

(node:13604) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected end of JSON input

Because the final bracket (]) has not been written to disk

/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52
resp.error = util.error(new Error(), error);
^

InvalidParameterException: User pool does not have SMS configuration to send messages.
at Request.extractError (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'InvalidParameterException',
time: 2022-08-12T17:17:28.468Z,
requestId: 'b0d0cee0-e499-46a6-9afd-542f0f10bc04',
statusCode: 400,
retryable: false,
retryDelay: 40.703989640319094
}

Your AWS version uses caret "aws-sdk": "^2.259.1", and the latest version of the SDK no longer has the shared_ini.js file.

I suggest updating to the latest SDK or changing your package.json to use only that specific version of the SDK.

There is a bug in src/cli/cli.ts (line 1) that causes the following error on MacOS:
env: node\r: No such file or directory

I think the CRLF line ending is causing it, but perhaps the shebang line could be removed altogether as it is not widely used.

After creating the backup, how can we import the same sub ids when we restore to a new pool? It generates new sub ids when you restore.

There are 312 users in the exported .json file but only a handful imported successfully with this exception thrown in the console:

$ cbr restore -f 'users.json' -p <profile> --pool <pool> -r <region>
✔ Users imported successfully to <poll>

/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52
  resp.error = util.error(new Error(), error);
                          ^

LimitExceededException: Exceeded daily email limit for the operation or the account. If a higher limit is required, please configure your user pool to use your own Amazon SES configuration for sending email.
    at Request.extractError (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/usr/local/lib/node_modules/cognito-backup-restore/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'LimitExceededException',
  time: 2022-09-07T02:29:15.790Z,
  requestId: '220b9b98-bc64-437b-8ba4-10df913201ac',
  statusCode: 400,
  retryable: false,
  retryDelay: 64.88455069958468
}

Node.js v18.8.0

Looking to do a backup and restore of a user pool. Unfortunately we require the {"Name":"phone_number","Value":"+1234567890"} and {"Name":"phone_number_verified","Value":"true"} fields. While I was able to manually add them to the resulting json, it would be great to have them added automatically to the export.

Thanks!

What if I just want to restore a backup from the prod account to the dev pool and don't need to send everyone a notification with the new password? I just want to import the result of the backup command.
This should be obvious when working with this software that notifications will be sent and only need to be done when someone explicitly sets this to true, otherwise it is not clear and may confuse users when they actually have nothing to worry about.

Getting timeout while execute the command, please assist.

cbr backup --pool ***** -r us-east-1 --dir ./cognito_emg_backup

connect ETIMEDOUT *****:443

I'm just trying to do a backup and restore to a dummy database for testing purposes and I can't have it sending emails to live users (which has already happened).

Also since it's using Cognito's own email system (i don't have SES) it fails after 50. Even after I have disabled "automatically verify users" in the UserPool.

According to the docs, setting MessageAction to "SUPPRESS" will suppress the welcome message.

It seems you only accept a key/password IAM credential. Using a role is a best practice. In the aws config file this is specified as follows (example from aws cli documentation):
[profile marketingadmin]
role_arn = arn:aws:iam::123456789012:role/marketingadminrole
credential_source = Ec2InstanceMetadata

It looks like accounts added from external oath sources (Google, Facebook, etc) won't keep that status on restore. As a test, I backed some of those users up, restored them, and then tried to sign in with that same account. Instead of forcing a password reset for that user, a new user was signed-up and added to the pool.

const data = await restoreUsers(cognitoProvider, 'asasasasasasa', './backups/data1.json','demooooo!@!!12121212')
I am using above code snippet to restore users to pool.Backup is done successfully but not able to restore to specific pool id.
I am using same region to import and export the users. restoreUsers function returning 200 success code but not importing data to user pool.

I am not able to use the command line tool because my account assumes a role and uses MFA.

It is possible to allow MFA like so

const token = '123456'
const profile = 'profile'
  const credentials = new AWS.SharedIniFileCredentials({
    profile,
    tokenCodeFn: (mfaSerial, done) => done(null, token),
  })
  AWS.config.credentials = credentials

Hi,
I have this "issue".
The firewall in my company inspect the SSL traffic, so in AWS CLI and other tools I have configured to use the firewall certificate (that is a self signed).
When running the tool I receive the "error": "* self signed certificate in chain" and nothing happens....I can't do the backup.

How can I solve this ?
Thanks
Dario

Is there a way to specify what the contents of the email are? I was getting a generic message which can be confusing to the users as they won't know the password for what has been reset. I'd like to add more information.

When restoring a users pool got the following error:

UnhandledPromiseRejectionWarning: InvalidParameterException: No email provided but desired delivery medium was Email
...
(node:61020) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:61020) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Both pools configured exactly the same with terraform creation.

Actually it seems doing the job one by one user and issue the error right after restoring it. It would be nice to have at least a "continue on error" option if some users are buggy

Cognito stores groups and user membership information which is critical for associating users with Roles and therefore with shared permissions. Even if the roles fell outside of the scope of Cognito User Pools (being in Identity pools, not user pools), having the group information and membership would be an important part of the functionality of Cognito that should be retained through backup, restore, and migration.

when the region is set in the environment - via profile, configuration, or environment variables - it would be useful to default to those settings