Bug report

Required System information

• Node.js version: v16.17.0
• NPM version: v8.15.0
• Source Strapi version: v3.5.0
• Target Strapi version: v4.5.5
• Source Database: MySQL
• Target Database: MySQL
• Operating system: Windows 10 64-bit
• Which script are you running: v3-sql-v4-sql (using the MySQL .env files)

Describe the bug

This appears to be the same issue as this bug from last year: #10

This issue was never properly solved and was closed due to inactivity. As all of the code that I'm using has been provided by the Strapi team, I'm hoping that this issue will provide a reproducible example, so the team can provide an explanation for the issue, and potentially a fix. For any repos / npm packages I use, I aim to provide commit IDs and version numbers.

This also looks like it might be related to #19

After using the Strapi example repo to migrate the demo strapi application from V3 to V4 via codemods (and following the livestream: https://www.youtube.com/watch?v=NSvdQKVvV9k), I've cloned the migration-scripts repo and followed the data migration livestream (https://www.youtube.com/watch?v=kdxivJjjVhY).

After filling in the env files and specifying my targets, I run the migration script, but then get the following error:

npm run start

> [email protected] start
> node index.js

Migrating Core Store
Migrating 59/88 items from core_store to strapi_core_store_settings
core_store batch #1
core_store batch #2
Migrating Admin
Migrating 3 items from strapi_role to admin_roles
strapi_role batch #1
Migrating 1 items from strapi_administrator to admin_users
strapi_administrator batch #1
Migrating 2 items from strapi_users_roles to admin_users_roles_links
strapi_users_roles batch #1
Migrating 100 items from strapi_permission to admin_permissions
strapi_permission batch #1
node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: insert into `admin_permissions` (`action`, `conditions`, `created_at`, `fields`, `id`, `properties`, `subject`, `updated_at`) select 'plugin::content-manager.explorer.create' as `action`, '[]' as `conditions`, 1595433593180 as `created_at`, '["name","restaurants"]' as `fields`, 1 as `id`, NULL as `properties`, 'api::category.category' as `subject`, 1595433593185 as `updated_at` union all select 'plugin::content-manager.explorer.create' as `action`, '[]' as `conditions`, 1595433593192 as `created_at`, '["author","review"]' as `fields`, 2 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593197 as `updated_at` union all select 'plugin::content-manager.explorer.create' as `action`, '[]' as `conditions`, 1595433593217 as `created_at`, '["content","note","author","likes","restaurant"]' as `fields`, 4 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593222 as `updated_at` union all select 'plugin::content-manager.explorer.read' as `action`, '[]' as `conditions`, 1595433593230 as `created_at`, '["name","restaurants"]' as `fields`, 5 as `id`, NULL as `properties`, 'api::category.category' as `subject`, 1595433593235 as `updated_at` union all select 'plugin::content-manager.explorer.read' as `action`, '[]' as `conditions`, 1595433593243 as `created_at`, '["author","review"]' as `fields`, 6 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593247 as `updated_at` union all select 'plugin::content-manager.explorer.read' as `action`, '[]' as `conditions`, 1595433593268 as `created_at`, '["content","note","author","likes","restaurant"]' as `fields`, 8 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593274 as `updated_at` union all select 'plugin::content-manager.explorer.update' as `action`, '[]' as `conditions`, 1595433593283 as `created_at`, '["name","restaurants"]' as `fields`, 9 as `id`, NULL as `properties`, 'api::category.category' as `subject`, 1595433593288 as `updated_at` union all select 'plugin::content-manager.explorer.update' as `action`, '[]' as `conditions`, 1595433593301 as `created_at`, '["author","review"]' as `fields`, 10 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593307 as `updated_at` union all select 'plugin::content-manager.explorer.update' as `action`, '[]' as `conditions`, 1595433593335 as `created_at`, '["content","note","author","likes","restaurant"]' as `fields`, 12 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593342 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593352 as `created_at`, NULL as `fields`, 13 as `id`, NULL as `properties`, 'api::category.category' as `subject`, 1595433593359 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593371 as `created_at`, NULL as `fields`, 14 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593377 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593386 as `created_at`, NULL as `fields`, 15 as `id`, NULL as `properties`, 'api::restaurant.restaurant' as `subject`, 1595433593391 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593403 as `created_at`, NULL as `fields`, 16 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593408 as `updated_at` union all select 'plugin::upload.read' as `action`, '[]' as `conditions`, 1595433593417 as `created_at`, NULL as `fields`, 17 as `id`, NULL as `properties`, NULL as `subject`, 1595433593422 as `updated_at` union all select 'plugin::upload.assets.create' as `action`, '[]' as `conditions`, 1595433593431 as `created_at`, NULL as `fields`, 18 as `id`, NULL as `properties`, NULL as `subject`, 1595433593435 as `updated_at` union all select 'plugin::upload.assets.update' as `action`, '[]' as `conditions`, 1595433593444 as `created_at`, NULL as `fields`, 19 as `id`, NULL as `properties`, NULL as `subject`, 1595433593448 as `updated_at` union all select 'plugin::upload.assets.download' as `action`, '[]' as `conditions`, 1595433593456 as `created_at`, NULL as `fields`, 20 as `id`, NULL as `properties`, NULL as `subject`, 1595433593460 as `updated_at` union all select 'plugin::upload.assets.copy-link' as `action`, '[]' as `conditions`, 1595433593469 as `created_at`, NULL as `fields`, 21 as `id`, NULL as `properties`, NULL as `subject`, 1595433593474 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '["admin::is-creator"]' as `conditions`, 1595433593646 as `created_at`, NULL as `fields`, 36 as `id`, NULL as `properties`, 'api::restaurant.restaurant' as `subject`, 1595433593651 as `updated_at` union all select 'plugin::upload.read' as `action`, '["admin::is-creator"]' as `conditions`, 1595433593671 as `created_at`, NULL as `fields`, 38 as `id`, NULL as `properties`, NULL as `subject`, 1595433593674 as `updated_at` union all select 'plugin::upload.assets.create' as `action`, '[]' as `conditions`, 1595433593681 as `created_at`, NULL as `fields`, 39 as `id`, NULL as `properties`, NULL as `subject`, 1595433593685 as `updated_at` union all select 'plugin::upload.assets.update' as `action`, '["admin::is-creator"]' as `conditions`, 1595433593691 as `created_at`, NULL as `fields`, 40 as `id`, NULL as `properties`, NULL as `subject`, 1595433593696 as `updated_at` union all select 'plugin::upload.assets.download' as `action`, '[]' as `conditions`, 1595433593703 as `created_at`, NULL as `fields`, 41 as `id`, NULL as `properties`, NULL as `subject`, 1595433593707 as `updated_at` union all select 'plugin::upload.assets.copy-link' as `action`, '[]' as `conditions`, 1595433593714 as `created_at`, NULL as `fields`, 42 as `id`, NULL as `properties`, NULL as `subject`, 1595433593718 as `updated_at` union all select 'plugin::content-manager.explorer.create' as `action`, '[]' as `conditions`, 1595433593750 as `created_at`, '["author","review"]' as `fields`, 44 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593754 as `updated_at` union all select 'plugin::content-manager.explorer.create' as `action`, '[]' as `conditions`, 1595433593774 as `created_at`, '["content","note","author","likes","restaurant"]' as `fields`, 46 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593778 as `updated_at` union all select 'plugin::content-manager.explorer.create' as `action`, '[]' as `conditions`, 1595433593786 as `created_at`, '["username","email","provider","password","resetPasswordToken","confirmed","blocked","role","reviews","likes","picture"]' as `fields`, 47 as `id`, NULL as `properties`, 'plugin::users-permissions.user' as `subject`, 1595433593790 as `updated_at` union all select 'plugin::content-manager.explorer.read' as `action`, '[]' as `conditions`, 1595433593812 as `created_at`, '["author","review"]' as `fields`, 49 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593817 as `updated_at` union all select 'plugin::content-manager.explorer.read' as `action`, '[]' as `conditions`, 1595433593836 as `created_at`, '["content","note","author","likes","restaurant"]' as `fields`, 51 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593839 as `updated_at` union all select 'plugin::content-manager.explorer.read' as `action`, '[]' as `conditions`, 1595433593847 as `created_at`, '["username","email","provider","password","resetPasswordToken","confirmed","blocked","role","reviews","likes","picture"]' as `fields`, 52 as `id`, NULL as `properties`, 'plugin::users-permissions.user' as `subject`, 1595433593852 as `updated_at` union all select 'plugin::content-manager.explorer.update' as `action`, '[]' as `conditions`, 1595433593871 as `created_at`, '["author","review"]' as `fields`, 54 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593876 as `updated_at` union all select 'plugin::content-manager.explorer.update' as `action`, '[]' as `conditions`, 1595433593896 as `created_at`, '["content","note","author","likes","restaurant"]' as `fields`, 56 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593900 as `updated_at` union all select 'plugin::content-manager.explorer.update' as `action`, '[]' as `conditions`, 1595433593908 as `created_at`, '["username","email","provider","password","resetPasswordToken","confirmed","blocked","role","reviews","likes","picture"]' as `fields`, 57 as `id`, NULL as `properties`, 'plugin::users-permissions.user' as `subject`, 1595433593913 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593920 as `created_at`, NULL as `fields`, 58 as `id`, NULL as `properties`, 'api::category.category' as `subject`, 1595433593925 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593933 as `created_at`, NULL as `fields`, 59 as `id`, NULL as `properties`, 'api::like.like' as `subject`, 1595433593937 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593946 as `created_at`, NULL as `fields`, 60 as `id`, NULL as `properties`, 'api::restaurant.restaurant' as `subject`, 1595433593950 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593958 as `created_at`, NULL as `fields`, 61 as `id`, NULL as `properties`, 'api::review.review' as `subject`, 1595433593963 as `updated_at` union all select 'plugin::content-manager.explorer.delete' as `action`, '[]' as `conditions`, 1595433593973 as `created_at`, NULL as `fields`, 62 as `id`, NULL as `properties`, 'plugin::users-permissions.user' as `subject`, 1595433593977 as `updated_at` union all select 'plugin::content-type-builder.read' as `action`, '[]' as `conditions`, 1595433593985 as `created_at`, NULL as `fields`, 63 as `id`, NULL as `properties`, NULL as `subject`, 1595433593990 as `updated_at` union all select 'plugin::upload.read' as `action`, '[]' as `conditions`, 1595433594038 as `created_at`, NULL as `fields`, 67 as `id`, NULL as `properties`, NULL as `subject`, 1595433594043 as `updated_at` union all select 'plugin::upload.assets.create' as `action`, '[]' as `conditions`, 1595433594051 as `created_at`, NULL as `fields`, 68 as `id`, NULL as `properties`, NULL as `subject`, 1595433594056 as `updated_at` union all select 'plugin::upload.assets.update' as `action`, '[]' as `conditions`, 1595433594063 as `created_at`, NULL as `fields`, 69 as `id`, NULL as `properties`, NULL as `subject`, 1595433594067 as `updated_at` union all select 'plugin::upload.assets.download' as `action`, '[]' as `conditions`, 1595433594076 as `created_at`, NULL as `fields`, 70 as `id`, NULL as `properties`, NULL as `subject`, 1595433594080 as `updated_at` union all select 'plugin::upload.assets.copy-link' as `action`, '[]' as `conditions`, 1595433594088 as `created_at`, NULL as `fields`, 71 as `id`, NULL as `properties`, NULL as `subject`, 1595433594093 as `updated_at` union all select 'plugin::upload.settings.read' as `action`, '[]' as `conditions`, 1595433594100 as `created_at`, NULL as `fields`, 72 as `id`, NULL as `properties`, NULL as `subject`, 1595433594105 as `updated_at` union all select 'plugin::content-manager.single-types.configure-view' as `action`, '[]' as `conditions`, 1595433594112 as `created_at`, NULL as `fields`, 73 as `id`, NULL as `properties`, NULL as `subject`, 1595433594116 as `updated_at` union all select 'plugin::content-manager.collection-types.configure-view' as `action`, '[]' as `conditions`, 1595433594123 as `created_at`, NULL as `fields`, 74 as `id`, NULL as `properties`, NULL as `subject`, 1595433594127 as `updated_at` union all select 'plugin::content-manager.components.configure-layout' as `action`, '[]' as `conditions`, 1595433594134 as `created_at`, NULL as `fields`, 75 as `id`, NULL as `properties`, NULL as `subject`, 1595433594138 as `updated_at` union all select 'plugin::users-permissionss.roles.create' as `action`, '[]' as `conditions`, 1595433594145 as `created_at`, NULL as `fields`, 76 as `id`, NULL as `properties`, NULL as `subject`, 1595433594149 as `updated_at` union all select 'plugin::users-permissionss.roles.read' as `action`, '[]' as `conditions`, 1595433594156 as `created_at`, NULL as `fields`, 77 as `id`, NULL as `properties`, NULL as `subject`, 1595433594160 as `updated_at` - SQLITE_ERROR: table admin_permissions has no column named fields] {
  errno: 1,
  code: 'SQLITE_ERROR'
}

Looking at previous issues which provide a similar error to this, people have recommended to open the target old_data.db file and drop the "fields" column in the V3 "strapi_permission" table. Doing so will allow the migration to run its course and "successfully" complete. However, if I then try to run "npm run develop", I get this:

[2023-01-05 16:45:11.427] debug: ⛔️ Server wasn't able to start properly.
[2023-01-05 16:45:11.428] error: Cannot read properties of null (reading 'fields')
TypeError: Cannot read properties of null (reading 'fields')
    at Array.<anonymous> (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\admin\server\services\content-type.js:167:21)    
    at Array.map (<anonymous>)
    at Object.cleanPermissionFields (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\admin\server\services\content-type.js:163:22)
    at Object.cleanPermissionsInDatabase (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\admin\server\services\permission\queries.js:163:26)
    at async module.exports (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\admin\server\bootstrap.js:88:3)
    at async Strapi.runLifecyclesFunctions (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\strapi\lib\Strapi.js:542:7)    
    at async Strapi.bootstrap (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\strapi\lib\Strapi.js:469:5)
    at async Strapi.load (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\strapi\lib\Strapi.js:478:5)
    at async Strapi.start (C:\MYOXYGEN-REPOS\PCC Backend V4\demomigration\node_modules\@strapi\strapi\lib\Strapi.js:212:9)

Steps to reproduce the behavior

1. Clone Strapi's v3 migration example: https://github.com/PaulBratslavsky/demoMigration (I cloned the following commit: a0a859fed0b628305f091f39c51beb5fda34b82a)
2. Seed the data using the script within the repo. For the seeder to run properly, you may need to apply my fix that I've specified in an issue within the demo repo: PaulBratslavsky/demoMigration#1 (comment)
3. Use codemods (I used npx when the npm package was at v1.3.0: https://www.npmjs.com/package/@strapi/codemods/v/1.3.0) and follow Strapi's migration walkthrough on YouTube (https://www.youtube.com/watch?v=NSvdQKVvV9k)
4. Clone the migration-scripts repo (I cloned at the following commit: d660a7e)
5. In the package.json of the v3-sql-v4-sql folder, you should specify "@vscode/sqlite3": "^5.1.2-vscode". I had to upgrade this package due to the following issue with this package: microsoft/vscode#152839.
6. npm install the rest of the packages in v3-sql-v4-sql, as normal.
7. Ensure you have a copy of the seeded v3 database; I've called this old_data.db.
8. Ensure you also have a fresh v4 database (I've left this as the defaultly-named data.db); I didn't create any admin users prior to running the script.
9. In the v3-sql-v4-sql, copy the .env.sqlite.example file and fill out the required info, including where your v3 and v4 database file is located. Save this copied file as .env.
10. In the v3-sql-v4-sql folder, run npm run start.
11. You should now see the first error I previously pasted.
12. Open the v3 database in a DB editor of your choice. Drop the fields column within the strapi_permission column. Re-run the migration script with npm run start. It should now claim it's successfully completed.
13. Try to run the V4 Strapi application with the migrated V4 database via npm run develop. You should now see the second error I've pasted.

Expected behavior

The database successfully migrates, all permissions are preserved, and I can use Strapi V4 with my migrated data.

Screenshots

Above is a screenshot of node_modules@strapi\admin\server\services\content-type.js:167:21, where the error after "successfully migrating" occurs. I'd like to draw attention to properties: { fields }. I'm unsure how deleting the fields column works for other users which face a similar issue to this, as Strapi then looks for the missing column

More screenshots can be provided on request.

Code snippets

Not applicable, but can provide on-request.

Additional context

Not applicable, but can provide on-request.

Bug report

This issue is only impactful because:
The properties field in 'strapi_permission' is not-required, and has a default value of {}, so realistically everyone should have this in their MongoDB. However, some may have migrated from older versions, and only have the 'fields' property. This could also silently cause other issues with long-time mongo databases being migrated

Required System information

• Node.js version: v14.21.1
• NPM version: v9.1.2
• Source Strapi version: 3.6.10
• Target Strapi version: 3.6.10
• Source Database: MongoDB
• Target Database: PostgreSQL
• Operating system: Mac
• Which script are you running: Mongov3->SQLv3

Describe the bug

When an attribute that has a default value is undefined, and the migration assistant copies it to the SQL, it will be copied as NULL.
Inside the SQL, there is no 'DEFAULT' specified for columns that have a 'default' value in the core_store entry

Steps to reproduce the behaviour

1. Go to your MongoDB (Strapi v3) database editor (MongoDB Compass)
2. Go to 'strapi_permission' collection
3. Choose an entry with a properties field, and delete that attribute.
4. Save/Replace/Update your changes on that entry
5. Run the migration assistant
6. Query SELECT * FROM strapi_permission WHERE properties IS NULL;
7. You will have at least 1 entry

Additional steps for why this impacts Strapi:

1. Run yarn install && yarn develop in your app directory
2. The run will fail & exit with code 1 - this is directly related to the 'properties' field being 'null' instead of '{}' (default value)
   Note: While this is unlikely, this could cause more hidden issues for other columns elsewhere that expect a default value.

Expected behaviour

Entries which have default values in core_store, will have their default values inserted into the SQL if it is otherwise left NULL during insertion.

Screenshots

Code snippets

Strapi_permission core_store structure - Note: relevant fields are 'attributes>properties|conditions', which has 'default: {}'

   "uid":"strapi::permission",
   "collectionName":"strapi_permission",
   "kind":"collectionType",
   "info":{
      "name":"Permission",
      "description":""
   },
   "options":{
      "timestamps":[
         "createdAt",
         "updatedAt"
      ]
   },
   "pluginOptions":{
      "content-manager":{
         "visible":false
      },
      "content-type-builder":{
         "visible":false
      }
   },
   "attributes":{
      "action":{
         "type":"string",
         "minLength":1,
         "configurable":false,
         "required":true
      },
      "subject":{
         "type":"string",
         "minLength":1,
         "configurable":false,
         "required":false
      },
      "properties":{
         "type":"json",
         "configurable":false,
         "required":false,
         "default":{
            
         }
      },
      "conditions":{
         "type":"json",
         "configurable":false,
         "required":false,
         "default":[
            
         ]
      },
      "role":{
         "configurable":false,
         "model":"role",
         "plugin":"admin"
      }
   }
}

Additional context

Note: This error only applies to long-term mongoDB users who's data has persisted long enough to not have a 'properties' field in strapi_permissions/have the old 'fields' column, and are attempting to migrate. This issue is for when an attribute does not have a 'non-required' attribute specified (that has a default value)

For long-time mongoDB users, they might have production databases without properties, or have the old 'fields' attribute.
When the properties field in mongoDB is undefined, it is migrated as null to SQL. Null values properties cause 'Type Error: Cannot convert undefined or null to object' error in /Path/To/Strapi/app/node_modules/strapi-admin/services/role.js:51:17

Bug report

Required System information

• Node.js version: v16.14.0
• NPM version: yarn 1.22.17
• Source Strapi version: 3.6.10 — Community Edition
• Target Strapi version: 4.3.9
• Source Database: Postgres 13
• Target Database: Postgres 13
• Operating system: macOS Monterey 12.5.1
• Which script are you running: yarn start

Describe the bug

I got this error when I run the command yarn start

/Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/migrate/migrateFiles.js:36
      if (item.uid.startsWith('application::')) {
                   ^

TypeError: Cannot read properties of undefined (reading 'startsWith')
    at /Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/migrate/migrateFiles.js:36:20
    at Array.reduce (<anonymous>)
    at /Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/migrate/migrateFiles.js:34:47
    at Array.map (<anonymous>)
    at migrateItems (/Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/migrate/helpers/migrateFields.js:24:16)
    at migrate (/Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/migrate/helpers/migrate.js:129:27)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.migrateTables (/Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/migrate/migrateFiles.js:33:3)
    at async migrate (/Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/migrate/index.js:63:5)
    at async f (/Users/mykolabreslavskyi/www/perfsol/migration-scripts/v3-sql-v4-sql/index.js:6:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Additional context

I added console.log here and additional check to if statment

Item is not an object, it is a string

Bug report

Taken 8hrs to get this far :-/

Required System information

- Node.js version: v16.15.0
- Source Strapi version: 3
- Target Strapi version: 4
- Source Database: postgres
- Target Database: postgres
- Operating system: nixos
- Which script are you running: v3-sql-v4-sql

Describe the bug

Migration failure

error: insert into "admin_permissions" ("action", "conditions", "created_at", "fields", "id", "properties", "subject", "updated_at") values ($1, $2, $3, $4, $5, $6, $7, $8), ($9, $10, $11, $12, $13, $14, $15, $16), ($17, $18, $19, $20, $21, $22, $23, $24), ($25, $26, $27, $28, $29, $30, $31, $32), ($33, $34, $35, $36, $37, $38, $39, $40), ($41, $42, $43, $44, $45, $46, $47, $48), ($49, $50, $51, $52, $53, $54, $55, $56), ($57, $58, $59, $60, $61, $62, $63, $64), ($65, $66, $67, $68, $69, $70, $71, $72), ($73, $74, $75, $76, $77, $78, $79, $80), ($81, $82, $83, $84, $85, $86, $87, $88), ($89, $90, $91, $92, $93, $94, $95, $96), ($97, $98, $99, $100, $101, $102, $103, $104), ($105, $106, $107, $108, $109, $110, $111, $112), ($113, $114, $115, $116, $117, $118, $119, $120), ($121, $122, $123, $124, $125, $126, $127, $128), ($129, $130, $131, $132, $133, $134, $135, $136), ......... - column "fields" of relation "admin_permissions" does not exist

Steps to reproduce the behaviour

Follow v3-sql-v4-sql guide and migrate.

Expected behaviour

A smooth transition from Strapi V3 to V4

Bug report

Required System information

• Node.js version: 16.19.1
• NPM version: 8.19.3
• Source Strapi version: 3.6.9
• Target Strapi version: 4.8.2
• Source Database: PostgreSQL
• Target Database: PostgreSQL
• Operating system: Ubuntu
• Which script are you running: v3-sql-v4-sql

Describe the bug

In the strapi v3 i have tables :
components_story_group_stories__stories with columns id | components_story_group_story_id | story_id
components_puzzle_group_puzzles__puzzles with columns id | components_puzzle_group_puzzle_id | puzzle_id

I migrated my strapi from v3 to v4 using codemods. And pointed strapi to new empty db and generated schema. After migrating the strapi following changes happen
components_story_group_stories_stories_links with the columns id | group_stories_id | story_id | story_order
components_puzzle_group_puzzles_puzzles_links with columns id | group_puzzles_id | puzzle_id | puzzle_order
**there are more like these tables

When i tried to migrate the data from v3 to v4 the column group_stories_id and group_puzzles_id are skipped
Showing this in the console -
WARNING - items of components_story_group_stories_stories_links was filtered ["group_story_id"]
WARNING - items of components_puzzle_group_puzzles_puzzles_links was filtered ["group_puzzle_id"]
In columns group_story_id and group_puzzle_id NULL value is showing

The other issue with the same process is no data is migarted for some tables:
in v3 i have a table components_game_group_games__game_sets with columns id | components_game_group_game_id| game-set_id
after migrating it to the v4 the table is renamed as components_game_group_games_game_sets_links with columns name as id | group_games_id | game_set_id | game_set_order
After running the migration script the table is still empty in v4 db and data for this table is not migrated from v3 to v4.

Expected behavior

All the data should have been migrated from v3 to v4 PostgreSQL.

Screenshots

Tables in v3

Tables in v4

Bug report

Describe the bug

When running the scripts I get DESTINATION TABLE does not exist for all my content types; I cannot find any code that would actually create the DESTINATION tables when trying to troubleshoot. Is the concept of the migration scripts that the content types first get created in a manual manner in Strapi v4 and the migration does not migrate schemas but only data?

Expected behavior

Destination tables being created for all content types based on source.

Bug report

Required System information

• Node.js version: v14
• NPM version: v6
• Source Strapi version: 3.6.11
• Target Strapi version: 4.10.5
• Source Database: Postgres
• Target Database: Postgres
• Which script are you running: migrate

Describe the bug

In some cases during migration, users core_store gets corrupted causing an inability to view collection types in the content manager. This currently only occurred for a collection type and not a single type.

Steps to reproduce the behavior

This occurred during a user's routine migration process. To our knowledge, there aren't any steps outside of what we prescribe in our guides that the users followed.

Screenshots

• Pre-migration
  

• Pos-migration
  

Additional context

To get past this, the user had to delete their core_store content manager config

Bug report

Required System information

N/A, this is a current typo

Describe the bug

Typo in code. Referencing 'attribute.moel' instead of 'attribute.model'

Steps to reproduce the behavior

1. Visit mongov3->sqlv3 tool, index.js, line 229

Expected behavior

Reference attribute.model

Screenshots

If applicable, add screenshots to help explain your problem.

Code snippets

const isOneWay = attribute.model && !attribute.via && attribute.moel !== '*';

Bug report

Required System information

• Node.js version: 14.20.0
• NPM version: 6.14.17
• Source Strapi version: 3.6.8
• Target Strapi version: 4.3.6
• Source Database: sqlite or mysql (verified on both)
• Target Database: sqlite or mysql (verified on both)
• Operating system: Ubuntu 22.04 LTS
• Which script are you running: v3-sql-v4-sql

Describe the bug

The conditions values in the admin_permissions tables are improperly formatted after running the migration script, this causes application crashes for admin users assigned to any role created pre-migration besides the super admin.

Steps to reproduce the behavior

• create a quickstart v3 project
• add an admin user to the project, assign them the editor or author role
• create a quickstart v4 project
• migrate the database from v3 to v4 using the migration script configured for the sqlite databases
• start the v4 application
• log in to the dashboard as the created user
• dashboard will crash and 500 errors observed on the server side

Expected behavior

The dashboard does not crash and the server does not error.

v3-sql-v4-sql (pg 10 to pg13)
3.6.8 -> 4.1.9

Describe the bug

All table names for content types in v3 use a plural name, but in v4 some of them use a singular name instead.

Error:

DESTINATION TABLE countries DOES NOT EXISTS
DESTINATION TABLE timezones DOES NOT EXISTS

In v4 the tables for these are country and timezone respectively.

Steps to reproduce the behavior

Migrate a content type with a plural name such as "countries" from v3 to a singular name in v4 "country".

Expected behavior / solution

Ability to map previous table names to new ones in config?

I'm not sure what the ideal solution is right now, I'm going to try and fix it in my script and I'll update this issue if I've made any progress.

Bug report

Required System information

• Node.js version: 16.15.0
• NPM version: 8.7.0
• Source Strapi version: 3.6.8
• Target Strapi version: 4.3.8
• Source Database: mysql
• Target Database: mysql
• Operating system: OSX
• Which script are you running: v3-sql-v4-sql

Describe the bug

When migrating relations, when one side of the relationship is a component, the id column of the component field in the link table is incorrectly generated in some cases. This is causing the migration of some component relations to be skipped.

For my specific case, my component name is teacher_notes and the relation column name in the link table is teacher_notes_id. However, the migration scripts is trying to migrate to a column in the link table called teacher_note_id (which doesn't exist) so is skipped.

Steps to reproduce the behavior

Hard to reproduce as it is project-specific.

Expected behavior

The relation column name in the link table for a component should be in the format {component_name}_id.

Screenshots

If applicable, add screenshots to help explain your problem.

Code snippets

^ This is where the column name for a relation is being generated.

Additional context

Within Strapi itself, the name of the relation column in a link table for a component is generated in the format: {relation_model_name}_id. However, in the migration scripts, for all cases, the column name for a relation in the link table is being formatted using the pluralize library, leading to column names for some component relations being incorrect.

Running Mongo -> Sql

[Error: insert into `haunts` (`Name`, `Slug`, `created_at`, `id`, `updated_at`) values ('SeaWorld San Diego’s Howl-O-Scream', 'sea-world-san-diego-s-howl-o-scream', '2021-03-14 22:04:41.060', 1, '2021-03-14 22:06:44.220') - SQLITE_CONSTRAINT: NOT NULL constraint failed: haunts.CurrentlyOperating] {
  errno: 19,
  code: 'SQLITE_CONSTRAINT'
}

Schema for that:

{
  "kind": "collectionType",
  "collectionName": "haunts",
  "info": {
    "name": "Haunts",
    "description": ""
  },
  "options": {
    "increments": true,
    "timestamps": true,
    "draftAndPublish": false
  },
  "attributes": {
    "Name": {
      "type": "string",
      "required": true
    },
    "Slug": {
      "type": "uid",
      "targetField": "Name",
      "required": true
    },
    "AlternativeNames": {
      "type": "component",
      "repeatable": true,
      "component": "common.alternative-names"
    },
    "CurrentlyOperating": {
      "type": "boolean",
      "default": true,
      "required": true
    },
    "Geo": {
      "type": "json"
    },
    "IsTempClosed": {
      "type": "boolean",
      "default": false,
      "required": true
    },
    "Park": {
      "model": "parks"
    },
    "NiceName": {
      "type": "string"
    },
    "MainPhoto": {
      "model": "file",
      "via": "related",
      "allowedTypes": [
        "images"
      ],
      "plugin": "upload",
      "required": false
    },
    "CoverPhoto": {
      "model": "file",
      "via": "related",
      "allowedTypes": [
        "images"
      ],
      "plugin": "upload",
      "required": false
    },
    "Gallery": {
      "collection": "file",
      "via": "related",
      "allowedTypes": [
        "images"
      ],
      "plugin": "upload",
      "required": false
    },
    "Sections": {
      "type": "dynamiczone",
      "components": [
        "text.section-header",
        "text.text"
      ]
    },
    "Region": {
      "model": "regions",
      "via": "Haunts"
    },
    "OpeningDate": {
      "type": "string"
    },
    "ClosedDate": {
      "type": "string"
    },
    "HauntYears": {
      "via": "Haunt",
      "collection": "haunt-years"
    },
    "Attribute": {
      "type": "component",
      "repeatable": true,
      "component": "attraction.attribute"
    }
  }
}

Bug report

Required System information

• Node.js version: 14.2
• Source Strapi version: 3.6.10
• Target Strapi version: 4.5.1
• Source Database: mysql
• Target Database: mysql
• Operating system: Linux
• Which script are you running: v3-sql-v4-sql

Describe the bug

I run data migration from 3.6.10 version to latest 4.5.1. The migration finished, but for localized entries no links were correctly created. The _localization_links tables are not empty, but the _inv field is always null. Like:

Bug report

Required System information

• Node.js version: v14.19.3
• NPM version: v6.14.17
• Source Strapi version: v3.6.6
• Target Strapi version: v4.5.6
• Source Database: Postgres
• Target Database: Postgres
• Operating system: macOS
• Which script are you running: v3-sql-v4-sql

Describe the bug

After running migration-scripts, when consulting the details of one entry, some fields appears twice.
Below for example for the Content-type : News-type, the field : 'type' appears twice

When the schema.json of News-type is :

  "kind": "collectionType",
  "collectionName": "news_types",
  "info": {
    "singularName": "news-type",
    "pluralName": "news-types",
    "displayName": "News-type",
    "name": "news-type"
  },
  "options": {
    "increments": true,
    "timestamps": true,
    "draftAndPublish": true
  },
  "pluginOptions": {
    "i18n": {
      "localized": true
    }
  },
  "attributes": {
    "type": {
      "type": "string",
      "required": true,
      "pluginOptions": {
        "i18n": {
          "localized": true
        }
      }
    }
  }
}

I also checked the database and there is no double column 'type'. So it is just a problem in the view. See below, we have the Config view edit and the field 'type' appears twice for some reason.

Interestingly the 2 other Content-types which are linked to News-type have some problem in their view as well :
Latest-news-and-resource

{
  "kind": "collectionType",
  "collectionName": "news_and_resources",
  "info": {
    "singularName": "latest-news-and-resource",
    "pluralName": "latest-news-and-resources",
    "displayName": "Latest-news-and-resource",
    "name": "latest-news-and-resource",
    "description": ""
  },
  ...
  "attributes": {
    "title": {
      "type": "string",
      "required": true,
      "pluginOptions": {
        "i18n": {
          "localized": true
        }
      }
    },
    ...
    "news_type": {
      "type": "relation",
      "relation": "oneToOne",
      "target": "api::news-type.news-type"
    },
    "primary_category": {
      "type": "relation",
      "relation": "manyToOne",
      "target": "api::category.category",
      "inversedBy": "primary_category_news_and_resources"
    },
    "other_categories": {
      "type": "relation",
      "relation": "manyToMany",
      "target": "api::category.category",
      "inversedBy": "other_categories_news_and_resources"
    },
    ...
    }
  }
}

For Latest-news-and-resources, the fields : news-type, primary_category, other_categories . All appear twice as in the image below :

And it is the same for Category :

{
  "kind": "collectionType",
  "collectionName": "categories",
  "info": {
    "singularName": "category",
    "pluralName": "categories",
    "displayName": "Category",
    "name": "category"
  },
  "options": {
    "increments": true,
    "timestamps": true,
    "draftAndPublish": true
  },
  "pluginOptions": {
    "i18n": {
      "localized": true
    }
  },
  "attributes": {
    "name": {
      "type": "string",
      "required": true,
      "pluginOptions": {
        "i18n": {
          "localized": true
        }
      }
    },
    "primary_category_news_and_resources": {
      "type": "relation",
      "relation": "oneToMany",
      "target": "api::latest-news-and-resource.latest-news-and-resource",
      "mappedBy": "primary_category"
    },
    "other_categories_news_and_resources": {
      "type": "relation",
      "relation": "manyToMany",
      "target": "api::latest-news-and-resource.latest-news-and-resource",
      "mappedBy": "other_categories"
    }
  }
}

It looks like this problem only appear for Content-types which have '-' hyphen in the name and is linked to other Content-types.
Once again the database and the model files are all fine, it is just in the view.

Steps to reproduce the behavior

1. Run migration-scripts : v3-sql-v4-sql with some Collection-types with '-' in the name and link them to Content-type with a relation
2. All data are fully migrated
3. Go to 'Content-Manager' of view the detail of one entry
4. See that some fields are displayed twice

Expected behavior

No fields should be displayed twice.

Additional context

The duplicated field through the Config view edit of the Content Manager. But we would like to avoid any manual action because we aren't necessarily aware where this may appear and we aren't sure if it will appear again.

v3 user_permissions user have a column "display_name".
This does not exist on a fresh v4

 - column "display_name" of relation "up_users" does not exist
    at Parser.parseErrorMessage 

which in my case was the development database, and not production I was trying to connect to. Took my a while before i realized what was happing 🥲

Bug report

Required System information

• Node.js version: 18
• NPM version: 8.6.0
• Source Strapi version: 3.6.8
• Target Strapi version: 4.1.5
• Source Database: strapi-production
• Target Database: strapi-production-v4
• Operating system: macos
• Which script are you running: v3-sql-v4-sql

Describe the bug

This output I got when trying to migrate

Migrating Core Store TBA
Migrating 59/79 items from core_store to strapi_core_store_settings
core_store batch #1
core_store batch #2
Migrating Admin
Migrating 3 items from strapi_role to admin_roles
strapi_role batch #1
Migrating 1 items from strapi_administrator to admin_users
strapi_administrator batch #1
Migrating 1 items from strapi_users_roles to admin_users_roles_links
strapi_users_roles batch #1
Migrating 135 items from strapi_permission to admin_permissions
strapi_permission batch #1
strapi_permission batch #2
strapi_permission batch #3
Migrating Users
Migrating 3 items from users-permissions_role to up_roles
users-permissions_role batch #1
Migrating 198/486 items from users-permissions_permission to up_permissions
users-permissions_permission batch #1
users-permissions_permission batch #2
users-permissions_permission batch #3
users-permissions_permission batch #4
Migrating 416 items from users-permissions_user to up_users
users-permissions_user batch #1
users-permissions_user batch #2
users-permissions_user batch #3
users-permissions_user batch #4
users-permissions_user batch #5
users-permissions_user batch #6
users-permissions_user batch #7
users-permissions_user batch #8
users-permissions_user batch #9
Migration custom  01_my-custom-migration.js
Migrate webhooks
Migrating 0 items from strapi_webhooks to strapi_webhooks
Migrate locales (i18n)
SOURCE TABLE i18n_locales DOES NOT EXISTS
Migrating components
Migrating files
Migrating 1881 items from upload_file to files
upload_file batch #1
upload_file batch #2
upload_file batch #3
upload_file batch #4
upload_file batch #5
upload_file batch #6
upload_file batch #7
upload_file batch #8
upload_file batch #9
upload_file batch #10
upload_file batch #11
upload_file batch #12
upload_file batch #13
upload_file batch #14
upload_file batch #15
upload_file batch #16
upload_file batch #17
upload_file batch #18
upload_file batch #19
upload_file batch #20
upload_file batch #21
upload_file batch #22
upload_file batch #23
upload_file batch #24
upload_file batch #25
upload_file batch #26
upload_file batch #27
upload_file batch #28
upload_file batch #29
upload_file batch #30
upload_file batch #31
upload_file batch #32
upload_file batch #33
upload_file batch #34
upload_file batch #35
upload_file batch #36
upload_file batch #37
upload_file batch #38
Migrating 1751 items from upload_file_morph to files_related_morphs
upload_file_morph batch #1
upload_file_morph batch #2
upload_file_morph batch #3
upload_file_morph batch #4
upload_file_morph batch #5
upload_file_morph batch #6
upload_file_morph batch #7
upload_file_morph batch #8
upload_file_morph batch #9
upload_file_morph batch #10
upload_file_morph batch #11
upload_file_morph batch #12
upload_file_morph batch #13
upload_file_morph batch #14
upload_file_morph batch #15
upload_file_morph batch #16
upload_file_morph batch #17
upload_file_morph batch #18
upload_file_morph batch #19
upload_file_morph batch #20
upload_file_morph batch #21
upload_file_morph batch #22
upload_file_morph batch #23
upload_file_morph batch #24
upload_file_morph batch #25
upload_file_morph batch #26
upload_file_morph batch #27
upload_file_morph batch #28
upload_file_morph batch #29
upload_file_morph batch #30
upload_file_morph batch #31
upload_file_morph batch #32
upload_file_morph batch #33
upload_file_morph batch #34
upload_file_morph batch #35
upload_file_morph batch #36
Migrating Models
Migrating 90 items from customer to customer
customer batch #1
/Users/javiercastro/git/migration-scripts/v3-sql-v4-sql/migrate/migrateModels.js:182
      const [createdAt, updatedAt] = modelDef.options.timestamps;
                                     ^

TypeError: boolean false is not iterable (cannot read property Symbol(Symbol.iterator))
    at /Users/javiercastro/git/migration-scripts/v3-sql-v4-sql/migrate/migrateModels.js:182:38
    at Array.map (<anonymous>)
    at migrateItems (/Users/javiercastro/git/migration-scripts/v3-sql-v4-sql/migrate/helpers/migrateFields.js:25:6)
    at migrate (/Users/javiercastro/git/migration-scripts/v3-sql-v4-sql/migrate/helpers/migrate.js:122:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async migrateModels (/Users/javiercastro/git/migration-scripts/v3-sql-v4-sql/migrate/migrateModels.js:181:5)
    at async migrate (/Users/javiercastro/git/migration-scripts/v3-sql-v4-sql/migrate/index.js:70:3)
    at async f (/Users/javiercastro/git/migration-scripts/v3-sql-v4-sql/index.js:6:3)

Node.js v18.0.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Steps to reproduce the behavior

Configure .env and do yarn start

Expected behavior

Just work.

Bug report

Required System information

• Node.js version: v16.15.0
• NPM version: 8.5.5
• Source Strapi version: v3.6.8
• Target Strapi version: v4.5.3
• Source Database: 8.5.1
• Target Database: 8.8.0
• Operating system: macOS
• Which script are you running: 'v3-sql-v4-sql'

Describe the bug

It starts the migration process, but after a while restoring records in a specific table it gets stuck and shows this error (regarding duplicate ids, but original data does not have duplicate ids of course since its the PK):

duplicate key value violates unique constraint "node_states_pkey"
    at Parser.parseErrorMessage (/strapi-v3-v4-db-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/strapi-v3-v4-db-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/strapi-v3-v4-db-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/strapi-v3-v4-db-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  length: 204,
  severity: 'ERROR',
  code: '23505',
  detail: 'Key (id)=(5440853) already exists.',
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: 'public',
  table: 'node_states',
  column: undefined,
  dataType: undefined,
  constraint: 'node_states_pkey',
  file: 'nbtinsert.c',
  line: '663',
  routine: '_bt_check_unique'
}
error Command failed with exit code 1.

Running the migration script duplicates the value of edit in core store plugin_content_manager_configuration_content_types::
for all collection types that have a dynamic component.

This is what my collection looks like in version 3 Strapi

As you can see it duplicates the collection view config values after running the migration script

The value of the json in field edit is duplicated for collections with dynamic components

"edit": [ [ { "name": "Title", "size": 6 }, { "name": "slug", "size": 6 } ], [ { "name": "start", "size": 6 }, { "name": "end", "size": 6 } ], [ { "name": "CalendarDates", "size": 12 } ], [ { "name": "PreviewMode", "size": 4 }, { "name": "AttendancePoll", "size": 4 }, { "name": "HomepageEvent", "size": 4 } ], [ { "name": "logo", "size": 6 }, { "name": "Background", "size": 6 } ], [ { "name": "Summary", "size": 12 } ], [ { "name": "Content", "size": 12 } ], [ { "name": "Title", "size": 6 } ], [ { "name": "Content", "size": 12 } ], [ { "name": "PreviewMode", "size": 4 }, { "name": "AttendancePoll", "size": 4 }, { "name": "HomepageEvent", "size": 4 } ], [ { "name": "Summary", "size": 12 } ], [ { "name": "ContentDate", "size": 4 }, { "name": "Background", "size": 6 } ], [ { "name": "CalendarDates", "size": 12 } ] ],

Bug report

Describe the bug

Can't handle all locales from single type after migration from strapi 3

Steps to reproduce the behavior

Fixed manually by run something like
insert into blog_page_localizations_links (blog_id, inv_blog_id) values (2, 1) insert into blog_page_localizations_links (blog_id, inv_blog_id) values (1, 2)

Bug report

Required System information

• Node.js version: 14.19.0
• NPM version: 6.14.16
• Source Strapi version: 3
• Target Strapi version: 4
• Source Database: postgresql
• Target Database: postgresql
• Operating system: docker
• Which script are you running: v3-sql-v4-sql

Describe the bug

Database schema in environment variables does not work properly.

Steps to reproduce the behavior

Use a schema other than public and run the v3-sql-v4-sql script

Expected behavior

To use the right schema

Screenshots

If applicable, add screenshots to help explain your problem.

Code snippets

UnhandledPromiseRejectionWarning: error: delete from "admin_permissions_role_links" - relation "admin_permissions_role_links" does not exist

Many of these errors when trying to run the script. If I prepend the table names in the code with the schema it works. The environment variable schema is "strapi"

v3-sql-v4-sql (pg 13 to pg13)
3.6.8 -> 4.1.9

Issue

Thanks for the migration scripts. However when executing the migration, I get the error:
Column »author« of relation »components_blog_blog_bases« does not exists..

In the mentioned v3 component we have a OneToOne relation to author like this (as described here for content types):

"author": {
      "model": "author"
    }

We migrated code to v4 like this:
"author": {
      "type": "relation",
      "relation": "oneToOne",
      "target": "api::author.author"
    }

Strapi creates the following tables in database (new table components_blog_blog_bases_categories_links on v4):

DBv3:
components_blog_blog_bases
components_blog_blog_bases__categories

DBv4:
components_blog_blog_bases
components_blog_blog_bases_author_links
components_blog_blog_bases_categories_links

Is it correct that strapi v4 creates the new table components_blog_blog_bases_categories_links for OneToOne relations?

It looks like, that you implemented this for models but not for components?
Can you please provide a solution to migrate relations in components?

Setup

Simple Component with a 1:1 Relation.

V3

A 1:1 Relation gets added as a separate column with the id of the target.

A 1:Many Releation (here categories) is a separate table

V4

V4 creates separate tables for 1:1 and 1:Many Relations

The script is however still looking for the author column in the v4 table, which of course now does not exist.

Tech

yarn 3.2.0
node 16.15.0
postgres 13

Bug report

Required System information

• Node.js version: v14.16.0
• NPM version: 6.14.11
• Source Strapi version: 3.6.8
• Target Strapi version: 4.1.1
• Source Database: Postgres
• Target Database: Postgres
• Operating system:
• Which script are you running: v3-sql-to-v4-sql

Describe the bug

The parameterized insert syntax for a json field in PostgreSQL contains invalid syntax. Specifically, - invalid input syntax for type json

In context:

(node:67148) UnhandledPromiseRejectionWarning: error: insert into "public"."services" ("created_at", "created_by_id", "description", "high_points", "id", "name", "published_at", "updated_at", "updated_by_id") values ($1, $2, $3, $4, $5, $6, $7, $8, $9), ($10, $11, $12, $13, $14, $15, $16, $17, $18), ($19, $20, $21, $22, $23, $24, $25, $26, $27), ($28, $29, $30, $31, $32, $33, $34, $35, $36), ($37, $38, $39, $40, $41, $42, $43, $44, $45), ($46, $47, $48, $49, $50, $51, $52, $53, $54) - invalid input syntax for type json
    at Parser.parseErrorMessage (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at Socket.Readable.push (internal/streams/readable.js:223:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:67148) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:67148) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Steps to reproduce the behavior

1. Create a JSON field
2. Input data
3. Try to migrate v3 to v4

Expected behavior

Migrate JSON data from v3 to v4

Code snippets

Additional context

Bug report

Required System information

• Node.js version: v16.17.0
• NPM version: v8.15.0
• Source Strapi version: v3.5.0
• Target Strapi version: v4.5.5
• Source Database: MySQL
• Target Database: MySQL
• Operating system: Windows 10 64-bit
• Which script are you running: v3-sql-v4-sql (using the MySQL .env files)

Describe the bug (TL;DR Version)

Please see here: https://github.com/strapi/migration-scripts/blob/main/v3-sql-v4-sql/migrate/helpers/relationHelpers.js#L49-L50

ManyToMany relationships in CollectionTypes with hyphens (e.g. Care-roles) are not migrated properly (screenshots in the full description below).

The following lines need to have the value of modelF and attributeF wrapped in snakeCase(), like so:

addRelation(
        {
          uid,
          model: collectionName,
          attribute: key,
          type: 'manyToMany',
          modelF: snakeCase(value.collection),
          attributeF: snakeCase(value.attribute),
        },
        relations
      );

Through testing on my side, wrapping just modelF seems to work. However, no better / worse behavior was observed when adding it to attribtueF, so ideally I'd need a second pair of eyes from the Strapi team to check for any consequences when adding it to the attributeF field.

Describe the bug (Full Version)

I have two CollectionTypes that interact with one another, articles and care-roles. They hold a many-to-many relationship, where many articles can hold many care-roles.

When I run the migration scripts, I seem to lose all relationships relating to the care-roles collection. I would expect to see something like this in the migration script console output, when run:

Migrating X items from articles_care_roles__care_roles_articles to articles_care_roles_links

However, no message appears. This means the migration script has skipped over it for some reason. I spent a few hours debugging the migration script in VSCode line-by-line, and found the cause of it.

As shown in this screenshot, this will mean that the migrations script is looking for a table called articles_care-roles__care-roles_articles (note the hyphens), which doesn't exist, and so is skipped over.

Whilst looking for potential fixes to this, I could see that the one-to-many addRelation() function uses snakeCase(). This sounded like something I needed for my many-to-many relationship. I added it like so and re-ran the script:

This seemed to fix the issue and all of my relationships migrated properly upon re-run! I ran this fix twice, once with just the modelF value snakecased, and once with both modelF and attributeF snakecased.

There appears to be no difference in behavior; both fixed the issue properly, which makes me wonder if I'm potentially causing any other issues under-the-hood with this.

It would be much appreciated if I could have this reviewed by someone on the Strapi team, as I've found the solution and would like this to be reviewed and a fix merged into the repo, but I've been trying to contact the Strapi team for ~3 weeks to raise this issue, to no avail.

Steps to reproduce the behavior

1. Create a Strapi V3 App.
2. Create two Collection Types called "Article" and "Care-role".
3. Form a many-to-many relationship between them.
4. Add some test data
5. Run CodeMods to upgrade it to V4.
6. Run the migration scripts on the DB to upgrade it to V4.
7. All relationships involving the Care-roles table should be skipped over, due to the reasons stated above.

Expected behavior

The Care-roles relationships should be migrated properly, outputting the following message in the console:

Migrating X items from articles_care_roles__care_roles_articles to articles_care_roles_links

Screenshots

Relevant screenshots included above, please see my messages in the Strapi discord for more screenshots:

• https://discord.com/channels/811989166782021633/963236405641887744/1063081595143204904
• https://discord.com/channels/811989166782021633/963236405641887744/1063142122833838125
• https://discord.com/channels/811989166782021633/963236405641887744/1063393231376744449
• https://discord.com/channels/811989166782021633/963236405641887744/1063410521447084155

Code snippets

Code samples AND pending solution posted above.

Additional context

This is pretty-much solved and I just need this solution reviewed and merged into the main repo by a member of the Strapi team. The only reason I haven't made this a PR is because I'm unsure whether attributeF requires the snake-case fix, so I'd like someone from the Strapi team to review the change to ensure no further issues happen with a result.

I'll be running this eventually on a real client's database, so having this fix officially merged in would be a real confidence-booster in running the script against their real database in a few weeks time.

On PostgreSQL (have not tried other backends), table references should use " instead of '.

yarn run v1.22.18
$ node index.js
Migrating Core Store
Migrating 18/32 items from core_store to strapi_core_store_settings
core_store batch #1
(node:64141) UnhandledPromiseRejectionWarning: error: SELECT SETVAL ('strapi_core_store_settings_id_seq', (SELECT MAX(id) + 1 FROM 'strapi_core_store_settings')) - syntax error at or near "'strapi_core_store_settings'"
    at Parser.parseErrorMessage (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/Users/colearendt/working/strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at Socket.Readable.push (internal/streams/readable.js:223:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:64141) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:64141) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Bug report

Required System information

• Node.js version: 16.16.0
• NPM version: 8.11.0
• Source Strapi version: 3.5.4
• Target Strapi version: 3.5.4
• Source Database: MongoDB
• Target Database: PostgreSQL
• Operating system: Ubuntu
• Which script are you running: v3-mongodb-v3-sql

Describe the bug

I am trying to migrate Strapi v3 with MongoDB to Strapi v3 Postgres. But I am getting the following error:

error: insert into "modules_components" ("component_id", "component_type", "field", "id", "module_id", "order") values (DEFAULT, $1, $2, $3, $4, $5) - null value in column "component_id" of relation "modules_components" violates not-null constraint
    at Parser.parseErrorMessage (/home/siddhesh/Projects/cssoch/migration-scripts/v3-mongodb-v3-sql/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/home/siddhesh/Projects/cssoch/migration-scripts/v3-mongodb-v3-sql/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/home/siddhesh/Projects/cssoch/migration-scripts/v3-mongodb-v3-sql/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/home/siddhesh/Projects/cssoch/migration-scripts/v3-mongodb-v3-sql/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  length: 282,
  severity: 'ERROR',
  code: '23502',
  detail: 'Failing row contains (130, Project, 1, components_project_projects, null, 38).',
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: 'public',
  table: 'modules_components',
  column: 'component_id',
  dataType: undefined,
  constraint: undefined,
  file: 'execMain.c',
  line: '1883',
  routine: 'ExecConstraints'
}

I am not getting the cause of the error. Whether it causing because by the script or because of data.

Enhancement Request

Describe the enhancement

Add additional logging options to have a verbose output of what is actually being done:

• What was copied
• What was skipped
• What errored out

Bug report

Required System information

• Node.js version: 16.13.0
• NPM version: 8.13.2
• Source Strapi version: 3.6.8
• Target Strapi version: 4.1.9
• Source Database: PostgreSQL 12.9
• Target Database: PostgreSQL 12.9
• Operating system: WSL2 / Ubuntu 20.04
• Which script are you running: v3-sql-v4-sql

Describe the bug

If the U&P plugin is missing from the target project, the script will crash.

error: delete from "up_permissions_role_links" - relation "up_permissions_role_links" does not exist

Steps to reproduce the behavior

Pretty straightforward, just try a migration, targeting a project without U&P installed.

Expected behavior

I would expect a warning message explaining that the U&P plugin is probably not installed, and the users' permissions were not migrated. Apart from that everything should work nonetheless.

Or you can choose to stop there and tell the user to add U&P to his/her app but then it implies that U&P is not really a plugin as it's required.

Bug report

Required System information

• Node.js version: 16.15.0
• NPM version: 8.5.5
• Source Strapi version: v3.6.9
• Target Strapi version: v4.1.9
• Source Database: postgres 12
• Target Database: postgres 12
• Operating system: Ubuntu
• Which script are you running: v3-sql-v4-sql

Describe the bug

I have the latest version with the Pull Request which fixed the relation issues #14 , but I still get an error that a column does not exist in a relation

error: insert into "components_slidetypen_template_twl_s_lights_links" ("components_slidetypen_template_twl_id", "light_id", "template_twl_id") values ($1, $2, $3), ($4, $5, $6), ($7, $8, $9), ($10, $11, $12) - column "components_slidetypen_template_twl_id" of relation "components_slidetypen_template_twl_s_lights_links" does not exist

The relation in question "lights" here is a 'hasMany' relation.

The source Model looks like this:

{
  "collectionName": "components_slidetypen_template_twl_s",
  "info": {
    "name": "template_TWL_S",
    "icon": "arrow-left",
    "description": ""
  },
  "options": {},
  "attributes": {
    "title": {
      "type": "string"
    },
    "subtitle": {
      "type": "string"
    },
    "lights": {
      "collection": "light"
    },
    "actorList": {
      "type": "component",
      "repeatable": true,
      "component": "sensors.actor-group"
    },
    "sensorList": {
      "type": "component",
      "repeatable": true,
      "component": "sensors.sensor-group"
    }
  }
}

The Target Model looks like this:

{
  "collectionName": "components_slidetypen_template_twl_s",
  "info": {
    "name": "template_TWL_S",
    "icon": "arrow-left",
    "description": ""
  },
  "options": {},
  "attributes": {
    "title": {
      "type": "string"
    },
    "subtitle": {
      "type": "string"
    },
    "lights": {
      "type": "relation",
      "relation": "oneToMany",
      "target": "api::light.light"
    },
    "actorList": {
      "type": "component",
      "repeatable": true,
      "component": "sensors.actor-group"
    },
    "sensorList": {
      "type": "component",
      "repeatable": true,
      "component": "sensors.sensor-group"
    }
  }
}

I also added the code snippets from #9. I thought it could have something to do with the Plural Naming, but it still does not work.

I am open for suggestions and questions.

We are pretty reliant on a v4 migration for the project to not launch it with an outdated version.

Bug report

Required System information

• Node.js version: v14
• NPM version: v6
• Source Strapi version: 3.6.11
• Target Strapi version: v4.10.5
• Source Database: Postgres
• Target Database: Postgres
• Which script are you running: Migrate

Describe the bug

While the scripts add data from the source to the destination, users seem to be reaching race conditions seemingly caused by PrimaryKey collisions in some database tables. The currently proposed solution from interaction with the user and observation is to add a setTimeout call between SQL create statements to slow down the process allowing the prior process to complete.

Steps to reproduce the behavior

This occurred during a user's routine migration process. To our knowledge, there aren't any steps outside of what we prescribe in our guides that the users followed.

Additional context

If more context is required, @kasonde or @derrickmehaffy would do their best gather the information required. For now, here's an example of the output they got from the scripts:

error: insert into "strapi_ecomm_<retracted>_v4"."admin_permissions" ("action", "conditions", "created_at", "properties", "subject", "updated_at") values ($1, $2, $3, $4, $5, $6) returning "id" - duplicate key value violates unique constraint "admin_permissions_pkey"
error: insert into "strapi_ecomm_<retracted>_v4"."admin_permissions" ("action", "conditions", "created_at", "properties", "subject", "updated_at") values ($1, $2, $3, $4, $5, $6) returning "id" - duplicate key value violates unique constraint "admin_permissions_pkey"

Anywhere you see <retracted> is information about the user we'd prefer to keep private but has no effect on the process. It is just three letters, i.e strapi_ecomm_abc_v4.

Bug report

Required System information

• Node.js version: v14.19.2
• NPM version: 8.9.0
• Source Strapi version: 3.6.10
• Target Strapi version: 4.2.2
• Source Database: PostgreSQL 12.11
• Target Database: same as above
• Operating system: Mac OS X 12.3.1 (Monterey)
• Which script are you running: v3-sql-v4-sql, pg

Describe the bug

During model migration the script omits existing "created_at" and "updated_at" fields/values,
thus migrated models in target database have empty "created_at" and "updated_at" fields.

Steps to reproduce the behavior

1. Setup a Strapi instance/database which uses "created_at" and "updated_at" fields on models for timestamps
2. Follow the instruction from README file
3. Inspect "created_at" and "updated_at" fields on migrated models in target database, their "created_at" and "updated_at" fields will be empty

Expected behavior

Migrated models in target database should have "created_at" and "updated_at" timestamps - matching the "created_at" and "updated_at" timestamps of models from source database.

Screenshots

n/a

Code snippets

see upcoming PR

Bug report

Required System information

• Node.js version: 16.16.0
• NPM version: 8.11.0
• Source Strapi version: 3.6.8
• Target Strapi version: 4.3.2
• Source Database: postgres 13
• Target Database: postgres 13
• Operating system: OSX
• Which script are you running: v3-sql-to-v4-sql

Describe the bug

The blog_post_categories_localizations_links table of one of my content types is not being properly populated. Only the first column blog_post_category_id column populated but not the inv_blog_post_category_id column. However for some content types with similar complexity this works (i also have other content types that don't work either).

Steps to reproduce the behavior

this is the v4 content type definition:

{
  "kind": "collectionType",
  "collectionName": "blog_post_categories",
  "info": {
    "singularName": "blog-post-category",
    "pluralName": "blog-post-categories",
    "displayName": "Blog Post Category",
    "description": ""
  },
  "options": {
    "draftAndPublish": false
  },
  "pluginOptions": {
    "i18n": {
      "localized": true
    }
  },
  "attributes": {
    "name": {
      "pluginOptions": {
        "i18n": {
          "localized": true
        }
      },
      "type": "string",
      "unique": true,
      "required": true
    }
  }
}

the v3 content type definition:

{
  "kind": "collectionType",
  "collectionName": "blog_post_categories",
  "info": {
    "name": "Blog Post Category",
    "description": ""
  },
  "options": {
    "increments": true,
    "timestamps": true,
    "draftAndPublish": false
  },
  "pluginOptions": {
    "i18n": {
      "localized": true
    }
  },
  "attributes": {
    "name": {
      "pluginOptions": {
        "i18n": {
          "localized": true
        }
      },
      "type": "string",
      "unique": true,
      "required": true
    }
  }
}

The v3 database shows the following records (interestingly the column is named blog-post_id and not as expected related_blog_post_id. maybe this is a no-issue, could be because of the - vs. _:

The v4 database:

Expected behavior

i'd expect the v4 table to be properly populated.

Additional context

might be related to #29

Bug report

Required System information

• Node.js version: 16.17.0
• NPM version: 8.15.0
• Source Strapi version: 3.6.8
• Target Strapi version: 4.3.4
• Source Database: Sqlite3
• Target Database: Sqlite3
• Operating system: MacOS
• Which script are you running: v3-sql-v4-sql

Describe the bug

Relation fields on the user model fails to migrate to V4 as schema is not loaded in migrateUser.js.

Steps to reproduce the behavior

1. Create relation field on user model in v3
2. Migrate strapi code from v3 to v4
3. Migrate sqlite3 db from v3 to v4
4. See error below

[Error: insert into up_users (all insert queries), SQLITE_ERROR: table up_users has no column named country] {
errno: 1,
code: 'SQLITE_ERROR'
}

Expected behavior

The script should create relations on the user model in the same way as custom made models.

Bug report

Required System information

• Node.js version: v16
• NPM version: v6
• Source Strapi version: 3.6.8
• Target Strapi version: 4.2.0
• Source Database: PG
• Target Database: PG
• Operating system: Linux POP OS 22.04
• Which script are you running: v3-sql to v4-sql

Describe the bug

insert or update on table "x_components" violates foreign key constraint "x_entity_fk"

Screenshots

Bug report

Required System information

• Node.js version: v14.18.3
• NPM version: 0.34.0
• Source Strapi version: 3.6.3
• Target Strapi version: 4.1.12
• Source Database: PostgreSQL 12.9 (Ubuntu 12.9-0ubuntu0.20.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
• Target Database: same as above
• Operating system: Linux Mint 20.2
• Which script are you running: v3-sql-v4-sql, pg

Describe the bug

I have an extra column on v3's strapi_permission table. At the moment I'm not sure if it's being used by the lib or the project. What would be the best way to port it to v4.

I get this error when trying to migrate SQL data, v3 to v4:

column "fields" of relation "admin_permissions" does not exist

Steps to reproduce the behavior

1.Follow the instruction from README file
2. See errors

Expected behavior

Not to have this error because afaik there isn't any fields column in v4's admin_permissions table.

Screenshots

n/a

Code snippets

Longer stack:

yarn start
yarn run v1.22.18
$ node index.js
Migrating Core Store TBA
Migrating 100/197 items from core_store to strapi_core_store_settings
core_store batch #1
core_store batch #2
Migrating Admin
Migrating 9 items from strapi_role to admin_roles
strapi_role batch #1
Migrating 109 items from strapi_administrator to admin_users
strapi_administrator batch #1
strapi_administrator batch #2
strapi_administrator batch #3
Migrating 109 items from strapi_users_roles to admin_users_roles_links
strapi_users_roles batch #1
strapi_users_roles batch #2
strapi_users_roles batch #3
Migrating 871 items from strapi_permission to admin_permissions
strapi_permission batch #1
(node:66281) UnhandledPromiseRejectionWarning: error: insert into "admin_permissions" ("action", "conditions", "created_at", "fields", "id", "properties", "subject", "updated_at") values ($1, $2, $3, $4, $5, $6, $7, $8),  ($393, $394, $395, $396, $397, $398, $399, $400) - column "fields" of relation "admin_permissions" does not exist
    at Parser.parseErrorMessage (strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (strapi-migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at Socket.Readable.push (internal/streams/readable.js:206:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:66281) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:66281) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Additional context

n/a

Bug report

Required System information

• Node.js version: v14.20.0
• NPM version: 6.14.17
• Strapi version: v4.3.8
• Database: PostgreSQL 14.5
• Operating system: Ubuntu 20.04

Describe the bug

After following the steps to migrate database from v3 to v4, the scripts fails with the following error:

error: delete from "up_users_role_links" - relation "up_users_role_links" does not exist

Steps to reproduce the behavior

1. Follow steps in https://github.com/strapi/migration-scripts/tree/main/v3-sql-v4-sql

Expected behavior

Database should be fully migrated to v4.

yarn start entire output

yarn run v1.22.19
$ node index.js
Migrating Core Store TBA
Migrating 17/29 items from core_store to strapi_core_store_settings
core_store batch strapi/strapi#1
Migrating Admin
Migrating 3 items from strapi_role to admin_roles
DBV4 ITEMS
strapi_role batch strapi/strapi#1
Migrating 5 items from strapi_administrator to admin_users
DBV4 ITEMS
strapi_administrator batch strapi/strapi#1
Migrating 12 items from strapi_users_roles to admin_users_roles_links
DBV4 ITEMS
strapi_users_roles batch strapi/strapi#1
Migrating 85 items from strapi_permission to admin_permissions
strapi_permission batch strapi/strapi#1
strapi_permission batch strapi/strapi#2
Migrating Users
Migrating 2 items from users-permissions_role to up_roles
DBV4 ITEMS
users-permissions_role batch strapi/strapi#1
Migrating 14/210 items from users-permissions_permission to up_permissions
users-permissions_permission batch strapi/strapi#1
Migrating 1069 items from users-permissions_user to up_users
(node:238514) UnhandledPromiseRejectionWarning: error: delete from "up_users_role_links" - relation "up_users_role_links" does not exist
    at Parser.parseErrorMessage (/mnt/480gb/casion-projects/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/mnt/480gb/casion-projects/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/mnt/480gb/casion-projects/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/mnt/480gb/casion-projects/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at Socket.Readable.push (internal/streams/readable.js:206:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:238514) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:238514) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Additional context

The error seems to originate in https://github.com/strapi/migration-scripts/blob/main/v3-sql-v4-sql/migrate/migrateUsers.js on line 64, where it tries to delete everything from the table up_users_role_links but this table doesn't exist.

Thanks so much for making these scripts!

v3 mongo=> v3 sql timestamps aren't converting for me.

I find it strange in the code that:

Also, the if isn't working for me when i do change around these lines, since here:

I'll make a PR after this report :)

TODO: Add information about sql to sql script for Notum (Credit/link to their site/ect)

Media entries in the upload_file table do get copied into the new files table but are not displayed in the strapi media library ui (I'm upgrading to strapi version 4.12.4). I found this could be fixed by adding folder_path: "/" below line 44 in https://github.com/strapi/migration-scripts/blob/30aedca3233ae28475dba301b963b4f106c90119/v3-sql-v4-sql/migrate/migrateFiles.js

Unfortunately I don't have the capacity to write a proper PR and do testing and all that but I thought I should open an issue in case anyone else runs into this and it's helpful to them or someone wants to properly fix this.

Ta!

Bug report

Required System information

• Target Strapi version: 4.4.3
• Which script are you running: v3-sql-v4-sql

Describe the bug

Strapi allows components to have the same name as collections but the migration script doesn't allow for this and results in link tables with only the component id set, with null for the collection.

If a component has a one to one relationship with a collection with the same name, Strapi uses the field name inv_{collectionname}_id in the links table to distinguish the names.

e.g.
If there is a collection named activity and a component also named activity, Strapi will generate a table, components_category_activities_activity_links with the fields

| id | activity_id | inv_activity_id |

I have created a PR, #67, which supports migrating into this kind of table. It's a bit of a letterbox fix but it works for my migration.

Bug report

Describe the bug

A new issue was introduced by this commit:
471c641

When a strapi v3 column has a hyphen in its name, "inv_" is being prepended to the strapi v4 column name, causing the migration to fail.

Steps to reproduce the behavior

See the comments added to the commit for the specific line causing the failure

Bug report

Issue Description

When running the script, if a column with a many-to-many relation is empty or contains null values, the script terminates prematurely. This issue occurs because the map method is called on a null value in such cases.

Steps to Reproduce

1. Run the migration script.
2. Encounter a column with a many-to-many relation that is empty or contains null values.

Expected Behavior

The script should gracefully handle cases where columns in many-to-many relations are empty or contain null values, allowing it to continue processing other data.

Temporary fix?

Found a solution for my specific use case adding guards like this
line 214 : const rows = (entry[key] ?? []).map((e, idx) => ({ ... })

line 270 : if (!entry[key] || !idMap.get(entry[key])) { continue; }

I don't know if i shoud submit a Pull Request with this fix or if something more robust will be added, let me know !

Bug report

Required System information

• Node.js version: 14.20.1
• NPM version: 6.14.17
• Source Strapi version: 3.6.10
• Target Strapi version: 4.3.9
• Source Database: SQL (MariaDB and SQLite)
• Target Database: SQL (MariaDB and SQLite)
• Operating system: Ubuntu 20.04
• Which script are you running: v3-sql-v4-sql

Describe the bug

In some cases, many-to-many relations are not properly migrated and data is missing.

It looks like it has something to do with which attribute is the dominant one, or with the alphabetical order of the referenced content-types, or with the name of the table that stores the relations.

Steps to reproduce the behavior

1. Run npx create-strapi-app@v3 my-project --quickstart to generate an empty Strapi v3 project
2. As described in the v3 Quick Start Guide, create the restaurant and categories content-types, but with one difference:
   Instead of adding the many-to-many relation into the category, add it to the restaurant
   
3. Create a restaurant and some categories and link them via the relation field
   
   
4. Notice that the name of the relation table is categories_restaurants__restaurants_categories
   
5. Run codemods to migrate code to v4 syntax
6. Remove package-lock.json, remove node_modules folder, change sqlite3 dependency to latest better-sqlite3 dependency for SQLite + Strapi v4 compatibility (source)
7. Run npm i
8. Configure new sqlite database path for Strapi v4 in .env
9. Run npm run build && npm run develop to start Strapi v4 and let it generate the empty tables, kill process afterwards
10. Execute the v3-sql-v4-sql script to migrate the database from v3 to v4
11. Run npm run develop to start Strapi v4
12. Notice that the many-to-many relations between restaurants and categories are missing:
    
    
    

Expected behavior

The restaurant_categories_links table contains the migrated many-to-many-relations.

Additional info

With the "How to reproduce" guide in mind this might look like an artificial issue. But we're currently migrating a v3 project which has been in production for 1.5 years to v4. That project contains some many-to-many relations of all sorts and changing how these relations are defined while ensuring data integrity is something I'm a little afraid of :)
By pure chance I found out that some data is missing. I hope there isn't more :(

If you try to reproduce this issue, pay attention to step 2:

When I create the many-to-many relation in the category as described in the Quick Start Guide, the many-to-many relation gets properly migrated. Only when I create it the other way round the migration does not happen.

I suspect it has something to do with either the alphabetical order of the content-type name, or with the dominant property of the attribute.

Although the dominant property would be weird, because the doc mentions that that one is used for NoSQL databases only.

When it's like this the migration will work:

// category.settings.json
{
...
  "attributes": {
    ...
    "restaurants": {
      "collection": "restaurant",
      "via": "categories",
      "dominant": true
    }
  }
}

// restaurant.settings.json
{
  ...
  "attributes": {
    ...
    "categories": {
      "via": "restaurants",
      "collection": "category"
    }
  }
}

But when it's like this the migration will not work:

// category.settings.json
{
...
  "attributes": {
    ...
    "restaurants": {
      "via": "categories",
      "collection": "restaurant"
    }
  }
}

// restaurant.settings.json
{
  ...
  "attributes": {
    ...
    "categories": {
      "collection": "category",
      "via": "restaurants",
      "dominant": true
    }
  }
}

Bug report

Required System information

• Node.js version: v14
• NPM version: v6
• Source Strapi version: 3.6.8
• Target Strapi version: 4.2.3
• Source Database: PG
• Target Database: PG
• Operating system: Linux Mint 21
• Which script are you running: v3-sql to v4-sql

Describe the bug

When old columns were not deleted (caused by: strapi/strapi#1114 most likely) the script will try to transfer data from these columns but since they don't exist on v4 the script crashes. We gracefully handle this for tables but not columns.

Steps to reproduce the behavior

1. Create a field in a content-type on v3
2. Delete the field (leaves trace of column in db due to issue 1114)
3. Try to migrate data from content-type to v4
4. See error (sanitized example below)

Expected behavior

We should gracefully handle fields that exist in v3 database that don't in v4 and ideally warn the user that the field was missing and data was not transferred for it.

Additional context

@martincapek If you need more info, if you are working on this, ping me on slack as I have a customer's project that I was troubleshooting and can help to show you how to reproduce this.

Hi. We are trying to migrate from strapi v3.x to v4.x using the migration scripts and codemods.

And when we attempted it, we get the error as shown in the screenshot:

UnhandledPromiseRejectionWarning: TypeError: Cannot read property ‘layouts’ of null
at /opt/ramco-cements-website-v4/migration-scripts/v3-sql-v4-sql/migrate/migrateCoreStore.js:51:17
at Array.map ()
at migrateItems (/opt/ramco-cements-website-v4/migration-scripts/v3-sql-v4-sql/migrate/helpers/migrateFields.js:24:16)
at Object.migrateTables (/opt/ramco-cements-website-v4/migration-scripts/v3-sql-v4-sql/migrate/migrateCoreStore.js:45:27)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async migrate (/opt/ramco-cements-website-v4/migration-scripts/v3-sql-v4-sql/migrate/index.js:63:5)
at async f (/opt/ramco-cements-website-v4/migration-scripts/v3-sql-v4-sql/index.js:6:3)
(Use node --trace-warnings ... to show where the warning was created)
(node:3437836) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see [Command-line API | Node.js v19.4.0 Documentation](https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode)). (rejection id: 1)
(node:3437836) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

We are unable to proceed with the migration because of this. Kindly help. We reached out to support, filed a ticket in the forum here: https://forum.strapi.io/t/data-migration-issue-from-v3-sql-to-v4-sql/24873 but weren't able to get a reply on the same.

Bug report

Required System information

• Node.js version: v14.19.3
• NPM version:
• Strapi version: From 3.6.10 to 4.2.2
• Database: psql13
• Operating system: MacOs

Describe the bug

Running v3-sql-v4-sql migration, got an error on table contentName_localizations migration.
For some data models : no error but invalid results (only contentName_id column is populated)
For some data models : get error : column "contentName_id" of relation "contentName_localizations_links" does not exist

Steps to reproduce the behavior

1. Run yarn start

Expected behavior

Should migrate localizations without errors

Code snippets

Migrating 2 items from companies__localizations to companies_localizations_links
companies__localizations batch #1
/scripts/strapi_migrations/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:287
        const message = name === 'notice' ? new messages_1.NoticeMessage(length, messageValue) : new messages_1.DatabaseError(messageValue, length, name);
                                                                                                 ^

error: insert into "companies_localizations_links" ("company_id", "related_company_id") values ($1, $2), ($3, $4) - column "related_company_id" of relation "companies_localizations_links" does not exist
    at Parser.parseErrorMessage (/scripts/strapi_migrations/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/scripts/strapi_migrations/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/scripts/strapi_migrations/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/scripts/strapi_migrations/migration-scripts/v3-sql-v4-sql/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:390:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:199:23) {
  length: 160,
  severity: 'ERROR',
  code: '42703',
  detail: undefined,
  hint: undefined,
  position: '60',
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'parse_target.c',
  line: '1066',
  routine: 'checkInsertTargets'
}
error Command failed with exit code 1.

i have this problem by migrate from v3 to v4
can somebody help me to fix this problem or does somebody have any idea ?
thanks in advance

Migrating 25786 items from users-permissions_user to up_users
users-permissions_user batch #1
(node:13348) UnhandledPromiseRejectionWarning: error: insert into "up_users" ("academy_department", "address_city", "address_country", "address_street", "address_zip", "billing", "blocked", "cities", "comments", "confirmation_token", "confirmed", "created_at", "created_by_id", "customer_id", "customer_number", "deletion_at", "description", "email", "email_confirmed", "id", "industry", "international_area_code", "language", "name", "organization_size", "partner_academy", "password", "phone", "privacy_consent_at", "provider", "reset_password_token", "slug", "stripe_customer_id", "subscriptions", "terms_consent_at", "token_version", "type", "updated_at", "updated_by_id", "username", "website_url", "welcomed_at")

Bug report

Required System information

• Node.js version: 16.13.2
• NPM version: 8.1.2
• Source Strapi version: 3.6.8
• Target Strapi version: 4.3.8
• Source Database: mysql
• Target Database: mysql
• Operating system: OSX
• Which script are you running: v3-sql-v4-sql

Describe the bug

When running the script, the script aborts with the following error:
const v3RelationTables = tables.filter((t) => t.includes("__"));
TypeError: Cannot read properties of undefined (reading 'includes')

Steps to reproduce the behavior

1. Go to 'nvm use 16'
2. Run $ yarn
3. Run $ yarn start
4. See error

Screenshots

Code snippets

const v3RelationTables = tables.filter((t) => t.includes("__"));

Additional context

Before starting the script, I had to adjust certain columns in the table to be compliant with the snake_case.

Bug report

Required System information

• Node.js version: 16.17.0
• NPM version: 8.15.0
• Source Strapi version: 3.6.8
• Target Strapi version: 4.3.4
• Source Database: pgs
• Target Database: pgs
• Operating system: alpine
• Which script are you running: v3-sql-v4-sql

Describe the bug

`const message = name === 'notice' ? new messages_1.NoticeMessage(length, messageValue) : new messages_1.DatabaseError(messageValue, length, name);
^

error: insert into "up_users" (...) - column "name" of relation "up_users" does not exist`

Bug report

When I try to migrate my Strapi v3 to v4, some many to many relations doesn't going right, after debug the code I found an issue.

Required System information

• Node.js version: 16.19.0
• NPM version: 8.19.3
• Strapi version: 4
• Database: PG
• Operating system: any
• Is your project Javascript or Typescript: Javascript

Describe the bug

So in script v3-sql-v4-sql in the below part o code, the verification is incorrect.
Many to many relations doesn't has a column to relate, they has it's own table to make the relations. (I know in Strapi V4 all relations has it's own table, but if in migration the relation it's doesn't identified as many to many your data is gonna be wrong)

The part of code here

So, to fix it locally I do: if (value.column) >>> if (!value.column) ...

Steps to reproduce the behavior

1 - Strapi V3 with PG
2 - Has many to many relations in strapi V3
3 - Do the steps of migration described in https://github.com/strapi/migration-scripts

Expected behavior

Your many to many relations gonna be empty.

Screenshots

Additional context

Maybe can be related with this other issue