Migrates Challenges, Resources, and Resource Roles from Informix to DynamoDB

JavaScript 89.44% Dockerfile 0.52% Shell 10.04%

legacy-challenge-migration-script's People

Contributors

Watchers

Forkers

mark-nakachon yoution thehashrocket codingdrone zarana-mehta12 srqway sathyajayabal rakibansary eisbilir

legacy-challenge-migration-script's Issues

Dirty groups data breaks the GET /challenges endpoint

We added the following to the ES query when the caller is not authenticated but because of dirty data (null and undefined string values) in ES, this makes the response always be an empty array.

{
   must_not: { exists: { field: 'groups' } }
}

This needs to be fixed here so we completely omit null and undefined values.

Migrate Scorecard and Review IDs

Create a migration for these two properties:

screeningScorecardId
reviewScorecardId

Setup Tasks:

Under src/scripts, create a folder for migrations
Name migrations 000-{descriptive-name}
Add an npm task to npm run migration {filename}

Tasks:

Create the migration script to query for 1000 rows at a time, to update those two properties on the legacy object in dynamo and es
Add these properties to the regular migration going forward (migrateAll, migrateOne)

Informix SQL given below to get all data. Please add a limit/offset to this so it doesn't pull all rows.

SQL (taken from https://github.com/appirio-tech/ap-challenge-microservice/blob/f4f8491b5a9b4fb21f19aeca69f51f8d844490ed/src/main/resources/sql/challenge/challenge_details.sql):

SELECT 
     p.project_id as legacyid,
     pc3.parameter AS screeningscorecardid,
     pc4.parameter AS reviewscorecardid
  FROM project p
     , project_status_lu pstatus
     , outer project_spec ps
     , outer project_studio_specification sps
     , outer project_mm_specification pmms
     , project_phase pp1 
     , project_phase pp2 
     , outer project_phase pp6 
     , outer project_phase pp15 
     , outer project_phase pp9 
     , outer ( project_phase pp3  
     , outer phase_criteria pc3 ) 
     , outer ( project_phase pp4 
     , outer phase_criteria pc4 )
     , project_info pn
     , project_info pidr
     , outer project_phase nd_phase
     , outer project_info pi70 
     , project_category_lu pcl
     , outer project_info pi4
     , outer project_info pi1
     , outer project_info pi51
     , outer project_info pi52
     , outer project_info pi78
     , outer project_info pi79
     , outer project_info pi56
     , outer project_info pi87
 WHERE 1=1
   AND p.project_status_id = pstatus.project_status_id
   AND p.project_id = pn.project_id
   AND pn.project_info_type_id = 6
   AND pp1.project_id = p.project_id
   AND pp1.phase_type_id = 1  
   AND pp2.project_id = p.project_id
   AND pp2.phase_type_id = 2  
   AND pp6.project_id = p.project_id
   AND pp6.phase_type_id = 6  
   AND pp15.project_id = p.project_id
   AND pp15.phase_type_id = 15 
   AND pp9.project_id = p.project_id
   AND pp9.phase_type_id = 9  
   AND pp9.project_phase_id = (SELECT MAX(project_phase_id) FROM project_phase WHERE project_id = p.project_id AND phase_type_id = 9)
   AND pp3.project_id = p.project_id
   AND pp3.phase_type_id = 3  
   AND pp3.project_phase_id = pc3.project_phase_id
   AND pc3.phase_criteria_type_id = 1
   AND pp4.project_id = p.project_id
   AND (pp4.phase_type_id = 4 OR (pp4.phase_type_id = 18 AND p.project_category_id = 38))
   AND pp4.project_phase_id = pc4.project_phase_id
   AND pp4.project_phase_id = (SELECT MAX(project_phase_id) FROM project_phase WHERE project_id = p.project_id AND phase_type_id IN (4,18))
   AND pc4.phase_criteria_type_id = 1 
   AND pidr.project_id = p.project_id
   AND pidr.project_info_type_id = 26  
   AND pi70.project_id = p.project_id
   AND pi70.project_info_type_id = 70  
   AND pi4.project_id = p.project_id
   AND pi4.project_info_type_id = 4  
   AND pi1.project_info_type_id = 1 
   AND pi1.project_id = p.project_id
   AND pi51.project_info_type_id = 51
   AND pi51.project_id = p.project_id
   AND pi52.project_info_type_id = 52 
   AND pi52.project_id = p.project_id
   AND pi78.project_info_type_id = 78 
   AND pi78.project_id = p.project_id
   AND pi79.project_info_type_id = 79 
   AND pi79.project_id = p.project_id
   AND pi56.project_info_type_id = 56
   AND pi56.project_id = p.project_id
   AND p.project_id = nd_phase.project_id
   AND nd_phase.phase_type_id = (SELECT MIN(phase_type_id) FROM project_phase WHERE project_id = p.project_id AND phase_status_id = 2 AND phase_type_id not in (13,14))
   AND p.project_category_id = pcl.project_category_id
   AND ps.project_id = p.project_id
   AND sps.project_studio_spec_id = p.project_studio_spec_id
   AND pmms.project_mm_spec_id = p.project_mm_spec_id
   AND pi87.project_info_type_id = 87
   AND pi87.project_id = p.project_id
AND ps.version = (SELECT max(version) FROM project_spec ps2 WHERE ps2.project_id = ps.project_id)

Challenge terms are not properly populated

Example data from challenge that was migrated using the script:

terms: [
{
legacyId: "20704"
},
{
legacyId: "20794"
},
{
legacyId: "20893"
},
{
legacyId: "21193"
}
],

Example data from challenge created using the API:

"terms": [
        {
            "id": "0dedac8f-5a1a-4fe7-936f-e1d04dc65b7d",
            "agreeabilityType": "Electronically-agreeable",
            "title": "Terms & Conditions of Use at TopCoder",
            "url": ""
        }
    ],

Support upsert

We need to add support for upserting already migrated challenges

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

topcoder-platform / legacy-challenge-migration-script Goto Github PK