Add required shopping list sync CLI options.
Unit tests
Make sure the order on sync all resources is respected (concurrency and dependencies needs to be taken into account)

Include lastSyncStatistics in LastSyncCustomObject#equals comparison

Depends on https://github.com/commercetools/commercetools-sync-java/issues/376.

When syncing states without any transitions configured (in our case null, using java),
the transition is set as empty array in the destination project.

We would expect that the transition field is null and not an empty array as in the source state.

Add an integration test for the case when project-sync runs multiple times on the same source and target project and in the source project there were changes that should be synced to the target project (delta sync).

We are syncing products between two environments, the source product has price tiers. These price tiers are not synced to the target environment.

Prices of source product:

"prices": [
    {
    "value": {
        "type": "centPrecision",
        "currencyCode": "EUR",
        "centAmount": 95000,
        "fractionDigits": 2
    },
    "id": "d67d52a8-9bfc-4655-8739-372d816d16cd",
    "country": "AT",
    "customerGroup": {
        "typeId": "customer-group",
        "id": "bbb93cf7-0379-47ba-8a9b-4629244fe151"
    },
    "tiers": [
        {
        "minimumQuantity": 3,
        "value": {
            "type": "centPrecision",
            "currencyCode": "EUR",
            "centAmount": 93000,
            "fractionDigits": 2
        }
        },
        {
        "minimumQuantity": 5,
        "value": {
            "type": "centPrecision",
            "currencyCode": "EUR",
            "centAmount": 92000,
            "fractionDigits": 2
        }
        },
        {
        "minimumQuantity": 10,
        "value": {
            "type": "centPrecision",
            "currencyCode": "EUR",
            "centAmount": 86100,
            "fractionDigits": 2
        }
        }
    ]
    }
],

This results in this price object in the target:

"prices": [
    {
    "value": {
        "type": "centPrecision",
        "currencyCode": "EUR",
        "centAmount": 95000,
        "fractionDigits": 2
    },
    "id": "1b16fe45-5bdc-4b6d-a9c7-5b55c8e13198",
    "country": "AT",
    "customerGroup": {
        "typeId": "customer-group",
        "id": "768252cf-fe31-450e-bd42-f65de6415e46"
    }
    }
],

In the target, the tiers are missing.

I'm trying to sync products (v3.12.0 and v3.11.0) between projects, but I'm getting this error:

{"timestamp":"2021-04-15T07:44:31.713Z","severity":"ERROR","loggerName":"com.commercetools.project.sync.product.ProductSyncer","message":"Error when trying to sync product. Existing key: <<not present>>. Update actions: []","stackHash":"1e0fe24d","stackTrace":"i.s.s.c.ErrorResponseException: detailMessage: Malformed parameter: where: Syntax error while parsing 'where'. Invalid input '-', expected comma or ')' (line 1, column 1194):\

With our recent change of java-sync to 2.0.0 the code coverage has dropped (#158).

The main reason is that we expanded warning and error callbacks to a lambda expression and somehow codecov started to count it as not tested. https://github.com/commercetools/commercetools-project-sync/pull/158/files#diff-623a82010d385f63b825f65e342a0eb0R64

We need to test also these warning and error callbacks to increase the coverage.

JRE has been enclosed inside docker image. It is more important to make sure docker container is installed instead.

Dependabot encountered the following error when parsing your .dependabot/config.yml:

Automerging is not enabled for this account. You can enable it from the [account settings](https://app.dependabot.com/accounts/commercetools/settings) screen in your Dependabot dashboard.

Please update the config file to conform with Dependabot's specification using our docs and online validator.

This test should replace reference IDs with keys: https://github.com/commercetools/commercetools-project-sync/blob/master/src/test/java/com/commercetools/project/sync/cartdiscount/CartDiscountSyncerTest.java#L34

However, there’s no reference key in a cart discounts that are used in the test: https://github.com/commercetools/commercetools-project-sync/blob/master/src/test/resources/cart-discount-key-1.json, https://github.com/commercetools/commercetools-project-sync/blob/master/src/test/resources/cart-discount-key-2.json

So this test is quite useless as it doesn’t really test that the references are replaced correctly.

Check also other tests as well.

Remove stacktraces from expected exceptions e.g. wrong/invalid input.

Currently Syncer order is listed as below

1. Type/ProductType/State
2. TaxCategory
3. Category
4. Product
5. CartDiscount
6. Inventory

Since CartDiscount, Inventory, Category do not rely on Product and on each other. Their Syncer can be executed in parallel.
Also TaxCategory doesn't rely on any other resources, therefore its Syncer can be executed in parallel with Type/ProductType/State.

Expected Order

1. Type/ProductType/State/TaxCategory
2. Category/CartDiscount/Inventory
3. Product

Reference Link
#52

Add required shopping list sync CLI options.
Unit tests
Make sure the order on sync all resources is respected (concurrency and dependencies needs to be taken into account)

The generated time stamp is persisted with the buffer (-2 minutes) which is wrong.

Only when querying from modified resources we should query using the -2 minutes buffer.

Situation
To limit the github action workflow concurrency we are using a github action step from https://github.com/softprops/turnstyle

This is needed because running build in parallel causing data conflicts related to the integration tests.

Complication
While we are building the first action flow, github was not supporting this feature, but now seems github introduced concurrency keyword to limit the concurrency in the action.
See: https://github.blog/changelog/2021-04-19-github-actions-limit-workflow-run-or-job-concurrency/

Resolution
Use the concurrency keyword to limit the concurrency of the workflow run.

Ensure you have changed all workflows in the folder: https://github.com/commercetools/commercetools-project-sync/tree/master/.github/workflows

CONTRIBUTING

Situation

Custom object sync excludes custom objects created by the project sync. However, there could be cases when the user might want to sync the custom objects created by the project sync.

Complication

The project-sync does not sync custom objects created by the project-sync itself.

Resolution

Enable project-sync to sync custom objects created by the project-sync itself.

Additional information

Add a flag to allow project-sync to sync custom objects created by the project-sync itself. By default the project sync will ignore those custom objects.

If you're using Java in Docker, memory params from Docker are not respected in Java 8, see https://developers.redhat.com/blog/2017/03/14/java-inside-docker/. Try to migrate to Java 9 to avoid this problem.

Bump commercetools-sync-java to sync products in any order

explain, how to pass env variables to docker.

It should be able to fetch tax categories from the source project and sync into the target project.

Description

To improve the developer experience and easen our support and training burden all existing references to *.sphere.io and .commercetools.co host names should be removed in favor of not defaulting to a specific region (a common complaint of US and AWS customers is that EU is defaulted everywhere) or, if needed for backwards compatibility, be replaced with the new *.europe-west1.gcp.commercetools.com notation.

Expected Behavior

full text search over the repository for ".sphere.io" and ".commercetools.co" should yield zero results

Context

https://docs.commercetools.com/release-notes#releases-2020-02-03-aws-and-new-hostnames

We have a multi channel setup and I'm trying to sync between production and staging system. The issue I always get (no matter if full or delta) is, that there is a duplicate value for the sku. Am I wrong or isn't it normal, that we have multiple inventory channels if there are mutliple channels? So why do I get this kind of error?

{"timestamp":"2021-06-23T10:37:16.927Z","severity":"ERROR","loggerName":"com.commercetools.project.sync.inventoryentry.InventoryEntrySyncer","message":"Error when trying to sync inventory entry. Existing key: <<not present>>. Update actions: []","stackHash":"e8fa2b3c","stackTrace":"i.s.s.c.ErrorResponseException: detailMessage: A duplicate value '\"5299\"' exists for field 'sku'. summary: POST https://api.europe-west1.gcp.commercetools.com/[project-id]/inventory failed with response code 400 with X-Correlation-ID[project-id]/b8ccdd76-2c53-4b93-828a-781e9bd0ad84on 2021-06-23T10:37:16.899508Z http response formatted body: { \"statusCode\" : 400, \"message\" : \"A duplicate value '\\\"5299\\\"' exists for field 'sku'.\", \"errors\" : [ { \"code\" : \"DuplicateField\", \"message\" : \"A duplicate value '\\\"5299\\\"' exists for field 'sku'.\", \"duplicateValue\" : \"5299\", \"field\" : \"sku\" } ] } http response: io.sphere.sdk.http.HttpResponseImpl@5fe04fb3[statusCode=400,headers={date=[Wed, 23 Jun 2021 10:37:16 GMT], content-length=[223], server=[istio-envoy], x-envoy-upstream-service-time=[8], Alt-Svc=[clear], access-control-allow-headers=[Accept, Authorization, Content-Type, Origin, User-Agent, X-Correlation-ID], x-correlation-id=[[project-id]/b8ccdd76-2c53-4b93-828a-781e9bd0ad84], x-http-status-caused-by-extension=[false], access-control-allow-methods=[GET, POST, DELETE, OPTIONS], Via=[1.1 google], access-control-allow-origin=[*], access-control-max-age=[299], content-type=[application/json; charset=utf-8], server-timing=[projects;dur=6]},associatedRequest=<null>,textInterpretedBody={\"statusCode\":400,\"message\":\"A duplicate value '\\\"5299\\\"' exists for field 'sku'.\",\"errors\":[{\"code\":\"DuplicateField\",\"message\":\"A duplicate value '\\\"5299\\\"' exists for field 'sku'.\",\"duplicateValue\":\"5299\",\"field\":\"sku\"}]}] SDK: 1.63.0 project: [project-id] endpoint: POST /inventory Java: 11.0.11 cwd: /app sphere request: InventoryEntryCreateCommandImpl[body=InventoryEntryDraftDsl[custom=<null>,expectedDelivery=<null>,quantityOnStock=0,restockableInDays=<null>,sku=5299,supplyChannel=<null>],creationFunction=io.sphere.sdk.inventory.commands.InventoryEntryCreateCommandImpl$$Lambda$830/0x00000008004cd440@1606666e,endpoint=/inventory,expansionModel=InventoryEntryExpansionModelImpl[pathExpressions=[]],expansionP... "}

Current Behaviour

Currently, users can pass to the CLI

java -jar commercetools-project-sync.jar -s types products

but the CLI would always process the first option only.

Improvement

Throw an error if more than one value is passed to the -s option.

Alternative Improvement

Process all the specified values, which means in the previous case it should first run the type sync and then product sync.

There were breaking changes in this version, so it needs to be manually updated.

See the effort of dependabot:
#158

Shopping list syncer class manages the whole process execution and also takes care of data transformation from the source to the target project thus it needs to be defined and add required tests.

Add cartDiscount sync option

Add an option to sync custom objects using commercetools-sync-java

• As a user of the CLI, I should be able to choose an option where I can run all the modules of the sync process one by one.
  • productType and type sync should run first together.
  • category sync
  • product sync
  • inventorySync

Currently the Inventory Sync is run after other syncers are done. However, It is also fine to run it in parallel to the product sync since there are no dependencies between the 2 resources.

When syncing products from a source commercetools project to a target project, some products could have attributes which are referencing other resources on the source project (e.g. categories, products, productTypes, etc..).

These references look like that:

{
 "id": "123e4567-e89b-12d3-a456-426655440000",
 "typeId": "category"
}

Such a uuid is project-specific and will not point to any category on the target project. Therefore, we need to resolve what this actual reference points to in the target project before syncing.

Now the commercetools-sync-java library expects the supplied drafts to have the keys instead of ids in the references. Which means the aforementioned references should look sth like:

{
 "id": "category-key",
 "typeId": "category"
}

The commercetools-sync-java takes care of resolving the key to an actual id in the target project.

Therefore, in order to transform the first reference (with the uuid) to the second one (with the key), we should fetch all the referenced resources keys and replace them in the drafts before supplying them to the sync.

Currently, we only need to support product, category, productType references.

The project sync can fail for a variety of reasons e.g. API Client with invalid scope etc. I would like to run the synchronisation as a scheduled job in a docker container and be notified if it fails.

The process currently returns a zero status code regardless of whether it ran successfully or not which hides whether or not the synchronisation was actually successful.

Hi,

The sync seems to need manage_project scope to run. Surely it should be able to run even using api credentials with lower privilege? For instance when syncing Products, it should be enough with view:products on the source project and manage:products on the destination project. So that we don't create api clients with unnecessarily permissive privileges.

To reproduce:

docker run \
  -e SOURCE_PROJECT_KEY=<SOURCE_PROJECT_KEY> \
  -e SOURCE_CLIENT_ID=<SOURCE_CLIENT_ID> \
  -e SOURCE_CLIENT_SECRET=<SOURCE_CLIENT_SECRET> \
  -e SOURCE_AUTH_URL=<SOURCE_AUTH_URL> \
  -e SOURCE_API_URL=<SOURCE_API_URL> \
  -e TARGET_PROJECT_KEY=<TARGET_PROJECT_KEY> \
  -e TARGET_CLIENT_ID=<TARGET_CLIENT_ID> \
  -e TARGET_CLIENT_SECRET=<TARGET_CLIENT_SECRET> \
  -e TARGET_AUTH_URL=<TARGET_AUTH_URL> \
  -e TARGET_API_URL=<TARGET_API_URL> \
  commercetools/commercetools-project-sync:4.0.3 -s products

The error:

{
  "timestamp": "2021-06-30T13:07:40.184Z",
  "severity": "INFO",
  "loggerName": "com.commercetools.project.sync.product.ProductSyncer",
  "message": "Starting ProductSync from CTP project with key 'source-project' to project with key 'dest-project'"
}

{
  "timestamp": "2021-06-30T13:07:40.870Z",
  "severity": "ERROR",
  "loggerName": "sphere.oauth",
  "message": "Can't fetch tokens.",
  "stackHash": "5c696068",
  "stackTrace": "i.s.s.c.UnauthorizedException: detailMessage: io.sphere.sdk.http.HttpResponseImpl@2a75603e[statusCode=400,headers={date=[Wed, 30 Jun 2021 13:07:40 GMT], content-length=[187], server=[istio-envoy], x-envoy-upstream-service-time=[17], Alt-Svc=[clear], access-control-allow-headers=[Accept, Authorization, Content-Type, Origin, User-Agent, X-Correlation-ID], x-correlation-id=[source-project/d39b6593-751e-42a4-8c17-727c12233c0d], access-control-allow-methods=[GET, POST, DELETE, OPTIONS], pragma=[no-cache], Via=[1.1 google], access-control-allow-origin=[*], access-control-max-age=[299], content-type=[application/json; charset=utf-8], server-timing=[auth;dur=16], cache-control=[no-store]},associatedRequest=HttpRequestImpl[body=FormUrlEncodedHttpRequestBody[data=[NameValuePairImpl[name=grant_type,value=client_credentials], NameValuePairImpl[name=scope,value=manage_project:source-project]]],headers={Authorization=[**removed from output**], User-Agent=[commercetools-jvm-sdk/1.63.0 (AHC/2.1) Java/11.0.11+9 (Linux; amd64) commercetools-sync-java/5.1.2], X-Correlation-ID=[source-project/d39b6593-751e-42a4-8c17-727c12233c0d], Content-Type=[application/x-www-form-urlencoded]},httpMethod=POST,url=https://auth.europe-west1.gcp.commercetools.com/oauth/token],textInterpretedBody={\"statusCode\":400,\"message\":\"Permissions exceeded\",\"errors\":[{\"code\":\"invalid_scope\",\"message\":\"Permissions exceeded\"}],\"error\":\"invalid_scope\",\"error_description\":\"Permissions exceeded\"}]\nhttp response: <unknown>\nSDK: 1.63.0\nproject: <unknown>\nJava: 11.0.11\ncwd: /app\nsphere request: <unknown>\nadditional notes: []\nJavadoc: https://commercetools.github.io/commercetools-jvm-sdk/apidocs/io/sphere/sdk/client/UnauthorizedException.html\n\n\tat i.s.s.c.TokensSupplierImpl.parseResponse(TokensSupplierImpl.java:124)\n\t... 16 common frames omitted\nWrapped by: i.s.s.c.InvalidScopeException: detailMessage: Invalid scope error\nsummary: POST https://auth.europe-west1.gcp.commercetools.com/oauth/token failed  with response code 400 with X-Correlation-I...\n"
}

We should structure the log entries since right now all our single log events are seen as multi-event logs.

• Current behavior:
  the project sync using defaults for timeouts:

https://github.com/commercetools/commercetools-project-sync/blob/master/src/main/java/com/commercetools/project/sync/util/ClientConfigurationUtils.java#L54

• Expected behavior:

It can happen some timeout configuration on the project should be changed, also integration tests should be added for handling timeout related errors.

Check related java-sync issue for more detailed context: commercetools/commercetools-sync-java#208

Hi :)

We are trying to sync the products between projects and the sync fails because our variants include prices for some customer groups, and the sync process is trying to reference the customer group in the target project using the customer group id, and thus, they product sync is failing.

Error:

[ForkJoinPool.commonPool-worker-0] ERROR com.commercetools.project.sync.product.ProductSyncer - Error when trying to sync product. Existing key: <<not present>>. Update actions: []
com.commercetools.sync.commons.exceptions.SyncException: Failed to process the ProductDraft with key:'335'. Reason: com.commercetools.sync.commons.exceptions.ReferenceResolutionException: Failed to resolve 'customer-group' reference on PriceDraft with country:'null' and value: 'USD 636'. Reason: Customer Group with key '1bf15ba9-1b1f-4e3f-a61d-57bc241cc3ba' does not exist.
	at com.commercetools.sync.products.ProductSync.lambda$syncDraft$17(ProductSync.java:375)
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	at io.sphere.sdk.utils.CompletableFutureUtils.lambda$transferResult$0(CompletableFutureUtils.java:84)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.util.concurrent.CompletionException: com.commercetools.sync.commons.exceptions.ReferenceResolutionException: Failed to resolve 'customer-group' reference on PriceDraft with country:'null' and value: 'USD 636'. Reason: Customer Group with key '1bf15ba9-1b1f-4e3f-a61d-57bc241cc3ba' does not exist.
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338)
	at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:911)
	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:953)
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
	... 10 more

Price in the source project

{
  "value": {
    "type": "centPrecision",
    "currencyCode": "USD",
    "centAmount": 63600,
    "fractionDigits": 2
  },
  "id": "f60c2647-9df8-4017-896f-ae7b1495acf1",
  "customerGroup": {
    "typeId": "customer-group",
    "id": "1bf15ba9-1b1f-4e3f-a61d-57bc241cc3ba"
  },
  "custom": {
    "type": {
      "typeId": "type",
      "id": "04495716-931a-4944-a3f0-74694812f49c"
    },
    "fields": {
      "discountAmount": "-20%",
      "discountLabel": "VIP Discount"
    }
  }
}

Customer group in source project

{
  "id": "1bf15ba9-1b1f-4e3f-a61d-57bc241cc3ba",
  "version": 3,
  "createdAt": "2019-04-11T08:45:50.967Z",
  "lastModifiedAt": "2019-05-21T14:10:35.574Z",
  "lastModifiedBy": {
    "isPlatformClient": true,
    "user": {
      "typeId": "user",
      "id": "0533a1ee-fd63-49e8-9a7e-223c9e42032a"
    }
  },
  "createdBy": {
    "isPlatformClient": true,
    "user": {
      "typeId": "user",
      "id": "35023530-6f47-4be0-a092-525a32160c5f"
    }
  },
  "name": "VIP",
  "key": "Vip"
}

Maybe, the customer group should be fetched by id in the source project, and referenced by key in the target project?

Add state sync option

The Sync of Tax Categories fails if source and target project have states configured in their Tax Categories.
Once you delete the Tax Rates with states in the target project, the sync is working again and the tax rates with states are created again. But this is not a good workaround to get the syncing work :)

Sample Tax Category to reproduce the error:
{ "id": "3aa1c850-0d8d-4d3b-8c0d-1dde8258f0a0", "version": 802, "createdAt": "2017-08-31T13:02:06.375Z", "lastModifiedAt": "2021-06-18T07:56:57.487Z", "lastModifiedBy": { "clientId": "70HeAtGffhvSNULfTSJ0J9yT", "isPlatformClient": false }, "name": "Riedel Retail Product Tax", "description": "Retail Taxrate for each Country/State", "rates": [ { "name": "IN", "amount": 0, "includedInPrice": false, "country": "IN", "id": "4nsc4BCu", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "id": "yfcYkCLk", "subRates": [] }, { "name": "AT", "amount": 0.2, "includedInPrice": true, "country": "AT", "id": "BnejW7cN", "subRates": [] }, { "name": "FR", "amount": 0.2, "includedInPrice": true, "country": "FR", "id": "roGGnJlN", "subRates": [] }, { "name": "DE", "amount": 0.19, "includedInPrice": true, "country": "DE", "id": "IMrNsCUa", "subRates": [] }, { "name": "IT", "amount": 0.22, "includedInPrice": true, "country": "IT", "id": "f1BZd_OU", "subRates": [] }, { "name": "LI", "amount": 0.077, "includedInPrice": true, "country": "LI", "id": "wWnrgJWz", "subRates": [] }, { "name": "NZ", "amount": 0.15, "includedInPrice": true, "country": "NZ", "id": "r91BBsfn", "subRates": [] }, { "name": "CH", "amount": 0.077, "includedInPrice": true, "country": "CH", "id": "mYRGOXen", "subRates": [] }, { "name": "UK", "amount": 0.2, "includedInPrice": true, "country": "GB", "id": "1VZWM5E8", "subRates": [] }, { "name": "BE", "amount": 0.21, "includedInPrice": true, "country": "BE", "id": "As16yu1J", "subRates": [] }, { "name": "BU", "amount": 0.2, "includedInPrice": true, "country": "BG", "id": "Bkw5TZ7K", "subRates": [] }, { "name": "HR", "amount": 0.25, "includedInPrice": true, "country": "HR", "id": "IPEU7fY-", "subRates": [] }, { "name": "CY", "amount": 0.19, "includedInPrice": true, "country": "CY", "id": "l3H_7_cS", "subRates": [] }, { "name": "CZ", "amount": 0.21, "includedInPrice": true, "country": "CZ", "id": "u8qClFlB", "subRates": [] }, { "name": "DK", "amount": 0.25, "includedInPrice": true, "country": "DK", "id": "VM34yGmw", "subRates": [] }, { "name": "ET", "amount": 0.2, "includedInPrice": true, "country": "EE", "id": "ieZfMcRI", "subRates": [] }, { "name": "FI", "amount": 0.24, "includedInPrice": true, "country": "FI", "id": "98xrn4RY", "subRates": [] }, { "name": "GR", "amount": 0.24, "includedInPrice": true, "country": "GR", "id": "z8aFuDUB", "subRates": [] }, { "name": "HU", "amount": 0.27, "includedInPrice": true, "country": "HU", "id": "DMlQvnsZ", "subRates": [] }, { "name": "IE", "amount": 0.23, "includedInPrice": true, "country": "IE", "id": "KGQhJm1p", "subRates": [] }, { "name": "LT", "amount": 0.21, "includedInPrice": true, "country": "LV", "id": "pLZTISYC", "subRates": [] }, { "name": "LV", "amount": 0.21, "includedInPrice": true, "country": "LT", "id": "aFKwLvjN", "subRates": [] }, { "name": "LU", "amount": 0.17, "includedInPrice": true, "country": "LU", "id": "FqrT5OWD", "subRates": [] }, { "name": "MA", "amount": 0.18, "includedInPrice": true, "country": "MT", "id": "S7Rd1h1B", "subRates": [] }, { "name": "NL", "amount": 0.19, "includedInPrice": true, "country": "NL", "id": "KzbElsJQ", "subRates": [] }, { "name": "NO", "amount": 0, "includedInPrice": false, "country": "NO", "id": "yez4urfj", "subRates": [] }, { "name": "PO", "amount": 0.23, "includedInPrice": true, "country": "PL", "id": "Zi4p52F8", "subRates": [] }, { "name": "PT", "amount": 0.23, "includedInPrice": true, "country": "PT", "id": "REl-B2Bi", "subRates": [] }, { "name": "RO", "amount": 0.2, "includedInPrice": true, "country": "RO", "id": "3VMI7BFF", "subRates": [] }, { "name": "SV", "amount": 0.2, "includedInPrice": true, "country": "SK", "id": "a8WQPbFa", "subRates": [] }, { "name": "SL", "amount": 0.22, "includedInPrice": true, "country": "SI", "id": "j3tZoy6-", "subRates": [] }, { "name": "ES", "amount": 0.21, "includedInPrice": true, "country": "ES", "id": "FB8qDQU-", "subRates": [] }, { "name": "VA", "amount": 0.22, "includedInPrice": true, "country": "VA", "id": "0A5t4a_Z", "subRates": [] }, { "name": "CA", "amount": 0.05, "includedInPrice": false, "country": "CA", "state": "AB", "id": "sieN0vc2", "subRates": [] }, { "name": "SE", "amount": 0.25, "includedInPrice": true, "country": "SE", "id": "6zXhsBU9", "subRates": [] }, { "name": "MC", "amount": 0.2, "includedInPrice": true, "country": "MC", "id": "nfmOQZUT", "subRates": [] }, { "name": "IS", "amount": 0, "includedInPrice": false, "country": "IS", "id": "x4CivvTO", "subRates": [] }, { "name": "CA", "amount": 0.12, "includedInPrice": false, "country": "CA", "state": "BC", "id": "lumlF5eB", "subRates": [] }, { "name": "CA", "amount": 0.12, "includedInPrice": false, "country": "CA", "state": "MB", "id": "wlKQ9UmF", "subRates": [] }, { "name": "CA", "amount": 0.15, "includedInPrice": false, "country": "CA", "state": "NB", "id": "RSjcfPpD", "subRates": [] }, { "name": "CA", "amount": 0.15, "includedInPrice": false, "country": "CA", "state": "NL", "id": "nHv5l25H", "subRates": [] }, { "name": "CA", "amount": 0.15, "includedInPrice": false, "country": "CA", "state": "NS", "id": "hvUnuZDm", "subRates": [] }, { "name": "CA", "amount": 0.05, "includedInPrice": false, "country": "CA", "state": "NT", "id": "-lz_opHC", "subRates": [] }, { "name": "CA", "amount": 0.05, "includedInPrice": false, "country": "CA", "state": "NU", "id": "BDvCDkz6", "subRates": [] }, { "name": "CA", "amount": 0.13, "includedInPrice": false, "country": "CA", "state": "ON", "id": "XD3WEpLB", "subRates": [] }, { "name": "CA", "amount": 0.15, "includedInPrice": false, "country": "CA", "state": "PE", "id": "_p2OSrAr", "subRates": [] }, { "name": "CA", "amount": 0.14975, "includedInPrice": false, "country": "CA", "state": "QC", "id": "NU4V3gyd", "subRates": [] }, { "name": "CA", "amount": 0.11, "includedInPrice": false, "country": "CA", "state": "SK", "id": "3P5jthul", "subRates": [] }, { "name": "CA", "amount": 0.05, "includedInPrice": false, "country": "CA", "state": "YT", "id": "5fkLvA6-", "subRates": [] }, { "name": "AE", "amount": 0, "includedInPrice": true, "country": "AE", "id": "LS_QwRGU", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "NSW", "id": "vi8p19Cv", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "QLD", "id": "WYYowrVn", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "SA", "id": "6QGXBZm2", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "TAS", "id": "-iKhtfrV", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "VIC", "id": "0wPwvJRb", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "WA", "id": "7xyTsLmN", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "ACT", "id": "kmVcnx0N", "subRates": [] }, { "name": "AU", "amount": 0.1, "includedInPrice": true, "country": "AU", "state": "NT", "id": "zP1-yP4_", "subRates": [] }, { "name": "GB", "amount": 0, "includedInPrice": false, "country": "GB", "state": "JE", "id": "fELfR8OJ", "subRates": [] }, { "name": "GB", "amount": 0, "includedInPrice": false, "country": "GB", "state": "GG", "id": "q8k-_m2U", "subRates": [] } ], "key": "standardRetailProduct" }

Used Docker Command:

docker run -e SOURCE_PROJECT_KEY=riedel-main-stage -e SOURCE_CLIENT_ID=xxx-0RXp -e SOURCE_CLIENT_SECRET=xxx -e TARGET_PROJECT_KEY=riedel-dev -e TARGET_CLIENT_ID=xxx -e TARGET_CLIENT_SECRET=xxx commercetools/commercetools-project-sync:4.0.3 -s taxCategories {"timestamp":"2021-06-18T07:50:41.636Z","severity":"INFO","loggerName":"com.commercetools.project.sync.taxcategory.TaxCategorySyncer","message":"Starting TaxCategorySync from CTP project with key 'riedel-main-stage' to project with ke y 'riedel-dev'"} {"timestamp":"2021-06-18T07:50:43.257Z","severity":"ERROR","loggerName":"com.commercetools.project.sync.taxcategory.TaxCategorySyncer","message":"Error when trying to sync tax category. Existing key: standardRetailProduct. Update acti ons: ReplaceTaxRate[taxRate=TaxRateDraftImpl[amount=0.05,country=CA,includedInPrice=false,name=CA,state=AB,subRates=[]],taxRateId=ScOM3b5y,action=replaceTaxRate],ReplaceTaxRate[taxRate=TaxRateDraftImpl[amount=0.1,country=AU,includedIn Price=true,name=AU,state=<null>,subRates=[]],taxRateId=rn63Bxf4,action=replaceTaxRate],AddTaxRate[taxRate=TaxRateDraftImpl[amount=0.0,country=GB,includedInPrice=false,name=GB,state=JE,subRates=[]],action=addTaxRate],AddTaxRate[taxRate =TaxRateDraftImpl[amount=0.0,country=GB,includedInPrice=false,name=GB,state=GG,subRates=[]],action=addTaxRate]","stackHash":"c408fba9","stackTrace":"i.s.s.c.ErrorResponseException: detailMessage: A duplicate value '{\"country\":\"CA\" ,\"state\":\"AB\"}' exists for field 'country'.\nsummary: POST https://api.europe-west1.gcp.commercetools.com/riedel-dev/tax-categories/3aa1c850-0d8d-4d3b-8c0d-1dde8258f0a0 failed with response code 400 with X-Correlation-ID riedel-
dev/909e544a-4394-4ccb-92c5-85a8266d2eb2 on 2021-06-18T07:50:43.138337Z\nhttp response formatted body: {\n \"statusCode\" : 400,\n \"message\" : \"A duplicate value '{\\\"country\\\":\\\"CA\\\",\\\"state\\\":\\\"AB\\\"}' exists for field 'country'.\",\n \"errors\" : [ {\n \"code\" : \"DuplicateField\",\n \"message\" : \"A duplicate value '{\\\"country\\\":\\\"CA\\\",\\\"state\\\":\\\"AB\\\"}' exists for field 'country'.\",\n \"action\" : {\n \"ac tion\" : \"replaceTaxRate\",\n \"taxRateId\" : \"ScOM3b5y\",\n \"taxRate\" : {\n \"name\" : \"CA\",\n \"amount\" : 0.05,\n \"includedInPrice\" : false,\n \"country\" : \"CA\",\n \"state\" : \"AB\",\n \"subRates\" : [ ]\n }\n },\n \"actionIndex\" : 1,\n \"duplicateValue\" : {\n \"country\" : \"CA\",\n \"state\" : \"AB\"\n },\n \"field\" : \"country\"\n } ]\n}\nhttp response: io.spher e.sdk.http.HttpResponseImpl@74024d7a[statusCode=400,headers={date=[Fri, 18 Jun 2021 07:50:38 GMT], content-length=[496], server=[istio-envoy], x-envoy-upstream-service-time=[9], Alt-Svc=[clear], access-control-allow-headers=[Accept, A uthorization, Content-Type, Origin, User-Agent, X-Correlation-ID], x-correlation-id=[riedel-dev/909e544a-4394-4ccb-92c5-85a8266d2eb2], x-http-status-caused-by-extension=[false], access-control-allow-methods=[GET, POST, DELETE, OPTIONS ], Via=[1.1 google], access-control-allow-origin=[*], access-control-max-age=[299], content-type=[application/json; charset=utf-8], server-timing=[projects;dur=8]},associatedRequest=<null>,textInterpretedBody={\"statusCode\":400,\"mes sage\":\"A duplicate value '{\\\"country\\\":\\\"CA\\\",\\\"state\\\":\\\"AB\\\"}' exists for field 'country'.\",\"errors\":[{\"code\":\"DuplicateField\",\"message\":\"A duplicate value '{\\\"country\\\":\\\"CA\\\",\\\"state\\\":\\\"A B\\\"}' exists for field 'coun...\n"} {"timestamp":"2021-06-18T07:50:43.307Z","severity":"INFO","loggerName":"com.commercetools.project.sync.taxcategory.TaxCategorySyncer","message":"Summary: 1 tax categories were processed in total (0 created, 0 updated and 1 failed to s ync).","statistics":{"updated":0,"created":0,"failed":1,"processed":1,"latestBatchProcessingTimeInDays":0,"latestBatchProcessingTimeInHours":0,"latestBatchProcessingTimeInMinutes":0,"latestBatchProcessingTimeInSeconds":0,"latestBatchP rocessingTimeInMillis":276,"latestBatchHumanReadableProcessingTime":"0d, 0h, 0m, 0s, 276ms","reportMessage":"Summary: 1 tax categories were processed in total (0 created, 0 updated and 1 failed to sync)."}}

There's a problem when syncing the updates between projects.

1. Create a new inventory entry:

    {
      "id": "30246812-78f4-4db4-a9e3-51834e6b7c32",
      "version": 155,
      "createdAt": "2017-02-01T15:40:55.087Z",
      "lastModifiedAt": "2017-12-27T18:31:01.719Z",
      "sku": "241070 1029 S_SCO",
      "quantityOnStock": 2,
      "availableQuantity": 2,
      "reservations": []
    }

1. Run project-sync with paramater "-s inventoryEntries" to sync the inventory to the other project
2. Update the inventory entry you created in step 1:

{
  "version": 1,
  "actions": [
    {
      "action": "changeQuantity",
      "quantity": 999
    }
  ]
}

1. Rerun project-sync with paramater -s inventoryEntries to sync the inventory to the other project
2. Check the result - inventory was not synced.

Situation

It is not possible to sync key-value-document (custom object) references of the product sync in the project-sync because the references are not transformed to the correct resolvable format.

Complication

With the release v2.2.0, sync-java supports the synchronization of the key-value-document attributes for the product sync, https://github.com/commercetools/commercetools-sync-java/blob/master/docs/RELEASE_NOTES.md#220---sep-25-2020
but the reference should be prepared to able to resolve, normally this part is not known by the sync-java and need to be prepared by the library user.

As the project sync is a client for the commercetools-sync-java library we need to make sure it's transformed into the right format.
Here the documentation from product sync (check the step 3): https://github.com/commercetools/commercetools-sync-java/blob/master/docs/usage/PRODUCT_SYNC.md#prerequisites

Resolution

The key-value-document attribute type is actually resolved to a commercetools custom object. We need both custom object container & custom object key to uniquely identify a custom object in the commercetools platform. Therefore, we expect for a given ProductDraft object, the value for the variant attribute of the key-value-document field "id" will be a text string. This string is composed in this format.

custom object container | custom object key

Setup Gradle Build

Currently the job will sync all resource in the source project.

It should be changed to sync only those which were changed since the last time the job was run. This can be done by persisting the time stamp of the last run.

The workflow of the algorithm can be found here: https://docs.google.com/drawings/d/1qWcYg2Sb42TWlNHVFMj-SJ5xSINGVIVJyArWxD10dOg/edit

The auth URL used here is https://auth.sphere.io. Can we have it to use the United states URL(https://auth.commercetools.co/) ?