Allow finer grained code upload about sfcc-ci HOT 17 OPEN

Proposal:

sfcc-ci code:push --path <sourcepath> --version <codeversion> --instance <instance>

with:

• -p,--path <sourcepath> (optional) being the path of the code source(s) to push, this may be a single file or a folder, in case of a folder the complete directory and all content (including directories) will be pushed recursively
• -v,--version <version> (optional) being the code version on the instance to push to, the version must exists, if no version is provided the code version is read from a dw.json file the current working directory, if no code version can be found in a dw.json file, the active code version on the instance is used
• -i,--instance (optional) being the Commerce Cloud instance to push to, if no instance is provided the instance is read from a dw.json file the current working directory, if no instance can be found in a dw.json file, the currently configured instance is used

Notes:

• There will be no check for overwriting code files on the instance (existing code is being overwritten)
• Pushing folders, whole packages (aka cartridges) or even whole versions (aka sets of packages) existing folders will not be overwritten, but merged with the code being pushed

Feedback welcome!

from sfcc-ci.

Any idea on how to approach those ?
I'm trying to deploy only artifact deltas as their weight is becoming a problem and deployments are currently taking several hours to settle down.
Thanks

from sfcc-ci.

We could imagine a CLI statement that allows us to upload only specific folders within the cartridges (even a specific folder found across all cartridges from a specific "root" folder, i.e. all static/ folders).
This way, we would be able to send only the frontend changes without the backend for example, or only the images and fonts folders without any code changes (as those resources are the more time-consuming during the deployment).
We could also imagine a way to ignore some files/folders from the deployment (like the scss folder and the client folder from SFRA as those are only used for development purpose)

Finally, you can imagine that you deploy all images/fonts during the night, and only deploy the code folders during the normal process of deployment.

from sfcc-ci.

No, basically I'm thinking on a diff at client level where changes are uploaded one by one (on a per file compiled/source/image basis even they are all transferred as single zip and later on positioned at the right folder) or removed (in case they've disappeared) from the code_version.
9X% of new deployments have minimal file differences in terms of deltas
It'd be transparent to the deployment (and maybe parametrisable if you want a full or incremental release). My aim was to find out current sfcc-ci support to achieve such behaviour.

from sfcc-ci.

@jjuanmendoza That has nothing to do with the ticket at hand :) which is about what Jordane explains.

I don't think it would help if you download the currently active code version to do your diff. So I think your build server has to rememeber the currently active code state and you diff the changes files from there.
I wouldn't put this in sfcc-ci though - rather an additional external script

from sfcc-ci.

@hnestmann
right now after diffing (yes from the latest deployed version) I'm launching curl -x COPY or DELETE on a file basis to achieve something similar
after reading this issue title i thought it's implementation would allow me to leverage sfcc-ci for incremental deployments (my final goal)

incidentally, the use of a script goes without saying, the potential support that sfcc-ci shall provide me in order to achieve incremental deployments (like "allowing finer grained code upload" but their might be others) was my whole point.
I can move my comment wherever it makes more sense but this is a recurrent problem that big implementations are hitting and that custom scripts are trying to fix by leveraging some old/modern tools (prophet watcher, dw grunt-gulp, sfcc-ci, etc).

from sfcc-ci.

Found a quick hack You can deploy single cartriges by putting those in a code.zip which has a normal folder structure like code/your_cartridge. The server will merge them

from sfcc-ci.

thanks @hnestmann !
correct me if I'm wrong cuz i walked that path some weeks ago and I might have overseen something important:
1.- such a merge behaviour comes directly from the zip command (https://linux.die.net/man/1/zip) search for "When given the name ..."
2.- for heavy code versions the whole .zip still needs to be uploaded even though only certain files from i.e. 2 cartridges have been added/changed/removed
Thanks

from sfcc-ci.

I think there is a misunderstanding

Say you have a sandbox with an existing 'code' version. Which contains cartridges 'app_storefront' and 'modules'. Now you use sfcc-ci to deploy to 'code' version with a zip that only contains 'code/plugin_serviceworker'. After the deployment, your sandbox will carry three cartridges.

• app_storefront
• modules
• plugin_serviceworker

from sfcc-ci.

great thanks, that answers my second question.
what about if that new code version contains (in addition to code/plugin_serviceworker) a 'modules' cartridge with less files that the one in the existing 'code' version ?

from sfcc-ci.

That I didn't try out, but you can if you want.

I would assume it works like windows file system. By overwriting existing files, adding new files and keep files which are not included in the zip.

from sfcc-ci.

right, that's why i was referring to std zip command's behaviour (as the default action is to add or replace zipfile entries from list)

That's one of the reasons why we need to add some script "glue" in order to use it for incremental ci (we need a preprocess to determine whether any files have been removed and if so either zip -d them or upload the whole new code version)

from sfcc-ci.

I don't connect the dots here it seems. The merging happens on unzipping on the sandbox. Not in the CI command directly.

from sfcc-ci.

You're right.
Unfortunately for a true incremental ci we need to be able to track such file removals (like the ones i was raising) as part of a new version of the existent code version. That causes to slow down the whole process as we need to compare it against the deployed version. Unless we discard deltas with file deletions I haven't been able to workaround such an expensive operation (compare against deployed code version) before deploying a new/updated code version.

from sfcc-ci.

In my opinion, for developer sandboxes this is a less of an issue and for actual test instances I would always push my clients to have a CI server which deploys everything and does do an integrity check for everything that needs to be deployed.
However what you can do write a little script that provides an index file of a given cartridge and deploys that onto the sandboxes
deployinfo.json
{"version": "1.2.3",files":[{"modules/server/server.js": "size": "12kb", "hash": "abc","lastModified": "2019-04-08T10:06:35.279Z"}]

However this is way out of scope for this CLI tool, and should rather be in the build-suite which has a build orchestration focus. We are happy to receive contributions

from sfcc-ci.

Thanks @hnestmann
Actually some deployments are taking us 4-5 hours (main part of it from transferring the heavy .zip file regardless changes/deletions/additions only affect 2-3% of the files) per pipeline hence we cannot go with the "deploy everything" path. The deployinfo.json is another solution we've been trying but has it's own downsides. Nevertheless I see your point and I agree this is out-of-scope for this CLI tool.
I'll let you know in case we end up with anything else beyond gluing/scripting sfcc-ci/build-suite

from sfcc-ci.

Thanks @hnestmann
Actually some deployments are taking us 4-5 hours (main part of it from transferring the heavy .zip file regardless changes/deletions/additions only affect 2-3% of the files) per pipeline hence we cannot go with the "deploy everything" path. The deployinfo.json is another solution we've been trying but has it's own downsides. Nevertheless I see your point and I agree this is out-of-scope for this CLI tool.
I'll let you know in case we end up with anything else beyond gluing/scripting sfcc-ci/build-suite

Thanks for the discussion. @jjuanmendoza Can we move the suggested incremental code push to a separate issue? I like the idea, but suggest we take a step by step approach here and first consider the single artefact push

from sfcc-ci.