shipitjs / shipit-deploy Goto Github PK

Hi!
I've got heavy git reposoitories, and it takes a lot of time to copy to the server.
It might be possible to directly clone git from the server with a command like:
shipit.remote.clone ('http://github.com/account/repo);

This would accelerate the deployment.

Thanks for your work ;)

When deploying, my shipit often freeze after deploy, and broke my "updated" event handler as a result. Sadly, that's where I'm doing some crucial work (symlink to front-end).

What can I do? Any idea? I have no error message.
Thanks!

Deploy doesn't work in Windows. Problem with paths to create an archive (extra slash at the beginning of the path. Problem in path2 module). It doesn't work in PowerShell (it doesn't support && command separator, only ;). Problem with SCP command.

Currently when I deploy, I am getting a folder within the current release which also has a release folder within it. I think it is happening during the deploy:update task.

After a few more deploys, I see that there are several folders of prior releases nested within the current folder.

Ok so this is not actually an issue for me anymore but I wanted to post it here in case someone else runs into it. I was getting the following error:

Checking out commit-ish "master"
Running "git checkout master" on local.
@ error: pathspec 'master' did not match any file(s) known to git.

and what was really causing it was the task before that ran this command:

Running "git fetch --depth=1 shipit -p --tags" on local.

the command did not create the actual .git/refs/heads/master as expected but put it in an unborn type of state.
To resolve this all you needed to do was run another git fetch in that script. How I know this is, if you run git remote show shipit I got the following:

• remote shipit
  Fetch URL: {REPO_URL}
  Push URL: {REPO_URL}
  HEAD branch: master
  Remote branch:
  master new (next fetch will store in remotes/shipit)

and running a git fetch again in the working directory added the master ref correctly.

ok so that helped me there in that state. But what REALLY solved my problem is that when realized my git wasn't doing something you guys were assuming it did, that my git version must be outdated. And it was. I was running 1.8.4. Since updating to 2.6.4, my shipit scripts worked immediately as intended.

That was a real pain to figure out for me. Hopefully that will help save someone else the time.

The branch config option says that it accepts tags as well as branches or commit hashes. However, the git fetch command that runs does not pull down all tags which can result in a deploy failing. To esnure that git fetch pulls down all tags, the command should be modified to git fetch -p --tags.

I have shitit set up using grunt-shipit to deploy to staging. However I cannot figure out how to change the branch for a different shipit environment. I have a different branch to merge staging and production, however I only see docs for defining a single branch for all deployment environments. Is it possible to change the branch for different environments?

It would be really nice to do something like this:

module.exports = function (grunt) {
    grunt.initConfig({
        shipit: {
            options: {
                workspace: '/tmp/deploy',
                deployTo: '/project/deploy',
                repositoryUrl: 'https://github.com/******',
                ignores: ['.git', 'docs', 'docs_build', 'tests'],
                keepReleases: 3,
                shallowClone: true
            },
            staging: {
                servers: ['node@***.***.***.***'],
                branch: "staging"
            },
            production: {
                servers: ['node@***.***.***.***']
                branch: "production"
            }
        }
    });

    grunt.loadNpmTasks('grunt-shipit');
    grunt.loadNpmTasks('shipit-deploy');

};

As discussed in shipitjs/shipit#66, we must use REVISION file to check if servers are synced to be able to rollback.

I'm trying to deploy an app and while Shipit has no trouble whatsoever with logging into to the server (and creating new folders, etc.), Rsync fails on authentication. Here's my config:

shipit.initConfig({
  default: { 
    workspace: '/temp/app',
    deployTo: '/home/deploy/www/',
    repositoryUrl: '[email protected]:antjanus/my-private-repo.git'
    keepReleases: 2,
    key: 'provisioning/files/ssh/id_rsa',
    shallowClone: true
  },
  beta: {
    servers: 'beta.my-site.com'
  }
});

The SSH key is included in the Repo (and that's on purpose). Using ssh [email protected] -i provisioning/files/ssh/id_rsa logs me in no problem; however, whenever I run shipit beta deploy, I get this error:

Copy project to remote servers.
Remote copy "/temp/app/" to "[email protected]:/home/deploy/www/releases/20150211200256"
@beta.my-site.com-err Permission denied, please try again.
@beta.my-site.com-err Received disconnect from 184.173.18.122: 2: Too many authentication failures for deploy
@beta.my-site.com-err rsync: connection unexpectedly closed (0 bytes received so far) [sender]
@beta.my-site.com-err rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.1]

Which I'm not understanding at all since the key is straightforward and declared in the app and I can already use it to log in from the command line.

I'm on a windows machine btw, so I wonder if the workspace variable is what is messing it up. I could not get a clear understanding as to where that directory is supposed to be, locally or remote. But it doesn't seem to matter.

Thoughts?

Hey Guys,

what`s the best method to update submodules during/vefore deploying via shipit?

I tried several methods, none of them seems to work?
Whats your suggestion?

Greets Marcus

For heavy repository, would be cool to know the progress of the transfer, like

[46%] 92Mo / 200Mo

Live updated ;)

So we could notify specifically when a rollback has finished

My setup:

• a git repo for shipit
• and another repo, with the project to deploy (options.repositoryUrl)

Due to the fact that setCurrentRevision() is not executed in the configured workspace, the REVISION file is wrongly filled with the git revision of the shipit project.

Hello,

First of of all this tools looks awesome, congratulations 👍

I would like to perform my tasks with an user different than the one I login into the servers.
Can you shed some light on how to do this ?

Regards,
Alfredo Palhares

Hi neoziro! First at all, excellent tool 👍 you are doing a good job!

I have a question about the deploy process.

Have we a way to make deploys without remoteCopy all the files? Like the tool rocketeer using "rocketeer update"

http://rocketeer.autopergamene.eu/#/docs/docs/IV-Usage/Deploying

Thanks!

We use this tool since some days for our deployments. Today I had to do my first rollback (no problem) and then my second: And I was confused for some minutes..

Reason: Rollback is currently to the previous deployment in time, and not the previous "stable" one.
Example:

• Deployment A is stable and deployed.
• I deploy B - doesn't work - I do rollback -> rollback to A => deployment fixed
• Devs try to fix..
• I deploy C - still doesn't work - I do rollback -> rollback to B => deployment still broken

After reading the code, I understand why this happens and I don't think that there is a easy way to mark a deployment as stable.
But, what do you think about an option to delete a deployment from the remote servers, when there is a rollback? (Should be rather easy, as the rollback code already knows from which deployment we rollback)

In my example, this would delete B after rollback to A, so that a rollback from C would also roll back to A..

Im having an issue when using shititjs to deploy to a vagrant VM.

Im using the same identity file that is set in my ~/.ssh/config which i use to connect to my VM.

key: '~/.vagrant/machines/default/virtualbox/private_key'

Error: Command failed: Permission denied (publickey,password).

    at ChildProcess.exithandler (child_process.js:648:15)
    at ChildProcess.emit (events.js:98:17)
    at maybeClose (child_process.js:756:16)
    at Socket.<anonymous> (child_process.js:969:11)
    at Socket.emit (events.js:117:20)
    at Pipe.close (net.js:465:12)

I'm interested in shipit-deploy, but am a mercurial user. Looking at the source, it seems it's git only. If so, it might be worth clarifying this in the readme. Since (I think) shipit supports mercurial, people may assume that this plugin does too.

Doh, just saw that git is mentioned in the description.

$ shipit staging rollback

node.js:815
    var cwd = process.cwd();
                  ^
Error: ENOENT, no such file or directory
    at Function.startup.resolveArgv0 (node.js:815:23)
    at startup (node.js:58:13)
    at node.js:906:3

Error message is kinda "blend".

So a user can avoid running the rollback tasks without releases available to rollback

Unhandled rejection Error: Command failed: /bin/sh -c ssh [email protected] "readlink /root/nodeapps/ship/current"

It's very common to deploy specific branch to environments (e.g., feature and bug branches to development and staging servers).

Example:

shipit staging deploy bug/someFix

or

shipit staging deploy --branch bug/someFix
shipit staging deploy -b bug/someFix

If the server has a directory current at shipit.releaseDirname, deploy/publish will fail silently. Instead of correctly replacing a current symlink with the new one, it'll symlink the release inside of the old current directory.

Ideally, IMO, updateSymbolicLink would be modified to make sure the directory doesn't exist and fail if it does.

Something along the lines of:

function updateSymbolicLink() {
  shipit.log('Publishing release "%s"', shipit.releasePath);

  var relativeReleasePath = path.join('releases', shipit.releaseDirname);

  return shipit.remote(
    'cd ' + shipit.config.deployTo + ' && ' +
    'if [[ -d target && ! (-L target) ]]; then ' +
    'echo nack; ' +
    'else ' +
    'ln -nfs ' + relativeReleasePath + ' current && echo ack;' +
    'fi'
  )
  .then(function (results) {
    var failedResults = results.filter(function(r) {
      return r.stdout !== 'ack'; // Or something
    });
    if(failedResults.length) {
      // Throw that the deployment failed telling what servers and why; IDK
    }

    shipit.log(chalk.green('Release published.'));
  });
}

Currently, if you have shallowClone: true, on every deploy the workspace gets wiped: https://github.com/shipitjs/shipit-deploy/blob/master/tasks/deploy/fetch.js#L43-L47

This can result in really slow deploys, especially if you have alot of things to build on deploy.

It would be nice to have an option to specify which directory within the workspace should be copied. Most of my repositories have a build folder which is all that I want to be uploaded to the server.

For some deployments, it is helpful to be able to customize 'shipit.releasesPath' and 'shipit.currentPath' variables...

Not having found solutions I created options in shipfile.js, like this :

    shipit.initConfig({
        default: {
            ...
            currentPath: 'mycurrent',
            releasesPath: 'mycurrent_releases',
          ...
   )};

And I boot these options in task/deploys/publish.js

    function updateSymbolicLink() {
      shipit.log('Publishing release "%s"', shipit.releasePath);

      var relativeReleasePath = path.join('' + shipit.config.releasesPath + '', shipit.releaseDirname);

      return shipit.remote('cd ' + shipit.config.deployTo + ' && ln -nfs ' + relativeReleasePath + ' ' + shipit.config.currentPath + '')
      .then(function () {
        shipit.log(chalk.green('Release published.'));
      });
    }

and in lib/init.js

module.exports = function(shipit) {
  shipit.currentPath = path.join(shipit.config.deployTo, '' + shipit.config.currentPath + '');
  shipit.releasesPath = path.join(shipit.config.deployTo, '' + shipit.config.releasePath + '');
  shipit.config.gitLogFormat = shipit.config.gitLogFormat || '%h: %s - %an';
  _.assign(shipit.constructor.prototype, require('./shipit'));

  return shipit;
};

It works well for deploy and rollback function.

Is there an easier way or do I complete my fork and propose a PR ?

It would be very useful

tasks/deploy/update.js uses "cp" with the parameter "-a" which is only available on NTFS file systems.
(Source: https://www.mkssoftware.com/docs/man1/cp.1.asp)
So the copy task fails from OSX with HFS+.
Should/could be cp -r instead of cp -a.

When deploying from a specified git branch, I get the following error:

Release path created.
Setting current revision and creating revision file.
Running "git rev-parse BRANCH" on local.
@ fatal: ambiguous argument 'BRANCH': unknown revision or path not in the working tree.

Obviously BRANCH is the place holder for my branch name.
I remove the branch option from the initConfig and it works fine.

Otherwise it's not possible to use 'em from tasks invoked out of deploy cycle.
Example below:

shipit.task('npm:build', function () {
  return shipit.remote('cd ' + path.join(shipit.releasesPath, shipit.releaseDirname) + ' && npm run build');
});

No way to run shipit deploy npm:build since shipit.releasesPath and friends are unset at that point.

I found an issue.

Some files was removed/changed in source repo. But when I updated it, they were recoverd and used the previouse release.

This confuse me.

how to build remotely like capistrano ?

I have a simple deploy which takes 16 minutes to complete. It just copies the directory, doesn't have any tasks

Hi. With shipit-deploy 1.1.1, we want to run an deploy:install task running npm install after the updated event has been emitted, that means before the symbolic link current is updated, so

var current = grunt.config('shipit.options.deployTo') + '/current';

does not work.

As that needs to be run from the latest release directory, how can that be figured out?

As a work-around, we run the deploy:install task after the publication of the published event.

This will be very awesome if you have a slow upload connection.

Bonus : compatible with private repository

hi. as far as i can see right now there are no flags to deploy from the repository including all submodules, which would be quite useful

We use shipit and really like it. Today, we ran into a bit of an edge case, and I'd like to suggest a change OR get advice on how to override the default deploy:fetch task.

What happened was this -- somehow my workspace was altered. Grunt-shipit executed during deploy:fetch a git checkout , but this kept my working directory dirty.

I'd like to insert an extra local command git reset --hard HEAD after the checkout but before the rsync. Would you be interested in having that in this project generally or is more of an edge case for my particular project?

shipitjs/shipit#61

the task deploy/update.js is trying to copy the previous release if present. But actually it is taking two releases back for copy.

I suppose that this is due to the fact that "previous release" is the previous release of the current release but this task is actually creating a new release.

I've been trying to write a test to show that behaviors but did not managed to fin time.

What if I want to exclude only the lib folder on the root?

Do you have help for composer on a deploy. Specifically:

1. At what point in the flow should I run an composer install
2. Do I need to authorise my server keys against github for private repos or is there an interactive mode I can use?

Cheers

In some projects, the files to be deployed are created for example by gulp build, creating the directory dist with all the files. Deployment in that setup is just to get the files in the folder to the server. Currently such a setup is not supported by shipit-deploy. Basically the fetch task has to be left out.

Is it wanted to support that use case? I could imagine not specifying the parameter repositoryUrl and setting the workspace directory to dist (workspace: dist) would be a possible way to integrate it with the current shipit-deploy configuration.

Would that work or do you see problems with this approach?

Hi,

I would like to say the module is awesome, thanks.

The problem I find is that since it will create a new release folder on the server every time, it will have to copy everything from local to server even if the local and remote has only minimal changes.

Since rsync can do a pretty good job to figure out what has been changed between local and remote, I think maybe it's a good idea to create a copy of previous release on the server rather than simply mkdir on the server.

Maybe this can be an another option so that the user can decide.
What do you think?

Shipit can't checkout from master to develop because git finds some local changes. It is strange. I make git clone and get absolutely clean repository without uncommitted files. I don't know where are its uncommitted files and how I can commit or stash (when I commit or stash in repository directory, git doesn't see uncommitted files, tmp folder is clean).

Running 'deploy:init' task...
Finished 'deploy:init' after 1.16 ms

Running 'deploy:fetch' task...
Create workspace "/tmp/project/client"
Workspace created.
Initialize local repository in "/tmp/project/client"
Running "git init" on local.
@ Reinitialized existing Git repository in /tmp/project/client/.git/
Repository initialized.
List local remotes.
Running "git remote" on local.
@ shipit
Update remote "[email protected]:test/project-client.git" to local repository "/tmp/project/client"
Running "git remote set-url shipit [email protected]/project-client.git" on local.
Remote updated.
Fetching repository "[email protected]:test/project-client.git"
Running "git fetch shipit -p" on local.
Repository fetched.
Checking out commit-ish "develop"
Running "git checkout develop" on local.
@ error: Your local changes to the following files would be overwritten by checkout:
@       projects/compiled/main/vendor/angularBootstrap.js
@       projects/compiled/main/vendor/angularStrap.js
@ Please, commit your changes or stash them before you can switch branches.
@ Aborting
'deploy:fetch' errored after 6.29 s
Error: Command failed: /bin/sh -c git checkout develop
error: Your local changes to the following files would be overwritten by checkout:
        projects/compiled/main/vendor/angularBootstrap.js
        projects/compiled/main/vendor/angularStrap.js
Please, commit your changes or stash them before you can switch branches.
Aborting

    at ChildProcess.exithandler (child_process.js:203:12)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at maybeClose (internal/child_process.js:817:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
Unhandled rejection Error: Command failed: /bin/sh -c git checkout develop
error: Your local changes to the following files would be overwritten by checkout:
        projects/compiled/main/vendor/angularBootstrap.js
        projects/compiled/main/vendor/angularStrap.js
Please, commit your changes or stash them before you can switch branches.
Aborting

    at ChildProcess.exithandler (child_process.js:203:12)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at maybeClose (internal/child_process.js:817:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)

I'm trying to deploy your tutorial but I had the following error on my dos console:
Could not create directory '/cygdrive/h/.ssh'.

Did you tested the module on windows?

Thanks

Doing a fresh checkout, then

npm i
npm test

All tests are failing for deploy:update task and pending:log task (timeout)

Travis builds are not accessible for recent pull requests...

The shipitfile.js example references an undocumented key option.

Does it exist ? What does it do ?

Is it possible to pause deployment until event listeners are handled?

Specifically, this code

shipit.on('updated', function() {
  return shipit.remote("cd " + shipit.releasePath + " && " + NPM_BIN + " install");
});

will run concurrent with the publish and clean tasks. Instead, I would like it to wait until npm install is done and then publish & clean.

Shipit-deploy uses path.join often and while this is correct when working on a single system, working cross system causes issues. So deploying for windows causes the path.join to join paths with a \ instead of / even when the target system is a Linux machine.

For some reason, I didn't have this problem before; however, whenever I deploy now, a folder is made in my SSH user's home directory equivalent to my deployTo without slashes like so:

deployTo: /home/deploy/www,

which in turn creates a folder in my deploy user's home directory called homedeploywww within which are my release folders (the time tag).

I think the reason for this was the target rsync directory which ended up being: [email protected]:/home/deploy/www/releases\012345678. This is because of this line:

shipit.releasesPath = path.join(shipit.config.deployTo, 'releases');

I temporarily resolved the issue on my system with a string concatenation:

shipit.releasesPath = shipit.config.deployTo + '/releases';

Any ideas on a different solution? I realize I'm probably the only Windows user here but I'd obviously love some help. The issues popped up, btw, with the new release of shipit-cli and shipit-deploy. I'm not sure what changed in between.

On a side note, is there any way to possibly force Node to use a different URL separator?