Automate release process about core HOT 16 CLOSED

Perhaps a first step would be identify the what's done during a release. When I showed up at FakeItEasy, the owners already had a system where each new release was accompanied by an issue with a checklist in it. It made my first releases quite easy. And also subsequent discussions about how to streamline the process, which have paid benefits in recent months.

See FakeItEasy/FakeItEasy#283 for an example checklist, if you'd like.

After we've something like that to point to, we can see how to make the process easier.

from core.

Hey,

thanks for kicking this off. While we have a fairly good build process, our release process has been largely manual and undocumented. Here's my attempt at listing all the steps involved:

1. For Windsor - make sure the project references the latest release version of Core
2. Go to Settings.proj and update the version number
3. Go to Changes.txt and BreakingChanges.txt and make sure they are updated
4. Commit, tag (with the release number) and push
5. When all the builds (there's one for each target runtime/version) in our TeamCity have successfully completed, download the artifacts
6. Prepare the nuget packages. Notice that one project in TeamCity produces multiple nuget packages, for example Windsor produces packages for Windsor and a handful of facilities. This usually involves:
   1. Getting the previous version of the package
   2. Editing the .nuspec file to update the package version, copyright, dependency version numbers
   3. Putting all the relevant files (indlucing .pdb) to the appropriate nuget folder for the package
   4. Putting the tagged version of the sourcecode to the /src folder
   5. Putting the extra files, like license, readme, changes, breaking changes etc in the package
   6. Building the packages( both binary and source) for each of the nugets to be pushed
7. Publish the nugets (in the right order) indlucing the source packages
8. Upload the release (that is the nugets minus the .nuspec and /src) to github
9. Announce the release to the users group
10. Update the website and docs

I think surely some of that can be automated. Ideally tagging, in step 4 would trigger the whole rest of the process (except step 10), step 9 surely can be automated with tools like IFTTT

from core.

Interesting. Thanks for the list. It looks to be quite comprehensive. This leads me to some thoughts and questions:

1. Getting to the point you would like, where tagging triggers everything up to (but not including) the website & docs updates sounds like quite a big undertaking. I think it would be best to tackle chunks. Worst case, we stop partway with a slightly-less-onerous release process. (Okay, worst case we break things so badly there's never another Castle release again. But that seems pretty unlikely.) Perhaps it would be best to use this issue to establish a list (perhaps one of the fancy GitHub checklist lists, with the []s) of subtasks, which could become their own issues.
2. You talk about Windsor and Core. Do you think it's mandatory to change the projects in lockstep, or would it be reasonable to work on (for example) Core first, and then port to Windsor as the changes proved themselves?
3. You mentioned the source NuGet packages, if I understand 6.vi. properly. I may be thick, but I can't find them. (That's kind of an aside, although it may affect my next point.)
4. There's a boatload of stuff in points 5–6: manually putting together the NuGet packages. I know in FakeItEasyLand, the TeamCity build creates the NuGet package. From a quick read, it seems like maybe implementing auto-NuGet-building may reduce a significant part of the pain, and is perhaps a good place to start?

Thanks for listening!

from core.

hey,

I don't really expect the process to automate updating the website or docs (althought that could be done too, but it'd be low priority), see the note at the end of the previous comment.

I completely agree that this should be tackled using divide and conquer.

The note about Windsor and Core basically means that Windsor should always be released targetting the latest Core version at that time.

About source packages, my bad - they are called symbol packages

I know that TeamCity can build nuget packages, but I never really used that beyond for what Octopus requires (and then I didn't have multiple target builds).

The problem with Castle seems to be that we have both, multiple builds ending up in a single nuget (.net 3.5 build, .net 4 build, silverlight 5 build etc) and single build ending up in multiple packages (Windsor and facilities for example), which I imagine can make things tricky to automate with TeamCity...

Has anyone actually tried doing something like that?

from core.

Oh, I realised that you didn't expect auto-updating of website or docs. Sorry I didn't express that well.

Ah. Symbol packages. I'd never heard of those either. Cool.

I don't know if anyone's tried NuGetting multiple DLLs in to one NuGet package as well as a single DLL ending up in multiple NuGets. I'm not sure why it would be a problem, but that's probably naivety talking.

I'll take some time over the next little while (perhaps about a week) and see if I can get something working just in my sandbox, maybe even as a separate batch file to be run after the build. If that seems to work, I'll submit a "for comment" PR and we can talk about pulling it into the TeamCity build?

from core.

Sounds great

Krzysztof Kozmic
sent from my phone
On 6 Apr 2014 20:45, "Blair Conrad" [email protected] wrote:

Oh, I realised that you didn't expect auto-updating of website or docs.
Sorry I didn't express that well.

Ah. Symbol packages. I'd never heard of those either. Cool.

I don't know if anyone's tried NuGetting multiple DLLs in to one NuGet
package as well as a single DLL ending up in multiple NuGets. I'm not sure
why it would be a problem, but that's probably naivety talking.

I'll take some time over the next little while (perhaps about a week) and
see if I can get something working just in my sandbox, maybe even as a
separate batch file to be run after the build. If that seems to work, I'll
submit a "for comment" PR and we can talk about pulling it into the
TeamCity build?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-39664329
.

from core.

Subtasks that we've identified so far:

• #49: create a tool to build NuGet packages - roughly corresponds to step 6 in @kkozmic's comment above
• take version numbers from Git tag (may require checking source out on agents, not server)
• use creation of Git tag to trigger build/release (may not be possible with lightweight tags)
• publish NuGet packages to public NuGet server
• upload release to GitHub (using octokit?)
• announce release to users group

from core.

Now that #49 has been accepted, and castleproject/Windsor#57 is well
underway, it may be time to explore tackle some other
optimizations. Would it be accurate to say that the release process
has become

1. For Windsor - make sure the project references the latest release version of Core
2. Go to Settings.proj and update the version number
3. Go to Changes.txt and BreakingChanges.txt and make sure they are updated
4. Commit, tag (with the release number) and push
5. When the "Pack" build in our TeamCity has successfully completed, download the artifacts
6. Publish the nugets (in the right order) including the source packages
7. Upload the release .zip file to GitHub
8. Announce the release to the users group
9. Update the website and docs

from core.

Do we want to tackle any more of the manual steps? Also, a bit late but huge thanks @blairconrad for the effort here!

from core.

Ha, thanks. It was a pleasure.

I'd be happy to help with some stuff, if you think I can. Schedules a bit funny now, but if you don't mind slow work, I could try something.

Not sure what's automatable any more…
Possibly (referencing my list 2 comments ago)

• 1 - finding out the latest Core build could be done on every Windsor build, I suppose. Or roll it into a command that will be executed below…
• 2ish (a human would need to pick the version number)
• 4 (semi-automated, the tag could be generated from Settings.proj)
• 5 (semi-automated - invoke curl or wget or something?)
• 6 we could conceivably have TeamCity publish, especially if it could figure out that it's building from a tag commit or something
• 7 - there's probably an Octokit function for this, so it could at least be a button push. Hmm. Is the only reason we're downloading the artifacts to push them? Then this step could obviate the need for the download in step 5?

3, 8, and 9 feel very humany to me. But I could be wrong.

What's yer big pain point? How would you like it to work?

from core.

Hey @blairconrad
I don't mind it taking a bit of time. I can't dedicate too much to doing it myself, so any help is welcome and appreciated :)

You're right, most of those still require some manual work.

In terms of how far it can be pushed I was thinking about a process somewhere along the lines of:

• set things up locally (Core reference for Windsor), update settings.proj, changes and breakingchanges files
• commit, tag and push
• Have TeamCity push to nuget and github (either implicitly, based on the tag, or explicitly as a manually triggered build in TC... although I'm not sure how well we can lock down the permissions here so that random people don't randomly push releases :)
• user group/twitter announcement could technically be automated with something like IFTTT I guess. Low priority
• updating the website would probably stay manual
• Doco is now part of the repo so hopefully would be brought up to date as part of the code changes.

Thoughts/ideas? /cc @castleproject/owners

from core.

I think trying to automate much more via TeamCity is going to be a lot of work, especially that there are quite a few manual steps in there. Next time I do a release I'm happy to put together a script that you can run on your own machine which could prompt you to complete manual tasks and prompt for credentials.

from core.

Sounds good.

On Fri, 11 Sep 2015 13:00 Jonathon Rossi [email protected] wrote:

I think trying to automate much more via TeamCity is going to be a lot of
work, especially that there are quite a few manual steps in there. Next
time I do a release I'm happy to put together a script that you can run on
your own machine which could prompt you to complete manual tasks and prompt
for credentials.

—
Reply to this email directly or view it on GitHub
#48 (comment).

sent from my phone
Krzysztof

from core.

Man, 19 days? Time flies.
@jonorossi, I see you've self-assigned, but if you like I could still try to pick off small pieces from time to time.
One thing I'd wondered about is the desired toolset. I'm used to (from FakeItEasy land) working with rake to set versions, create tags, update releases, and whatnot, but am happy to work with something else.
It seems that .cmd files are the preferred ways to build. Would we like something cmdy that we could have
"commit, tag and push", to steal one of @kkozmic's steps?
In that case, we may need to take other (development) dependencies for tools, specifically to read versions out of settings.proj and whatnot.
Or we could build our own tiny utilities.

An alternative, if we're still considering performing releases from Windows, would be Powershell scripts, which would let us leverage XML readers and so on.

Thanks.

from core.

@jonorossi, I think #241 is going to subsume this issue. Shall we close?

from core.

Yep, closing.

from core.