CI and Build server about housekeeping HOT 81 CLOSED

From @micdenny on November 18, 2014 21:48

We should highly consider the free ci build server AppVeyor

from housekeeping.

From @micdenny on January 11, 2015 13:47

@mauroservienti we need to use a nuget package for System.Windows.Interactivity or embed as dll in the repo, because otherwise it is not possible to install the blend interactivity for v4.5 because it is not distributed as an sdk anymore, and comes just with visual studio, and probably not on the free visual studio version that does not have blend, and this is a problem for the CI build server that has this software pre-installed: http://www.appveyor.com/docs/installed-software

I've already created the chocolatey package for Blend3 and Blend4 sdk, but can't for 4.5 because doesn't exist.

I think the easiest way is to embed or using a nuget package for that, could you arrange it?

from housekeeping.

From @micdenny on January 12, 2015 12:44

@mauroservienti please have a look to PR #119 and see if we can go with that solution, to best work with build server as well.

from housekeeping.

From @micdenny on January 12, 2015 18:8

Build scheduled let's see if it works.

In the meanwhile here's the badge that we'll can add to our repo readme file:

from housekeeping.

From @micdenny on January 12, 2015 18:12

I also found that AppVeyor is able to understand that a PR was sent, and it starts a build from that PR, before that you actually have merge it.

This is an invaluable feature of AppVeyor, because it can tells you if the build failed (+tests) before you actually merge it.

from housekeeping.

Lovely, what limitations have? I mean, given that it is free are we limited in some way?

Sent from my Amazing not anymore Yellow Lumia, typos are guaranteed ;-)

From: Michael Dennymailto:[email protected]
Sent: ý12/ý01/ý2015 19:12
To: RadicalFx/radicalmailto:[email protected]
Cc: Mauro Servientimailto:[email protected]
Subject: Re: [radical] CI and Build server (#93)

I also foundhttps://github.com/EasyNetQ/EasyNetQ/issues/344 that AppVeyor is able to understand that a PR was sent, and it starts a build from that PR, before that you actually have merge it.

This is an invaluable feature of AppVeyor, because it can tells you if the build failed (+tests) before you actually merge it.

—
Reply to this email directly or view it on GitHubhttps://github.com/RadicalFx/Radical/issues/93#issuecomment-69616177.

from housekeeping.

From @micdenny on January 12, 2015 18:29

In the free edition you have limitation just on calculation power and concurrent jobs (see the pricing offical page).

What is "concurrent job"?

Every push to the repository starts a new build with a single or multiple jobs. If your current plan includes only 1 concurrent job and there is another build job currently running all consequent jobs will be queued until the first job finishes. With multiple concurrent jobs your team will be getting results faster or you can use build matrix and parallel testing to build and test, for instance, Debug/Release or x64/x86 configurations in parallel.

What hardware is used to run builds?

Every build job runs on a pristine virtual machine provisioned just for this build. "Basic" and "Free" plans run builds on "Azure" environment where each build worker has 1 CPU core and 1.75 GB of RAM ("Small" instance). "Pro" and "Premium" plans run builds on dedicated hardware with faster CPUs, SSD drives, 1 GB network uplink and use Hyper-V for creating isolated build workers.

Are there any build time restrictions?

All plans have maximum build job execution time of 40 minutes.

from housekeeping.

From @micdenny on January 12, 2015 22:12

and finally it works, check the pushed artifacts 😄 of course the artifacts will be also pushed to the nuget.org gallery, but this is the last step, and we should discuss more on it.

this is just a starting point, I had to add a new chocolatey package for NuBuild, and AppVeyor just installed without any problem with a simple choco install NuBuild ;)

Now I have to manage the versioning and we're quite there.

I was wondering if it's better to avoid all those post-build action to copy assemblies, and just add a .nuspec file for each project we would like to publish, and leave the build server do the rest, actually it will also be clear for whom it may build directly from source where to look at "bin\release" and when you have to create a manual nuget package, just run run nuget pack from the console... I'm just thinking out loud... maybe a branch with a PR could be better to discuss.

from housekeeping.

From @micdenny on January 13, 2015 9:46

I was wondering if it's better to avoid all those post-build action to copy assemblies

I thought about it, and I think I will leave your structure for the moment, and proceed step by step, first step just use your script+nubuild to build and push packages, then we will see if it worth remove the post-build script, maybe the first time we need to change, we remove 😄

from housekeeping.

How do we define when and where packages are pushed?
I mean:

• can we say that we want to push on NuGet only on master commits/push with a specific tag for example?
• can we say that each push to develop shoul be packaged and pushed to myget?

Any other idea?

from housekeeping.

From @micdenny on January 14, 2015 10:22

• can we say that we want to push on NuGet only on master commits/push with a specific tag for example?

I don't see that feature, there's a "tag" promoting scenario, but only for pushing to GitHub releases.

I think we should start easy, something like "when commit to master, build and instantly push to nuget as stable", as opposite to "when commit to develop, build and instantly push to nuget as prerelease".

I would also say, that from my experience, when you have small community, having just one master branch always pushed as stable gives you more feedback on the project, and the project itself evolve/fix more quickly, it sounds risky I agree, but I saw it work remarkably well. If you have a pre-release but no one cares about it, bugs won't get noticed until the release version and then they'll have to be fixed in there anyway.

What do you think about choosing one of the following process?

• Leave master / develop branch, build them and instantly create stable and prerelease on nuget. Prerelease of course could be pushed on myget, but at the moment I would push to nuget, and use myget when you would like to differentiate nightly-build from promoted alpha/beta (but again, that's more work!).
• Remove develop branch, leave only master, on every commit, build and instantly create stable release on nuget. Every other branch from master, is on developing or waiting to be merged, and could be promoted as a prerelease package but only when it worth, means someone is there to heavily test it.

from housekeeping.

• Leave master / develop branch, build them and instantly create stable and prerelease on nuget. Prerelease of course could be pushed on myget, but at the moment I would push to nuget, and use myget when you would like to differentiate nightly-build from promoted alpha/beta (but again, that's more work!).

Like this one, go ahead. Good stuff.

from housekeeping.

In light of #15 do you think that it gets simpler? I suspect yes :-)

from housekeeping.

From @micdenny on June 7, 2015 12:21

for my reference... a way to patch the assembly and nuspec version, setting the version just once in the appveyor config file:

http://help.appveyor.com/discussions/problems/1785-publish-prerelease-nuget-package-with-version-from-yml-file

a working sample:

https://github.com/FeodorFitsner/nuget-test

from housekeeping.

moving back to the backlog, to ambitious for the next release, I hope to be able to complete #15 and we can concentrate on this

from housekeeping.

From @micdenny on June 27, 2015 13:18

ok great!

from housekeeping.

Easier than expected. #15 is now done.

All the develop branches, and most of the master ones are now aligned. Each package has its own repo.

You can start crafting this one, I suggest that if there is something to be done at the repo level issues should be opened there, linking back to this one.

from housekeeping.

From @micdenny on June 29, 2015 8:5

ok will create issue for every change I need to do, basically the appveyor config file for sure, anyway will work as usual in branch+issue

thanks!

from housekeeping.

From @micdenny on June 29, 2015 8:11

Just a recap for my reference:

• Separate each component (Radical, Radical.Windows, etc...) in its own repo.
• Create a build for each component: each time we push, a build is started and deployed to a feed.
• Differentiate publish by branch: if we push to develop then we publish as beta to Myget, if we push to release then we publish to NuGet. (optional)

from housekeeping.

yes. all correct. What may miss how to generate version numbers

from housekeeping.

From @micdenny on June 29, 2015 8:20

I've posted something regard that in this comment: RadicalFx/Radical#93 (comment)

I've just tested this scenario on a simple project and it worked.
Repo: https://github.com/FeodorFitsner/nuget-test
nuspec: https://github.com/FeodorFitsner/nuget-test/blob/master/MyNuGetLib/...
appveyor.yml: https://github.com/FeodorFitsner/nuget-test/blob/master/appveyor.yml
build results: https://ci.appveyor.com/project/FeodorFitsner/nuget-test/build/1.0....

from housekeeping.

From @micdenny on June 29, 2015 8:27

Basically in that way, you will just need to patch the version variable of the appveyor.yml, so every time you are merging a PR you just ensure that the version is correctly patched (non-breaking or breaking), and that's it, after you commit the merge the build process start with the correct version.

from housekeeping.

From @micdenny on June 29, 2015 8:30

then to separate the beta from the release, it should be a PR in the middle, with the merge of all the changes, and the version patch removing the pre-build tag (ie.: -rc1), and that's it. This is the mainly idea, let's see to realize something, and then see if we like it.

from housekeeping.

From @micdenny on June 29, 2015 16:45

This is a good way to differentiate publish by branch

If you use git flow you may want to have a different build configuration (e.g. deploying to a different environment) in a feature branch. Changing appveyor.yml in a feature branch becomes an issue when you merge it into master overriding appveyor.yml and breaking master builds.

To solve this problem AppVeyor allows having multiple per-branch configurations in a single appveyor.yml.

Multiple configurations are defined as a list with branches section in every item that:

# configuration for "master" branch
# build in Release mode and deploy to Azure
-
  branches:
    only:
      - master

  configuration: Release
  deploy:
    provider: AzureCS
    ...

# configuration for all branches starting from "dev-"
# build in Debug mode and deploy locally for testing
-
  branches:
    only:
      - /dev-.*/

  configuration: Debug
  deploy:
    provider: Local
    ...

# "fall back" configuration for all other branches
# no "branches" section defined
# do not deploy at all
-
  configuration: Debug

from housekeeping.

From @micdenny on June 29, 2015 18:49

first automatic build done successfully 😄

I'm on a personal repo/branch and a separated appveyor build project to do all my tests

and my first package as been deploy on MyGet (micdenny-radical feed): https://www.myget.org/F/micdenny-radical/api/v2

from housekeeping.

🎉 👍

from housekeeping.

From @micdenny on June 30, 2015 18:16

Lesson learnt: appveyor does not like the way you add spaces around method call 😄

It took a while to understand why the assembly version patching was working on AssemblyFileVersion and AssemblyInformationalVersion but not for the most important AssemblyVersion, and it turns out that probably they have a poor regex replace pattern 😄

[assembly: AssemblyVersion( "1.0.0.0" )]

now replaced with this that work:

[assembly: AssemblyVersion("1.0.0.0")]

I've opened a discussion on appveyor: http://help.appveyor.com/discussions/problems/2407-c-patching-assembly-version-doesnt-work-if-theres-spaces-around-version-parameter

from housekeeping.

From @micdenny on June 30, 2015 18:34

Habemus Papam!

Automatic release based on the branch!

commit on development and you get a package build with a pre-release version number eg.: 1.0.15-alpha1 that is pushed on MyGet.org

commit on master and you get a package build with a release version number eg.: 1.0.16 that is pushed on the public gallery on NuGet.org

we will just need to remember to patch the version in the appveyor.yml file before commit the changes.

you can preview the changes on my repo on the brach deploy (simulates the master branch) and deploy-development (simulates the development branch), of course I will PR you the right configuration to build against master and development branches.

from housekeeping.

From @micdenny on June 30, 2015 18:50

@mauroservienti should we pass to semantic versioning? like microsoft is doing also for framework packaging.

In MyGet.org they adding also the build number:

and then when push to nuget they remove build number:

Actual Radical packages will become as follow:

Pre-release on myget: 1.5.2-alpha1-{build}
Pre-release on nuget: 1.5.2-alpha1
Release on nuget: 1.5.2

Basically we just remove build version when we push on nuget.

what do you think?

from housekeeping.

@mauroservienti should we pass to semantic versioning? like microsoft is doing also for framework packaging.

We are already doing our best to follow SemVer so it should not be a problem at all. 👍 to strict SemVer.

Regarding version numbers take a look at these:

• http://particularlabs.github.io/GitVersion/
• https://github.com/GitTools/GitVersion
• https://www.nuget.org/packages/GitVersionTask

Since we are already adopting GitFlow the above should be very easy, it is based on branch naming conventions.

from housekeeping.

From @micdenny on July 1, 2015 9:39

interesting GitVersion, need to give a try to understand how it works, it uses tag to understand the version to bump assembly info?

Can you describe the flow do you except to have included versioning?

with what I configured until now it looks like this:

when integrating on develop branch

• change the version on develop section of appveyor.yml
• commit to develop
• build start
• all the assemblies and the nuget package are bumped with the version specified
• nuget are pushed to myget

when integrating on master branch

• change the version on master section of appveyor.yml
• commit to master
• build start
• all the assemblies and the nuget package are bumped with the version specified
• nuget are pushed to nuget

You want to add some kind of tag commitment, so the build on master start only when there's a tag? and what about development then? Sorry to bother you, but I want to understand what's your idea, because there's so many ways to do this kind of stuff 😄

from housekeeping.

Can you describe the flow do you except to have included versioning?

Nothing different for what you described, except that the versioning is deducted by GitVersion based on conventions (TAG + branch + Git Flow).

Testing it now in the public Radical repo, found some issues due the existing TAGs, in touch with the GitVersion crew to see how to move forward.

from housekeeping.

From @micdenny on July 1, 2015 12:17

Testing it now in the public Radical repo, found some issues due the existing TAGs, in touch with the GitVersion crew to see how to move forward.

so basically you're just trying the console app against the repo and GitVersion reply with the json of what it thinks the next version should be?

from housekeeping.

No, I'm using the MSBuild TASK. Pull the repo, try to build it locally,
take a look at the generated assembly.
Also take a look at the AssemblyInfo files, they are missing the
Assembly*Version attributes. Injected at build time by GitVersion.
It is not working as expected, right now, the generated build number is not
expected.

Here the ongoing discussion:

• https://gitter.im/GitTools/GitVersion?at=5593c780b57c03f7556c37e1
• GitTools/GitVersion#470 (comment)

On Wed, Jul 1, 2015 at 2:17 PM, Michael Denny [email protected]
wrote:

Testing it now in the public Radical repo, found some issues due the
existing TAGs, in touch with the GitVersion crew to see how to move forward.

so basically you're just trying the console app against the repo and
GitVersion reply with the json of what it thinks the next version should
be?

—
Reply to this email directly or view it on GitHub
RadicalFx/Radical#93 (comment).

Mauro Servienti

from housekeeping.

All the components are splitted into their own repo. Despite GitVersion not yet usable due to our legacy commit history we can move this forward in my opinion.

Thoughts?

from housekeeping.

From @micdenny

I think is just a matter of power and control, Microsoft start on appveyor then pass on teamcity and I think just because they installed on bigger machines on azure :) and versioning will be anyway a PITA

from housekeeping.

+1, go on with AppVeyor

from housekeeping.

Any progress on this? Do we have an ETA?

No pressure at all, just to understand. Thanks 😄

from housekeeping.

I'm back from my holidays, so I think I'm going to look at this next week (or maybe this weekend), I will push a first working attempt, then we can evolve if it doesn't fit our requirements.

from housekeeping.

from housekeeping.

in progress...

from housekeeping.

there's some problem with the nuget v3.1 that is used by default from appveyor, but I figure it out temporary adding nuget.exe 2.8.x directly in the repo.

at the moment I'm still working on my fork I will PR when I have a complete working solution.

from housekeeping.

there's some problem with the nuget v3.1

now works! it was a nuget.org problem with the new v3 gallery:

from housekeeping.

this is what I've done so far to automatically build... versioning is a PITA!

the sad part:

• {build} number always increment (appveyor manages), also when building pull request
• to follow the semver the version is set to be 1.1.{build} for releases and 1.1.{build}-alpha1 for pre-releases
• need to change the version in two section of the appveyor.yml file (release and pre-release configuration)
• when changing major or minor version the {build} number has to be manually reset from appveyor settings UI page, for example to pass from 1.2.45 to 1.3.0 correctly.

the happy part:

• need just to bump the version on appveyor.yml when a major or minor version changes, so only on breaking changes
• pull requests builds, does not trigger package deploy, only artefacts
• we can publish release package to nuget.org when building from master
• we can publish pre-release package to myget.org when building from develop

@mauroservienti what do you think about that?

In the meanwhile I will experiment another way, that maintains version in AssemblyInfo.cs then with a PS script extracts the version and use it to set the version of appveyor build and of the nuget package.

from housekeeping.

I'm developing a nicer experience, just change the version in a semver way, and the script does the rest ;)

stand by....

from housekeeping.

What if we schedule a short Skype/Zoom call to get this going?
I can show you what I'm doing with GitVersion and see if we can seamlessy
integrate it solving the SemVer issue as well.

On Wed, Aug 19, 2015 at 7:31 PM, Michael Denny [email protected]
wrote:

I'm developing a nicer experience, just change the version in a semver
way, and the script does the rest ;)

stand by....

—
Reply to this email directly or view it on GitHub
#23 (comment)
.

Mauro Servienti

from housekeeping.

yes we can arrange it

from housekeeping.

• Create AppVeyor account for the radical-bot user
• Add radical-bot user to the Radical Build team on GitHub
• Grant access to the Radical Build to the relevant projects
• Add Radical Build team as AppVeyor admin joining the radical-bot account
• Create NuGet environment on AppVeyor adding API Key as radical NuGet user
• Create MyGet account as radical user
• Create MyGet feed as radical-unstable
• Create MyGet environment on AppVeyor adding API Key as radical MyGet user

from housekeeping.

All done. Ping me if I missed/screwed something 😄

from housekeeping.

Great, if that is all what I need, you should see my PR soon ;)

from housekeeping.

@mauroservienti do you think build team can have admin access to MyGet and NuGet feed? Is there a way to use my personal account to manage MyGet and NuGet? Exactly as it is for appveyor where build team as admin rights.

from housekeeping.

@mauroservienti there's some problem building Radical, I added the project on radical-bot, but can't start it, and no error is shown.

I think we need support from appveyor, probably something had messed up with the Radical project and appveyor, because before the project was added into my personal appveyor user space.

I'm going to open a ticket with appveyor.

from housekeeping.

issue opened: http://help.appveyor.com/discussions/problems/2859-cant-start-a-new-build-of-a-newly-imported-project-radical

from housekeeping.

👍 thanks!

from housekeeping.

for project Radical, I will need to temporary remove Gitversion, 'cause yes, it's working with appveyor seamlessly, but taking a strange version as you said to me this morning, and also if you don't mind, I would removed the nuget packager project, that I don't need for creating the package, because I'm using nuget pack directly from appveyor, otherwise I need to choco install that packager, and it will just slow down the build for nothing.

from housekeeping.

Drop all what you need to drop, no worries. Nuget projects can be safely
deleted, make only a copy (or leave them in the repo) of the nuspec files.

On Sat, Aug 22, 2015 at 7:04 PM, Michael Denny [email protected]
wrote:

for project Radical, I will need to temporary remove Gitversion, 'cause
yes, it's working with appveyor seamlessly, but taking a strange version as
you said to me this morning, and also if you don't mind, I would removed
the nuget packager project, that I don't need for creating the package,
because I'm using nuget pack directly from appveyor, otherwise I need to
choco install that packager, and it will just slow down the build for
nothing.

—
Reply to this email directly or view it on GitHub
#23 (comment)
.

Mauro Servienti

from housekeeping.

and please can you check that the webhook url you see here is added to the radical github webhook, thanks!

from housekeeping.

make only a copy (or leave them in the repo) of the nuspec files.

The nuspec will remain in place, I need that. I will only remove the nuget project.

from housekeeping.

Web hook added for all events, if we need less events tell me which one and I'll fix it.

from housekeeping.

Web hook added for all events, if we need less events tell me which one and I'll fix it.

just push and pull_request events please, thanks

from housekeeping.

push and pull_request

done

from housekeeping.

webhook is not working, can you check why the latest PR and push has not been delivered, you should see the load sent from the webhook page in github

from housekeeping.

@mauroservienti if you give me temporary ownership of radicalfx, I can try to fix it by myself

from housekeeping.

Done, added as owner.

from housekeeping.

@mauroservienti do you think build team can have admin access to MyGet and NuGet feed? Should we share the radical and radical-bot credentials, or we can use our personal account like we do with appveyor?

On nuget I think we have to change the signature of nuspec to add us as admin, but as far as I see from other packages they prefer to add just the "organization" account, and then I suppose they share the credential to the build team.

from housekeeping.

removed generic script appveyor-buildversion.ps1 from every repo, and created a common repo Radical.Build where we can add common scripts, this leaves the repo much more cleaner, and simplifies fixes of scripts, that now has to be pushed just one time to fix them all 😄

I'm starting to be very happy with this build system. Next step will be adding GitVersion, but I suggest to first taking confidence with this way, then end-over to Tags.

from housekeeping.

also added the possibility to push the master branch as a pre-release to the stable feed (nuget.org), this is possible because I introduced the env variable "use_build_number_in_packages", that by default is false, so that the master branch will never push a package with build version on it.

This could be helpful when working on new project or major release, where at some point you want to push a pre-release to nuget.org and let people play with it, without having to look at unstable feed on myget.org.

from housekeeping.

everything is in place, up and running!!

when we will merge to master, the build will temporary push to myget.org to be sure everything is fine before messing up the undoable nuget.org 😄 if everything will be ok, we will just change the appveyor config file to push to the nuget feed environment.

from housekeeping.

also all the build badges are in place 😄

from housekeeping.

simply awesome 😄 can we close this one?

from housekeeping.

One more question are all the changes on develop only? should we merge to master?

from housekeeping.

Last: we need some docu around the entire job u did.
A PR here: https://github.com/RadicalFx/documentation/blob/master/docs/infrastrucure/ci-build.md
Will render here: http://docs.radicalframework.com/en/latest/infrastrucure/ci-build/

It is time to start #5

from housekeeping.

One more question are all the changes on develop only? should we merge to master?

yes, changes are only on develop, I waited to merge in master just because I don't know if every develop branch are ready to be released.

Last: we need some docu around the entire job u did.

don't know how to best structure it, is there any example out there? what do you like to see in this doc? I think the most important part is the versioning, should I write it there?

from housekeeping.

One more question are all the changes on develop only? should we merge to master?

yes, changes are only on develop, I waited to merge in master just because I don't know if every develop branch are ready to be released.

OK, starting next week I'm on vacation for 2 weeks, as I'm back we can plan a new release.

Last: we need some docu around the entire job u did.

don't know how to best structure it, is there any example out there? what do you like to see in this doc? I think the most important part is the versioning, should I write it there?

Yes, I'd say 2 things to start with:

• Versioning
• How to setup a new project: imagine to create a new project from scratch, document all the detailed steps

Thoughts?

from housekeeping.

@micdenny what is the status of this?

AFAIK all the repos are ready but everything is on develop only and we are not yet publishing to NuGet by only to MyGet, correct?

from housekeeping.

@micdenny what is the status of this?

@mauroservienti I think we can close this one, and open issue for documenting this on docs (https://github.com/RadicalFx/documentation/blob/master/docs/infrastrucure/ci-build.md).

AFAIK all the repos are ready but everything is on develop only and we are not yet publishing to NuGet by only to MyGet, correct?

You're right! Now we should merge in master and see if the stable packages are created correctly in MyGet, if they're fine you can push from MyGet to NuGet directly from myget portal, if everything is fine we can commit the change to push master commits directly into NuGet.

That's the plan, is that ok for you?

from housekeeping.

I think we can close this one, and open issue for documenting this on docs

👍 go

That's the plan, is that ok for you?

👍 go.

from housekeeping.

working on....

from housekeeping.

Issue for documenting opened: RadicalFx/documentation#20

from housekeeping.

merging from develop to master branch starting with Radical.Design project that is a low-impact library :)

from housekeeping.

merge done!!

everything worked just fine, so for the next stable release we can change the appvejor config file to push directly on nuget.

from housekeeping.