microsoft / azure-pipelines-agent-cookbook Goto Github PK

View Code? Open in Web Editor NEW

35.0 35.0 24.0 114 KB

Chef Cookbook to install and manage Azure Pipelines Agent for Build and Release tasks

Home Page: https://supermarket.chef.io/cookbooks/vsts_agent

License: MIT License

Ruby 99.69% HTML 0.31%

chef azure-devops azure-pipelines azure-pipelines-agent cookbook

azure-pipelines-agent-cookbook's Introduction

Visual Studio Team Services Build and Release Agent Cookbook

Installs and configures Visual Studio Team Services Build and Release Agent

Please check Wiki for more examples

Requirements

Chef 12.5.0 or higher

Platforms

The following platforms are tested and supported:

Debian 8 x64 (Jessie)
Debian 9 x64 (Stretch)
Ubuntu 16.04 (Xenial Xerus)
Ubuntu 18.04 (Bionic)
Ubuntu 17.10 (Artful Aardvark)
CentOS 7
Windows 10
Windows Server 2012 R2
OS X 10.10.5
OS X 10.11.4

Attributes

node['vsts_agent']['binary']['version'] - set version of package to install
node['vsts_agent']['prerequisites']['redhat']['install'] - control dependencies installation for redhat based distros(redhat, centos etc.) . Default true
node['vsts_agent']['prerequisites']['debian']['install'] - control dependencies installation for debian based distros(debian, ubuntu etc.). Default true

Resource/Provider

vsts_agent

This resource installs and configures the vsts build and release agent

Actions

:install: Install and configure the agent
:remove: Remove the agent and unregister it from VSTS
:restart: Restart the agent service

Parameters

agent_name: name attribute. The name of the vsts agent
deploymentGroup: deploy the agent into the deployment group. Default 'false'
- deploymentGroupName: name of the deployment group. Only applies if deploymentGroup==true
- projectName: name of the vsts/tfs project where to deploy the agent. Only applies if deploymentGroup==true
- collectionName: name of the vsts/tfs collection where to deploy the agent. Only applies if deploymentGroup==true. Dafault value is DefaultCollection
- deploymentGroupTags: a comma-separated list of tags to set for the agent. Only applies if deploymentGroup==true
version: an agent version to install. Default version from an attribute
install_dir: A target directory to install the vsts agent
path: Overwrite system PATH environment variable values. Linux and macOS only
env: Additional environment variables. Linux and macOS only
user: Set a local user to run the vsts agent
group: Set a local group to run the vsts agent
runasservice: run agent as a service. Default 'true'
windowslogonaccount: Set a user name to run a windows service. Possible values are "NT AUTHORITY\NetworkService", "NT AUTHORITY\LocalService" or any system valid username
windowslogonpassword: Set password for windowslogonaccount unless it is equal to NetworkService or LocalService
vsts_url: url to VSTS instance
vsts_pool: A pool to connect an agent
vsts_auth: Authentication type. Default PAT auth. Valid options are:
- PAT - Personal Access Token (requires vsts_token),
- Negotiate - Kerberos or NTLM (requires vsts_username and vsts_password),
- ALT - Alternate Credentials (requires vsts_username and vsts_password),
- Integrated - Windows default credentials (doesn't require any credentials).
vsts_token: A personal access token for VSTS. Used with PAT auth type. See
vsts_username: A user to connect to VSTS. Used with Negotiate and ALT auth
vsts_password: A user to connect to VSTS. Used with Negotiate and ALT auth
work_folder: Set different workspace location. Default is "install_dir/_work"
proxy_url: Set the forward proxy (example value: http://192.168.1.1:8080)
proxy_sslcacert: Set the root certificate which is used for communicating through the forward proxy (example value: /usr/local/share/ca-certificates/Internal-CA.crt)

Examples

Install, configure, restart and remove a build agent. Check windows, linux or osx tests for more examples.

include_recipe 'vsts_agent::default'

if platform_family?('windows')
  dir = 'c:\\agents'
else
  dir = '/tmp/agents'
end

vsts_agent 'agent_01' do
  install_dir dir
  user 'vagrant'
  group 'vagrant'
  path '/usr/local/bin/:/usr/bin:/opt/bin/' # only works on nix systems
  env('M2_HOME' => '/opt/maven', 'JAVA_HOME' => '/opt/java') # only works on nix systems
  vsts_url 'https://contoso.visualstudio.com'
  vsts_pool 'default'
  vsts_token 'my_secret_token_from_vsts'
  windowslogonaccount 'builder' # will be used only on windows
  windowslogonpassword 'Pas$w0r_d' # will be used only on windows
  action :install
end

vsts_agent 'agent_01' do
  action :restart
end

vsts_agent 'agent_01' do
  vsts_token 'my_secret_token_from_vsts'
  action :remove
end

Install, configure, restart and remove a deployment agent.

vsts_agent 'deployment_agent_01' do
  install_dir dir
  deploymentGroup true
  deploymentGroupName 'project1-deployment-group'
  projectName 'project1'
  collectionName 'DefaultCollection'
  deploymentGroupTags "web, db"
  user 'vagrant'
  group 'vagrant'
  path '/usr/local/bin/:/usr/bin:/opt/bin/' # only works on nix systems
  env('M2_HOME' => '/opt/maven', 'JAVA_HOME' => '/opt/java') # only works on nix systems
  vsts_url 'https://contoso.visualstudio.com'
  vsts_token 'my_secret_token_from_vsts'
  windowslogonaccount 'builder' # will be used only on windows
  windowslogonpassword 'Pas$w0r_d' # will be used only on windows
  action :install
end

How to contribute

Check Contribution Guide and Testing Guide

azure-pipelines-agent-cookbook's People

Contributors

Stargazers

Watchers

azure-pipelines-agent-cookbook's Issues

Default agent version?

Should the default agent be the latest pre-release: 2.125.1, or the latest release: 2.124.0?

Authentication error

Using Chef 12.14.89 I get the following error when it is trying to authenticate to VSTS:

Chef::Exceptions::ValidationFailed: Required argument :principals is missing!

Any help on resolving this would be greatly appreciated.

vsts_agent 'agent_02' do
install_dir dir
vsts_url 'https:/my_company.visualstudio.com'
vsts_pool 'default'
vsts_token 'my_PAT'
windowslogonaccount 'corp\User' # will be used only on windows
windowslogonpassword 'my_password' # will be used only on windows
action :install
end

Thanks!!!

Bug: NoMethodError: undefined method `win_friendly_path'

Error Encountered

================================================================================
             Error executing action `install` on resource 'vsts_agent[WIN-D8TTNH071C2-ECMY-1]'
             ================================================================================

             NoMethodError
             -------------
             undefined method `win_friendly_path' for #<#<Class:0x000000000539ec48>:0x000000000a3f0020>

[2018-08-17T11:11:09-05:00] FATAL: NoMethodError: my_tfs_agent[default] (my_tfs_agent::default line 26) had an error: NoMethodError: vsts_agent[WIN-D8TTNH071C2-ECMY-1] (C:/Users/vagrant/AppData/Local/Temp/kitchen/cache/cookbooks/my_tfs_agent/resources/my_tfs_agent.rb line 71) had an error: NoMethodError: undefined method `win_friendly_path' for #<#<Class:0x000000000539ec48>:0x000000000a3f0020>

Dependency versions:

  ark (4.0.0)
    build-essential (>= 0.0.0)
    seven_zip (>= 0.0.0)
  build-essential (8.1.1)
    mingw (>= 1.1)
    seven_zip (>= 0.0.0)
  mingw (2.1.0)
    seven_zip (>= 0.0.0)
  seven_zip (3.0.0)
    windows (>= 0.0.0)
  vsts_agent (3.0.0)
    ark (>= 0.0.0)
    seven_zip (>= 2.0.0)
    windows (>= 0.0.0)
  windows (5.0.0)

Relevant Lines of Code

Questions

Is it really necessary to use win_friendly_path? Chef and powershell are perfectly happy with the / form of path separator.
Are you missing an include and Windows.Helper. namespaces in these calls? See this stack overflow

Cookbook doesn't install service properly on Chef 13

log "Trigger service installation for agent #{new_resource.agent_name}" do
        notifies :enable, service_id, :immediately if new_resource.runasservice
      end

This code doesn't notify service creation when chef version 13.
Need investigations.

Missing .service file on Windows 2012R2

I can't seem to make it work on Windows 2012R2 (official AWS AMI). I get this error:

Recipe looks like this:

vsts_agent 'agent_01' do
  user 'Administrator'
  group 'Administrators'
  install_dir 'c:\\agents'
  vsts_url 'https://mycorp.visualstudio.com'
  vsts_pool 'Default'
  vsts_token 'XXXXXXX'
  windowslogonaccount "NT AUTHORITY\NetworkService"
  action :install
end

And error is always this:

vsts_agent_service[agent_01] (C:/Users/ADMINI~1/AppData/Local/Temp/kitchen/cache/cookbooks/vsts_agent/providers/default.rb line 25) had an error: Errno::ENOENT: No such file or directory @ rb_sysopen - c:\agents/.service

Test kitchen fails on macOS during 'Configuring agent' execution

The log indicates the problem is located on line 97 of /providers/default.rb here:

execute "Configuring agent '#{new_resource.agent_name}'" do
  cwd "#{new_resource.install_dir}/bin"
  sensitive true if respond_to?(:sensitive)
  command vsagentexec(args)
  action :run
end

From the look of it, it appears to be a sensitive data issue, as seen below. Note that I have replaced the PAT key with asterisks for security.


             ================================================================================
             Error executing action `run` on resource 'execute[Configuring agent 'osx_osx-basic-10105_01']'
             ================================================================================

             Mixlib::ShellOut::ShellCommandFailed
             ------------------------------------
             Command execution failed. STDOUT/STDERR suppressed for sensitive resource

             Resource Declaration:
             ---------------------
             suppressed sensitive resource output

             Compiled Resource:
             ------------------
             suppressed sensitive resource output

             System Info:
             ------------
             chef_version=13.0.118
             platform=mac_os_x
             platform_version=10.10.5
             ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-darwin14]
             program_name=chef-client worker: ppid=732;start=12:44:07;
             executable=/opt/chef/bin/chef-client


           ================================================================================
           Error executing action `install` on resource 'vsts_agent[osx_osx-basic-10105_01]'
           ================================================================================

           Mixlib::ShellOut::ShellCommandFailed
           ------------------------------------
           execute[Configuring agent 'osx_osx-basic-10105_01'] (/tmp/kitchen/cache/cookbooks/vsts_agent/providers/default.rb line 97) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

           Resource Declaration:
           ---------------------
           # In /tmp/kitchen/cache/cookbooks/osx-basic/recipes/default.rb

            19: vsts_agent agent1_name do
            20:   install_dir "#{agents_dir}/#{agent1_name}"
            21:   user 'admin'
            22:   group 'staff'
            23:   vsts_url node['vsts_agent_test']['vsts_url']
            24:   vsts_pool node['vsts_agent_test']['vsts_pool']
            25:   vsts_token node['vsts_agent_test']['vsts_token']
            26:   action :install
            27: end
            28:

           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cache/cookbooks/osx-basic/recipes/default.rb:19:in `from_file'

           vsts_agent("osx_osx-basic-10105_01") do
             action [:install]
             updated true
             updated_by_last_action true
             default_guard_interpreter :default
             declared_type :vsts_agent
             cookbook_name "osx-basic"
             recipe_name "default"
             vsts_token "******************************************************"
             install_dir "/Users/admin/agents/osx_osx-basic-10105_01"
             user "admin"
             group "staff"
             vsts_url "office.visualstudio.com"
             vsts_pool "OXO Hub Eng Mac Pool"
             agent_name "osx_osx-basic-10105_01"
           end

           System Info:
           ------------
           chef_version=13.0.118
           platform=mac_os_x
           platform_version=10.10.5
           ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-darwin14]
           program_name=chef-client worker: ppid=732;start=12:44:07;
           executable=/opt/chef/bin/chef-client


       Running handlers:
       [2017-05-04T12:48:06-07:00] ERROR: Running exception handlers
       [2017-05-04T12:48:06-07:00] ERROR: Running exception handlers
       Running handlers complete
       [2017-05-04T12:48:06-07:00] ERROR: Exception handlers complete
       [2017-05-04T12:48:06-07:00] ERROR: Exception handlers complete
       Chef Client failed. 13 resources updated in 03 minutes 59 seconds
       [2017-05-04T12:48:06-07:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2017-05-04T12:48:06-07:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2017-05-04T12:48:06-07:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
       [2017-05-04T12:48:06-07:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
       [2017-05-04T12:48:06-07:00] ERROR: vsts_agent[osx_osx-basic-10105_01] (osx-basic::default line 19) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[Configuring agent 'osx_osx-basic-10105_01'] (/tmp/kitchen/cache/cookbooks/vsts_agent/providers/default.rb line 97) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
       [2017-05-04T12:48:06-07:00] ERROR: vsts_agent[osx_osx-basic-10105_01] (osx-basic::default line 19) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[Configuring agent 'osx_osx-basic-10105_01'] (/tmp/kitchen/cache/cookbooks/vsts_agent/providers/default.rb line 97) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
       [2017-05-04T12:48:06-07:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
       [2017-05-04T12:48:06-07:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Improve integration testing

Need to add integration tests (inspec or serverspec)

On macOS, launchd service does not work over ssh.

Chef's transport of choice for macOS, SSH does not allow a user to start or stop launchd services. In theory, if the user that owns the service is logged into the GUI, they should be able to issue launchtl commands over SSH, but I haven't gotten that to work for some reason. Basically this means that svc.sh start, svc.sh stop are broken for this cookbook on macOS.

For example, any time a Service.stop is called: https://github.com/Microsoft/vsts-agent-cookbook/blob/267358dbac2aff64e081911917ac38e006002105/resources/default.rb#L223

The way this is currently set up, when the node is reconverged the resource attempts to stop a service it can't stop. Really not sure how to get around this one. If you stop, it will break. If you don't stop, it could break because you might change files associated with a running daemon process when you reconverge. I'm thinking that there should be a way to make launchd work over ssh, but my research indicates that it should already work...

Add UnitTests for travis-ci builds

#8 introduced several issues which could be caught by unit tests in travis.
ChefSpec is a standard for Chef unit testing.

No need to convert environment to string.

https://github.com/Microsoft/vsts-agent-cookbook/blob/ac16dbdef613254b9685359addd80d9d2278835e/libraries/service.rb#L81

Execute is expecting a Hash-type, not a string.

Cookbook throws an error when we specify new agent pool

I was able to install an agent into an existing pool. However, When I specify a new agent pool to vsts_pool property it is throwing an error saying that pool doesn't exist. it is supposed to creating the agent pool and then configure the agent into the pool right?

Pinning of seven_zip dependency using ~> prevents users from selecting version 3.0.0

The current seven_zip dependency declaration is impacting our ability to use the most current version of the seven_zip cookbook.

Consider using >= 2.0.0 instead of ~> 2.0.0

Add CHANGELOG.md file to repo

Add CHANGELOG.md file to repo with release notes for different versions. Makes it easier to see changes in chef supermarket as well as in the repo here without having to go through commits.

On-Prem compatibility?

Any plans to accomodate On-Premises rollouts?

Feature Request: Add support for deployment groups

I have a team at work that is using the new TFS Deployment groups. Is it possible support for this feature could be added to the cookbook. Below is an example configuration from the config.cmd:

.\config.cmd --deploymentgroup --deploymentgroupname "test group" --agent $env:COMPUTERNAME --runasservice --work '_work' --url 'https://tfs.example.com/tfs/' --collectionname 'DefaultCollection' --projectname 'Example Company'

Version is not a state value

I was just trying to update the version of the agents on my build servers and none of them would update. Checking the code I see version is not part of the state.
Is that deliberate? Or can it be added in a new version?

Test kitchen fails on macOS using Chef Client 13.0.118

I was receiving an error using the declared omnibus version of 12.4.3, so I modified .kitchen.yml to use the latest version. Now receiving the following error during the agent removal test:

         * vsts_agent[osx_osx-basic-10105_01] action remove

           ================================================================================
           Error executing action `remove` on resource 'vsts_agent[osx_osx-basic-10105_01]'
           ================================================================================

           NameError
           ---------
           uninitialized constant Chef::Resource::VstsAgent

           Cookbook Trace:
           ---------------
           /tmp/kitchen/cache/cookbooks/vsts_agent/providers/default.rb:14:in `load_current_resource'

           Resource Declaration:
           ---------------------
           # In /tmp/kitchen/cache/cookbooks/osx-basic/recipes/default.rb

            13: vsts_agent agent1_name do
            14:   vsts_token node['vsts_agent_test']['vsts_token']
            15:   action :remove
            16: end
            17:

           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cache/cookbooks/osx-basic/recipes/default.rb:13:in `from_file'

           vsts_agent("osx_osx-basic-10105_01") do
             action [:remove]
             default_guard_interpreter :default
             declared_type :vsts_agent
             cookbook_name "osx-basic"
             recipe_name "default"
             vsts_token "nkxk67z764a3wz5p7km3xhh3ykvofupta3nywzzpbzkjtxbkfevq"
             agent_name "osx_osx-basic-10105_01"
           end

           System Info:
           ------------
           chef_version=13.0.118
           platform=mac_os_x
           platform_version=10.10.5
           ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-darwin14]
           program_name=chef-client worker: ppid=643;start=12:13:44;
           executable=/opt/chef/bin/chef-client

The full log is here

FYI, I've revoked the token shown.

chef 14 error: cannot load such file -- chef/mixin/language

Getting error loading vsts_agent cookbook on chef-client 14. This line seems to break it: https://github.com/Microsoft/vsts-agent-cookbook/blob/master/libraries/service.rb#L4

Starting Chef Client, version 14.3.37
resolving cookbooks for run list: ["pl_roundhouse_deployment_role"]
Synchronizing Cookbooks:
  - bash-completion (0.1.0)
  - dstat (0.1.1)
  - logrotate (2.2.0)
  - nano (1.0.3)
  - vim (2.0.2)
  - trusted_certificate (3.2.0)
  - chef-sugar (4.0.0)
  - timezone_iii (1.1.6)
  - sshd (1.3.1)
  - build-essential (8.1.1)
  - pl_katello_client (1.0.5)
  - redhat_subscription_manager (1.0.4)
  - selinux_policy (2.1.0)
  - powershell (6.1.2)
  - windows (4.2.5)
  - line (1.0.6)
  - nfs (2.5.1)
  - sudo (5.4.0)
  - sysctl (1.0.5)
  - systemd (3.2.3)
  - seven_zip (3.0.0)
  - ohai (5.2.3)
  - selinux (2.1.1)
  - chef_hostname (1.0.1)
  - mingw (2.1.0)
  - ms_dotnet (4.2.0)
  - vsts_agent (2.0.3)
  - pl_active_directory (1.2.1)
  - pl_install_certificates (1.0.0)
  - pl_tfs_agent (1.1.0)
  - pl_roundhouse_deployment_role (1.0.0)
  - audit (7.0.1)
  - ark (4.0.0)
  - chef-client (11.0.0)
  - chef_client_updater (3.4.0)
  - cron (6.2.0)
  - firewall (2.6.5)
  - ntp (3.6.0)
  - postfix (5.3.1)
  - pl_active_directory_wrap (1.2.1)
  - pl_base_role (1.3.5)
Installing Cookbook Gems:
Compiling Cookbooks...

================================================================================
Recipe Compile Error in C:/chef/cache/cookbooks/vsts_agent/libraries/service.rb
================================================================================

LoadError
---------
cannot load such file -- chef/mixin/language

System Info:
------------
chef_version=14.3.37
platform=windows
platform_version=10.0.14393
ruby=ruby 2.5.1p57 (2018-03-29 revision 63029) [x64-mingw32]
program_name=C:/opscode/chef/bin/chef-client
executable=C:/opscode/chef/bin/chef-client


Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 13 seconds

username and password arguments should not be led with '--'

https://github.com/Microsoft/vsts-agent-cookbook/blob/ac16dbdef613254b9685359addd80d9d2278835e/libraries/helpers.rb#L72

When this happens it is rendered as ----username and ----password, which the agent config does not recognize. Snipping the '--' out here should work just fine.

Add proxy Support for vsts agent

We need to run the agent behind the proxy. We need an option to specify proxy-url. Right now cookbook won't support proxy.