Some more detail on this thread; we updated the chef client on ~130 nodes today from 12.18 to 13.6.4 without issue. However, ~6 nodes failed to check back in at all. As others have reported the Chef_upgrade scheduled task exists still and fails when executed. Running the upgrade script fails due to a locked file, chef-log.dll which is located in C:\opscode\chef\embedded\lib\ruby\gems\2.3.0\gems\chef-12.18.31-universal-mingw32\ext\win32-eventlog, otherwise the rest of the \opscode\chef directory has been deleted already. Because that dll is locked by the svchost that hosts the EventLog service, a reboot is the only way I've been able to workaround this.

We use the chef client cookbook to configure the chef client. We are not doing anything special in terms of trigger the chef client. But, we do set the log_location to :win_evt which I suspect is somehow related to this issue.

from chef_client_updater.

Hey there @dheerajd-msys , I am still experiencing this problem on Windows 10, chef-client 15.5.17 downgrading to 15.5.16 via [email protected]. Even after observing the "upgrade chef" scheduled task run with a 0x1 return code, observing the chef_upgrade.log looks successful, waiting for 30 minutes, rebooting, I end up with 'chef-client' is not a recognized as an internal or external command, operable` and a non working chef-client.

I'm wondering if you could provide recommendations on next steps?

from chef_client_updater.

You are interrupting the upgrade mid-way. The upgrade on Windows is not very elegant. You should not interfere this process.

Wait for the scheduled task to do its thing. It will keep triggering until no file locks exist anymore in the c:\opscode\chef folder. After that, the new version is installed.

This can take up to 15 minutes. So don't touch it for at least that time.

chef-client is not a valid command during that time on purpose. This prevents you to run the old version during the upgrade and interfere the upgrade process.

from chef_client_updater.

Well, I waited for much longer than 15 minutes initially. The second test (different system), I left overnight, with the same result.

After the upgrade runs, chef never works again until it is manually installed and the system rebooted.

from chef_client_updater.

What version are you upgrading from?

from chef_client_updater.

Any 12.X version. We have different versions in our environment. I can try one at random and let you know the version, but it happens on all of them.

from chef_client_updater.

Yesterday, I had the same behavior trying to upgrade from 13.5.3 to 13.6.4.

I waited far longer than 15 minutes. I noticed a scheduled task was created for upgrading the client, but it showed that it had errored; executing it again yielded the same result. I had to completely remove the Opscode folder and reinstall chef client.

from chef_client_updater.

We had a PR merged and released that fixes an issue with Powershell execution. It would be great if the folks here could give that a try and report back.

from chef_client_updater.

@tas50

================================================================================
Error executing action update on resource 'chef_client_updater[Install latest]'

SystemExit

exit

Cookbook Trace:

c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:176:in exit' c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:176:in run_post_install_action'
c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:295:in block (2 levels) in class_from_file' c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:294:in block in class_from_file'

Resource Declaration:

In c:/chef/cache/cookbooks/fisher-server-cheffiles/recipes/default.rb

10: chef_client_updater 'Install latest' do
11: version '13'
12: end
13:

Compiled Resource:

Declared in c:/chef/cache/cookbooks/fisher-server-cheffiles/recipes/default.rb:10:in `from_file'

chef_client_updater("Install latest") do
action [:update]
updated true
updated_by_last_action true
retries 0
retry_delay 2
default_guard_interpreter :default
declared_type :chef_client_updater
cookbook_name "fisher-server-cheffiles"
recipe_name "default"
version "13"
post_install_action "kill"
exec_command "c:/opscode/chef/bin/chef-client"
exec_args []
end

System Info:

chef_version=12.21.26
platform=windows
platform_version=6.3.9600
ruby=ruby 2.3.5p376 (2017-09-14 revision 59905) [x64-mingw32]
program_name=c:/opscode/chef/bin/chef-client
executable=c:/opscode/chef/bin/chef-client

from chef_client_updater.

How large is your c:\opscode\chef folder after running the chef_upgrade task?

from chef_client_updater.

545MB

from chef_client_updater.

After the upgrade:
What version does Add/Remove programs state?
Do you still have a c:\opscode\chef_upgrade.ps1 on disk?
What is the last exit code of the chef_upgrade scheduled task?
Do you have errors in your Chef log after the last run on the old Chef version throws the system exit?

from chef_client_updater.

I'm also seeing this problem on a Windows Server 2008 R2 box upgrading to 13.6.4.

What version does Add/Remove programs state? - 13.6.4
Do you still have a c:\opscode\chef_upgrade.ps1 on disk? - YES
What is the last exit code of the chef_upgrade scheduled task? - 1
Do you have errors in your Chef log after the last run on the old Chef version throws the system exit? - NO

from chef_client_updater.

Sorry Robert, I had a lot of other stuff going on, I will update you with that info shortly.

from chef_client_updater.

What output do you see when you run the chef_upgrade.ps1 manually in an elevated Powershell console?

from chef_client_updater.

ModuleType Version Name ExportedCommands

Script 0.0 Omnitruck {Get-ProjectMetadata, Install-Project, install}
Installing chef from C:\Users\USER~1\AppData\Local\Temp\chef-client-13.6.0-1-x64.msi
msiexec was not successful. Received exit code 1603
At C:\opscode\chef_upgrade.ps1:447 char:5

• throw "msiexec was not successful. Received exit code $($p.ExitCo ...
  

• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : OperationStopped: (msiexec was not... exit code 1603:String) [], RuntimeException
  • FullyQualifiedErrorId : msiexec was not successful. Received exit code 1603

from chef_client_updater.

You are downgrading from 13.6.4 to 13.6.0.
This is not supported on Windows in the current release.

I do have a pull request open to fix this though, check out #77

from chef_client_updater.

Yep, I was using the knife bootstrap command which was installing the latest client (13.6.4) and then my cookbook specified version 13.6.0. I avoided issue by specifying the version in the bootstrap command "--bootstrap-version 13.6.0". Thanks Robert!

from chef_client_updater.

@RobbertJanSW

Before: 12.19.36

After the upgrade:
What version does Add/Remove programs state? 12.19.36
Do you still have a c:\opscode\chef_upgrade.ps1 on disk? Yes
What is the last exit code of the chef_upgrade scheduled task? Not sure where to find this. It says in scheduled tasks that it ran successfully.
Do you have errors in your Chef log after the last run on the old Chef version throws the system exit? Not sure what you mean. Let me describe the process:

1. Chef 12.19.36 is installed, runs fine every time.

2. Add the following to the cookbook:

chef_client_updater 'Install latest Chef 13.x' do
  version '13'
end

3. Run chef. Fails:

- upgrade chef-client 12.19.36 to 13.6.4[2017-11-22T13:29:36-05:00] WARN: New chef-client installed. Forcing chef exit!

 ================================================================================
 Error executing action `update` on resource 'chef_client_updater[Install latest Chef 13.x]'
 ================================================================================

 SystemExit
 ----------
 exit

 Cookbook Trace:
 ---------------
 c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:176:in `exit'
 c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:176:in `run_post_install_action'
 c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:295:in `block (2 levels) in class_from_file'
 c:/chef/cache/cookbooks/chef_client_updater/providers/default.rb:294:in `block in class_from_file'

 Resource Declaration:
 ---------------------
 # In c:/chef/cache/cookbooks/fisher-server-websec/recipes/default.rb

  11: chef_client_updater 'Install latest Chef 13.x' do
  12:   version '13'
  13: end

 Compiled Resource:
 ------------------
 # Declared in c:/chef/cache/cookbooks/fisher-server-websec/recipes/default.rb:11:in `from_file'

 chef_client_updater("Install latest Chef 13.x") do
   action [:update]
   updated true
   updated_by_last_action true
   retries 0
   retry_delay 2
   default_guard_interpreter :default
   declared_type :chef_client_updater
   cookbook_name "fisher-server-websec"
   recipe_name "default"
   version "13"
   post_install_action "kill"
   exec_command "C:/opscode/chef/bin/chef-client"
   exec_args []
 end

 Platform:
 ---------
 x64-mingw32

4. Wait 20 minutes
5. Refresh program list. Chef is still listed as 12.19.36.
6. Look at scheduled tasks - there is a chef_upgrade task that says run result "succeeded" every minute for the past 20 minutes.
7. Try to run chef client manually. Result:

chef-client : The term 'chef-client' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ chef-client
+ ~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (chef-client:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

8. Give up

from chef_client_updater.

@RobbertJanSW

FYI, after reboot, chef works and is upgraded.

Is this intended? I would prefer not to have to reboot 200 systems to update the chef client. I believe that it is not necessary to reboot if you simply run the MSI yourself, correct?

from chef_client_updater.

@blackdynamite595

No, definitely not intended. No reboot is needed.
I've got the feeling your upgrade is performed right after your reboot by the scheduled task.
This would mean it fails every minute until your reboot. But that would mean your Scheduled Task would have a non-0 last result exit code (please double-check the exit code number)

You can also try and run the chef_upgrade.ps1 manually in an elevated powershell console to see what output it returns. This should tell you why the upgrade fails.

Most of the time; this is due to locked files. Although these should normally clear within 20 minutes, it might explain why the upgrade succeeds after a reboot. Are you by any change triggering the Chef client very often in an unusual way?

from chef_client_updater.

@rdean716
I haven't looked into if this is a problem, but I'm experiencing the same issue and I have log_location set to WinEvt as well.

from chef_client_updater.

@RobbertJanSW

Sorry, this really left my radar for a few months.

I just ran the cookbook again in hopes that a newer version would work for me. Unfortunately it does not. Here are my circumstances:

Currently on chef client v13.6.4, Server 2012R2.

I see "upgrade chef-client 13.6.4 to 13.8.5 Warn: new chef-client installed and exit is allowed. Forcing exit...."

Then the chef run fails, saying a batch file cannot be found.

Trying chef-client after tells me it is not recognized. If I look in programs and features, chef 13.6.4 is still installed.

Now chef is broken even after reboot and I still get "chef-client : The term 'chef-client' is not recognized as the name of a cmdlet, function, script file, or operable program."

from chef_client_updater.

@blackdynamite595

Could you please provide this info:

• Do you (still) have a scheduled task called Chef_upgrade?
• What is the last exit code / status of this scheduled task?
• Do you use winevt logging or logging to logfile?
• Do you still have a c:\opscode\chef_upgrade.ps1 ?
• If possible, please run this test on one node: open a powershell console and run the c:\opscode\chef_upgrade.ps1 manually. What does it show on your console?

from chef_client_updater.

I have a similar issue. I'm logging to file but use nxlog for event log capture. This is holding a lock to chef-log.dll and so prevents the deletion of \opscode\chef. Restarting nxlog is sufficient to release the lock. Renaming the chef directory also allows the upgrade to complete. It might be an idea to do a rename (to something unique), then delete that folder. Leaving fragments behind isn't nice but better than a zombie server.

NB: This is with cookbook version 3.2.5

from chef_client_updater.

@RobbertJanSW thanks for your continued responses!

Do you (still) have a scheduled task called Chef_upgrade?

Yes

What is the last exit code / status of this scheduled task?

Task Scheduler successfully finished "{0068c131-abbc-4021-b8df-0718e8097cd4}" instance of the "\Chef_upgrade" task for user "NT AUTHORITY\SYSTEM". Operational Code (2)

Do you use winevt logging or logging to logfile?

Our chef logs are in C:\chef\log\ or do you mean in general?

Do you still have a c:\opscode\chef_upgrade.ps1 ?

Yes

If possible, please run this test on one node: open a powershell console and run the c:\opscode\chef_upgrade.ps1 manually. What does it show on your console?

Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby\gems\2.3.0\gems\chef-12.21.26-universal-mingw32\ext\win32-eventlog\chef-log.dll: Access to the path 'chef-log.dll' is denied.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : PermissionDenied: (chef-log.dll:FileInfo) [Remove-Item], UnauthorizedAccessException
    + FullyQualifiedErrorId : RemoveFileSystemItemUnAuthorizedAccess,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby\gems\2.3.0\gems\chef-12.21.26-universal-mingw32\ext\win32-eventlog: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (win32-eventlog:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby\gems\2.3.0\gems\chef-12.21.26-universal-mingw32\ext: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (ext:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby\gems\2.3.0\gems\chef-12.21.26-universal-mingw32: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (chef-12.21.26-universal-mingw32:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby\gems\2.3.0\gems: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (gems:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby\gems\2.3.0: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (2.3.0:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby\gems: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (gems:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib\ruby: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (ruby:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded\lib: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (lib:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef\embedded: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (embedded:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\opscode\chef: The directory is not empty.
At C:\opscode\chef_upgrade.ps1:8 char:11
+           Remove-Item "c:/opscode/chef" -Recurse -Force
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (C:\opscode\chef:DirectoryInfo) [Remove-Item], IOException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand

from chef_client_updater.

Is this issue resolved? I am using the latest version of chef_client_updater cookbook (3.2.6) and upgrade runs fine but fails to run chef-client service.

from chef_client_updater.

As an aside I've used this technique in order to stop my nxlog service if a chef update is set to run. I couldn't subscribe to the chef_client_updater resource as the hard exit prevents notifications (but not delayed resources)

if node.key?('chef_client_updater')
  log 'Make powershell script run at end of run' do
    level :debug
    notifies :run, 'powershell_script[Check for Chef upgrade]', :delayed
  end
  powershell_script 'Check for Chef upgrade' do
    code <<-EOH
    schtasks /Query /FO LIST /V /TN Chef_upgrade | out-null
    if ($LastExitCode -eq 0)
    {
      Stop-Service nxlog -ErrorAction SilentlyContinue
    }
    exit 0
    EOH
    action :nothing
  end
end

from chef_client_updater.

Closing this issue as it seems fixed in #149.

from chef_client_updater.

Using chef_client_updater version 3.8.2 oOn windows : upgrading from chef-client-15.5.17-1-x64 to chef-client-15.6.10-1-x64. It updates then errors :

Recipe: chef_client_updater::default

• chef_client_updater[update chef-client] action update

  • chef_gem[mixlib-install] action install

    • install version ~> 3.11 of package mixlib-install

  • remote_file[C:\chef\cache/handle.zip] action create (skipped due to not_if)

  • powershell_script[name] action runSUCCESS: The scheduled task "Chef_upgrade" has successfully been created.

    • execute "C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NonInteractive -NoProfile -ExecutionPolicy Bypass -InputFormat None -File "C:/Users/xxx/AppData/Local/Temp/chef-script20200127-74840-1gx0nco.ps1"
  • upgrade chef 15.5.17 to 15.6.10[2020-01-27T10:52:31+00:00] WARN: New chef-client installed and exit is allowed. Forcing chef exit!

Running handlers:
[2020-01-27T10:52:31+00:00] ERROR: Running exception handlers
Running handlers complete
[2020-01-27T10:52:31+00:00] ERROR: Exception handlers complete
Chef Infra Client failed. 2 resources updated in 01 minutes 10 seconds
[2020-01-27T10:52:32+00:00] FATAL: Stacktrace dumped to C:/chef/cache/chef-stacktrace.out
[2020-01-27T10:52:32+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
The batch file cannot be found.

chef_upgrade.log

[] Destroying open Chef handles.
[] Completed destroying open Chef handles.

ModuleType Version Name ExportedCommands

Script 0.0 Omnitruck {Get-ProjectMetada...

Only folder in opscode :

C:\opscode\chef.upgrade

Tried running the Chef_upgrade scheduled task not resolved.
To recover I deleted the conntents of C:\opscode\ & the Chef_upgrade task then ran a repair on the previous chef-client-15.5.17-1-x64.msi install.

Any ideas?

from chef_client_updater.