bbr deployment backup fails with SSH errors when using ssh+socks5 scheme about bosh-backup-and-restore HOT 13 CLOSED

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/160167687

The labels on this github issue will be updated when the story is started.

from bosh-backup-and-restore.

Hello @markround thanks for opening the issue. This used to be bug which we recently fixed, you can see the related tracker story here.

Can you please verify that you are seeing this issue with the latest BBR 1.2.7 and not any previous cli version? We will investigate further if this is the case. Thanks

from bosh-backup-and-restore.

Hi,

This is with BBR 1.2.7 from PivNet. This is how it shows up in the Concourse step:

availability | All Users
-- | --
controlled | false
description | BOSH Backup and Restore binary
eccn | 5D002
end_of_availability_date |  
end_of_guidance_date |  
end_of_support_date | 2019-03-31
license_exception | TSU
release_date | 2018-08-15
release_notes_url | https://docs.pivotal.io/bbr/bbr-rn.html
release_type | Minor Release
version | 1.2.7

from bosh-backup-and-restore.

Thanks, we will look into it.

from bosh-backup-and-restore.

Hello @markround

What version of OpsManager are you using and how large is the environment (i.e. number of VM's)?

Thanks,
Jake & @gmrodgers

from bosh-backup-and-restore.

Hey @markround,

We just cut v1.2.8 which should make the connection reset by peer issue disappear. Can you give it a try?

Closing this but feel free to re-open if the issue persists.

from bosh-backup-and-restore.

Hi,

Just tried it with 1.2.8 and it now fails with the following error:

[bbr] 2018/09/25 23:09:33 INFO - Finished validity checks -- for job director on bosh/0...
	
[bbr] 2018/09/25 23:09:33 INFO - Finished validity checks -- for job bbr-credhubdb on bosh/0...
	
[bbr] 2018/09/25 23:34:17 INFO - Finished copying backup -- for job blobstore on bosh/0...
	
[bbr] 2018/09/25 23:34:17 INFO - Starting validity checks -- for job blobstore on bosh/0...
	
[bbr] 2018/09/25 23:49:01 INFO - Backup created of 192.168.0.10 on 2018-09-25 23:09:32.52464317 +0000 UTC m=+12.091158177
	
[bbr] 2018/09/25 23:49:01 INFO - Cleaning up...
	
[bbr] 2018/09/25 23:49:01 ERROR - Backup artifact clean up failed
	
2 errors occurred:
	
error 1:
	
1 error occurred:
	
error 1:
	
Unable to calculate backup checksum: ssh.Run failed: ssh.Stream failed: ssh.Dial failed: EOF
	

	
error 2:
	
Deployment '192.168.0.10' failed while cleaning up with error: 1 error occurred:
	
error 1:
	
Unable to clean up backup artifact: ssh.Run failed: ssh.Stream failed: ssh.Dial failed: EOF

from bosh-backup-and-restore.

:( :( :( .... reopening

from bosh-backup-and-restore.

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/160820873

The labels on this github issue will be updated when the story is started.

from bosh-backup-and-restore.

Hey Mark,

We are looking into this again. You are using the bbr cli version 1.2.8 but what version of OpsManager are you using? Also in which IAAS is your PCF foundation deployed?

Thanks,
Glen && @alamages

from bosh-backup-and-restore.

Hi,

OpsMan 2.2.6 on Azure. We've noticed that it always seems to happen during the validity checks where there's a long pause while the checksums are computed. Our guess is that as there's no output/data transferred for a long while (there's around 56GB of data in the BOSH blobstore to checksum), the SSH tunnel closes.

We've worked around this by setting up the SSH tunnel beforehand and just using a plain socks5:// scheme, and in this case we also saw a similar behaviour so had to add -o ServerAliveInterval=60 to the SSH command to keep the tunnel open while the validation step was completing.

from bosh-backup-and-restore.

Hello @markround

Thanks for the information. We were able to reproduce the issue and we are working on a fix. You can follow our progress the tracker story.

Emmanouil & @jamesjoshuahill

from bosh-backup-and-restore.

Hi @markround

This issue is different to the original issue resolved in v1.2.8. So we are closing this issue and have created a new issue: #22.

Please could you check the workaround is correct?

Best wishes
Josh and @alamages

from bosh-backup-and-restore.