<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clip

As far as I understood from <a href="https://github.com/container-storage-interface/sp

Was fixed with <a class="issue-link js-issue-link" data-error-text="Failed to load tit

AttachVolume.Attach: Don’t return internal error when server is locked about csi-driver HOT 5 CLOSED

hetznercloud commented on June 28, 2024 2

AttachVolume.Attach: Don’t return internal error when server is locked

from csi-driver.

Comments (5)

serhii-shnurenko commented on June 28, 2024 1

Yeah just faced this issue but with detach behavior.

Assume the reason is our pod have 2 mounted csi volumes and while 1-st of them detaching server becomes locked for short amount of time.

level=debug ts=2019-09-26T16:12:07.571214176Z component=grpc-server msg="handling level=debug ts=2019-09-26T16:12:07.571214176Z component=grpc-server msg="handling request" req="volume_id:\"volume_id\" node_id:\"server_id_here\" "
level=info ts=2019-09-26T16:12:07.571564951Z component=api-volume-service msg="detaching volume from server" volume-id=volume_id server-id=server_id_here
level=info ts=2019-09-26T16:12:07.705827634Z component=api-volume-service msg="failed to detach volume" volume-id=volume_id err="cannot perform operation because server is locked (locked)"
level=error ts=2019-09-26T16:12:07.705971793Z component=grpc-server msg="handler failed" err="rpc error: code = Internal desc = failed to unpublish volume: cannot perform operation because server is locked (locked)"

@thcyron
Assume it should be done here. Or do you have a better idea to fix this bug for both Detach and Attach actions?

csi-driver/driver/controller.go

Lines 196 to 207 in 8da058c

 if err := s.volumeService.Detach(ctx, volume, server); err != nil { 

 code := codes.Internal 

 switch err { 

 case volumes.ErrVolumeNotFound: 

 code = codes.NotFound 

 case volumes.ErrServerNotFound: 

 code = codes.NotFound 

 case volumes.ErrLockedServer: 

 code = codes.Aborted 

 } 

 return nil, status.Error(code, fmt.Sprintf("failed to unpublish volume: %s", err)) 

 }

from csi-driver.

serhii-shnurenko commented on June 28, 2024 1

As far as I understood from this specification

ABORTED code means that action already invoked and running for this volume. But "server_locked" error doesn't mean action already running for this volume, it can be another volume currently attaching/detaching to/from the server causing it to be locked.

I have no experience in go, but for me, it looks like if we returning ABORTED code Publish/Unpublish action will not be retried, and will mean volume is detached.

I believe it should work next way:
verifying the server isn't locked and wait till server not locked -> unpublish/publish volume -> in case got server locked error, go to the beginning, of course with some meaningful retries limit.

A little test
Decided to test if error reproducible using hcloud cli tool. For this purpose created 1 server and 3 volumes in the empty project we are using for testing (IP redacted because some time ago hetzner had assigned IP of the previously destructed instance to the new machine):

$ hcloud server list
ID        NAME                        STATUS    IPV4             IPV6                     DATACENTER
3360348   testing-volume-concurency   running   116.203.***.**   2a01:***:***:****::/64   nbg1-dc3
$ hcloud volume list
ID        NAME        SIZE    SERVER   LOCATION
3323542   test-vol1   10 GB   -        nbg1
3323544   test-vol2   10 GB   -        nbg1
3323545   test-vol3   10 GB   -        nbg1

Here the code snippet I used to verify this error can occur due to concurent opperations running against same instance:

hcloud volume attach --server <server_id> <vol_name_1> &
hcloud volume attach --server <server_id> <vol_name_2> &
hcloud volume attach --server <server_id> <vol_name_3> &
wait

And what I've got in the output:

$ hcloud volume attach --server 3360348 test-vol1 &
[1] 8153
$ hcloud volume attach --server 3360348 test-vol2 &
[2] 8154
$ hcloud volume attach --server 3360348 test-vol3 &
[3] 8155
$ wait
hcloud: cannot perform operation because server is locked (locked)
hcloud: cannot perform operation because server is locked (locked)
[1]   Exit 1                  hcloud volume attach --server 3360348 test-vol1
[2]-  Exit 1                  hcloud volume attach --server 3360348 test-vol2
   1s [====================================================================] 100%
Volume 3323545 attached to server testing-volume-concurency
[3]+  Done                    hcloud volume attach --server 3360348 test-vol3

Results: Only test-vol-3 got mounted on the server, vol1 and vol2 attachments rejected by the hetzner doe to "server is locked" error.

To not make this message too long, want to mention the same behavior observed while detaching multiple volumes in a concurrent manner.

from csi-driver.

thcyron commented on June 28, 2024

According to the code you posted, we already return error code aborted in case the server is locked:

 	case volumes.ErrLockedServer: 
 		code = codes.Aborted

That that doesn’t seem to work. Would need to debug this.

from csi-driver.

thcyron commented on June 28, 2024

I think you’re right. We don’t use the ABORTED error code correctly. I’ve creatd #63 to address this.

from csi-driver.

LKaemmerling commented on June 28, 2024

Was fixed with #84

from csi-driver.

AttachVolume.Attach: Don’t return internal error when server is locked about csi-driver HOT 5 CLOSED

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	if err := s.volumeService.Detach(ctx, volume, server); err != nil {
	code := codes.Internal
	switch err {
	case volumes.ErrVolumeNotFound:
	code = codes.NotFound
	case volumes.ErrServerNotFound:
	code = codes.NotFound
	case volumes.ErrLockedServer:
	code = codes.Aborted
	}
	return nil, status.Error(code, fmt.Sprintf("failed to unpublish volume: %s", err))
	}