[Idea/Suggestion] Use private IP with rancher in different network about docker-machine-driver-hetzner HOT 23 OPEN

I wrote down my setup here: https://tech.mecodia.cloud/2021/01/12/running-a-tightly-integrated-hetzner-cloud-kubernetes-with-rancher/

from docker-machine-driver-hetzner.

Hi,

sorry if my questions are silly, but I have 0 experience with ranger. So you are basically suggesting that when --hetzner-use-private-network is active, the machine's IP should be the private address but the SSH hostname should still be set to the public IP?
Implementing this should not be a problem, but I would suggest adding another flag --hetzner-force-public-ssh or something instead, so we don't have a breaking change here.

That being said, I have no idea whether docker-machine supports this scenario for provisioning. At least commands like docker-machine env seem to fail when the machine's port 2376 is not reachable, which would be the case, if you are in a completely different network. Being able to create a machine, where only removal, starting, stopping and ssh-ing into works, seems to kind of defy the purpose of docker-machine IMO.

So the question really is, is this within scope for a docker-machine driver or is this something that should happen after docker provisioning?

@mxschmitt you are the ranger expert here, any thoughts?

from docker-machine-driver-hetzner.

New flag would make more sense in this case yes. I forgot about 2376 port, but that port could be open just like 22 etc, since it's protected with cert authentication it shouldn't be a big security concern.

Rancher / kubernetes would then use the private IP for etcd/controplane communication and other services which would make firewall management easier. There are some workarounds right now namely the one done here but in this case the traffic still goes through the public interface.

from docker-machine-driver-hetzner.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from docker-machine-driver-hetzner.

Hi, any update on this? I opened a related issue in mxschmitt/ui-driver-hetzner#104 about this.
It would be awesome if we could use the private network for communication between nodes, but not for the connection between Rancher and the nodes, so we could keep things in separate projects. Thanks!

from docker-machine-driver-hetzner.

@boris-savic HI, did you find a workaround? Thanks

from docker-machine-driver-hetzner.

I think this is the relevant line?

Why does it use the private IP if the private network is enabled? It looks like changing that line to use the public IP would be enough? @JonasProgrammer @mxschmitt

from docker-machine-driver-hetzner.

@boris-savic HI, did you find a workaround? Thanks

Unfortunately I did not.

from docker-machine-driver-hetzner.

@boris-savic HI, did you find a workaround? Thanks

Unfortunately I did not.

OK. I am not familiar with Go to be honest but I am going to fork and try changing that line if the flag --hetzner-force-public-ssh is passed.

from docker-machine-driver-hetzner.

Hi, as stated before I'm still unsure whether this is something the lower-level docker provisioner (i.e. docker-machine) should care about, but contributions are always welcome! You can try and see if this fixes your use case, but if you change the line right there, the ip of the machine will the public one, if ever queried (i.e. as it would have been without --hetzner-use-private-network.

As @boris-savic suggested, perhaps adding a flag to force only getSSHHostName() to return the public ip might be an idea, but I'm still not convinced, whether this will actually help with the ranger configuration upstream.

from docker-machine-driver-hetzner.

Hi @JonasProgrammer what are the implications if we return the public IP on that line - only if ForcePublicSSH or some flag is set to true?

from docker-machine-driver-hetzner.

If the private IP is returned there, Rancher will connect to the nodes using the private IP, meaning that Rancher and all the clusters need to be in the same private network and thus same project. If we just return the public IP there I think the issue is solved.

from docker-machine-driver-hetzner.

The IPAddress field is not part of the hetzner specific driver part, but of the base code for all docker machine drivers. Existing APIs, such as GetIP() rely on this. If you hard-code it to return the public IP, it will be used everywhere -- as if --hetzer-use-private-network was never given.

The PR's author, that introduced the line in question, originally inteded the flag to be used so you have a 'provisioning machine' within the network IIRC, so yes, even returning it as the SSH host to connect to makes sense for this use case. IMHO just having a flag to override the SSH host is the way to go,

from docker-machine-driver-hetzner.

I've added a flag and am doing this:

func (d *Driver) GetSSHHostname() (string, error) {
	if d.ForcePublicSSH {
		return d.PublicIPAddress, nil
	} else {
		return d.GetIP()
	}
}

Would this be OK?

from docker-machine-driver-hetzner.

I have added the public IP property too. I am new to Go please let me know if there is a more elegant way.

from docker-machine-driver-hetzner.

Uhm I made that change and was going to test it that way, but I just checked the code at https://github.com/mxschmitt/ui-driver-hetzner and I don't understand how it works because it doesn't seem to set flags for the driver.... So how/where does this happen? 🤔

from docker-machine-driver-hetzner.

The values here https://github.com/mxschmitt/ui-driver-hetzner/blob/7932c861aeaa7ded4873dce3ba0c323afc7662dc/component/component.js#L48-L55 get converted to kebab case and then passed to the driver as CLI parameters.

You want to add a new text field?

from docker-machine-driver-hetzner.

Hi @mxschmitt thanks for the clarification. Those variables seemed in a different format so I was confused :) I just need to pass a boolean flag "forcePublicSSH" according to the changes I want to test in the driver. How should I name the variable then? "forcePublicSSH" or "forcePublicSsh"?

from docker-machine-driver-hetzner.

Update: I first changed the driver so it only uses the public IP for the SSH hostname as you suggested, it worked in the beginning when creating the cluster but then it failed because something (I think Rancher) was trying to connect to the port 2376 on the private IPs. Just to try, I changed the code to always return the public IP of the machines and everything worked just fine. The cluster was deployed with Rancher in a different project, and I could configure Kubernetes to use the private interface for the traffic between nodes only.

So... is it possible to ensure that the connections to the port 2376 are done to the public IP (when the aforementioned flag is set)? If yes, how? That would fix the remaining issue.

Thanks @JonasProgrammer @mxschmitt

from docker-machine-driver-hetzner.

I gave up because I want to use the node driver now. I would have preferred to keep things in separate projects but I can leave with a single project for now. @LKaemmerling also recommends this...

I set up Rancher and a cluster with the docker driver and the node driver as they are now and everything seems to work.

from docker-machine-driver-hetzner.

Looking at: https://github.com/rancher/machine/blob/094ab1c372c598f945c1e9f2bbb1d0816dc68751/drivers/google/google.go#L277

It seems that we have to return the Public IP in getIP and getURL when we set the flag.
The little documentation that is available also suggests so:

https://github.com/rancher/machine/blob/master/libmachine/drivers/drivers.go

from docker-machine-driver-hetzner.

Together with: https://github.com/hetznercloud/hcloud-cloud-controller-manager
I think this should work as expected. In that case, we don't even have to set the private network first, since the hetzner-cloud-controller-manager will set the internal and external IPs. Which makes this PR pretty "überflüssig".

from docker-machine-driver-hetzner.

Finally everyone, I think I found a solution that works for the use case we all want. Use the external IP for Rancher Communication while using the private Network and the internal IP for in Cluster Communication.

Setup a Node Template with a private Network, but don't set the private network as first. Then setup a cluster using the Hetzner Cloud-Controller and define the used network in the used Secret: I added a pull request for this usecase.
hetznercloud/hcloud-cloud-controller-manager#144
You can also use Rancher user addons when deploying the Cluster: https://rancher.com/docs/rke/latest/en/config-options/add-ons/user-defined-add-ons/

You don't have to use the pod network feature of the cloud controller in that case but it will set the external and internal IPs of the nodes correctly. @vitobotta I hope this is what you were looking for.

This way we can use a central Rancher Instance that will talk with the public IPs and the cluster will use the Hetzner Network internally. Please correct me if I am wrong, I am exhausted by reading shitty docs and spaghetti Open Source Code all weekend. But thank god for open source.

from docker-machine-driver-hetzner.