Remote Docker Container Execution
Hi all,
I have an entire DSS ecosystem installed on various VirtualBox VMs (design node, automation node, api deployer, api nodes, etc.). I also created a separate node to test out remote docker container execution, but am having some trouble getting this to work.
I understand this would be difficult to troubleshoot without having access to every single config file / package version / etc., what I'm looking for though is if someone can give me an idea of if these high level steps are correct or if I'm doing anything obviously wrong.
My steps:
- Installed docker on my design node (ip: 10.0.2.5)
- Built the base image on my design node
- Installed docker on my server where I want to do remote docker execution (ip: 10.0.2.10)
- (Also, I already know these servers can ping each other, so I know it isn't a VBox network issue, and neither box has a firewall enabled)
- Enabled the docker daemon on my design node to be a docker swarm manager (so I can create overlay docker networks which I assume is necessary for docker daemons on different servers to communicate over a network? Big assumption here)
- Created an "overlay" docker network on the design node named "dss-net"
- Connected to this dss-net overlay docker network from the remote execution server (10.0.2.10)
At this point, when I go to Settings -> Containerized Execution and leave the "new config" blank, I get a message that says my test passes and my DSS base image is fine. But I'm not interested at all in running in a docker container on the same server as my design node. I want to run on a remote server. I can't seem to get any combination of configuration settings working to set up remote docker container execution.
Questions:
- Did I get this all backwards? Was my remote execution box (10.0.2.10) supposed to be where I set up the overlay network? And then should I have connected to that from the docker daemon on my design node box (10.0.2.5)?
- Are there any other glaring mistakes present in what I've laid out as my approach so far?
- What other "gotchas" might I need to check?
- What other information would I need to provide to continue with the next troubleshooting steps?
Thank you,
-Taylor
Answers
-
Hi Taylor,
My head hurts just reading about your config.
Working with VBox VM's and network connectivity is always painful. I think your first bullet is right, the overlay network should be on the Docker Server. You can eliiminate DSS from your testing and just verify that you can submit a container to the Docker server from the DSS node from command line. If it works that way, DSS should work. At least from my experience in other areas.
Good luck. I'm curious to see your solution.
-
Hi Taylor,
I assume you have your good reasons for such a setup
To use remote docker you don't need the overlay network.
Make sure that:- your vms can reach the remote docker server, as long as containers running on it. Both ways, from DSS to containers and vice-versa;
- To use remote docker you will still need a registry, because the images that you build on the design nodes needs to be available to the remote docker. Your remote server can provide the registry too, it's not an intensive service. Have a look here;
- On the remove docker what you need to do is make sure docker is listening on all network interfaces;
- In DSS, you have to configure a "docker" containerized configuration. Provide your docker registry address (IP:PORT) and the remote docker (also IP:PORT);
- Normally, "docker network" is host (literally the word host), and "docker runtime" is empty (I am assuming you are not going too exotic);
- Push the images to the registry and then see if you can offload a recipe to the config you just created.
If you have issues, chances are they are non the network side, so triple check:
- vms and containers can talk to each other (note that ping might not be enough in this case, use telnet instead);
- there is no firewall and the communication is open on all ports;
Take care,
Omar
Architect @ Dataiku
