feat(monitors): expose a configurable control socket for each monitor - #850
feat(monitors): expose a configurable control socket for each monitor#850Anamika1608 wants to merge 6 commits into
Conversation
✅ Deploy Preview for urunc ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Expose each monitor's control socket in the normal boot flow, so the runtime can keep talking to the VMM after the guest starts. Every monitor boots exactly as before; the only change is that its control socket stays open and reachable: - Firecracker launches with --api-sock instead of --no-api, keeping --config-file so the guest still boots from the config file. - QEMU exposes a QMP Unix socket in server mode, configured not to wait for a client before booting, alongside the disabled human monitor. - Cloud Hypervisor exposes its REST API socket (--api-socket). The socket location is configurable through a new socket_path option under a monitor's configuration, wired through MonitorConfig, ExecArgs and the state.json annotation passthrough, with a per-container default of /tmp/<id>.sock behind a DefaultSocketDir constant and a shared resolveSocketPath helper. After changeRoot, urunc creates the socket path's directory inside the monitor rootfs, so any custom path works; it fails only if the location is invalid, such as a file already existing at one of the path's components. Extend the QEMU BuildExecCmd tests to cover the new argument and document the socket_path option. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>
48afaad to
3e02d31
Compare
cmainas
left a comment
There was a problem hiding this comment.
Hello @Anamika1608 ,
thank you for the PR. I have added some comments in the code. Also, some generic comments:
- There is still the case where an admin configures the urunc to run the monitors without any socket. This might be for security reasons or because they do not require it. Therefore, if the socket path is not in the configuration, then there should be no socket (no default value).
- We need to cleanup the socket path, because in case of a container restart (e.g. pod) the monitor might fail to use the same path.
Remove the default /tmp/<id>.sock socket path. A monitor now gets a control socket only when socket_path is set in config; with no socket_path it launches with no control socket, exactly like upstream. Firecracker restores --no-api in that case. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>
Move the control socket directory creation from between changeRoot and setupUser to right after setupUser, so it runs as the monitor's user and a non-root monitor can create and use it. Also remove a stale socket left at the same path by a previous instance before the monitor binds, so a restart reusing the same socket_path does not fail to bind. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>
Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>
Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>
The QEMU QMP flag string "server,nowait" trips cspell in qemu.go and qemu_test.go. Signed-off-by: Anamika Aggarwal <anamikaagg18@gmail.com>
|
can you take a look at the new commits i have pushed, done all three changes as you said. @cmainas |
cmainas
left a comment
There was a problem hiding this comment.
Hello @Anamika1608 ,
thank you for the updates. I have added some more comments. One thing that we need to have in mind is that the socket path can be anywhere, but it should still be in an accessible places for any user and more importantly do not collide with any other file.
| monitor runs with no control socket at all; an operator can leave it unset if | ||
| they do not need the socket, or to keep a smaller attack surface. When it is | ||
| set, the monitor creates the socket inside its own (pivoted) rootfs; `urunc` | ||
| creates the directory of a custom `socket_path` there for you, so the path |
There was a problem hiding this comment.
nit: remove "for you", the documentation should use third person only.
There was a problem hiding this comment.
Also, the path can not be really anywhere. It should be in a directory that is accessible from all users (because we create it after user setup) and it should not be over existing files/directories.
| // (--api-sock <path>) so the control socket stays open for use after | ||
| // the guest has started. | ||
| // - With no socket_path, launch with --no-api (upstream default) so no | ||
| // control socket is exposed. |
There was a problem hiding this comment.
nit: comment too verbose without any benefit and in the wrong place. It should be before the if else
|
|
||
| // UsesControlSocket reports whether a monitor exposes a control socket whose | ||
| // path (socket_path) urunc must make reachable before the monitor launches. | ||
| func UsesControlSocket(vmmType VmmType) bool { |
There was a problem hiding this comment.
We should replace this with a specific call to each VMM interface implementation. If we add a new monitor in the future, we will definitely forget to update this. As an example check #850
| } | ||
| // Remove a stale socket left by a previous instance (e.g. a restart | ||
| // reusing the same socket_path) so the monitor can bind it again. | ||
| if err = os.Remove(vmmArgs.SocketPath); err != nil && !os.IsNotExist(err) { |
There was a problem hiding this comment.
The removal of the socket should not take place here, but in delete.
| // can bind its socket there. | ||
| if hypervisors.UsesControlSocket(hypervisors.VmmType(vmmType)) && vmmArgs.SocketPath != "" { | ||
| sockDir := filepath.Dir(vmmArgs.SocketPath) | ||
| if err = os.MkdirAll(sockDir, 0o755); err != nil { |
There was a problem hiding this comment.
We can be stricter here and use 0o700
Description
Expose each monitor's control socket in the normal (config-file / CLI) boot flow, so the runtime can talk to the VMM after the guest starts (graceful shutdown now, snapshots later). Each monitor still boots exactly as before — this only keeps its control socket open and reachable:
--api-sockwhile keeping--config-file, so the guest still boots from the config file and the socket stays open (drops only--no-api).-qmp unix:<path>,server,nowaitalongside the disabled human monitor.--api-socket path=<path>.The socket path is configurable via a
socket_pathmonitor field, shared across all monitors through a commonResolveSocketPathhelper, defaulting to a per-container/tmp/<id>.sock. AfterchangeRoot, urunc creates the socket path's directory inside the monitor rootfs so a custom path works; it fails only if the location is invalid (a file already exists on the path).Related issues
How was this tested?
go build ./...,go test ./pkg/... ./internal/...,gofmt,make lint, and cspell pass; the QEMUBuildExecCmdtests cover the new-qmpargument (Ubuntu 24.04 aarch64 VM, KVM).socket_path; QEMU (chttp-qemu-linux-aarch64) boots, the guest serves HTTP 200, launches with-qmp unix:/tmp/qemu.sock,server,nowait, and the QMP socket answersquery-statuswithrunning.LLM usage
claude code (opus 4.8) for the understanding of codebase, approach decisions and reviews
Checklist
make lint).make test_ctr,make test_nerdctl,make test_docker,make test_crictl).