Lima's dependency stability concern

[KevinLiAWS]

Hi, I would like to ask about how Lima deal with dependency stability, e.g. Lima depends on QEMU, and if installed via homebrew, by default it will install all dependencies of their latest version, and homebrew is not support install a historical version. In this case, by any chance QEMU introduced some breaking changes or security bugs, Lima will be impacted.
For this scenario, I would like to hear people's thoughts on the dependency management, and if dependency breaks, what will be the action for Lima. Does it mean only if dependency package is fixed, Lima will be back to healthy? Or is there other way to increase the stability? Thanks.

[AkihiroSuda]

Lima follows the latest Homebrew version of QEMU.
When there is a breaking change we release a new version of Lima ASAP.

[jandubois]

The releases of lima-and-qemu are just the tarballs of lima with all the dependencies: https://github.com/rancher-sandbox/lima-and-qemu/releases

These releases are used by Rancher Desktop to bundle lima (and qemu), effectively pinning the version.

The build script for lima-and-qemu uses dtrace to collect only the files needed to run lima, so the size is a lot smaller than a complete qemu with all prerequisites, which iirc would add up to a full gigabyte.

You are of course free to use these releases for anything you want, but they are custom-made for Rancher Desktop releases, so don't always perfectly match the lima releases.

[KevinLiAWS]

Hi @jandubois , really thank you for your instroduction for lima-and-qemu, which seems to be our ideal option. We have some follow up questions regarding the lima-and-qemu:

1. In Rancher, how the env PATH or symlink is configured? such as how lima can find qemu binary and same with other deeper level dependencies?
2. If the user have qemu installed via homebrew previously, will the pinned version inside of lima-and-qemu has conflict with the previous installed one via homebrew?

[jandubois]

Rancher Desktop always puts the directory that limactl is installed into first on the PATH:

https://github.com/rancher-sandbox/rancher-desktop/blob/main/src/backend/lima.ts#L677-L678

The 2 qemu executables qemu-img and qemu-system-* are installed in the same directory, so limactl will invoke these versions.

The bundled qemu binaries are built so they resolve their dylib dependencies relative to the path of the executable instead of using the /usr/local/bin path used by Homebrew:

$ ls -l /Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/lima/bin/
total 57248
-rwxr-xr-x  1 jan  admin  14016032  5 Oct 18:08 limactl
-rwxr-xr-x  1 jan  admin   1913712  5 Oct 18:08 qemu-img
-rwxr-xr-x  1 jan  admin  13377424  5 Oct 18:08 qemu-system-x86_64

$ otool -L /Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/lima/bin/qemu-system-x86_64
/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/qemu-system-x86_64:
	/System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor (compatibility version 1.0.0, current version 1.0.0)
	@executable_path/../opt/pixman/lib/libpixman-1.0.dylib (compatibility version 41.0.0, current version 41.0.0)
	@executable_path/../opt/snappy/lib/libsnappy.1.dylib (compatibility version 1.0.0, current version 1.1.9)
	@executable_path/../opt/lzo/lib/liblzo2.2.dylib (compatibility version 3.0.0, current version 3.0.0)
[...]

That means the whole directory can be installed anywhere, and the libraries will still be located correctly.

The bundled version of lima and qemu should not interfere with the Homebrew version as long as you don't put the internal Rancher Desktop directory on the PATH. And Rancher Desktop will always use the bundled versions of the tools, so everything should be completely separate (by design).

Rancher Desktop also sets LIMA_HOME so the VM is invisible from the Homebrew limactl and vice versa, and has therefore its own set of override.yaml, default.yaml, networks.yaml configurations.

We want the user to be able to independently install and uninstall Homebrew Lima and Rancher Desktop in any order, and they should never conflict with each other, or share any kind of configuration.

[vsiravar]

Thanks for providing this detailed approach to upstream dependency management. We are trying to use a similar approach in our project to snapshot dependencies.

The build script for lima-and-qemu uses dtrace to collect only the files needed to run lima

I was curious about how you managed to run dtrace on ARM64 self hosted runner? Did you have to disable SIP manually on the cloud VM on the self hosted runner. Thanks!

[jandubois]

I was curious about how you managed to run dtrace on ARM64 self hosted runner? Did you have to disable SIP manually on the cloud VM on the self hosted runner.

Yes, you have to disable SIP, so it is running on a self-hosted machine. It is also triggered manually because I found that dtrace can lock up the machine, which is inconvenient because the machine is still remote from me. So I manually reboot it (remotely) before each build...

I wish this could be automated better, but we don't rebuild this release too frequently, so it has been ok for now.

[afbjorklund]

You could use a LTS distribution to install Lima and dependencies, instead of something rolling like Homebrew ?

For some platforms, that might include rolling your own (distro), though. If packages are not available.

[AkihiroSuda]

Btw I guess it is ok to have limactl install-qemu command that installs a recommended version of QEMU and its deps (from Homebrew snapshots or somewhere)

This is not a silver bullet, though, as we can't pin the versions of the OS components such as Hypervisor.framework and vmnet.framework (and Virtualization.framework, in the future)