Moving toward native sandboxes

[jonct]

I've been studying @Jip-Hop's other investigation and brain dump, on enabling native nspawn sandboxes. I focus on the gap between today's Jailmaker and tomorrow's TrueNAS SCALE — to grok what adjustments might be helpful and practical during the transition period.

The study seems to be written directly to an audience of iX developers, who have full control over the root filesystem. But the scripts mostly concern themselves with details of retrofitting native-like behavior onto the filesystem as it exists today. (Which is entirely appropriate; I just note this tension between the two goals as prologue.)

My big picture takeaways

The native usage pattern with machinectl and friends is similar to Jailmaker in a lot of ways. But it's maybe not quite similar enough to justify retraining current Jailmaker users on something new before iX shows their cards and the cycle repeats.

We could keep the familiar CLI personality essentially as-is, though the transition — and still help prepare for native usage via some of the back-end changes that we discussed.

There's no longer need for any ambitious long-view feature ideas or rich template libraries. At least not from Jailmaker.

Triage and focus

If that description is anything like a consensus, then I think a lot of complexity falls away! Jailmaker continues in its role of interfacing with the system, until iX takes over and documents another approach.

As a convenient side-effect: any complications with systemctl edit and such do not apply for our purposes. Those get put to the side as post-transition issues. To be addressed by people with decision power over the base filesystem mounts. And largely staying out of their way.

I'll propose some [possibly conflicting] implementation ideas in separate replies. I don't want to turn people off of discussing the general overview, just because I'm also muttering on the side about unvetted small-grain details.

[jonct]

Let me get my stupidest idea out of the way. 🙄 I'm not proposing this, but just reacting to the trickiest complication as noted in the project:

# Supporting `systemctl edit` is not trivial, because /etc/systemd/system contains files unrelated to Sandboxes
# This would probably require a pre-upgrade backup and post-upgrade restore in the SCALE upgrade process
# Or a process monitoring /etc/systemd/system for changes and syncing files we want to persist into $SANDBOXES_DIR

This sounds like it would be super complicated and error-prone. It might be only modestly complicated and error-prone?

During Pre-Init you could make /etc/systemd a git repository with an immediate commit. Then use git status to itemize later changes. Whitelist specific areas of interest using .gitignore. For persistent changes, squirrel away a patch somewhere — that's when it might get tricky. But everything up to that point is harmless, and convenient for observation.

Of course you could use .path units to trigger on any changes, if that proved necessary.

[Lockszmith-GH]

what you are describing sounds like what chezmoi is doing, check it out.

[jonct]

Very cool! 🤍 But again: they're applying traditional continuous management workflow to a persistent directory — generating historical record and providing remote sync. Despite sharing one building block with them, this would do none of that.

Here, initializing a new transient git repo at each boot would not inherit the UUID to share history with the previous session. It would only supply a convenient local audit of changes since boot. (Which is also the necessary input to your hypothetical symlinking script.)

I see this as more comparable to, say, tripwire. But with the additional option to replay changes after a restart.

The restart, of course, resets all of /etc/systemd back to a safe clean slate. A pre-init startup script (if run at all) could either 1) resume the last session and its running jails, or 2) start a new session where they're available but not enabled. Simply by whether or not it applies the patch representing the previous state of /etc/systemd.

[Lockszmith-GH]

generating historical record and providing remote sync

that is does not do, it can do it with git as it's backend, but at it's core, it's a tempalting engine - you have a tamplated copy in one location and it provides syncing mechanism.

From what I understand, you're interested only in the 'apply' from the template to the etc directory. (which is similar to what tripwire does)
So the idea of keeping a safe copy somewhere and with a single command applying the preserved state to the volatile location - seems like a great match

[jonct]

From what I understand, you're interested only in the 'apply' from the template to the etc directory. (which is similar to what tripwire does)

I keep describing that action as the optional, audacious next step. But since tripwire only ever compares hashes and doesn't actually copy anything, I'll say yes: you've essentially got it. Thanks for your patience with me.

[Lockszmith-GH]

The reason I'm suggesting an existing tool, even if it's not going to be used - it can be very easy to 'wireframe' and/or demo it with some fiddling instead fo trying to reinvent something.

[jonct]

Now to perhaps a less stupid idea. It's also about directories and persistence. And is again temporary until iX has their own long-view plan ready and available.

Bootstrapping

Let's assume that there's a tool or a script in charge of preparing for sandbox usage. Maybe that's Jailmaker; maybe it's sandboxes-patch or something else.

One thing that this will need to do is to get items into the search path. There are many ways to do this. Jip-Hop presented an outstanding approach of overlay-mounting /usr/local/bin atop a private directory.

$PREFIX/share

Note that it is not just /usr/local/bin that needs this treatment. Any build-and-install tool is expected to add parts under $PREFIX/bin, $PREFIX/share/man/man1, $PREFIX/sbin, $PREFIX/lib, and many others. So I think that it's important to treat the entire $PREFIX tree as one cohesive unit, and to mount it as such.

Me, I use /root/.local for this. That's a stable target path, convenient for root administration. (It is, after all, ~/.local for root.) Note that linked binaries in /usr/local/lib etc. are not expected to survive a major system upgrade or rebuild anyway. But several of you have made solid arguments that everyone should create a new directory on a data pool. So I conclude that the script and support materials should strongly encourage setting a custom path to your own dataset — while falling back to the known location of /root/.local until you do.

# $SANDLOCAL is a persistent directory to be fused behind the system's
# read-only /usr/local tree. If you install a $SANDLOCAL/bin/cooltool
# your shell will find it in your search path at /usr/local/bin/cooltool
# -- and so will root's, so that "sudo cooltool" will work as expected.
#
# The default /root/.local works fine, but to survive major upgrades
# prepare a dataset on your data pool. Owned by root, and executable.

SANDLOCAL=/root/.local

Managed images

Once there is such a tree mounted, then /usr/local/lib/machines becomes available as a recognized stable persistent location for Jailmaker-managed containers. I suggest that Jailmaker should be the sole owner and operator of $SANDLOCAL/lib/machines.

Separately: the user's direct native interactions with machinectl pull-tar and such (i.e. outside of Jailmaker) will make serious, complex changes under /var/lib/machines. They will also create or remove symlinks deep inside /etc/systemd. Quoting machinectl(1):

Machine images are preferably stored in /var/lib/machines/, but are also searched for in /usr/local/lib/machines/ and /usr/lib/machines/. … Note that images stored below /usr are always considered read-only. It is possible to symlink machines images from other directories into /var/lib/machines/ to make them available for control with machinectl.

In other words: the user doesn't directly manipulate images in /usr/local/lib/machines; that's where Jailmaker reigns. Jailmaker's own curated images, then, are also naturally available as prepared models to be used by the user's native commands such as machinectl enable.

Non-managed images

If you follow that, then we still have the matter of where to keep non-Jailmaker sandbox data. TrueNAS SCALE currently starts with nothing in the /var/lib/machines directory, so we can simply mount a dataset there.

# $MACHINES is a persistent directory to be mounted at /var/lib/machines
# to which machinectl will download container images and search for them.
# Each container will host its own root filesystem in a directory.
#
# Prepare a dataset on your data pool. Owned by root, and executable.

MACHINES=/mnt/pool/sandbox/machines

Whew!

And that, I think, is the bulk of it. Two mounts, from two datasets. All other user sandbox matters are under /etc/systemd, which we're addressing in other threads.

If I may quickly double back to $SANDLOCAL for a moment: say the user sets up a Pre-Init or Post-Init in the TrueNAS GUI. Configured to run the command /mnt/tank/SecretSandBox/admin/sbin/jlmkr-prepare. That script could safely conclude that it's intended to use /mnt/tank/SecretSandBox/admin as its $SANDLOCAL value.

I hope some of the above makes sense, and I look forward to your feedback. 🔬