Skip to content

design

Jump to bottom
Justin Clift edited this page Jan 16, 2017 · 7 revisions

Design of poudriere

!! This is not updated for 3.1 !!

Poudriere is a port building tool, it has two main frontends:

  • testport
  • bulk

Both are designed to build packages in a clean environment, and are able to handle both the legacy pkg_install and pkg(8).

The code is written mostly in shell and has very few dependencies outside of the FreeBSD base system.

Optional dependencies that might be useful:

Port Optional Description
devel/subversion yes Used to be able to get the sources and ports from the FreeBSD svn
devel/git yes Used to be able to get the ports from the FreeBSD github mirror
ports-mgmt/portlint yes Used for testport or bulk -t to run portlint prior to building the packages
ports-mgmt/pkg yes Used for sanity checking a pkg(8) repository (incremental building)

Poudriere is composed of jails and ports tree, it associates them to build packages.

Data

Jails

The poudriere jail is designed to handle the jails themself.

Configuration of a jail is stored in /usr/local/etc/poudriere.d/jails//*

With the following files in the directory:

File Description
arch Architecture of the jail
fs ZFS dataset path if ZFS is in use
method the method used to get the "sources"
mnt the path to the root of the jail on the host filesystem
version the version of the jail

Creating a jail

Creating a jail can be done using different methods: ftp, allbsd, gjb, svn, svn+http, svn+ssh, svn+https, svn+file, and csup.

Whatever method is used the principle is the same: download the binaries or the sources for a given version/architecture of FreeBSD, and prepare a minimal setup so that it results in a usable jail.

The jails will never be touched again except when specifically requested for an update via "poudriere jail -u".

The jails created will then be considered as a reference of a clean vanilla jail.

The minimal setup consists of the following:

Modify login.conf to add the following environment variables:

  • UNAME_r
  • UNAME_v
  • OSVERSION

Which will set what is excepted of the jail in order to make sure the build correctly discovers the release, version and OSVERSION of the jail they are building on.

Modify the make.conf to add the following variables:

  • USE_PACKAGE_DEPENDS=yes
  • BATCH=yes
  • WRKDIRPREFIX=/wrkdirs

Only on an i386 jail when the host is amd64:

  • ARCH=i386
  • MACHINE=i386
  • MACHINE_ARCH=i386

If the jail is created on top of a ZFS filesystem, then a @clean snapshot will be created.

Via HTTP/FTP

This is the default method, it fetches the prebuilt sets from http://ftp.FreeBSD.org (default host can be tweaked via poudriere.conf).

The following set will be installed:

For FreeBSD < 9.0 jails:

  • base
  • dict
  • src
  • games
  • lib32 (only if the jail is amd64)

For FreeBSD >= 9.0 jails:

  • base
  • src
  • games
  • lib32 (only if the jail is amd64)

Via gjb/allbsd

This method is the same as ftp except that it fetches the sets out of the gjb's snapshots or the allbsd snapshots.

Via svn

This method will build the jails from sources fetching them using svn (all svn possible methods are supported).

Only world is built, and the following features are supported: ccache and parallel builds via -j.

The build jail can be tuned using src.conf: If a src.conf file exists in /usr/local/etc/poudriere.d/ on the host, it will used as the src.conf for the jail. If a <jailname>-src.conf file exists in /usr/local/etc/poudriere.d/ on the host, it will be appended to the src.conf used to build the jail.

The rest is just "normal" building of world:

make buildworld
make installworld
make distrib-dirs
make distribution

Via csup

This method is considered as deprecated and will be removed soon, it works the same way as the svn method except the sources are fetched using csup.

Updating/Upgrading jails

Via ftp

The update of the jail is done by calling freebsd-update from inside the jail. A parameter can be passed to specify a release the user wants to jump to, in that case freebsd-update will be called to perform an upgrade to the said version.

Via gjb/allbsd

No update/upgrade is supported if the jail has been created using the method.

Via svn

Only updates on the same branch are supported, sources will be updated, the world rebuilt and installed, once the new world is in the old files will be cleaned out:

make delete-old
make delete-old-libs

Ports trees

The ports tree management can be done using the following methods:

  • portsnap
  • svn+*
  • git

The list of configured ports trees is located in: /usr/local/poudriere.d/ports/<ptname>.

With the following files in it:

File Description
fs ZFS dataset path if ZFS is in use
method the method used to get the ports
mnt the path to the root of the ports tree on the host filesystem

Creating a ports tree

Ports trees are named to allow having different ports trees, for example one per patch to test.

If no name is specified then "default" is used.

Via portsnap

This is the default method as it requires no external dependencies. A directory which will contain the ports tree is created, in this directory a .snap hidden directory is created to receive all the portsnap temporary files.

Creation is done using the following command:

/usr/sbin/portsnap -d ${PTMNT}/.snap -p ${PTMNT} fetch extract

PTMNT being the path to the ports tree that has been created.

Via svn

As for the jails, the svn method can support the following submethods:

  • svn
  • svn+http
  • svn+https
  • svn+ssh
  • svn+file

The following variable can be tweaked in poudriere.conf: SVN_HOST which by default is svn.FreeBSD.org

Via git

Same as the two previous methods except that the ports tree will be fetched from git.

The following variable can be tweaked in poudriere.conf: GIT_URL which by default is: git://github.com/freebsd/freebsd-ports.git

Updating the ports tree

The following command will update the ports tree:

poudriere ports -u -p <name>

According to the method used to create the command it will run the following commands:

Using portsnap

portsnap -d ${PTMNT}/.snap -p ${PTMNT} fetch update

or if run from crontab:

portsnap -d ${PTMNT}/.snap -p ${PTMNT} cron update

Using svn

svn -q update

Using git

git pull

The build process

General principles

Whatever build is run, the principle is the same.

First a reference jail is created by copying the original jail (chosen via the -j option) into the reference destination.

If the original jail has been created on top of ZFS then the reference jail is created as a ZFS clone of the @clean snapshot of the original jail.

Otherwise the jail is duplicated using pax(1).

If USE_TMPFS is set to "all" in poudriere.conf then the reference jail is mounted as tmpfs and the original jail duplicated using pax(1).

Once the reference jail is created, the following directories are created inside the reference jail:

  • /proc
  • /dev
  • /compat/linux/proc
  • /usr/ports
  • /wrkdirs
  • /usr/local (or custom localbase is set)
  • /distfiles
  • /packages
  • /new_packages
  • /ccache
  • /var/db/ports

Then the following filesystems are mounted in it:

  • devfs is mounted in /dev with the following rules applied:
    • everything hidden
    • null unhide
    • zero unhide
    • random unhide
    • urandom unhide
    • stdin unhide
    • stdout unhide
    • stderr unhide
    • fd unhide
    • fd/* unhide
  • The ports dir is nullfs mounted in /usr/ports in readonly
  • The packages dir is nullfs mounted in /packages in readonly
  • The distfiles dir is nullfs mounted in /distfiles in readonly

The option directories that can be found under /usr/local/etc/poudriere.d will be mounted if they exist respecting that order:

  1. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-<setname>-options
  2. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-options
  3. /usr/local/etc/poudriere.d/<jailname>-options
  4. /usr/local/etc/poudriere.d/options

It will be nullfs mounted as readonly in /var/db/ports inside of the jail directory.

The reference jail is then populated:

  1. WITH_CCACHE_BUILD=yes is added to the jail's make.conf if CCACHE_DIR is defined
  2. MAKE_ENV+= CCACHE_DIR=/ccache is added to the jail's make.conf if CCACHE_DIR is defined
  3. PACKAGES=/packages is appended
  4. DISTDIR=/distfiles is appended

The content of the following is appended if present:

  1. /usr/local/etc/poudriere.d/make.conf
  2. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-make.conf
  3. /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
  4. /usr/local/etc/poudriere.d/<jailname>-<portstreename>-<setname>-make.conf

To finish the file corresponding to the variable RESOLV_CONF in poudriere.conf is copied inside the jail as /etc/resolv.conf

The jail is now ready to be used.

Dependency and sanity checking

The queue calculation is done on the reference jail directly, inside the /poudriere directory.

If TMPFS_DATA or USE_TMPFS is set to all then /poudriere will be a tmpfs directory (highly recommanded).

The following directories are created inside the /poudriere directory:

  • /poudriere/building # this is the pool of currently building ports
  • /poudriere/pool # this is the pool of ready to build ports (no dependencies left to built)
  • /poudriere/deps # this is the pool of left ports which at least depend on one port not already built
  • /poudriere/rdeps # this is a reference of reverse dependencies. Each package dir contains a link to each package depending on it
  • /poudriere/var/run # This will receive the pids of the running builders
  • /poudriere/var/cache/origin-pkgname #cache to quickly figure out package name out of an origin
  • /poudriere/var/cache/pkgname-origin #cache to quickly figure out origin out of a packagename

It initializes the log directory in:

  • ${POUDRIERE_DATA}/logs/bulk/<MASTERNAME>/<STARTTIME>

$POUDRIERE_DATA being /usr/local/poudriere/data by default

$MASTERNAME being if setname is set: <jailname>-<portstreename>-<setname> otherwise: <jailname>-<portstreename> $STARTTIME begin the time of the beginning of the build is the following format: date +%Y-%m-%d_%H:%M:%S

The different statistic files are initialized in the log directory:

file default value
.poudriere.stats_queued 0
.poudriere.stats_built 0
.poudriere.stats_failed 0
.poudriere.stats_ignored 0
.poudriere.stats_skipped 0
.poudriere.ports.built ""
.poudriere.ports.failed ""
.poudriere.ports.ignored ""
.poudriere.ports.skipped ""

The dependency calculation itself is done in parallel (using one process per core by default).

For each port listed to be built en entry is created in /poudriere/deps corresponding to the package name.

All the dependency are extracted from the port via:

make -VPKG_DEPENDS -VBUILD_DEPENDS -VEXTRACT_DEPENDS -VLIB_DEPENDS -VPATCH_DEPENDS -VFETCH_DEPENDS -VRUN_DEPENDS

For each dependency an entry corresponding to the dependency pkgname is added to /poudriere/deps/<pkgname>/<depname>. An entry is also added to /poudriere/rdeps/<depname>/<pkgname> (reverse dependency tracking).

The above process is also applied to the depname.

The above will populate /poudriere/deps with all the packages that needs to be built each package will have entry for each dependency it depends on.

Now that the full dependency tree is populated, poudriere will sanity check the repository of packages if there are already packages in it In parallel (one process per core) each package is checked and deleted if:

  1. The origin of the package is not present in the ports tree any longer.
  2. The version in ports is newer than the version of the package.
  3. If CHECK_CHANGED_OPTIONS is set, the package is deleted if the options activated in the ports tree for the given port are different from the built package.

After that poudriere will process all existing packages and will delete each one for which a dependency is missing and loop through until each package has been checked.

To finish with the sanity check, all the dead symlinks are deleted.

The dependency calculation process ends by looping over all the entries in /poudriere/deps, and each time it checks if the package already exists in the package directory then the entry is removed from the queue.

All the ports with no dependencies to be built (empty directories in /poudriere/deps) are moved to /poudriere/pool/unbalanced so they can be built straight away.

Bulks

Before starting the bulk itself, the reference jail is marked "prepkg." On ZFS without tmpfs, a new ZFS snapshot, @prepkg, is created of the jail. On other filesystems a mtree file tracking all files in the jail is created.

The builds are started in parallel, with one build jail per core.

The reference jail is cloned using zfs clone or duplicated via pax(1).

The following filesystems are mounted in each jail:

  • devfs is mounted in /dev with the following rules applied:
    • everything is hidden
    • null unhide
    • zero unhide
    • random unhide
    • urandom unhide
    • stdin unhide
    • stdout unhide
    • stderr unhide
    • fd unhide
    • fd/* unhide
  • The ports dir is nullfs mounted in /usr/ports as readonly
  • The packages dir is nullfs mounted in /packages as readonly
  • The distfiles dir is nullfs mounted in /distfiles
  • fdescfs is mounted in /dev/fd
  • procfs is mounted in /proc
  • linprocfs is mounted in /compat/linux/proc
  • The specified ccache dir if any is nullfs mounted in /ccache
  • /wrkdirs is mounted as tmpfs or mdfs according to configuration if needed
  • The option directory is nullfs mounted readonly using the same rule as the reference jail

For each port in /poudriere/pool a job is started on one of the builders created above:

  1. The /poudriere/pool/<pkgname> dir is moved to /poudriere/building/<pkgname>
  2. Umount all tmpfs within the builders to cleanup their content
  3. Rollback the jail file system to the '@prepkg' state (using zfs rollback or using mtree -r + some pax hack)
  4. Cleanup /wrkdirs from any previous build
  5. Install all pkg, fetch, extract, patch, build and lib dependencies if any inside a jail(8) with no network
  6. Run make check-config in a jail(8) with no network
  7. Run make fetch in a jail(8) with network access
  8. Run make checksum in a jail(8) with network access
  9. Copy the distfiles required by the port and only them into /portdistfiles and modify /etc/make.conf so that the port will pickup its distfiles only in this directory
  10. Run make extract in a jail(8) with no network
  11. Run make patch in a jail(8) with no network
  12. Run make configure in a jail(8) with no network
  13. Run make build in a jail(8) with no network
  14. Run make run-depends in a jail(8) with no network
  15. Run make install-mtree in a jail(8) with no network
  16. If in test mode mark the filesystem as preinst (mtree file created)
  17. Run make install in a jail(8) with no network
  18. Modify /etc/make.conf to that is will create its packages in /new_packages and create that directory
  19. Run make package in a jail(8) with no network
  20. If in test mode run make deinstall in a jail(8) with no network
  21. If in test mode check for leftovers by comparing the state of the filesystem with the one in the preinst mtree
  22. If all the above went ok copy the created packages from /new_packages to the final package directory
  23. Remove all the /poudriere/deps/*/<pkgname> and move all the new empty directoris into /poudriere/pool/unbalanced so that they can be picked up in a jail for building.
  24. The /poudriere/building/<pkgname> is removed

Pool priority

The pool queue is designed to prioritize building packages that are depended on first. This allows the ready-to-build pool to more quickly get all packages in a ready-to-build state. This can help avoid bottlenecks where large dependencies must build before moving on, which may be a problem with large CPU sets.

  1. A /poudriere/pool/N pool is created for the number of POOL_BUCKETS (default is 10).
  2. Packages are initially added into /poudriere/pool/unbalanced.
  3. Packages are then moved to /poudriere/pool/N where N is the number of packages that directly depend on it. This is capped at POOL_BUCKETS-1.
  4. The build queue reads from the highest priority pool down to 0.
  5. As builds finish, they add new packages ready-to-build into /poudriere/pool/unbalanced and then call balance_pool to prioritize those.

Once finished (all packages built)

If the built packages are pkg(8) packages, a repository is created (signed with a key if it is specified in poudriere.conf).

If the built packages are pkg_install packages, a INDEX file is created out of the packages in the repository.

Clone this wiki locally