Charlène's

... it's worse than your datacenter.



Cross compiling ports with distcc on OpenBSD

Last Update: 2020-05-21

One thing is sure; compiling software on OpenBSD/macppc requires a lot of patience, even with several machines and distcc. I can't add machines indefinitely ... but since the move to clang, it's now possible to use an amd64 machine as a macppc distcc helper, reducing greatly build times.

It's not the perfect solution or setup, i'm detailing the caveats at the end.

Note that while macppc is targeted here, it can be used on any arch that has clang as the base-compiler and is super slow (hello armv7).

Pre-requisites

All commands prepended by # must be run as root.

Preparing the macppc machine

At first, install and setup distcc:

# mkdir /var/distcc
# chown _pbuild /var/distcc
# pkg_add distcc

Now you need to tell /etc/mk.conf that you want to use distcc, so add to it:

COMPILER_WRAPPER=/usr/bin/env \
    DISTCC_IO_TIMEOUT=600 \
    DISTCC_DIR=/var/distcc \
    DISTCC_HOSTS="ip_amd64_machine/8" \
    DISTCC_VERBOSE=0 \
    distcc

The /8 represents the number of jobs you want the helper machines to do.

Some ports will fail with distcc, so it's needed to not use distcc for them:

$ cd /usr/ports
$ for port in devel/cmake graphics/cairo lang/ruby/2.6 graphics/lcms2 net/samba; do \
        echo "COMPILER_WRAPPER=" >> "${target}/usr/ports/${port}/Makefile"; \
  done

Preparing the amd64 machine

At first we're going to create the chroot, you'll need proot from our ports tree. Note that you want your chroot to be in a wxallowed mounted partition:

# /usr/ports/infrastructure/bin/proot \
    -B /some/where/my_macppc_chroot \
    actions=chown_all
# cp /etc/installurl > /some/where/my_macppc_chroot

Enter in the chroot

# chroot /some/where/my_macppc_chroot

So basically what we're doing here is removing the base compiler binaries and replace them with cross-compilation wrappers:

# pkg_add llvm distcc-server
# cd /usr/bin
# rm cc c++ clang clang++ clang-cpp
# cat > cc
exec /usr/local/bin/clang  \
--target="powerpc-unknown-openbsd" \
"$@"
^D
# for i in c++ clang clang++ clang-cpp; do ln -s cc $i; done
# chmod 755 cc

Distcc, unless you use --make-me-a-botnet, only runs whitelisted commands, so we're generating a white list with our compilers:

# mkdir -p /etc/distcc
# for i in cc c++ clang clang++ clang-cpp; do \
    echo /usr/bin/$i >> /etc/distcc/commands.allow; done

The next step will be configuring distccd itself, we'll need to add an environment variable, something that would require a separate login class if done perfectly. Instead we're creating our own service:

# cp /etc/rc.d/distccd{,_local}
# vi /etc/rc.d/distccd_local

Modify the below lines to get something like this:

daemon="/usr/bin/env DISTCC_CMDLIST=/etc/distcc/commands.allow /usr/local/bin/distccd"
[...]
daemon_flags="--daemon --allow 127.0.0.1 --allow ip_powerbook"

Now is the time to run distccd:

# rcctl enable distccd_local
# rcctl start distccd_local

You can leave the chroot.

Building ports

Sit back and have fun, ports are now built transparently with distcc, you can use dpb as well, which is recommended as distcc works better with 2 or more concurrent compilations.

You won't need to touch the chroot until the next base-clang ABI change, and can control the chroot'd distccd with:

# chroot /some/where/my_macppc_chroot rcctl {start,stop} distccd_local

By default, logs are in var/log/daemon.

Security?

A distcc premise is that you should only use it on a trusted network.

Filtering by IP address does not protect you from spoofing, and commands whitelisting is great, but compilers can still be used without any authentification.

To address this you may use distcc with ssh or GSS-API, but these solutions are more complex to set up given that the _distcc user would require to be setup as something more akin to a regular user.

Jobs?

Actually a PowerBook G4 A1138 (1.67GHz) can provide only 6 jobs at best to the amd64 machines.

On top of that some ports like lang/gcc bootstrap themselves and can't be built via distcc. distcc is just a compiler wrapper, ports that don't compile anything are run in the macppc machine, and so are the configure stage, linking, and ports not using the base compiler.

Putting a high number of jobs clogs the machine in these cases and jobs are not sent to the amd64 distcc helper.

As such, i'm using 4 dpb jobs as a good tradeoff, but you can put more if you're building a single port with all its depends built.

It still allows me to build cad/oce within 5 hours, instead of 53 hours in the macppc bulk cluster, webkitgtk4 takes only 4 hours.

Back to the top