- Overview of the setup
- Setting up a build server
- Running builds
- Optionally using QEMU/KVM/libvirt instead of VirtualBox
The F-Droid build server isolates the builds for each package within a clean, isolated and secure throwaway virtual machine environment. Building thousands of apps, especially with automated and/or unattended processes, could be considered a dangerous pastime from a security perspective. This is even more the case when the products of the build are also distributed widely and in a semi-automated (“you have updates available”) fashion.
Assume that an upstream source repository is compromised. A small selection of things that an attacker could do in such a situation:
- Use custom build steps to execute virtually anything as the user doing the build.
- Access the keystore.
- Modify the built APK files or source tarballs for other applications in the repository.
- Modify the metadata (which includes build scripts, which again, also includes the ability to execute anything) for other applications in the repository.
Through complete isolation, the repurcussions are at least limited to the application in question. Not only is the build environment fresh for each build, and thrown away afterwards, but it is also totally isolated from the signing environment.
Aside from security issues, there are some applications which have strange requirements such as old versions of the NDK. It would be impractical (or at least extremely messy) to start modifying and restoring the SDK on a multi-purpose system, but within the confines of a throwaway single-use virtual machine, anything is possible.
All this is in addition to the obvious advantage of having a standardised and completely reproducible environment in which builds are made. Additionally, it allows for specialised custom build environments for particular applications.
Overview of the setup
This is how to set up a working build server, starting from a
completely clean minimal Debian/stable install. This HOWTO assumes
you have already
setup fdroidserver. Running
the fdroidserver tools directly out of
~/fdroidserver/fdroid build org.adaway), will likely be
the easiest for now since the build server setup scripts are not
really ready for proper packaging. Also, it will likely only work on
Debian, Ubuntu and other Debian-derivatives since F-Droid only uses
Debian in its infrastructure (we welcome porting contributions!).
The base server needs to be at minimum Debian/jessie, or there will need to be some heavy tweaking. If you run Ubuntu or derivative distro, you can get any packages missing from your version, like vagrant-cachier, from this PPA: https://launchpad.net/~fdroid/+archive/ubuntu/buildserver/
First, install the necessary packages and create a new user to run the whole process here, e.g. fdroid. These are only the packages required by all builds, you might need to install additional packages to build apps, for example, mercurial or subversion. Once the packages are installed and the fdroid user is created, nothing else in this process should be run using root or sudo.
root:~# apt-get install vagrant virtualbox git python3-certifi \ python3-libvirt python3-requestbuilder python3-yaml \ python3-clint python3-vagrant python3-paramiko python3-pyasn1 \ python3-pyasn1-modules root:~# adduser --disabled-password fdroid root:~# su fdroid
Clone source code and configure the buildserver settings, running as fdroid user:
fdroid:~$ cd ~ fdroid:~$ git clone https://gitlab.com/fdroid/fdroidserver.git fdroid:~$ cp fdroidserver/examples/makebuildserver.config.py fdroidserver/ fdroid:~$ sed -i "s@^baseboxurl.*@baseboxurl = \"https://f-droid.org/jessie64.box\"@" fdroidserver/makebuildserver.config.py
You also have to make sure your
ANDROID_HOME environment variable is
set up correctly.
For your convenience you optionally may add the fdroid executable to your path:
fdroid:~$ echo "PATH=\$PATH:$HOME/fdroidserver" >> ~/.bashrc
Create the base buildserver image… (downloading the basebox and all the sdk platforms can take long time).
fdroid:~$ cd fdroidserver fdroid:~/fdroidserver$ ./makebuildserver
Get all of the app build metadata from the fdroiddata repo…
fdroid:~/fdroidserver$ cd ~ fdroid:~$ git clone https://gitlab.com/fdroid/fdroiddata.git fdroid:~$ cp fdroidserver/examples/config.py fdroiddata/ fdroid:~$ sed -i "s@^[# ]*build_server_always.*@build_server_always = True@" fdroiddata/config.py
Setting up a build server
In addition to the basic setup previously described, you will also need a Vagrant-compatible Debian/jessie base box called ’jessie64’.
You can use a different version or distro for the base box, so long as you don’t expect any help making it work. One thing to be aware of is that working copies of source trees are moved from the host to the guest, so for example, having subversion v1.6 on the host and v1.7 on the guest would fail.
Creating the Debian base box
The output of this step is a minimal Debian VM that has support for remote login and provisioning.
There are two possible ways to get a debian base box for use with
Using the Debian provided vagrant boxes
Use the prebuilt vagrant boxes built by the Debian team at https://wiki.debian.org/Teams/Cloud/VagrantBaseBoxes and adapt them to fit F-Droid’s requirements.
Install the following vagrant plugins, the first one automatically installs the virtualbox guest additions for us, the second one can resize the virtual hard disk:
vagrant plugin install vagrant-vbguest vagrant plugin install vagrant-disksize
Then create a file named
Vagrantfile with the following contents:
Vagrant.configure("2") do |config| config.vm.box = "debian/jessie64" # Disable default /vagrant folder rsync config.vm.synced_folder ".", "/vagrant", disabled: true config.disksize.size = '60GB' config.vm.provision "shell", inline: <<-SHELL sudo apt-get update && sudo apt-get install -y parted sudo swapoff /dev/sda5 # disable swap so we can then remove the partition sudo parted /dev/sda rm 5 rm 2 # remove swap and extended partition # Extend sda1 to the full disk. Yes, this needs three dashes. See # https://unix.stackexchange.com/a/365657/126364. sudo parted ---pretend-input-tty /dev/sda resizepart 1 yes 100% sudo resize2fs /dev/sda1 # enlarge filesystem to fill the whole partition sudo dd if=/dev/zero of=/swap bs=1024K count=500 # create a swapfile sudo chmod 600 /swap sudo mkswap /swap sudo swapon /swap sudo sed -i '/swap/d' /etc/fstab # and update fstab with the swapfile echo "/swap none swap sw 0 0" | sudo tee -a /etc/fstab SHELL end
In the same directory run
vagrant up --provider virtualbox.
This will download and import the debian/jessie64 base.box. It will then bring up the box once and do some initial setup including installing the virtualbox guest additions, resizing the hard disk, resizing the partition and filesystem and adjusting the fstab.
After that is done we can repackage the box and move it to the expected
fdroid cache location. We also want to make sure there is no left over
jessie64 box known to vagrant.
vagrant package mv package.box ~/.cache/fdroidserver/jessie64.box # Remove any previous versions (if one exists) of the 'jessie64' machine # created by makebuildserver. Otherwise it will reuse the old version. vagrant box remove jessie64
Building a Debian base box from scratch
Create one from scratch form verified standard Debian installation media. Documentation for creating a base box can be found at https://www.vagrantup.com/docs/boxes/base.html.
In addition to carefully following the steps described there, you should consider the following:
It is advisable to disable udev network device persistence, otherwise any movement of the VM between machines, or reconfiguration, will result in broken networking.
For a Debian/Ubuntu default install, just
touch /etc/udev/rules.d/75-persistent-net-generator.rulesto turn off rule generation, and at the same time, get rid of any rules it’s already created in
Unless you want the VM to become totally inaccessible following a failed boot, you need to set
GRUB_RECORDFAIL_TIMEOUTto a value other than -1 in
/etc/grub/defaultand then run
Creating the F-Droid base box
The next step in the process is to create
./examples/makebuildserver.config.py as a reference - look at the settings and
documentation there to decide if any need changing to suit your
environment. There is a path for retrieving the base box if it doesn’t
exist, and an apt proxy definition, both of which may need customising
for your environment. You can then go to the
and run this:
This will take a long time, and use a lot of bandwidth - most of it spent installing the necessary parts of the Android SDK for all the various platforms. Luckily you only need to do it occasionally. Once you have a working build server image, if the recipes change (e.g. when packages need to be added) you can just run that script again and the existing one will be updated in place.
The main sdk/ndk downloads will automatically be cached to speed things
up the next time, but there’s no easy way of doing this for the longer
sections which use the SDK’s
android tool to install platforms,
add-ons and tools. However, instead of allowing automatic caching, you
can supply a pre-populated cache directory which includes not only these
downloads, but also .tar.gz files for all the relevant additions. If the
provisioning scripts detect these, they will be used in preference to
running the android tools. For example, if you have
buildserver/addons/cache/platforms/android-19.tar.gz that will be used
when installing the android-19 platform, instead of re-downloading it
android update sdk --no-ui -t android-19. It is possible to
create the cache files of this additions from a local installation of
the SDK including these:
cd /path/to/android-sdk/platforms tar czf android-19.tar.gz android-19 mv android-19.tar.gz /path/to/buildserver/addons/cache/platforms/
If you have already built a buildserver it is also possible to get this files directly from the buildserver:
vagrant ssh -- -C 'tar -C ~/android-sdk/platforms czf android-19.tar.gz android-19' vagrant ssh -- -C 'cat ~/android-sdk/platforms/android-19.tar.gz' > /path/to/fdroidserver/buildserver/cache/platforms/android19.tar.gz
Once it’s complete you’ll have a new base box called ’buildserver’ which
is what’s used for the actual builds. You can then build packages as
normal, but with the addition of the
--server flag to
to instruct it to do all the hard work within the virtual machine.
The first time a build is done, a new virtual machine is created using
the ’buildserver’ box as a base. A snapshot of this clean machine state
is saved for use in future builds, to improve performance. You can force
discarding of this snapshot and rebuilding from scratch using the
--resetserver switch with
When using the buildserver, running
fdroid directly from a git checkout of fdroidserver will be the easiest. If you don’t already have the fdroidserver tools installed and setup, you will need to do that next: Installing the Server and Repo Tools. That provides all of the dependencies needed to run fdroidserver from git.
Now you are ready to run builds. Test by building the latest fdroid version:
fdroid:~/fdroidserver$ cd ~/fdroiddata fdroid:~/fdroiddata$ ~/fdroidserver/fdroid build org.fdroid.fdroid -l --server
Optionally using QEMU/KVM/libvirt instead of VirtualBox
It is also possible to QEMU/KVM guest VMs via libvirt instead of the
default VirtualBox. VirtualBox is still the recommended setup since
that is what is used by f-droid.org, but there are cases where it is
not possible to run VirtualBox, like on a machine that is already
running QEMU/KVM guests. In order to make the libvirt image files
directly readable by
vagrant package, libvirt’s QEMU needs to be
configured to always set the ownership to
root:~# apt-get install vagrant vagrant-mutate vagrant-libvirt ebtables dnsmasq-base \ python3-libvirt libvirt-clients libvirt-daemon-system qemu-kvm qemu-utils git root:~# cat << EOF >> /etc/libvirt/qemu.conf user = "libvirt" group = "libvirt" dynamic_ownership = 1 EOF root:~# service libvirtd restart
Then create a makebuildserver.config.py next to makebuildserver and add:
vm_provider = 'libvirt'
Debian/stretch and Ubuntu/xenial
root:~# adduser fdroid libvirt root:~# adduser fdroid libvirt-qemu
older Debian and Ubuntu
root:~# adduser fdroid libvirtd root:~# adduser fdroid kvm
Advanced nested KVM Setup:
This section is not relevant for using F-Droid in a normal setup. If you
want to run
fdroid build --server flag inside a KVM, this chaper will
help you getting started.
Consider following basic nesting setup:
bare metal host (l0) \- F-Droid VM (l1) \- F-Droid builder VM (l2)
The steps above describe how to setup (l1) and makebuildserver sets up (l2).
First of all you’ll have to check if you cpu support the vmx (or svm on amd) instruction set. You can use this command to list details about your cpu:
root:~# cat /proc/cpuinfo
On (l0) you have to check that nesting is enabled:
root:~# cat /sys/module/kvm_intel/parameters/nested
If it’s not enabled you can turn it on by running:
echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf
You’ll need to reboot to for this to take effect.
Next you’ll need to make sure that your (l1) vm configuration forwards cpu features required for nesting. So open up your configuration for the VM /etc/libvirt/qemu/my-vm.xml and insert a cpu block inside your domain-tag. (virt-manager also provides a user-interface for this operation.)
<cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <vendor>Intel</vendor> <feature policy='require' name='vmx'/> </cpu>
The actually required configuration here depends on your cpu. You can find details in libvirts manual. The important part is that you forward vmx (or svm on amd) to the guest system.
This is the setup that is used in the Continuous Integration builds as part of the reproducible builds effort. You can see this in action on the Debian Project’s jenkins setup: