APIs for All The Things!Posted on 2021-02-05 by eighthave
For F-Droid to be trusted, we need to be transparent. Making it easy for people to pay attention to all the processes and systems we use helps us stay secure. One thing that Debian and other GNU/Linux distros have proven over the past decades is that transparency is an effective ingredient for delivering software securely. This ensures that the apps and packages published on f-droid.org are only the sources from upstream developers. And reviewing the sources of upstream developers helps those developers deliver their code securely. These reviews are also fundamental to how F-Droid represents users first when publishing software. Towards those ends, we have been making a big push to publish JSON APIs with key information about all parts of the process of publishing trusted free software that respects users.
The repo index
F-Droid is built around a signed index of all the apps and packages that are available. This is a JSON file, though more of an index than an API since all the information is in a single file. It is available for any use, and there are some project that take advantage of that. fossdroid.com is an alternate web view on all the apps available in f-droid.org. Aurora, F-Droid Classic, G-Droid, and M-Droid are alternate Android client apps.
The repo index is in the form of a signed JAR file that contains
index-v1.json. The best method is to verify the JAR signature, then extract
index-v1.json if it validates. The fdroidserver Python library provides
fdroidserver.download_repo_index() to make this easy to do. It is also
possible to just read index-v1.json using any ZIP library.
App build metadata
Every app in f-droid.org has a corresponding
build metadata file in
fdroiddata. In order to encourage
more creative reuse, we keep our data in standard formats in easy to find
places. The whole app collection is in YAML format. The file can either be directly accessed
by Application ID using the GitLab Raw URL
The repo index and the GitLab Repository Files
API can be used to query
which Application IDs are available.
f-droid.org build server status
The four main steps of the build cycle are: checkupdates, build, update, and deploy. There is a JSON file for each one of those with status information. It is posted once that step completes running. The step that is currently running is posted to running.
Two examples of how these JSON APIs can be used are the F-Droid Monitor website and the F-Droid Build Status app.
Mirror and repo monitoring
mirror-monitor and repo-monitor are automated jobs that check various stats about f-droid.org mirrors and third party repos respectively. They each provide a single, large JSON log of all the times the checks where run:
- f-droid.org mirror status: https://fdroid.gitlab.io/mirror-monitor/report.json
- third-party repo status: https://fdroid.gitlab.io/repo-monitor/report.json
verification.f-droid.org is a rebuilder that rebuilds the official releases
from f-droid.org, then checks to see if they were reproducibly
built. There is a JSON file per APK that has
been checked, where the filename follows the pattern
<ApplicationId>_<VersionCode>.apk.json, for example:
Binary transparency logs
Binary transparency is the idea that all released packages should be logged as they are published. This provides a way to check if a given binary was produced by the publisher, or came from somewhere else, e.g. as an exploit. fdroidserver has built in tools for managing a binary transparency log of the index files as part of the release process. This has been enabled on the Guardian Project repo: https://github.com/guardianproject/binary_transparency_log
There is also a prototype for running a binary transparency log for f-droid.org. https://gitlab.com/eighthave/f-droid.org-binary-transparency
Since Gradle and the Google Android Tools team does not publish one, F-Droid has done it. The basic API is a JSON file with a listing of all URLs known to have be published. Each URL then contains a list of SHA-256 checksums that the log process has seen. There are also other files tracked, based on what the publisher provides.
issuebot runs on app merge requests and requests for packaging (RFP) to provide information to reviewers about the app. Much of that information is also published in JSON format. Each report is made of data from modules that run on a specific app. There are two entry points:
Usage data is gathered from various F-Droid websites and published with no personally identifiable information (PII). It is published with one JSON file per-week, with a JSON index file per website. For example, here is a week of search metrics: https://fdroid.gitlab.io/metrics/search.f-droid.org/2021-01-18.json
Each git repo has lots of interesting metadata. The GitStats project generates nice charts from that data. This is not quite an API, but it is related since it is using public project data. Pre-generated pages for each of the central git repos are visible here: https://fdroid.gitlab.io/.