[RFC] Overlay repos

Forums Development [RFC] Overlay repos

This topic contains 13 replies, has 2 voices, and was last updated by  pfalcon 1 year, 7 months ago.

Viewing 14 posts - 1 through 14 (of 14 total)
  • Author
    Posts
  • #1398

    pfalcon
    Member

    I bet some folks thought about forking F-Droid, so did I. I never really was going to do that by various reasons: 1) I don’t really have time to maintain it; 2) I don’t really want to bother with hosting it (complete F-Droid repo is pretty big already); 3) I really think that F-Droid is great project and is fully worth collaboration and effort to make it better in mainline, not in own’s corner.

    That said, I could really use a personal “overlay” style repo, which would contain my own packages which could be installed in addition to mainline repo. There’re various usecases for that:

    1. Staging area for new builds: it’s hard to be sure that your recipe works well without enough testing, but I want to do that testing comfortably (deploy with F-Droid, let other folks test it easily).
    2. Include wider selection of packages, which didn’t get or were removed from F-Droid. Don’t get me wrong – I’m in full support of CiaranG’s thoroughness regarding selection of apps included. F-Droid should be trusted source, and thus shouldn’t allow questionable compromises. The opposite should be also true – it’s a privilege to be included in F-Droid, so some random crippleware, even with source attached, shouldn’t get in. It’s just life’s short and I’m pragmatic, so for pretty tolerate some free-as-beer but not as-speech services.
    3. Host binary blobs in organized manner – it’s hard to get around them either.

    So, is anyone interested in hosting an overlay repo? Anyone already does? What problems are spotted? F-Droid itself already has basic support for such usage, for example you can add more than one repo to the client. But it can be rough in places and some small details would require polishing or addition, I guess. For example, update.py currently warns if there’s no binary for each metadata recipe, and client doesn’t seem to allow to check from which repo a particular package comes.

    #1399

    CiaranG
    Moderator

    The main problem would be that the current UI for managing repositories is terrible. You really need to be able to manage them better – particularly setting the priority of them – i.e. if the same app exists in two of your configured repos, which takes priority?

    Regarding the update.py thing, you would really only want the metadata for your repo on your server. Ideally the metadata should be separated from the server code itself, in a different git repo, to facilitate this. I’ve never worried about it, because so few people run their own, but I could do it easily enough if someone asks.

    #1401

    pfalcon
    Member

    The main problem would be that the current UI for managing repositories is terrible. You really need to be able to manage them better – particularly setting the priority of them – i.e. if the same app exists in two of your configured repos, which takes priority?

    Yes, sure, supporting more than 1 repo in general case is not that easy. That’s why I want to start with just simple overly scenario – custom repository contains (few) number of packages which are not in f-droid. Of course, that gets complicated soon even for the usecases I mentioned above. For example, if I add a new version for package which already in F-Droid and installed, I won’t be able to install new version without uninstalling F-Droid’s, because keys are different. Anyway, git to start with some simple scenario at least.

    Regarding the update.py thing, you would really only want the metadata for your repo on your server. Ideally the metadata should be separated from the server code itself, in a different git repo, to facilitate this.

    Well, that gets too complicated. Or rather, that’s more complicated than what I have in mind so far. For me, the main usecase is still a) staging area for recipes to be submitted to F-Droid mainline and b) being able to deploy packages to my devices ASAP as I made a recipe, not waiting while it’s merged upstream. I don’t really won’t a fork of F-Droid, more like superset, i.e. git fork repo follows upstream close, and there’s a branch which overlays on top of it, with stuff from from overlay being submitted for merge whenever possible.

    So, thanks for understanding such usecases, I’ll be making changes to support them then, and will try to document that in one way or another.

    #1417

    pfalcon
    Member

    Ok, so I now set up a SourceForge project for my overlay repo: https://sourceforge.net/projects/fdroid-staging/ . The package repository itself is hosted in downloads area. To add repository to your client:

    1. In client’s main screen, open menu, choose “Manage Repositories”
    2. You should see entry for F-Droid mainline repository
    3. Open menu again, choose “New Repository”
    4. Enter following:

    http://master.dl.sourceforge.net/project/fdroid-staging/repo

    . Note there’s no slash at the end.
    5. Leave repository list screen, answer yes to question about updating repositories.

    Unfortunately, currently there’s no way to tell which package comes from which repository. However, you can temporarily disable any repository and update package list to see package belonging to remaining repository, then turn them back on afterwards.

    #1419

    pfalcon
    Member

    And here’s “first customer” of new staging repo – RsyncDroid, which happen to be shipping rsync as binary blob and thus violating GPL.

    And that’s just perfect example why I wanted to have such staging repo – that package can’t go into mainline, but it’s nice to have it handy while we’re solving task of building native binaries.

    #1435

    pfalcon
    Member

    Ok, figured how to easily tell packages in staging repo from mainline – just give them category suffixes for example. Few more stuff in http://master.dl.sourceforge.net/project/fdroid-staging/repo

    #1437

    CiaranG
    Moderator

    It would probably be a good idea to use a signed index for that repo if you care about security.

    #1439

    pfalcon
    Member

    Maybe, when I figure that out ;-). But that’s again more intended as devel sandbox for testing stuff before pushing to F-Droid than a real repo.

    #1445

    CiaranG
    Moderator

    Just generate a key and set repo_keyalias in your config file (see the sample config). It’s the same kind of key, and stored in the same keystore, as the app keys.

    You shouldn’t need to do anything else – even if the client is configured to use an unsigned repository, it should see that the signed one is available (you can tell by the presence of index.jar) and switch to it. Obviously none of this helps much if the repo was compromised *before* you installed, but it’s a bit of added safety for almost no cost.

    #1446

    pfalcon
    Member

    Ah, sorry, that’s not me ;-). “Figure out” there meant not “figure out what magic button to press”, but “from where there security comes”. Because, well, signing a file with random keypair and letting people download random signed file and random public key to verify it sounds like a nice checksum, not anything which verifies identity.

    #1452

    CiaranG
    Moderator

    It’s not verifying identity, it’s some protection against the scenario where the repo server is compromised, which makes it impossible add or modify the applications. Modified binaries will be rejected by the client because they don’t match the checksum in the index. You could get around that by simply modifying the index, but if it’s not signed with the original key, the client won’t accept it.

    #1473

    pfalcon
    Member

    Ok, so did I understand it right – security works in the same way as for Android packages: on a first install, it accepts any key, but on the next update of the same package, if key won’t match the original, it will refuse to update. So, is this the same for signed package index? How exactly client behaves when key mismatch is detected? It’s worth documenting details of this (in manual/README), as this is important if we really talk about security.

    And it would be nice to solve README/fdroid.texi dichotomy ;-).

    And final side-thought – Android’s package keying works against F-Droid (or its users, depending how look a that), and I have a feeling that some developers are dissatisfied with F-Droid because of this. That’s worth separate discussion.

    #1784

    pfalcon
    Member

    FYI, https://sourceforge.net/p/fdroid-staging/ was updated to new repo after the fdroiddata repository split (had some issues with doing that earlier).

    #5891

    pfalcon
    Member

    Just a small update: after some hiatus, I hope to hack on F-Droid again, and as the first step, I reinitialize the staging repo from scratch, as many apps are now in mainline, etc.

     

Viewing 14 posts - 1 through 14 (of 14 total)

You must be logged in to reply to this topic.

Posted in