Updating of system apks
If an apk was installed with the ROM, and F-Droid is installed as a regular (non-privileged) apk, then it is not able to update the system apks. While not ideal, it is even worse because this is not currently conveyed to the user appropriately.
Tell the user that the apk in F-Droid and the one on their phone are signed by different keys:
"The new version is signed with a different key to the old one. To install the new version, the old one must be uninstalled first. Please do this and try again. (Note that uninstalling will erase any internal data stored by the application)."
While technically true, this is not really what the user wants to hear. The reason I say this, is because they are not able to remove the system apk in order to install the F-Droid one.
Add new incompatible reason
It might be nice to add a new "Incompatible reason" that can be shown to the user in the AppDetails screen. For example, where some apks are incompatible because of "Wrong android version" or "NFC required", this could be "Can't replace system apk" or something like that.
The problem with this is that not everybody will have "Show incompatible apks" enabled (for good reason, phones without many features will have heaps of incompatible apps).
Feedback about the origin of each apk
Also known as "proper multi repo support", this would make it clear that some apks came built from f-droid, others from the ROM, and others from, e.g. guardianproject or other F-Droid repos. Then, the intuitiveness of the error messages may make more sense to the user:
"To install the version of this apk from repo BLAH, you will first need to uninstall the one from repo SNEH".
Ignore system apks
Either automatically ignore system apks from prompting for updates (I don't like this idea), or show a different message to the user, giving them the option to "Ignore all updates" if it is identified as a system apk.
There are no doubt several other ways to make the UX better when dealing with F-Droid provided apks which replace system apks. Please discuss here.