Version update process

[handschuhMember]

While looking trough the fdroid souorcecode, I noticed, that it checks the market version and then performs build of the current source if a new version is available.

I am sorry, but I think this is a bad behavior for several reasons:
– no support for programs that are not available in the market
– fdroid uses an US-identifier to check market data such that programs that are not in the US-android market are excluded
– it is very likely that the fdroid server builds the wrong version since it checks out the current source which does not have to be the source of the current release

So I think the fdroid server should check the AndroidManifest.xml in the programs repository and do not contact the market at all.

CiaranG, you wrote in your blog, that you do not like the android market because of its proprietary nature but still the fdroid server ONLY depends on the version-data that comes from the market.

[pfalconMember]

You probably mix up something. To build all F-Droid packages, you don’t need any market access. And if CiaranG has auxiliary scripts which monitor market to catch application updates, that’s only because market provides centralized place to get such information from. It’s up to whoever to either use such auxiliary scripts or not, if CiaranG finds it useful up to the level of having coded those scripts, that’s good, not bad.

[handschuhMember]

What I tried to talk about was the process of fetching information of software updates, not building the packages.

In therms of a centralized place: why is such a thing needed? The version information is stored in the repositories itself (in the AndroidManifest.xml). Moreover, from the repository you also get the correct revision that has been published. This is something the android market cannot tell you.

So here is what could happen:
Somebody makes a release in the market after he/she published the sources correctly at 6:00 in the morning. During that day, the developer makes several commits that contain unstable features. At the evening, the code is checked out (containing the unstable features) and published.

At least, that is how I understood the code.

[pfalconMember]

In therms of a centralized place: why is such a thing needed?

Well, if we’re talking about having an app distro (and I’d like FDroid to be(come) just that), it must be update for new version regularly, and CiaranG seems to deal quite well with that, so whatever tools he uses, good for him. And obviously, it’s easier to harvest for new versions of hundred of apps in one place, than in hundreds of place. If you’d be interested to work on harvesting that info on individual project pages, that’s cool, and “patches welcome”. But that wouldn’t be easy idea (think AI), as there’s no consistent means to announce and mark a new release (while market exactly provides just that).

At least, that is how I understood the code.

Which code exactly? I had a look at marketcheck/test.java (heh, should be renamed, I guess), and what it does is updating “Market Version” & “Market Version Code” fields of metadata. But those fields are purely informational, and not related to build process. Instead, CiaranG apparently manually eyeballs for such changes, then digs in specific project’s repo for proper revision to build against, then does tests builds, etc, etc. It would be nice if AI did that instead, so yeah, if you’re interested…

[handschuhMember]

Code: the code that is used to write the latest version information to the metadata-files.

I checked several projects that are in fdroid and their version information could always be retrieved by one of the three heuristics:
– revision of the AndroidManifest.xml
– tag with the version name
– a download package (compare debian watch files)
If there are projects that do not match these things, I am sure the upstream authors will add files to match the heuristics.

I repeat again to make my point clearer: you have to use proprietary software (android market) to get your application into fdroid. That seems to me as a bit inconsistent to me.

Some time ago, I suggested this thing directly to CiaranG but there was no response so I assume that there is no interest in that. (I just wanted to know if he would consider adding this, if I provide a patch) Of course, this is perfectly fine, I just wanted to know how others think about this.

[pfalconMember]

I noticed myself that CiaranG doesn’t always respond quickly, for example, I have merge request for few days now w/o any comment. But that’s of course because he already spends lots of time maintaining the project, and his time is limited, as everyone else’s.

For forums in particular, I guess the idea is that FDroid community members discuss stuff here, so just in case I share my opinion while CiaranG may be busy with other things .

I repeat again to make my point clearer: you have to use proprietary software (android market) to get your application into fdroid.

That’s not true.

I am sure the upstream authors will add files to match the heuristics.

You must be kidding. If you looked inside FDroid metadata file, you saw dozens of notes “Disabled: Unclear license” or “!No source code in the repo”. I.e. “upstream authors” don’t even care to specify a license for their piece of code, or suddenly close up their code, having got their “open source” advertisement.

OpenSource software for Android is a mess, and I see FDroid as almost only attempt (well, there were few earlier attempts, they’re dead now!) to organize it and let community enjoy it. There’s for sure lot of things to improve, so if you have ideas for that and can back them by code or upstream communication, just contribute!

[handschuhMember]

So how does one get (as a developer) version updates into the fdroid repository? I do not see a way but maybe you know one, and I would be happy if you could share this.

If the license is not clear than the application should not be in the repository. Fullstop.

I am posting the ideas here to discuss about them. If they are not wanted then this is definitely no problem. I also do not want to spend my time on doing something that nobody wants.

• This reply was modified 1 year, 9 months ago by  handschuh. Reason: grammar

[pfalconMember]

One just adds a new “Build Version:” line – manually. You can see this for example in one of recent CiaranG’s commits:
https://gitorious.org/f-droid/fdroidserver/commit/c32843a0a2122aea4230655cc444b9172d9ff8cd

As you see, there’re quite a few market version updates done by automated tool, by only few recipes actually have new version spec added – ones which CiaranG bothered to look up. And you can see that it’s truly manual process by comments like ” #Still guessing, see previous comment”.

I guess it’s very good thing to discuss ideas here – it helps to shape up them better, be in loop on possible future developments, and of course, to shape up community .

[handschuhMember]

I don’t think the commit that you mentioned has been created manually. This is what I said before, there is a script in the repository that fetches the new version information from the market and automatically writes them into the metadata files. Have a look an the marketcheck folder in the fdroidserver repository.

So in probably would be possible to manually add the lines, make a merge request and wait for the request to be merged. So indeed, you are right one could update an fdroid version w/o a market release by putting quite some effort into this (and causing more work for CiaranG).

With the heuristics, I suggested this could be done automated for a lot of projects without getting using the market data, I think.

[CiaranGMember]

Sorry for not getting involved in this discussion earlier, I’ve been very busy elsewhere. There are a number of misunderstandings of the source code and process here. I think pfalcon has understood and covered most of the points I would want to make, but just for clarification I’ll confirm the following…

1. The market checking script is used only as an indicator of what the latest version might be. It’s not required for an application to be in the Google’s Market for it to be in F-Droid, or for it to be updated – it’s just a convenient place to check what *the author says* is the “current” version, if it happens to be there.

Checking an application’s source repository does not give you this information at all. Either a) there may be newer unstable versions in there, which is very common, and/or b) the code for the real latest version may not be there at all, again very common.

2. It’s impossible for it to ‘build the wrong version’ – nothing new is ever built as a result of a script and after checking what’s been changed.

3. It’s also impossible for it to build the wrong version because the version numbers and codes are checked in the compiled APK files.

4. The “Build Versions” lines are always added to the metadata files manually, currently.

There are definitely some options here for making things better, but it is a complicated issue and many different approaches need to be taken depending on the development process of the application in question. Before embarking on anything like this, someone would need to at least understand the existing process!

I have various unwritten plans for how to improve this stuff in future (obviously, because I deal with it every day), so thanks to Handschuh for bringing the subject up for discussion. I’ll write some more about this later – just wanted to clear up the misunderstandings for now.

[handschuhMember]

Thank you for the clarification and sorry for causing trouble with my lack of insight.

Here is what caught my attention and was the motivation behind the hole thing: look at com.googlecode.chartdroid:
– you built revision 294 (maybe I am wrong here, please correct me in case)
– revision 294 is from January the 17th in 2011
– version 2.0.0 has been published on November 22nd in 2010 in the market and the projects download area
– November 22nd in 2010 the revision of this project was revision ~266

There were no changes to this specific subfolder in the later revisions, which I just realized.
My assumption was, that you built a version that was 2 months older than the actual release. The version codes can be the same but the revision can be different.

So to summarize the hole thing: you are doing a lot of things manually? Kudos for that!

A last Question: How does a developer inform the fdroid repository without a market release? Merge-requests or EMails maybe or is there some recommended way?

Again, sorry for the trouble.

[CiaranGMember]

No need to apologise – it’s great that you’re looking into this stuff, and being forced to explain it means the information is there for others to read too. Please keep doing it.

[daithib8Moderator]

I’d like to bump this thread, to see what progress there has been since the last post. I have an idea: CiaranG does a checkupdates once a week and commits the result, leaving it to others to update the build versions. It would give a rhythm to the process and people without git set up could post build version lines in the forum threads.

Besides this all sorts of things happen: source code disappears (a problem if you build your own), donation links come and go, repos change location, descriptions become irrelevant. We need to make it as easy as possible for folks to let us know.

[daithib8Moderator]

The wiki goes a large way towards fixing this. Now there is a very small number of apps in the list that need updating. But there are still many that need to be checked manually and we need to work out a way of scheduling this. There is a new update mode that checks Tags. This works on git but it isnt always clear when to choose this over another method. The other repo types still rely on Market update checking, so we are as reliant as ever on it.

[daithib8Moderator]

So I would say in order of preference Tags, Market, RepoManifest. Tags works very well except in the cases that app tag betas. If the tags are regular, there is an auto update setting that can add a new build line when the current version is updated, but 99% of apps needed to be updated manually as there are often problems. Market is ok, but not working always: but I’m sure that Google will find a better way of presenting version info for device-variable versions. RepoManifest is better than nothing.

As for new update methods, rss feeds of the downloads page on Google Code works, but is not very automatable. I use it for checking for K9 stable updates. Pity there’s nothing that I can see like that on GItHub.

[Author]

Posts