GSoC comes to an end in a short while so the last week was reserved for testing APT in all its glory. Not much to say about it beside that I wrote a lot of shellscripts to build packages, build archives from it, sign them, ship them over file as well as http method, can even install packages with dpkg in the test environment and so on. This thankfully showed a few small bugs in the process of writing the tests so many unstable and testing users will not be hit by them 🙂 Yeap, unstable as the releaseteam yesterday gave their go on uploading APT to unstable after many months in experimental. Over the week was also a bugreport on the deity list which made it today to a proper bugreport and blocked the upload until we reassigned it to curl… The bug is strange so it needed a while until I managed to understand why.

Friday was a bit busy as Michael was on holiday while he was requested to fill in some sort of final evaluation: Getting him on phone was a bit complicated, but all worked out well in the end. 🙂

So all in all a boring week. 😉

And this week will be even more boring as I need to prepare for an exam on Thursday…

From time to time I am asked why I do such a boring thing as working on a many years old application, dealing with code and bugs from the last decade, an even more boring commandline only application and finally why all this for free and for this crazy "linux" thing which isn't even called Ubuntu…

It's sometimes hard to answer this in a good way as you need to educate the questioner all the way through GNU, free beer vs free software, linux, debian, package management to APT. In two minutes and most of them get distracted by "free beer"… and there are many of them. So many that you could sometimes come to the (false) conclusion that you are the only one. Okay, in debian that is unrealistic with > 900 DDs, many DMs and a few more contributors without an official status, but the tendency to feel a bit "alone" is still everytime you look around in a class and see everywhere non-debian and mostly even non-linux operation systems running. And even if you made it so far to find an debian-related user you still have the way down to APT… its not completely uncommon to hear: "Oh yeah, fine, but pretty oldschool isn't? Good that someone developed a replacement for it. Its called $graphical-apt-frontend!" And you know that you talk with an expert if he says "I use aptitude exclusively. I have even purged apt with it…"

So everything you expect in the morning opening your emails is maybe a bunch of bugreports - but not today, today a lot of messages started with "thanks"! Nice difference and a very cool idea! So a big thanks for this thank-project. I hope this project helps a bit in motivating contributors to continue their work and might even motivate more people to start contributing 🙂

Contribution is the perfect keyword for the real content of the post: A few upgrade-related bugs needed to be debugged like #591882 there we are still in the process to understand all failed package upgrades - beside that a few small fixes like a corrected LongDesc handling and improving my testframework.

The upgrade "fix" we call "FixByInstall" is btw a quite interesting thing as it looks like an error to commit it. Very few lines of code, but they introduce an autoinstall enabled MarkInstall call in the resolver which is a premiere… and dangerous but it fixes a problem and maybe all the thinking about ways it could possibly break which can't be produced in testcases by now is not needed… We will see.

The rest of the week was spent with writing in nearly endless amount of emails to bugreports, discussions and even to APT2 announcements, threads and ITPs. So much, that i don't want to comment that further expect that you can find most of it in my crosspost to merge the independent threads: http://lists.debian.org/deity/2010/08/msg00107.html

I recommend to try running the posted APT3 brainfuck code btw, just save the code as apt3 and run it with beef (apt-get install beef before of course if needed).

For my own reference, the APT3 brainfuck code is:

++++++++++
[>+>+++>++++++>++++++++>++++++++++>+++++++++++<<<<<<-]
>>>+++++.>.++++.<<++.>>>++.>+.+++.<-.>++++.<.>----.<<<<+.<.

And no, APT3 was not a direct reaction to APT2 - it was an initial a bad joke I told a few people before and on the UDS in Dallas 2009 referring to cupt written in perl… And because people tend to misunderstand me then I say such stuff: I have nothing against cupt nor APT2 (expect name) and I even like them for giving interesting ideas what to do next… and next is currently to finalize APT in the hope that it will migrate to unstable and testing some time in the future. Also, GSoC starts to come to an end… and this week report has also reached his end. See next week for more…

As said, last week I was away with "my kids" - 21 + three more advisor + myself. Long travel - ~15 hours - in a special train for all 1000 participations from the diocese Limburg, to finally meet ~55.000 people on the Saint Peter's Square. A cool week full of program and fun but also quite demanding…

I am feeling like my backpack sticker below right now fighting against an enormous backlog, watching some debconf material and trying to speed up again on APT bugfixing. See you hopefully refreshed next week 🙂

Jepp, I am late but working against a week of backlog is hard 😉 I intended to publish this post even before my leave but some unexpected requests blocked that, so the long-awaited post for the week before I left…

Another week passed away and so you get the weekly report… a bit earlier as i am on the way to the train-station for my week of in Roma with "my kids", but let us look in the past first:

The week started with a nice bugreport which acquired most of the weeks work: #590438: Removes pseudo-essential package. Okay, the title is a bit misleading as APT doesn't remove it without replacement and it is not limited to (pseudo) essentials but it is most obvious with them. 🙂 The problem was that APT removed mawk before installing gawk which both provide awk which is a pseudo essential. So everything needed is "just" to move the installation and configuration of gawk before the removal of mawk. Easy, isn't it? 😉

The other thing wasting time without end was to push APT from debian/experimental into Ubuntu Maverick to get it in before they feature freeze and as a testcase for the transition to debian/sid and /testing which will hopefully happen better sooner than later. Some tools have really strange assumptions about the behavior of APT… (yes, I am looking at you sbuild!)

I also spent a bit of time in preparing "my" debconf presentation which will be held by Arthur Liu as I am not around in person nor online at all at that time.

(The last-minute request was btw to drop text from the slides into the speaker notes)

Last week was relatively short. I had to pass an exam on Friday in Net Centric Systems, our bachelor practice asks for a lot of attention (still writing documentation and now even a bit of JSP) and last not least "my kids" (you know, not "my" - I am just the young group leader) starting to get nervous about next week (week 11) in Roma and also started to clean and prepare the old party basement. The basement was closed for at least two years now and looked exactly like this: Bad musty smell, mildew and a lot of (death) bugbears: spiders and alike. Better now but still a lot to do. Anyway, i guess that this is not of common interest. 😉

So, what do you did APT related? As usual: answered a few mails (mostly off list this time so no pointers). The biggest point code wise is the enhancement of the apt-cache (r)depends commands - they do share code now instead of maintaining their own code-copies. I did this as I need the commands a bit for my second big think this week which isn't finished yet: A shell-scripted test framework to be able to automate what currently is done manual or never. Its still missing a lot: e.g. setting up environment for dpkg to test complete runs, an easy way to deploy real test deb-packages and a way to test also http connection with a local web server as file is easy to test but not always comparable to the http method which is used far more often in real world - e.g. in the new compressed index feature.

I also mentally start to prepare "my" presentation as each GSoC project is present on DebConf for 5 minutes. I can't be around for it myself as i need to stay here in town respective i am away with my kids next week already as described above. Maybe next time… 🙂

GSoC Half-time for me (and all the others). Good time to look a bit back and forward. I tried that in the evaluation we were required to fill with some interesting and some strange questions.

I failed to make a copy of questions/answers as I thought Google wouldn't take it offline - I was wrong. 😕 Anyway, I remember still the tone of the strangest question: "How do you describe the quality of work you have produced so far?" [ ] Exceeds my personal expectations [ ] Meet my personal expectations [ ] Doesn't meet my personal expectations

Michael said that is a "german people" problem, so let me explain why I find that strange: The question is: "What I have expected?" - "I will write crap". Mhh okay, I "meet" my expectations… Hopefully not. I don't even want to imagine what "exceeds" in this context would mean. 😉 "I expect the best possible code" - "I made a few mistakes, so I doesn't meet it all time" mhh okay, nobody is perfect (my name is nobody) so I guess that is okay and I "meet" my expectations. Exceed is completely impossible by definition…

Other interesting questions were asked, most of them related to communication and interaction with the mentor. How often, how much and about what do we communicate? Do I hate him for that? Is he an idiot and will I stop working on the project better now than later? Just try to imagine what I answered to these questions (which were formulated a bit different obviously). 🙂

Beside that, I didn't do "much" as the multiarch conflict handling on providers were a bit hard to track down and a lot of talk was needed - with Michael as well as with the evaluation and stuff. Just in time to motivate me to do it was btw Cyril Brulebois's thank you mail on the xorg-upgrade stuff. I chose "communication" as the interesting as well as most demanding part of GSoC so that was a good example. 🙂 And while I haven't posted something to the discussions about "apt vs. aptitude" and "sense of priorities" so far I followed them closely. Maybe, depending on how they will evolve… We will see it in the next week. 🙂

If you hear the word: "Dynamic" what do you thing?

APT includes a DynamicMMap to build his package cache. If the user added enough sourcelist entries the user saw the DynamicMMap in full action: It was so dynamic that it even ran out of room. I don't know about you, but this drove me crazy for a long time. Nearly a year ago i implemented a Grow() method which tried to grow the DynamicMMap if needed. That worked in theory - and this even for kfreebsd & co which do not have mremap available for moving - for these systems it uses a char[] array and realloc.

The only problem was: While the binary cache tries hard to be reusable in a subsequent call - otherwise the cache would make much sense after all - while building the cache it couldn't be remapped. All sorts of iterators pointed into the MMap so just enabling MOVEABLE resulted in a huge SEGFAULT probability…

Why I am writing all this and why I try to talk in past tense here? 5 bugs (and counting) are already talking about it - and they are marked as done now. 🙂 The theory is simple: Each pointer into the MMap registers itself and every time the MMap gets a new location some pointer arithmetic is used to fix it.

The theory is easy, in practice it was a bit harder to achieve as the methods causing a grow need to be identified and also all the pointers into the MMap - gdb is a good friend now - and I managed to disprove some bug report participators saying it would be impossible and/or too hard. Yeaha! 🙂

Anyway Mid term evaluation is now, so I suspect I will be a bit more verbose next week and so I am better of saving the reader now from all the boring details. 😉

Finally, I managed to close an existing (and not self-made) bug again 🙂 #196021: Order of package names in argument list shouldn't be relevant

The preparation for it happened the last few weeks with the CacheSets stuff and while sounding completely unspectacular I love it as it results in a bit saner (in my eyes) code and I could use funky c++ stuff like std::for_each for it. 🙂

What I want now is to change the display code for all the Lists a bit as some of them e.g. still print out the pseudo packages unfortunately. Also, the pseudo packages confuse a bit the trivial action code and last but not least some code in apt-get is a bit buggy which could be fixed now that we have more information (which packages are requested, which are extra?).

As a completely different topic I experiment with NotEquals for the MultiArch-Breaks dependencies. I have forgotten why I used two dependencies (Less and Greater) for this before… Anyway, it is active now again and we will see what breaks with these Breaks. Breaks are in general an interesting topic, as they were the last week with the Xorg upgrade, I in the mean time understand that the handling of Breaks in APT is… lets say limited. I still don't know exactly why APT wants to remove xorg in this situation but I have somehow given up on it so far. I saved my testcase, maybe I will go back to it by the time it is not as hot as it is now all day long… That is maybe also the reason why the Grow-Code for MMap still doesn't want to work…

Beside that, at Friday I finally revised my Google Sommer of Code welcome letter 🙂 Unfortunately the customs office chooses to open this letter - maybe in the hope to find drugs and big amounts of cash - but it includes "only" a notebook (the pen&paper thing), two stickers, a three pages welcome letter/payment instructions and a subletter including even more instructions and the gift card. Google claims to have included also a pen in the listing attached to the letter. That was NOT included, so either Google made a mistake or someone at FedEx or the customs office needed a new pen… Maybe I will find the time to mail them today to ask for it… I mean, I have a notebook now, but without a pen… I NEED ONE. Douglas Adams included a nice theory about these pens in one of his books, I have the strong feeling this is not Sci-Fi (as with the rest of the book) …

See/read you next week 🙂

Please, please, don't punch me too hard to post the weekly report too late! I have a good excuse after all. 😉

Petter Reinholdtsen did and still does some cool upgrade tests from lenny to squeeze and is hit by a few problems: The first is what APT wants to remove in the upgrade. Some are fine and nobody will miss them, but i am not sure if all users will be happy if the xserver is removed in the upgrade. 😉

My normal approach to track such thinks down is based on guesswork and luck: Find a (at best) minimal test case for this problem as you don't want to perform dependency resolving by hand on 1000 packages. The problem is, to find a minimal test case you need to understand the problem, but if you would, you wouldn't search for one…

You might think that this can consume a lot of time. You are wrong. It consumes even more time. 😉

Thankfully, after a lot of work and time you find a test case and shortly after even a solution. The solution is mostly a very short change which gives the false impression it was easy. It becomes strange if you know how to work around it but you can't find the actual cause as seen now in the Xorg upgrade. I really don't know why I tried what I did - I assume it was an enlightenment from the dependency-god - but it worked out. At least I have now something to look for…

The second thing is even nicer: APTs downloader can end in an endless loop in some undefined circumstances. This exists for a LONG time, but it never happened to me, never under a debug build or even with debug options enabled. So it is again something you need to guess: How to reproduce it in a controlled (minimal) environment. We haven't made it so far…

Same code different topic is the apt-get improvement idea - why such threads always have such a bad name? Scheduling package downloads across different (available) sources. I am not completely sure if I like that: It makes downloads faster - at least for now as nobody uses it. I am a bit frighted that users will all add the five big (round-robin) mirrors in their sourcelist which would only increase the speed penalty for all users maybe… I am more for a more intelligent choice of a mirror than using 10 at the same time (as this requires also 10 times the metadata download).

But I have also done some "real" work, you know, but it feels mostly like writing yet more emails: Improve support for installing 32-bit libraries on 64-bit systems. 😉 Still I also committed a few things: Improving the GlobalError class was a long-standing todo list item and I am happy that I finally came around to do it as it was needed for my CacheSets.

As I was recently told again that people care for LoCC (Lines of Code Changes) here is the one for this week: 19 files changed, 763 insertions(+), 361 deletions(-)

So, enough for last week, i am already behind schedule for this week. 😉

Last week was full of distractions: On one hand I was required to do some work on our bachelor lab which is placed under the Code Recommenders umbrella. This includes mostly discussions and writing documentation which is quite boring and sometimes even frustrating as the requirements change every once in a while -- not the best conditions if you should work according to the Unified Process… Some topic different track was the four hours presentation training. I learned a bit, especially that I seem to have a "melodic voice" which is a pleasure to listen to - at least this is what my coach said. I personally can't confirm that as I do not listen to me, so I recommend to meet me in person to validate or disprove it. 🙂

On the other hand the World Cup sucks up some time. Most games are quite boring unfortunately - especially if a "odds-on favorite" (at least before the World Cup) is one of the playing teams, but every time I can't watch a game I see later that it was a good game… PAR:SVK for example looked like a good game in the replay. BRA against CIV yesterday on the other hand had glory moments but was all in all pretty lame. Especially "glory" was it to use a double-"hand of god" and lying to the referee. If the players would invest more of their time into playing good instead of giving the referee headaches… but I am dreaming.

Good that I had also a few more other distractions like the Strawberry Festival in my home town, my little cousin (who becomes my daughter at the festival for 5 years now - one of the more popular urban myths…), that my other cousin is now the vinaceous queen of your town and in general all the small and medium events happening now and then around strawberries, -bowle and wine…

So you might wonder if I have done ANYTHING and I can say: Yes, but the changes are for now invisible and groundwork for Week 5. In my branch for example is apt-get already half-ported to the CacheSets. Interesting is now how to handle cases like e.g. virtual packages in a way that the Sets can be used in apt-get as well as apt-cache… Another thing which fall into my mind today morning in the train is as simple as: Should the responses to the queries in apt-cache not be in the same order as the request? The answer is yes, so I will need CacheLists also. Hopefully this can be done as easy as I currently think. 🙂

And finally the pro forma MultiArch content: Setting Candidate as well as Reinstall in the DepCache now accounts for Pseudo packages (that is bigger and more important than it sounds). 🙂 Slightly related is Adobes flash drop for linux-amd64. Hopefully the MultiArch future will protect us a bit more. I am personally not affected as I have gnash now for a while up and running and it works for all what I need (basically nothing), but this move made me think of how much many people depend on closed source software, how badly especially flash is maintained by upstream, how easy it is for a company to ignore the needs of a "niche" like linux-amd64 and what will happen if they decide tomorrow to drop also linux-i386 with the promise to publish a new version "in the future". Remember, even "Duke Nukem Forever" is/was promised to release "in the future". "It's done then it's done" is fine as long as you have no customers or a previous version which is still supported, but this move is more like debian would announce shortly after squeeze release: "Sorry fellows, we came to the conclusion that squeeze on amd64 has a few bugs we don't want to fix and instead want work on squeeze+1. We have therefore removed squeeze-amd64 from our mirrors. We encourage you to wait for squeeze+1 which will be released 'in the future'. Stay tuned."

I hope this will ever happen. Free Software for the win.