APT for package self-builders

One of the main jobs of a package manager like apt is to download packages (ideally in a secure way) from a repository so that they can be processed further – usually installed. FSVO "normal user" this is all there ever is to it in terms of getting packages.

Package maintainers and other users rolling their own binary packages on the other hand tend to have the packages they want to install and/or play-test with already on their disk. For them, it seems like an additional hassle to push their packages to a (temporary) repository, so apt can download data from there again… for the love of supercow, there must be a better way… right?

For the sake of a common start lets say I want to modify (and later upload) hello, so I acquire the source via apt source hello. Friendly as apt is it ran dpkg-source for me already, so I have (at the time of writing) the files hello_2.10.orig.tar.gz, hello_2.10-1.debian.tar.xz and hello_2.10-1.dsc in my working directory as well as the extracted tarballs in the subdirectory hello-2.10.

Anything slightly more complex than hello probably has a bunch of build-dependencies, so what I should do next is install build-dependencies: Everyone knows apt build-dep hello and that works in this case, but given that you have a dsc file we could just as well use that and free us from our reliance on the online repository: apt build-dep ./hello_2.10-1.dsc. We still depend on having a source package built previously this way… but wait! We have the source tree and this includes the debian/control file so… apt build-dep ./hello-2.10 – the later is especially handy if you happen to add additional build-dependencies while hacking on your hello.

So now that we can build the package have fun hacking on it! You probably have your preferred way of building packages, but for simplicity lets just continue using apt for now: apt source hello -b. If all worked out well we should have now (if you are on a amd64 machine) also a hello_2.10-1_amd64.changes file as well as two binary packages named hello_2.10-1_amd64.deb and hello-dbgsym_2.10-1_amd64.deb (you will also get a hello_2.10-1_amd64.buildinfo which you can hang onto, but apt has currently no way of making use of it, so I ignore it for the moment).

Everyone should know by now that you can install a deb via apt install ./hello_2.10-1_amd64.deb but that quickly gets boring with increasing numbers, especially if the packages you want to install have tight relations. So feel free to install all debs included in a changes file with apt install ./hello_2.10-1_amd64.changes.

So far so good, but all might be a bit much. What about install only some debs of a changes file? Here it gets interesting as if you play your cards right you can test upgrades this way as well. So lets add a temporary source of metadata (and packages) – but before you get your preferred repository builder setup and your text editor ready: You just have to add an option to your apt call. Coming back to our last example of installing packages via a changes file, lets say we just want to install hello and not hello-dbgsym: apt install --with-source ./hello_2.10-1_amd64.changes hello.

That will install hello just fine, but if you happen to have hello installed already… apt is going to tell you it has already the latest version installed. You can look at this situation e.g. with apt policy --with-source ./hello_2.10-1_amd64.changes hello. See, the Debian repository ships a binary-only rebuild as 2.10-1+b1 at the moment, which is a higher version than the one we have locally build. Your usual apt-knowledge will tell you that you can force apt to install your hello with apt install --with-source ./hello_2.10-1_amd64.changes hello=2.10-1 but that isn't why I went down this path: As you have seen now metadata inserted via --with-source participates as usual in the candidate selection process, so you can actually perform upgrade tests this way: apt upgrade --with-source ./hello_2.10-1_amd64.changes (or full-upgrade).

The hello example reaches its limits here, but if you consider time travel a possibility we will jump back into a time in which hello-debhelper existed. To be exact: Right to the moment its maintainer wanted to rename hello-debhelper to hello. Most people consider package renames hard. You need to get file overrides and maintainerscripts just right, but at least with figuring out the right dependency relations apt can help you a bit. How you can feed in changes files we have already seen, so lets imagine you deal with with multiple packages from different sources – or just want to iterate quickly! In that case you want to create a Packages file which you would normally find in a repository. You can write those by hand of course, but its probably easier to just call dpkg-scanpackages . > Packages (if you have dpkg-dev installed) or apt-ftparchive packages . > Packages (available via apt-utils) – they behave slightly different, but for our proposes its all the same. Either way, ending up with a Packages file nets you another file you can feed to --with-source (sorry, you can't install a Packages file). This also allows you to edit the dependency relations of multiple packages in a single file without constant "fiddle and build" loops of the included packages – just make sure to run as non-root & in simulation mode (-s) only or you will make dpkg (and in turn apt) very sad.

Of course upgrade testing is only complete if you can influence what is installed on your system before you try to upgrade easily. You can with apt install --with-source ./Packages hello=2.10-1 -s -o Dir::state::status=/dev/null (it will look like nothing is installed) or feed a self-crafted file (or some compressed /var/backups/dpkg.status file from days past), but to be fair that gets a bit fiddly, so at some point its probably easier to write an integration test for apt which are just little shellscript in which (nearly) everything is possible, but that might be the topic of another post some day.

Q: How long do I have to wait to use this?

A: I think I have implemented the later parts of this in the 1.3 series. Earlier parts are in starting with 1.0. Debian stable (stretch) has the 1.4 series, so… you can use it now. Otherwise use your preferred package manager to upgrade your system to latest stable release. I hope it is clear which package manager that should be… 😉︎

Q: Does this only work with apt?

A: This works just the same with apt-cache (where the --with-source option is documented in the manpage btw) and apt-get. Everything else using libapt (so aptitude included) does not at the moment, but potentially can and probably will in the future. If you feel like typing a little bit more you can at least replicate the --with-source examples by using the underlying generic option: aptitude install -s hello-dbgsym -o APT::Sources::With::=./hello_2.10-1_amd64.changes (That is all you really need anyhow, the rest is syntactic sugar). Before you start running off to report bugs: Check before reporting duplicates (and don't forget to attach patches)!

Q: Why are you always typing ./packages.deb?

A: With the --with-source option the ./ is not needed actually, but for consistency I wrote it everywhere. In the first examples we need it as apt needs to know somehow if the string it sees here is a package name, a glob, a regex, a task, … or a filename. The string "package.deb" could be a regex after all. And any string could be a directory name… Combine this with picking up files and directories in the current directory and you would have a potential security risk looming here if you start apt in /tmp (No worries, we hadn't realized this from the start either).

Q: But, but, but … security anyone?!?

The files are on your disk and apt expects that you have verified that they aren't some system-devouring malware. How should apt verify that after all as there is no trustpath. So don't think that downloading a random deb suddently became a safe thing to do because you used apt instead of dpkg -i. If the dsc or changes files you use are signed and you verfied them through, you can rest assured that apt is verifying that the hashes mentioned in those files apply to the files they index. Doesn't help you at all if the files are unsigned or other users are able to modify the files after you verified them, but apt will check hashes in those cases anyhow.

Q: I ❤︎ u, 🍑︎ tl;dr

Just 🏃︎ those, you might 😍︎ some of them:

apt source hello
apt build-dep ./hello-*/ -s
apt source -b hello
apt install ./hello_*.deb -s
apt install ./hello_*.changes -s
apt install --with-source ./hello_*.changes hello -s
apt-ftparchive packages . > ./Packages
apt upgrade --with-source ./Packages -s

P.S.: If you have expected this post to be published sometime inbetween the last two months… welcome to the club! I thought I would do it, too. Lets see how long I will need for the next one… I have it partly written already, but that was the case for this one as well… we will see.

APT for DPL Candidates

Today is a special day for apt: 20 years ago after much discussion in the team as well as in the Debian project at large "APT" was born.

What happened in all these years? A lot! But if there is one common theme then it is that many useful APT features, tricks and changes are not as known to the general public or even most Debian Developers as they should be.

A few postings are unlikely to change that, but I will try anyhow and this post is the start of a mini-series of "APT for …" articles showing off things: For the first installment I want to show nothing less but the longest running vote-rigging scheme in the known free (software) world. But lets start from the beginning:

Humans pride themselves having evolved over simple animals following their instincts by having a mind of their own to form decisions. On this concept, humans hold votes to agree upon stuff, including the Debian Project Leader for the next year.

DPL candidates are encouraged to provide a platform and a discussion between the candidates and the voters ensures that everyone can form a well informed opinion on the candidates in question and based on this information choose a candidate with their own free will.

Or, That is at least the theory Debian Developers want to believe in.

The following table tallies each leader vote since 1999 with the information if the candidate (68 over all, 37 unique) had contributed to APT (31/13), dpkg (29/17) or both (18/9) for cases I could identify (if I missed anything feel free to get in touch). The candidate in bold has won the election in the given year, candidates in italics won a later election:

Year Candidate APT? dpkg?
1999 Joseph Carter no no
Ben Collins no1 yes
Wichert Akkerman no yes
Richard Braakman no yes
2000 Ben Collins no1 yes
Wichert Akkerman no yes
Joel Klecker no yes
Matthew Vernon no yes
2001 Branden Robinson yes2 yes
Anand Kumria no yes
Ben Collins yes1 yes
Bdale Garbee yes3 no
2002 Branden Robinson yes2 yes
Raphaël Hertzog yes4 yes
Bdale Garbee yes3 no
2003 Moshe Zadka no no
Bdale Garbee yes3 no
Branden Robinson yes2 yes
Martin Michlmayr yes5 no
2004 Martin Michlmayr yes5 no
Gergely Nagy no no
Branden Robinson yes2 yes
2005 Matthew Garrett no no
Andreas Schuldei no yes
Angus Lees no no
Anthony Towns yes6 yes
Jonathan Walther no no
Branden Robinson yes2 yes
2006 Jeroen van Wolffelaar yes7 yes
Ari Pollak no no
Steve McIntyre yes8 no
Anthony Towns yes6 yes
Andreas Schuldei no yes
Jonathan (Ted) Walther no no
Bill Allombert no yes
2007 Wouter Verhelst no no
Aigars Mahinovs no no
Gustavo Franco no no
Sam Hocevar yes10 yes
Steve McIntyre yes8 no
Raphaël Hertzog yes4 yes
Anthony Towns yes6 yes
Simon Richter yes9 yes
2008 Marc Brockschmidt no no
Raphaël Hertzog yes4 yes
Steve McIntyre yes8 no
2009 Stefano Zacchiroli yes11 no
Steve McIntyre yes8 no
2010 Stefano Zacchiroli yes11 no
Wouter Verhelst no no
Charles Plessy yes12 yes
Margarita Manterola no yes
2011 Stefano Zacchiroli yes11 no
2012 Wouter Verhelst no no
Gergely Nagy no no
Stefano Zacchiroli yes11 no
2013 Gergely Nagy no no
Moray Allan no no
Lucas Nussbaum no no
2014 Lucas Nussbaum no no
Neil McGovern no no
2015 Mehdi Dogguy no no
Gergely Nagy no no
Neil McGovern no no
2016 Mehdi Dogguy no no
2017 Mehdi Dogguy no no
Chris Lamb yes13 yes
2018 Chris Lamb14 yes13 yes

We can see directly that until recently it was nearly mandatory to have contributed to apt or dpkg to be accepted as a DPL candidate. The recent no streak before Chris entered the table gets doubly weird if we factor in that I joined Debian and the APT project around 2009… We might get to the bottom of this coincident in the future, but for now lets get back to the topic at hand:

DDs have no free will

It is hard to accept for a human, but the table above shows that DDs aren't as free in their choice as they think they are; they follow a simple rule:

If at least one of the candidates contributed to APT, an APT contributor wins the election.

You can read this directly from the table above (20 votes total, including 6 votes without an apt candidate). Interestingly, the same rule for dpkg does not hold at all. In fact there are years in which the defining difference for the winning candidate is that he15 hasn't contributed to dpkg but only to apt (e.g. 2002).

Praying via bugreports and sacrifices in the form of patches in the Pantheon of the Supercow, the deity@ mailinglist, can have profound effects: Take the very first elections as an example: After being an unsuccessful candidate in 1999 and 2000, candidate Ben implemented the rsh transport method for apt and as a result became DPL in 2001.

And as if that wouldn't be enough, being on good terms with Supercow has additional benefits:

Contribution beats Re-Election

While it seems like DPLs are granted a second term if they wish the recent 2017 election shows that contributing to APT is stronger. Two other non-re-elections are on record: In 2003 where the bugreporter Bdale lost against the patchprovider Martin, so contribution size and recency seem to play a role as well, but that might not be everything there is to infights between contributors as shown in 2007 where Anthony lost against Sam in the biggest vote so far with 5 out of 8 candidates supported by apt (including the present and future DPL in this year).

The "Super" rubs of on the DPL

Many DPLs run for two terms, but only a single one managed to run a third time: After unsuccessful campaigning in 2009 Stefano not only worked on apt in 2010 and the following year(s) but also consulted with a certain highpriest of the Supercow netting a record three-year stint as DPL as a result.

It is to be seen if intercession of a highpriest is needed for long DPL terms, but it certainly doesn't hurt – and I make myself of course selflessly available (for a reasonable monetary offering) as said highpriest should a DPL (candidate) be in need of my divine bovine help (again)…

Every contribution matters – for real!

No contribution is too small, everything counts & supercow sees everything. Even "just" downgrading the severity of a bug counts10. Supercow values all contributors: Join their rank today and win the next DPL election in 2019!

Of course, supercow likes big and groundbreaking patches as much as the next project, but while other projects are just talking about how they like testers, bugreporters, translators and documentation improvers we in the apt project have the election-rigging data of 20 years to proof it! Supercow really cares for its contributors!

So to all past, present and future DPL candidates: Thanks for your contributions to APT (and Debian)!

That… this… what the f…edora?!?

Look at the calendar: Its not only easter sunday, its also the beginning of voting period for DPL 2018. What better day would there be for some fun about humans, genesis, elections and the 20th birthday of apt.

I promise that future installments in the series will be more practically useful but until then enjoy the festive days (if applicable) around apts birthday, have fun, take care and make sure to contribute to apt!


  1. Contributed the RSH/SSH method in 2000. Won the following election after two unsuccessful rounds. 

  2. Credited in AUTHORS for "Man Page Documentation" 

  3. Early tester as shown in e.g. #45050 

  4. Bugreporter and provider of draft patch, e.g. #793360 

  5. Bugreporter and patch provider, e.g. #417090 

  6. Multiple patches over the years since 2005 including the latest reimplementation of rred 

  7. Bugreporter and patch provider: #384182 

  8. Bugreporter and tester, e.g. #218995 

  9. Bugreporter, tester and patch provider, e.g. #509866 

  10. RC bug triager, e.g. #454666 

  11. Multiple bugreports and patches, including pushing & documenting EDSP 

  12. Documentation patches, e.g. #619088 

  13. Tester and patch provider, e.g. #848721 

  14. The vote hasn't happened yet, but NOTA is by definition not an apt contributor and hence can't win as outlined in this post. 

  15. To this day Debian had no DPL for which "he" does not apply. With this information, you might be able to change this in future elections! 

Winning the Google Open Source Lottery

I don't know about you, but I frequently get mails announcing that I was picked as the lucky winner of a lottery, compensation program or simply as "business associate". Obvious Spam of course, that never happens in reality. Just like my personal "favorite" at the moment: Mails notifying me of inheritance from a previously (more or less) unknown relative. Its just that this is what has happend basically a few weeks ago in reality to me (over the phone through) – and I am still dealing with the bureaucracy required of teaching everyone that I had absolutely no contact in the last two decades with the person for which I am supposed to be one of the legal successors now, regardless of how close the family relation is on paper… but that might be the topic of another day.

On the 1st March a mail titled "Google Open Source Peer Bonus Program" looked at first as if it would fall into this lottery spam class. It didn't exactly help that the mail was multipart HTML and text, but the text really only the text, not mentioning the embedded links used in the HTML part. It even included a prominent and obvious red flag: "Please fill out the form". 20% Bayes score didn't come from nothing. Still, for better or worse the words "Open Source" made it unlikely to be spam similar to how the word PGP indicates authenticity. So it happened, another spam message became true for me. I wonder which one will be next…

You have probably figured out by now that I didn't know that program before. Kinda embarrassing for a previous Google Summer of Code student (GSoC is run by the same office), but the idea behind it is simple: Google employees can nominate contributors to open source stuff for a small monetary "thank you!" gift card. Earlier this week winners for this round were announced – 52 contributors including yours truly. You might be surprised, but the rational given behind my name is APT (I got a private mail with the full rational from my "patron", just in case you wonder if at least I would know more).

It is funny how a guy who was taken aback by the prospect of needing a package manager like YaST to use Linux contributed just months later the first patch to apt and has roughly 8 years later amassed more than 2400 commits. It's birthday season in my family with e.g. mine just a few days ago, so its seems natural that apt has its own birthday today just as if it would be part of my family: 19th years this little bundle of bugs joy is now! In more sober moments I wonder sometimes how apt and I would have turned out if we hadn't meet. Would apt have met someone else? Would I? Given that I am still the newest team member and only recently joined Debian as DD at all…

APT has some strange ways of showing that it loves you: It e.g. helps users compose mails which end in a dilemma to give a recent example. Perhaps you need to be a special kind of crazy1 to consider this good, but as I see it apt has a big enough userbase that regardless of what your patch is doing, someone will like it. That drastically increases the chances that someone will also like it enough to say so in public – offsetting complains from all those who don't like the (effects of the) patch which are omnipresent. And twice in a blue moon some of those will even step forward and thank you explicitly. Not that it would be necessary, but it is nice anyhow. So, thanks for the love supercow, Google & apt users! 🙂︎

Or in other words: APT might very well be one of the most friendly (package manager related) project to contribute to as the language-specific managers have smaller userbases and hence a smaller chance of having someone liking your work (in public)… so contribute a patch or two and be loved, too! 💖︎

Disclaimer: I get no bonus for posting this nor are any other strings attached. Birthdays are just a good time to reflect. In terms of what I do with my new found riches (in case I really receive them – I haven't yet so that could still be an elaborate scam…): APT is a very humble program, but even it is thinking about moving away from a dev-box with less than 4 GB of RAM and no SSD, so it is happily accepting the gift and expects me to upgrade sooner now. What kind of precedence this sets for the two decades milestone next year? If APT isn't obsolete by then… We will see.

  1. which even ended up topping Hacker News around New Year's Eve… who would have thought that apt and reproducibility bugs are top news ;) 

the new apt-transport-tor

It happened: Now that I am an uploading DD for a few months I finally made my first upload of a package – mind you, not of apt, but of a package I declared my intend to "steal" from another person a few weeks ago on deity@ and later also in a bugreport (#835128).

The result is that apt-transport-tor which used to be maintained by Tim Retout as a modified copy of apt code is now maintained by the APT team (with him and me as uploaders) using the apt code directly via a few symlinks.

That brings along a bunch of changes which I mentioned in the list/bug as well, but for completeness:

I had tried a few times to get people to provide feedback, but there wasn't much. I guess this is good as it means nobody has any complains about it. We will see if that will change now that it is on its way to archive, buildds, mirrors and users: Brace for impact in any case!

Week 12: Happy birthday Debian!

From time to time I am asked why I do such a boring thing as working on a many years old application, dealing with code and bugs from the last decade, an even more boring commandline only application and finally why all this for free and for this crazy "linux" thing which isn't even called Ubuntu…

It's sometimes hard to answer this in a good way as you need to educate the questioner all the way through GNU, free beer vs free software, linux, debian, package management to APT. In two minutes and most of them get distracted by "free beer"… and there are many of them. So many that you could sometimes come to the (false) conclusion that you are the only one. Okay, in debian that is unrealistic with > 900 DDs, many DMs and a few more contributors without an official status, but the tendency to feel a bit "alone" is still everytime you look around in a class and see everywhere non-debian and mostly even non-linux operation systems running. And even if you made it so far to find an debian-related user you still have the way down to APT… its not completely uncommon to hear: "Oh yeah, fine, but pretty oldschool isn't? Good that someone developed a replacement for it. Its called $graphical-apt-frontend!" And you know that you talk with an expert if he says "I use aptitude exclusively. I have even purged apt with it…"

So everything you expect in the morning opening your emails is maybe a bunch of bugreports - but not today, today a lot of messages started with "thanks"! Nice difference and a very cool idea! So a big thanks for this thank-project. I hope this project helps a bit in motivating contributors to continue their work and might even motivate more people to start contributing 🙂︎

Contribution is the perfect keyword for the real content of the post: A few upgrade-related bugs needed to be debugged like #591882 there we are still in the process to understand all failed package upgrades - beside that a few small fixes like a corrected LongDesc handling and improving my testframework.

The upgrade "fix" we call "FixByInstall" is btw a quite interesting thing as it looks like an error to commit it. Very few lines of code, but they introduce an autoinstall enabled MarkInstall call in the resolver which is a premiere… and dangerous but it fixes a problem and maybe all the thinking about ways it could possibly break which can't be produced in testcases by now is not needed… We will see.

The rest of the week was spent with writing in nearly endless amount of emails to bugreports, discussions and even to APT2 announcements, threads and ITPs. So much, that i don't want to comment that further expect that you can find most of it in my crosspost to merge the independent threads: http://lists.debian.org/deity/2010/08/msg00107.html

I recommend to try running the posted APT3 brainfuck code btw, just save the code as apt3 and run it with beef (apt-get install beef before of course if needed).

For my own reference, the APT3 brainfuck code is:


And no, APT3 was not a direct reaction to APT2 - it was an initial a bad joke I told a few people before and on the UDS in Dallas 2009 referring to cupt written in perl… And because people tend to misunderstand me then I say such stuff: I have nothing against cupt nor APT2 (expect name) and I even like them for giving interesting ideas what to do next… and next is currently to finalize APT in the hope that it will migrate to unstable and testing some time in the future. Also, GSoC starts to come to an end… and this week report has also reached his end. See next week for more…

This blog has no centralized comment section. Feel free to comment on it in your own blog, some other page or channel you are on to discuss stuff and/or contact me by mail. I am happy to add pointers and updates as needed.