APT for package self-builders
One of the main jobs of a package manager like apt is to download packages (ideally in a secure way) from a repository so that they can be processed further – usually installed. FSVO "normal user" this is all there ever is to it in terms of getting packages.
Package maintainers and other users rolling their own binary packages on the other hand tend to have the packages they want to install and/or play-test with already on their disk. For them, it seems like an additional hassle to push their packages to a (temporary) repository, so apt can download data from there again… for the love of supercow, there must be a better way… right?
For the sake of a common start lets say I want to modify (and later upload)
hello, so I acquire the source via apt source hello. Friendly as
apt is it ran dpkg-source for me already, so I have (at the time of writing)
the files hello_2.10.orig.tar.gz, hello_2.10-1.debian.tar.xz and
hello_2.10-1.dsc in my working directory as well as the extracted tarballs
in the subdirectory hello-2.10.
Anything slightly more complex than hello probably has a bunch of
build-dependencies, so what I should do next is install build-dependencies:
Everyone knows apt build-dep hello and that works in this case, but given
that you have a dsc file we could just as well use that and free us from
our reliance on the online repository: apt build-dep ./hello_2.10-1.dsc.
We still depend on having a source package built previously this way…
but wait! We have the source tree and this includes the debian/control file so…
apt build-dep ./hello-2.10 – the later is especially handy if you happen to
add additional build-dependencies while hacking on your hello.
So now that we can build the package have fun hacking on it! You probably
have your preferred way of building packages, but for simplicity lets just
continue using apt for now: apt source hello -b. If all worked out well
we should have now (if you are on a amd64 machine) also a
hello_2.10-1_amd64.changes file as well as two binary packages named
hello_2.10-1_amd64.deb and hello-dbgsym_2.10-1_amd64.deb (you will also
get a hello_2.10-1_amd64.buildinfo which you can hang onto, but apt has
currently no way of making use of it, so I ignore it for the moment).
Everyone should know by now that you can install a deb via apt install
./hello_2.10-1_amd64.deb but that quickly gets boring with increasing numbers,
especially if the packages you want to install have tight relations.
So feel free to install all debs included in a changes file with
apt install ./hello_2.10-1_amd64.changes.
So far so good, but all might be a bit much. What about install only some
debs of a changes file? Here it gets interesting as if you play your cards
right you can test upgrades this way as well. So lets add a temporary
source of metadata (and packages) – but before you get your preferred
repository builder setup and your text editor ready: You just have to add an
option to your apt call. Coming back to our last example of installing packages
via a changes file, lets say we just want to install hello and not
hello-dbgsym: apt install --with-source ./hello_2.10-1_amd64.changes
hello.
That will install hello just fine, but if you happen to have hello installed
already… apt is going to tell you it has already the latest version installed.
You can look at this situation e.g. with apt policy --with-source
./hello_2.10-1_amd64.changes hello. See, the Debian repository ships a
binary-only rebuild as 2.10-1+b1 at the moment, which is a higher version than
the one we have locally build. Your usual apt-knowledge will tell you that you
can force apt to install your hello with apt install --with-source
./hello_2.10-1_amd64.changes hello=2.10-1 but that isn't why I went down this
path: As you have seen now metadata inserted via --with-source participates
as usual in the candidate selection process, so you can actually perform
upgrade tests this way: apt upgrade --with-source
./hello_2.10-1_amd64.changes (or full-upgrade).
The hello example reaches its limits here, but if you consider time travel a
possibility we will jump back into a time in which hello-debhelper existed.
To be exact: Right to the moment its maintainer wanted to rename
hello-debhelper to hello. Most people consider package renames hard. You
need to get file overrides and maintainerscripts just right, but at least with
figuring out the right dependency relations apt can help you a bit. How
you can feed in changes files we have already seen, so lets imagine you deal
with with multiple packages from different sources – or just want to iterate
quickly! In that case you want to create a Packages file which you would
normally find in a repository. You can write those by hand of course, but its
probably easier to just call dpkg-scanpackages . > Packages (if you have
dpkg-dev installed) or apt-ftparchive packages . > Packages (available via
apt-utils) – they behave slightly different, but for our proposes its all the
same. Either way, ending up with a Packages file nets you another file you can
feed to --with-source (sorry, you can't install a Packages file). This also
allows you to edit the dependency relations of multiple packages in a single
file without constant "fiddle and build" loops of the included packages – just
make sure to run as non-root & in simulation mode (-s) only or you will make
dpkg (and in turn apt) very sad.
Of course upgrade testing is only complete if you can influence what is
installed on your system before you try to upgrade easily. You can with apt
install --with-source ./Packages hello=2.10-1 -s -o Dir::state::status=/dev/null
(it will look like nothing is installed) or feed a self-crafted file (or some
compressed /var/backups/dpkg.status file from days past), but to be fair that
gets a bit fiddly, so at some point its probably easier to write an integration
test for apt which are just little shellscript in which (nearly) everything is
possible, but that might be the topic of another post some day.
Q: How long do I have to wait to use this?
A: I think I have implemented the later parts of this in the 1.3 series. Earlier parts are in starting with 1.0. Debian stable (stretch) has the 1.4 series, so… you can use it now. Otherwise use your preferred package manager to upgrade your system to latest stable release. I hope it is clear which package manager that should be… 😉
Q: Does this only work with apt?
A: This works just the same with apt-cache (where the --with-source option is
documented in the manpage btw) and apt-get. Everything else using libapt (so
aptitude included) does not at the moment, but potentially can and probably
will in the future. If you feel like typing a little bit more you can at least
replicate the --with-source examples by using the underlying generic option:
aptitude install -s hello-dbgsym -o APT::Sources::With::=./hello_2.10-1_amd64.changes
(That is all you really need anyhow, the rest is syntactic sugar).
Before you start running off to report bugs: Check before reporting duplicates
(and don't forget to attach patches)!
Q: Why are you always typing ./packages.deb?
A: With the --with-source option the ./ is not needed actually, but for
consistency I wrote it everywhere. In the first examples we need it as apt
needs to know somehow if the string it sees here is a package name, a glob, a
regex, a task, … or a filename. The string "package.deb" could be a regex after
all. And any string could be a directory name… Combine this with picking up
files and directories in the current directory and you would have a potential
security risk looming here if you start apt in /tmp (No worries, we hadn't
realized this from the start either).
Q: But, but, but … security anyone?!?
The files are on your disk and apt expects that you have verified that they
aren't some system-devouring malware. How should apt verify that after all as
there is no trustpath. So don't think that downloading a random deb suddently
became a safe thing to do because you used apt instead of dpkg -i. If the dsc
or changes files you use are signed and you verfied them through, you can rest
assured that apt is verifying that the hashes mentioned in those files apply to
the files they index. Doesn't help you at all if the files are unsigned or other
users are able to modify the files after you verified them, but apt will check
hashes in those cases anyhow.
Q: I ❤ u, 🍑 tl;dr
Just 🏃 those, you might 😍 some of them:
apt source hello
apt build-dep ./hello-*/ -s
apt source -b hello
apt install ./hello_*.deb -s
apt install ./hello_*.changes -s
apt install --with-source ./hello_*.changes hello -s
apt-ftparchive packages . > ./Packages
apt upgrade --with-source ./Packages -s
P.S.: If you have expected this post to be published sometime inbetween the last two months… welcome to the club! I thought I would do it, too. Lets see how long I will need for the next one… I have it partly written already, but that was the case for this one as well… we will see.