David Kalnischkieshttps://david.kalnischkies.de/blog/tags/gsoc2016/staticsite2019-05-19T14:00:49Zapt-get a life: gsoc2016GSoC 2016: Summaryhttps://david.kalnischkies.de/blog/2016/gsoc_2016_summary2016-08-18T07:22:51Z2016-08-18T07:22:51Z<h2>Intro, Outro and tl;dr</h2>
<p>I participated again as a student in this years edition of the <a href="https://summerofcode.withgoogle.com/">Google
Summer of Code</a> with
<a href="https://debian.org/">Debian</a> on the project <a href="https://wiki.debian.org/SummerOfCode2016/Projects/APTDpkgCommunicationRework">APT↔dpkg communication
rework</a>.
My <a href="https://wiki.debian.org/SummerOfCode2016/StudentApplications/DavidKalnischkies">initial proposal on the
wiki</a>
details me and the plan, while this post serves as link hub and
explanation of what I did. You can also find personal week-to-week posts
starting with <a href="/blog/2016/day_0_that_was_a_mistake">Day 0</a> right here on this
blog, too.</p>
<p>The code for this project was already
<a href="https://anonscm.debian.org/git/apt/apt.git/">merged</a> and <a href="https://tracker.debian.org/pkg/apt">uploaded to
Debian</a> unstable multiple times over
the course of the summer, so everything described here later can be
experienced directly. The official GSoC2016 final is
<a href="https://anonscm.debian.org/git/apt/apt.git/tag/?h=1.3_rc2">1.3~rc2</a>,
but APT always moves forward and I have no intention of leaving it
alone, so this tag just marks the end of the GSoC2016 period and my
return to "normal" contribution levels.</p>
<p>On a social front I <a href="https://nm.debian.org/public/person/donkult">finally applied for and shortly after
received</a> "Debian
Developer, uploading" status. This is also the moment I want to thank
Michael Vogt (mentor, apt), Julian Andres Klode (apt), Manuel A.
Fernandez Montecelo (aptitude), Guillem Jover (dpkg), Enrico Zini (<abbr
title="Application Manager">AM</abbr>), the <a href="https://wiki.debian.org/Teams/Outreach">Debian Outreach
team</a> and the countless people
I should have mentioned here, too, who have all helped me in many ways
over the course of this GSoC and my entire Debian journey up to this
point.</p>
<p>It was an overall great experience to work again on something as
important as APT in Debian on a full-time basis. After two (very
different) GSoCs in 2010 and now 2016 I can full heartily recommend to any
student with a passion for open-source to apply next year. Perhaps in
Debian and maybe in an APT project? We are waiting for YOU!</p>
<h2>Statistics</h2>
<p>My first commit as part of GSoC was made on 25. April titled <a href="https://salsa.debian.org/apt-team/apt/commit/3c7567a586690042fdd899c7e82543cb31a3b853">edsp: ask
policy engine for the pin of the version
directly</a>
(minor bugfix), the last commit I will be counting on 17. August titled
<a href="https://salsa.debian.org/apt-team/apt/commit/d1bdb73a96d01896ec8e213a0f14abc38d19a929">methods: read config in most to least specific
order</a>
(regression fix). Not all of them are directly related to the GSoC
project itself (the first is), but "just" in the timeframe (like the
last) but were handled as part of general emergencies or for similar
reasons described later and/or in the weekly reports. This timeframe of
115 days saw a total of <a href="https://anonscm.debian.org/git/apt/apt.git/log/?id=3c7567a586690042fdd899c7e82543cb31a3b853&qt=author&q=David+Kalnischkies">222
commits</a>
authored by me + 9 commits committed by me for others (translations,
patches, …). The timeframe saw 336 commits as a whole making me
responsible for a bit shy of ⅔ of all APT commits in this timeframe
with on average of nearly 2 commits each day. A diffstat run over my
commits says "322 files changed, 11171 insertions(+), 5847 deletions(-)"
consisting of code, documentation and tests (this doesn't include
automatic churn like regeneration of po and pot files, which deludes the
global statistic). As a special mention our tests alone changed by:
"109 files changed, 2759 insertions(+), 1063 deletions(-)". In my weekly
reports here on this blog I used ~10574 words (not including this post),
another ~23555 words in the IRC channel #debian-apt and sometimes very
long mails to deity@ and bugreports (~100 mails) [Not counting private
chit-chat with mentor via IRC/mail].</p>
<h2>APT External Installation Planner Protocol (EIPP)</h2>
<p>The meat of the GSoC project was the ability to let libapt talk to (external)
executables (called planners) which are tasked with creating a plan for the
installation (and removal) of packages from the system in the order required by
their various dependency relations, similar to how libapt can talk to external
dependency solvers like aspcud via <a href="/blog/2011/External_dependency_solvers_in_APT">EDSP</a>.
The protocol
(<a href="https://anonscm.debian.org/git/apt/apt.git/plain/doc/external-installation-planner-protocol.txt">current</a>,
<a href="https://anonscm.debian.org/git/apt/apt.git/plain/doc/external-installation-planner-protocol.txt?h=1.3_rc2">archive</a>)
details how apt and a planner can communicate. APT ships such an external
planner already in the form of 'apt' which is "just" using the internal always
existing planner implementation, but reads and talks proper EIPP. The major
benefit is testing here as it is now possible to generate an EIPP request, feed
it to different versions and compare results to find regressions and similar.
It also helps in bugreports as such a request is now auto-generated and logged
so that it can be easily attached to bugreports and a triager can use that file
to reproduce the problem. Previously recreating the system state a user had
before the failed upgrade was a very involved, error prune and time consuming
task (actually fixing the problem still is, but at least the first step got a
lot easier).</p>
<p>APTs good old planner implementation saw also <a href="https://salsa.debian.org/apt-team/apt/commit/28557f94578602f9ce0011501a2259bd98ab0688">the activation (and
fixing) of many previously experimental
options</a> intended to optimize
the process blocked previously by items of the next paragraph, which
makes it look like a new planner now. Entirely new planners exist as
prototypes, but they aren't suitable for real use yet due to not
handling "edgecases" and being effected by bugs in dpkg. Summary:
Everyone can create and work on his own planner in the programming
language of choice and run it against realworld cases directly opening
a competition space for the invention of future improvements.</p>
<h2>APT↔dpkg communication</h2>
<p>The other major building block and donor of much of the projects name. Assuming
a planner has figured out a plan there is still much left to do which is of no
concern for each planner but handled centrally in libapt: The actual calling of
dpkg and interpreting its replies. That sounds easy enough, but if you imagine
the need of thousand of packages to be installed/configured at once you fear
hitting something as barebones as the kernels maximum allowed commandline
length. That happened once in a while in the past so finding better
solutions to that problem within easy reach (as in: <a href="https://salsa.debian.org/apt-team/apt/commit/f495992428a396e0f98886c9a761a804aa161c68">existing in dpkg
already</a>, new interfaces for
possible future use are a different matter) is in order. Other problems
included the <a href="https://salsa.debian.org/apt-team/apt/commit/d3930f8716f439c229cd3d11813823d847a2ecff">overuse of --force
options</a>, not <a href="https://salsa.debian.org/apt-team/apt/commit/e7d4e0cf4f6d79810e4b4e7de505729e759213dd">communication
the purge/removal intentions to
dpkg</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/83e5cffc2015aa809acac84737756d292d7bf106">insufficient
crossgrade handling</a> and
<a href="https://salsa.debian.org/apt-team/apt/commit/7ec343309b7bc6001b465c870609b3c570026149">avoiding losing user configuration on conffile moves involving
packages to be purged</a> just to
name a few. But also listening to dpkg in terms of how it processes
triggers and how all this should be <a href="https://salsa.debian.org/apt-team/apt/commit/8b21456acba3cbbc28f35eba684d8f21b0c290b4">reported in the form of progress
reports</a> to the user
especially if <a href="https://salsa.debian.org/apt-team/apt/commit/9ffbac99e52c91182ed8ff8678a994626b194e69">some</a>
<a href="https://salsa.debian.org/apt-team/apt/commit/bfc0933abedcd57e0a5bd07e282f3a50ba1fa5b2">steps</a> aren't explicitly
planned anymore by a planner, but left to dpkg to do <a href="https://salsa.debian.org/apt-team/apt/commit/4326680d2ed23d597f45ca8872a7054368560acc">at some
point</a>.</p>
<p>The result is mostly invisible to the user, expect that it should all be
now slightly faster as e.g. <a href="https://salsa.debian.org/apt-team/apt/commit/0dd19915f44df0048d594d527797c62ab6195cc6">triggers are run
less</a> and most "strange"
errors a thing of the past.</p>
<h2>Side projects, emergency calls and random bufixes</h2>
<p>Not actually on my plan for GSoC and at best only marginally related if
at all I ended up working on these to deal with important bugs on a "as
long as we have a full-time developer" basis.</p>
<p>This includes hunting for <a href="https://salsa.debian.org/apt-team/apt/commit/0e071dfe205ad21d8b929b4bb8164b008dc7c474">strange errors if rred is involved in
updating indexes</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/e8e5d464623f1c2e1ef96b14e622728bbf4b89af">further
preparing for a binary-all
future</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/3af3ac2f5ec007badeded46a94be2bd06b9917a2">fixing SRV
support</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/0b45b6e5de1ba4224ced67a9952e009d0f4139a0">being the master
of time</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/d03b947b0ce4f87d7d5cc48d4d274ab3bd0b289a">improving security
by allowing it to be sidestepped
sometimes</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/952ee63b0af14a534c0aca00c11d1a99be6b22b2">improving
security by potentially breaking backward-compatibility
a bit</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/1d742e01470bba27715a8191c50adde4b39c2f19">stumble into
libstdc++6 bugs</a>, <a href="https://salsa.debian.org/apt-team/apt/commit/61db48241f2d46697a291bfedaf398a1ca9a70e3">implement
SOCKS5 proxy support</a> and
<a href="https://salsa.debian.org/apt-team/apt/commit/30060442025824c491f58887ca7369f3c572fa57">generic config fallback for acquire
methods</a> to be able to propose
the <a href="https://lists.debian.org/deity/2016/08/msg00012.html">mergeback of
apt-transport-tor</a>
among very many other things.</p>
<p>A complete list can be found with the previously shared git-branch
browsing starting at <a href="https://anonscm.debian.org/git/apt/apt.git/log/?id=3c7567a586690042fdd899c7e82543cb31a3b853&qt=author&q=David+Kalnischkies">my first
commit</a>
in the GSoC timeframe (see also statistics above).</p>
<h2>Future</h2>
<p>I would love to keep working on APT full-time, but that seems rather
unrealistic and all good things need to come to an end I guess, so the
next weeks will have me adjust to a more "normal" contribution level of
"just" in my (extended) free time again. I will also be casually
"shopping" for a new money source in the form of a small job while
returning to university which hasn't seen a lot of me the last few
months and picking up some community work I had delayed for after GSoC.
That means I will surely not keep my daily commit average up, but my
journey here is far from over:</p>
<p>After many years in APT and Debian there is still something new in it to
explore each week as both are evolving continuously – but most of it
hidden in plain sight and unnoticed by the general public: Around the
start of GSoC I was talking on #gsoc with an admin of another org who
couldn't imagine that Debian participated at all as all projects Debian
could offer would be bitsized in nature: It is just a distribution,
right, not a real org producing value (slightly exaggerated for drama).
I would like to concur with this view of course for obvious reasons.</p>
<p>My life would be massively different if I hadn't started to contribute
to Debian and APT in particular ~7 years ago – even through
I thought I wouldn't be "good enough" back then. I think its fair to say
that I showed my past self that in fact I am. Now it is your turn!</p>Week 16: Constant Optimizationhttps://david.kalnischkies.de/blog/2016/week_16_constant_optimization2016-08-15T19:54:51Z2016-08-15T19:54:51Z<p>This week saw the release of 1.3~rc1 sporting my tor-changes disguised as
general acquire methods changes I mentioned last week as well as the revamp
of apt talking to dpkg (and back) I worked on the last couple weeks as part
of GSoC. It doesn't include any new planner, a stub is still lying in a wip
branch, but our good old planner looks and behaves slight different, so it
feels like a new one – and surprising as it is: So far no bugreport related to
this. Probably all user systems caught instantly fire! <span class="emoji" data-unicode="1F642" title=":)">🙂︎</span></p>
<p>So, the week was further used to <a href="https://juliank.wordpress.com/2016/08/10/porting-apt-to-cmake/">get used to cmake</a>,
build and run apt on various porterboxes to fix testcase failures, fixing other
incomings and especially pull some hair out while debugging the bug of the week
which lends the title to this blogpost: A one word fix for an <a href="https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1473674">issues
manifesting itself only in optimization level -O3 on
ppc64el</a>.
Optimizations are evil…</p>
<p>Beside causing quite some time waste for this week as well as in previous years
it is also closing a loop: I introduced this problem myself while being a GSoC
student… in 2010. Time really flies. And I have no idea what I was thinking either…
I could be describing more of these "tiny" bugs, but the commit messages tend
to do a reasonable job and if you are really that damn interested: Feel free to
ask. <span class="emoji" data-unicode="1F642" title=":)">🙂︎</span></p>
<p>This next week will be the last official in GSoC from a students POV as I am
supposed to clean up all bases & submit my work for the final evaluation – this
submit will be as a blogpost describing & linking to everything, which equals
miles long and relatively soon, so that I purposefully have kept this one a
very short one so you will have enough energy to bear with me for the next one.</p>Week 15: Onion orderinghttps://david.kalnischkies.de/blog/2016/week_15_onion_ordering2016-08-08T09:34:26Z2016-08-08T09:34:26Z<p>The week started badly: I had for a long while now a dead 'e' key on my
keyboard, but I didn't care that much… I just remapped CAPSLOCK,
retrained my fingers and be done with it [I have some fucked up typing
style anyhow]. All good, but at this week additional keys started to
give up. You have no idea how annoying it is to not be able use the
arrow keys. Many things I work with have at least the vim-keybindings,
but even in vim picking an autocompletion becomes a nightmare (or
navigating shell history)… So, replacement keyboard please! That took
a while, especially replacing it as my laptop makes that extra hard it
seems but oh well. All working again now! The c-key is actually working
a bit too good (you have to only touch it now, which had me worried as
it started out with producing and endless stream of 'c' out-of-the-box
before I removed the cap once) but so be it for now.</p>
<p>As you might guess that wasn't the ideal work-environment and slowed me
down (beside being annoying), so what I intended to do as a sideproject
turned out to be covering most of the week. <a href="https://lists.debian.org/deity/2016/08/msg00012.html">Mergeback of
apt-transport-tor into
apt?</a> Yes, no,
maybe? The first few responses by mail & IRC are in regards to the plan,
but that still had the need for a lots of code to be written and
refactored. I have to say, implementing SOCKS5 proxy support in apt was
kinda fun and not nearly as hard as I had imagined. Slightly harder it
was to get a setup working in which I could test it properly. Everyone
knows netcat, but that really targets more text-based protocols, not
binary ones like SOCKS5. Still, I managed to figure out how to do it
with socat eventually, resulting in a testscript we can at least run
manually (as it requires a specific port. Not every tool is as nice as
our webserver which can be started on port 0 and reports the port it
eventually picked for real). Playful as I am I even compared my
implementation to others like curl, which our https method is using,
where I ended up reporting <a href="https://github.com/curl/curl/issues/944">minor
bugs</a>.</p>
<p>But why and why now you might ask: apt-transport-tor can be (surprise
surprise) used to let apt talk to Tor network. Conceptionally that isn't
incredibly hard: The Tor software provides a SOCKS5 proxy an application
can connect to be done. Two years ago then apt-transport-tor was
introduced only our curl-backed https method could do that & the
intention was be able to make backports of that transport available,
too, so even through I wasn't all that happy about it, we ended up with
a modified copy of our https method named tor in the archive and as it
is with most modified copies of code, they aren't really kept in sync
with the original. I really want to get this resolved for stretch, so it
slowly gets hightime to start this as if it turns out that I need to
take over maintenance for it without previous-maintainer consent there
is quiet a bit of waiting involved stuff like this should really not be
changed last-minute before the freeze… you will find more details in the
mentioned mail on the reasons for proposing this solution in the mail.</p>
<p>Beside, SOCKS support is actually a so much requested feature that the
internet actually believes apt would support it already via
Acquire::socks::proxy … which is and will also be in future wrong as
there is no socks method we would configure a proxy for – if at all you
configure a method like http to use a socks proxy…</p>
<p>Of course, bundled with this comes a bunch of other things like better
redirection handling across methods and stuff, but that isn't really
user visible, so I skip it here and instead refer you to the git
branches if you are really interested. A few things I will surely also
mention then the relevant code is in the archive so that interested
peers can test…</p>
<p>On my actual battle front, the ordering progress was insignificant.
I got lots of manual testing and review done, but not much new stuff.
The problem is mostly that ordering is easy as long as the problem is
easy, but as soon as Pre-Depends are entering the picture you suddenly
have to account for all kinds of strange things like temporal removals,
loops conflicting or-groups, … stuff you don't want to loose hair over
while losing hair over your broken keyboard already. <span class="emoji" data-unicode="1F609" title=";)">😉︎</span></p>
<p>This week for realz, although target is now really more to merge the
current stuff for apt 1.3. A new ordering algorithm is as detailed in
the initial proposal buster material anyhow – and given all the changes
in terms of trigger delaying and pending calls you are likely not to
recognize our "old" ordering anymore, but more on this in the next two
weeks as that will be the end of GSoC and hence I am going to look back
at "the good old times" before GSoC compared to what we have now. <span class="emoji" data-unicode="1F642" title=":)">🙂︎</span></p>
<p>P.S.: This weeks weekend marks the start of a <a href="http://www.wiesbaden.de/microsite/rheingauer-weinwoche-en/index.php">big wine festival in our
state
capital</a>
my family is one of the founding members of. I am "just" going to help
building the booth through, so no giant timesink this time – just
a couple hours – just in case you hear me saying something about wine
again on IRC.</p>Week 14: This is CROSS!https://david.kalnischkies.de/blog/2016/week_14_that_is_cross2016-08-01T08:34:26Z2016-08-01T08:34:26Z<p>Picture me as a messenger kicked into an endless pit of complexity by
what was supposed to be an easy victim. It wasn't /that/ bad, but
massaging apt to treat crossgrades right took some time – and then some
more to <a href="https://bugs.debian.org/832972">request that dpkg would handle some of
it</a>, too, as
it gets confused by multi-instance packages. Much like apt although that
was just effecting apts progress reporting so not that bad… perhaps I am
<a href="https://bugs.debian.org/832176">play-testing too
much</a>…</p>
<p>In unrelated news I dealt with the two acquire bugs which started last
week as such bugs are annoying & carry the risk of being security
problems which would require immediate attention. Thankfully non of them
seems to be one, but #831762 had me seriously worried for a while.
Trivial in retrospective, but getting to a point in which you consider
the possibility of that happening at all…</p>
<p>But back to the topic: There was one thing still needed to get our
current internal planner a bit "smarter" by enabling options which
existed for a while now, but were never activated by default, and that
thing was simulation. While its kinda appealing to have the simulation
only display what the planner explicitly told use to do, ignoring what
would implicitly be done by the --pending calls we can't really do that
as this would be an interface break. There are surely scripts out there
doing funny things with this output so having it be incomplete is not an
option, which in turn means that what I did internally for the progress
reporting (and hook scripts) must also be done in the simulation. Easier
said then done through as the implementation of it followed a direct
approach running the simulation of each action as soon as the action was
called rather than collecting all the actions first to post-process them
(as I want to do it) and execute them only then. Add to this that this
is a public class, so ABI is a concern… the solution I arrived at is
slightly wrong, but is going to satisfy all existing callers (which is
only aptitude in the archive thanks to codesearch.d.n) and reuses the
"dpkg specific" code, which is a layer violation, but reuse is better
than copy&paste without breaking ABI, so I am happy all things
considered.</p>
<p>So, with that out of the way glory awaits: Changing the default of
PackageManager::Configure from "all" to "smart"… and tada: It works! The
simulation shows everything, the dpkg invocations are much shorter,
trigger executions delayed, we rely more on --pending calls and progress
reporting is properly moving forward as well! Not 100% production ready but
good enough for a public <a href="https://git.kalnischkies.de/apt/log/?h=wip/gsoc">wip branch for
now</a> (= wip aka: going to be
rebased at will).</p>
<p>This also includes the barebones 'dpkg' planner I mentioned last week,
based on that I was playing with ideas to find a more viable
implementation (= handling Pre-Depends) but nothing of particular note
produced yet. Maybe I can get something working this week – it is at
least part of the plan beside polishing my wip branch – after leaving
that pit that is… Hello, is anyone up there? Hello? Hello? …</p>Week 13: Progress reportinghttps://david.kalnischkies.de/blog/2016/week_13_progress_reporting2016-07-25T07:00:39Z2016-07-25T07:00:39Z<p>On the public side of things I did a bunch of things this week which
weren't exactly related to the GSoC project which were triggered by
incoming mails (and IRC highlights) with bugreports and interesting
whichlists, which isn't completed yet as there are two new responses
with debug logs waiting for me next week.</p>
<p>I say that as I brushed up a smallish commit for merge this week which
is supposed to deal better with triggers in progress reporting. That
sounds boring – writing progress reporting stuff – but many people care
deeply about it. While this one is more of a cosmetic change its
conceptionally a big one: With the actions the planner proposed, apt
builds for each package a list of states it will pass through while it
is installed/upgraded/removed/purged. Triggers rain in this parade as we
don't know beforehand that a package will be triggered. Instead,
a status message from dpkg will tell us that a package was triggered, so
if we don't want to end up in a state in which apt tells via progress
report that it is done, but still has a gazillion triggers to run we
have to notice this and add some progress states to the list of the
triggered package – easy right? The "problem" starts with packages which
are triggered but are destined to be upgraded to. A triggered package
will loose its trigger state if it is unpacked, so our progress report
has to skip the trigger states if the package is unpacked – we can't
just exclude packages if they will be unpacked as it can easily be that
a package is triggered, the trigger is acted upon and is upgraded "ages"
later in this apt run.</p>
<p>My wip branch contains many more progress related commits as there is
a big gotcha in the description above: I said "with the actions the
planner proposed", so what about the actions the planner isn't proposing
but will happen as part of dpkg --configure/--remove/--purge --pending
calls? And what about hook scripts like apt-listbugs/changes which get
told the actions to perform their own magic?</p>
<p>The solution is simple: We don't tell dpkg about this, but for our own
usage we do trivial expansions of the --pending commands and use these
for progress report planning as well as for telling the hookscripts
about them. That sounds like a very simple and optional thing, but it
was actually what blocked the activation of various config options I had
implemented years ago which delay trigger execution, avoid explicit
configuration of all packages at the end and all that which I could now
all enable – a bit more on that after this hits the master branch. <span class="emoji" data-unicode="1F642" title=":)">🙂︎</span></p>
<p>I also looked into supporting crossgrades better. In apts conception
a crossgrade is the remove of a package of arch A and the installation
of a new package (with the same name) of arch B. dpkg on the other hand
deals with it like a 'normal' upgrade, just that the architecture of the
package changes. The issue with that isn't gigantic usually, but it
becomes big with essential packages like if you try to crossgrade dpkg
itself with apt: APT refuses to do that by default, but with enough
force it will tell dpkg to remove dpkg:A and then it tells dpkg to
unpack dpkg:B – just that there is no dpkg anymore which could unpack
itself. At least in that case we can skip the remove of dpkg:A, but we
can't do it unconditionally as that might very well be some part of an
order requirement, so progress reporting should be prepared for either
to (not) happen… That isn't finished yet and will surely leak into next
week.</p>
<p>Next week will also see my freshly built planner 'dpkg' get a proper
tour: With all the --pending calls it seems like a good idea to try to
be extra dumb and have a planner just unpack everything in one go, let
the rest be covered by --pending calls and see what breaks: Obviously
the harder stuff, but I have two directions I would like to explore
based on this minimal planner to make it viable. First I have to finish
the crossgrading through, my usual self-review of commits and the
bugreports I participated in this week want to trigger further actions,
too… see you next week!</p>Week 11+12: Where did all the time gohttps://david.kalnischkies.de/blog/2016/week_11+12_where_did_all_the_time_go2016-07-19T09:11:39Z2016-07-19T09:11:39Z<p>Julian said that in his talk about the last year in APT in regards to
(further) fixes to the performance of pdiff and apt in general. I want
to say that about the last two week in general through. I was two days
at the university (last two weeks of lectures even), a day helping
moving, a day away with the young group, a combined day busy with mental
and organisational buildup and teardown of that away-day, 5 days
building, tearing down and in between manning our wine booth as well as
other booths at a festival… Somehow that left only 4 days for day-to-day
business like gsoc work… not ideal, especially if the plan isn't working
out as it should either. I had to deal with a few issues uncovered by
our testsuite caused (on non-amd64 architectures) more or less by the
EIPP merge. Mind you, not actual hardcore bugs, but mostly: For some
esoteric reason like additional lines in output the testcases are
unhappy, but its not reproducible always and absolutely not on your
local system.</p>
<p>Turns out that one of them was a deeper issue with how compression
methods can be dynamically configured and how the testcases do this to
force testing of certain aspects which interfered with me hardcoding
xz-compression for EIPP – time-dependent of course as EIPP tends to run
faster than /bin/false (no joke, as the errors were pipe write errors
which only happen if the program we called closes the pipes aka exits).
Sounds simple, but the fix is split over 6 logical commits (which all
just change a few lines, but finding them is the trick…). Another dpkgs
inability to change the architecture it believes to be native, while apt
can do that easily and uses it in the testcases frequently.</p>
<p>Another issue was an a lot bigger bug in which apt ended up changing
permissions, ownership & access/modification times on files it should
actually only read through symlinks created by the file:// method.
Finding and fixing this was an interesting walk in the kernel/libc API
park as even if it isn't strictly needed to change the ownership of
a symlink, it feels better too avoid to much of special casing if it can
be helped and what we really don't want is singling out the Linux kernel
against other kernels – I did that once accidentally (#738567) and try
to avoid it ever since (that report is related by the way as that call
would have been handy if it could have been used here…).</p>
<p>With the weeks so grossly derailed I purred some time into an issue with
our relatively new SRV records support which was working only if the
first connect would be successful, which is kinda against the point of
SRV. Testing this was fun as I was actually convinced I would need
a fullblown DNS server setup (much like we need a fullblown webserver
setup so much that we implemented our own), but it turned out that its
dead simple to add SRV records via dnsmasq, so that was easy to test
(after I figured that out that is) at least interactively.</p>
<p>Also I implemented support for "install ./foo.changes" partly in
response to the possibility of securing the download of individual debs
question in the APT talk – now you can distribute a bunch of debs (and
dscs) via a changes file, which can be signed (which the USER has to
verify!) and includes the hashes of said files (which apt checks
against). Mostly I was doing it for me through as I do local rebuilds
at times (like dpkg with a fix for #830267) and picking the individual
debs can be a bit annoying and with changes its slightly less bad.</p>
<p>Apart from that I looked into the progress reporting thing as intended
and tried a few options, but that isn't all too easy given that apt is
expecting to know EVERY action performed… I started with this, but
I will probably tell you more about that next week as I fully intend to
quickly make up for last two weeks, so I better get started right about
now…</p>Week 10: That is plane wrong!https://david.kalnischkies.de/blog/2016/week_10_that_is_plane_wrong2016-07-04T10:11:39Z2016-07-04T10:11:39Z<p>As a kid I liked Asterix as comic as well as in the animation films
– I am not so fond of the actor films and haven't looked at the newer
comics, but that is beside the point. I want to quote Obelix here:
"These Brits are crazy!". Not because of the recent vote to leave the EU
or because the english soccer team decided that this vote applies also
to their play. No, I am thinking of the scene in the film in which the
hero's walk down a street in search for a house and all of the houses in
the street, the block, even the district look the same, but luckily they
have the number, find the one and beat up the inhabitant only to be told
that an 'I' is missing in the number…</p>
<p>After last weeks merge, I was told that "planer" doesn't mean what
I thought it does. A planer is actually a type of machine to make stuff
plane – which is again not making something into a plane to fly with it,
but to make something plane so you don't get splinters (if its wood),
all of this can be done according to a plan btw. What I meant is
a "planner" – aka I was missing an 'n'. Perhaps only because I have
light dyslexia, but I hate these language things. German isn't all
around better in this: "The 'Planer' makes a 'Plan'." but the stuff can
still be made 'plan' (even through 'planar' is what you usually hear and
it hasn't the wood-meaning, you can just do this to the ground). But at
least we don't drag planes or plants into this, the later being
a masterpiece of word meaning all by itself… but enough of a "rant",
I am pretty sure it all makes sense if I would look up history more than
I did and in the end its just me trying to hide my embarrassment over
such trivial mistakes being made by me for weeks…</p>
<p>Anyway, on the real topic of this post as I did more than just
correcting this mistake (which did take a while through). Beside various
things which are only slightly related or even not at all like: Not
overriding an atomic file if the new file had failures in creation (oh
my god, how did we miss that for so long?!?), various testing things, an
overhaul of the autoremover code + implementing 'protect only the latest
same-source provider' which has explicit benefits e.g. in terms of
out-of-tree kernel modules (if built e.g. with module-assistant instead
of dkms) as well as in other cases, fixing the handling of local deb
files if it was previously already installed in that version as not
behaving like a downgrade and finally a bunch of apt-key changes to make
its badness more visible… which all went directly to git/master.</p>
<p>I started another of my wip branches for the things I am actually paid
for to do: The plan was to tackle some "how apt invokes dpkg" things.
Silly things like not setting --force-remove-essential on each and every
remove/purge (if possible, there are non-default apt options which
prevent apt from knowing which packages are essential in dpkgs view).</p>
<p>Also worked on codifying the --recursive usage for --unpack which sets
up a symlink farm in a temporary directory and tells dpkg said directory
instead of explicitly listing each and every package on the commandline
explicitly (which can turn out to be quite long, see weeks earlier), but
as mentioned back then this doesn't work yet unconditionally as dpkg has
<a href="https://anonscm.debian.org/cgit/dpkg/dpkg-tests.git/commit/?id=8ed23e4495ad5e87d08e3e784a23c517efa2b15c">a
bug</a>
in its ordering code, so for the time being apt needs to be nice and
still help dpkg out here (that is the type of stuff I mentioned in my
proposal as likely hindrance of using another planner for stretch
already).</p>
<p>I also began saving, clearing and restoring dpkg selections before/after
running dpkg so that a planner can eventually outsource the removal
orders to dpkg. That is quite a dance as these selections are user
settings "for the future" which we can't just discard, but we also can't
just ignore them as they can include purge selections and if we would
not unset them before calling dpkg in a new world order apt would
implicitly instruct dpkg to perform removals the user hadn't agreed to…
frankly, its a freaking mess, but the interface exists so I have to make
due with what we have rather than what we "will like to be have had" if
only there was a timemachine.</p>
<p>I am somewhere deep down in this clusterfuck, but I see light at the end
of the tunnel – or perhaps that is just an oncoming train. Hopefully
I am out of this by next week. I also want to give --no-triggers
a chance again, a few years back I implemented support for it, but it
never really saw the daylight thanks to bugs in dpkg and elsewhere, but
perhaps now is a good moment for retrying. It does deal the final blow
to sensible progress reporting through, so I would need to figure
something out for that, too… we will see; planning plane plans isn't
always working out for me.</p>
<p>(This weekend I have a daytrip with ~70 kids with the usual last minute
emergencies and much needed decompress afterwards, so reports might be
delayed and my replies/work even more off business hours than usual).</p>Week 9: Evaluate me pleasehttps://david.kalnischkies.de/blog/2016/week_9_merging_EIPP2016-06-27T09:11:39Z2016-06-27T09:11:39Z<p>Last week was as mentioned midterm evaluation, so my mentor is required
to judge me and I am required to judge my mentor. I still remember
<a href="/blog/2010/week_8_midterm">last time</a> so I was actually a bit
surprised to see just a handful of questions and my "favorite" wasn't
one of them anymore. The freeform answers even had a sizelimit of 2000
characters. Basically the same question topics through: How is the
mentor & how are you doing. Most interesting perhaps is that most of the
answers as I understood it aren't shared with the Orgas the students
work for. Just the mentor judging is. Still, the form claims to be
fillable in 10 minutes, I took considerably more time constantly
rewriting and hunting for better words.</p>
<p>Not unlike what I did the rest of the week to the EIPP code, brushing
over it, reordering and merging commits into potentially more logic
units and finally testing and also fixing the bugs I ended up
(re)introducing with my constant code shuffling – but enough is enough,
I just pushed that stuff into master for everyone to see and work with
(it was available last week in my wip branch as said, so that is just
the more public-public. Still, master is always a big deal as it
suddenly becomes "code which is supported" instead of "code I can freely
change at will as nobody cares").</p>
<p>So, with the next upload of apt to experimental you will have funky
stuff to play with, which you already know all about from the last
weeks, but lets summarize how to opt into becoming my testlab monkey:</p>
<p>Individual apt(-get) calls [simulation or not] can be performed
externally with the commandline switch --planner. The only available one
at the moment is 'apt' which does the same as the internal one, just
externally (okay, I am lying, there are two as there is also 'dump', but
that isn't really a good planner…). Externally planed or not the EIPP
scenario of the last request is stored (xz-compressed) in
/var/log/apt/eipp.log.xz, so if you end up running into a ordering bug,
you can now attach this file for perfect reproducibility which was
considerably harder in the past with directing the user to grab a backup
of the dpkg/status file from before the failure and hope that the
mirrors are still in a state to reproduce it. Now, this file can be
piped into the planner: <code>/usr/lib/apt/apt-helper cat-file eipp.log.xz
| /usr/lib/apt/planners/apt</code> and out comes hopefully a bunch of progress
messages (which in case of apt aren't displayed yet as I haven't really
found a good place were it would also make sense & not sure if there is
much point after all) and various actions to be performed via a protocol
described in all its glory details in <a href="https://anonscm.debian.org/cgit/apt/apt.git/tree/doc/external-installation-planner-protocol.txt">the
documentation</a>.</p>
<p>(That isn't all I did, but its the only thing you can play with … the
rest is playing with you my little monkeys <span class="emoji" data-unicode="1F61B" title=":-P">😛︎</span> )</p>
<p>That should keep you busy for a week allowing me to continue investing
some time into how dpkg itself is called by apt (with the plan in hand).
There are 'old' ideas of how to change/improve it, some implemented but
disabled, some not and some blocked by dpkg bugs. EOE will need some of
these to work at all but that will be explained later… see you next
week!</p>
<p>P.S.: It was also very hot & sometimes very rainy this week, a small
british majority made me, themselves and the rest of the world feel sad
and the french have <a href="https://bugs.debian.org/828011">strange number formats</a>, but I can't and don't
want to talk about everything in a little status update, can/should I…</p>Week 8: I have a new e-mail addresshttps://david.kalnischkies.de/blog/2016/week_8_I_have_a_new_email_address2016-06-20T21:37:39Z2016-06-20T21:37:39Z<p>So, today I got a new <a href="https://nm.debian.org/public/person/donkult">e-mail
address</a>. That is an
interesting development as I wasted a bit of time into stetting it all
up, but as that is basically already Week 9 material, lets recap last
week:</p>
<p>As suspected it was a bit slower than last week, due to setting up and
"hanging out" at our wine booth (tear down tomorrow), so I was happy to
had some buffer. Actually, I ended up not doing what I planed but
"wasted" most of the week on debugging <a href="https://salsa.debian.org/apt-team/apt/commit/99968cf75b46210bded1662d34c4c2b0ef07be04">http
strangeness's</a>,
<a href="https://salsa.debian.org/apt-team/apt/commit/1d742e01470bba27715a8191c50adde4b39c2f19">avoiding libstdc++6
bugs</a>
and most importantly work on acquire system in relation to repositories
apt considers insecure or weakly secured and how packages with the same
"problems" are dealt with. Sounds boring and it pretty much is, but
<a href="https://lists.debian.org/debian-devel/2016/06/msg00264.html">people</a>
run into this <a href="https://bugs.debian.org/827758">issue</a> (there is
a certain joy in fixing a bug before it is "officially" reported) and
while we talked about it for a while, nobody really did it…</p>
<p>Anyway, that meant that my EIPP merge I was hoping for isn't done yet
– it merges cleanly on top of master & all tests pass, but I hadn't the
time to review it again (and again and again – attempted perfectionism
isn't even close in describing what I tend to do with patches; and they
are still riddled with bugs…)… so, I pushed it to a <a href="https://git.kalnischkies.de/apt/log/?h=wip/eipp">wip
branch</a> for now as it
is midterm evaluation week now.</p>
<p>As said earlier that branch is concerned with adding EIPP itself (20
files changed, 1120 insertions(+), 22 deletions(-)) only, a lot of code
already moved to master in earlier weeks as part of EDSP fixes, ordering
code changes and stuff (a bit harder to calc as its intermixed with
other changes I did in my free time on other things I have to subtract,
but the neutralized running total in that branch should be around: 73
files changed, 2423 insertions(+), 1002 deletions(-)).</p>
<p>So, now that I mentioned it, with midterm this week I will have to pass
some formalities (filling questionnaires mostly) I will probably talk
about in the next report, I will surely get my EIPP merge done early on
& start the last part of my GSoC project: Introducing the new planner
'eoe' (acronym suggested by Michael, meaning too be revealed later). Oh,
and I will probably play a bit with my new account given that I should
have more time this week. <span class="emoji" data-unicode="1F642" title=":)">🙂︎</span></p>Week 7: Hey, you are leaking!https://david.kalnischkies.de/blog/2016/week_7_hey_you_are_leaking2016-06-13T08:37:39Z2016-06-13T08:37:39Z<p>Remember how I talked last week about how things should be different
planners rather than option? Well, while writing I was doubting my self
already and only a few hours later I actually did what I said I would
probably not do. EIPP has now two options: A boolean
Allow-Temporary-Removal-of-Essentials (guess what it does) and
a tri-state Immediate-Configuration, which defaults to being unset, so
a planner can do whatever it wants to, while yes and no activate or
disable it for all packages.</p>
<p>That seems inconsequential, but while these options will not make sense
for every planner (which is fine), for "general purpose planners" like our
internal one its either this or 6 slightly differently named planners
which could get messy…</p>
<p>Messy… remember that I said progress would be printed? Turns out, I was
actually wrong, that wasn't progress. Or, well, it was, but not of the
planner but of the apt instance itself. You see, the EDSP/EIPP binaries
created by me all drop their privileges soon after they are called. They
simple don't need root-rights. Its not much in terms of security, but it
can prevent mistakes, so that is a plus. And that progress actually was
a mistake prevented in normal execution by the dropping, but our
testcase environment is special enough that an unprivileged user has
rights well above the usual (like writing to the pretend /). So what it
tried here was updating the binary caches with the protcol data… not
a problem as the next apt run will see these caches as invalid and
recreate them (which was the cause for the progress line), but it is
a pointless action and easy to prevent. As that worked so well I also
made it so that EDSP-like protocols default to privilege dropping as
supposedly non of them really needs rights, but dropping is hard to
implement… it is configurable through so if one of them needs rights
(like dump) it can keep them a bit longer.</p>
<p>So, with our internal solver mostly working as an external one I shifted
slowly to working on the surroundings. EDSP-like protocols can now
generate and save a (compressed) copy of a scenario – in case an
external solver/planner is used its input is saved like as if it would be
called via <em>tee</em> and if the internals are used a scenario is generated
and put on disk directly. That can be handy for all sorts of things, but
mostly for debugging and bug reports. It was always possible to call
a special 'dump' solver/planner who streamed its input to a file and
produced an EDSP-understandable failure, but while for EDSP this is okay
as a solution you don't like can be NACKed and the setup repeated as
often as you want to (which is why logging isn't enabled for it) a bug
in EIPP means usually a failure in the installation process which has
modified the problem space already… so storing the scenario here allows
for easy retrieval of this earlier state.</p>
<p>I also started on changing the way apt calls dpkg "for real". The planner
is responsible for the order, but I decided that this is still rather
"abstract" (e.g. no difference between remove and purge). The first
thing I did is rather simple: As apt calls "dpkg --configure -a" at the
end of its operation since 'recently' (= last stable release) it could
stop explicitly configuring the last batch of packages before that,
which tends to be a giant list. So giant, that apt actually has code to
check if this all "fits" into the call as commandline have a size limit.
On my system it reports ~2 MB (try e.g. with "xargs --show-limits")
which seems like enough for everyone™, but that value could be (a lot)
smaller and there are old stories of this happening… as with splitting
it might fit again, but the chance of exploding also skyrockets.
Anyway, the 1.3~exp2 release (in [surprise surprise] experimental) has
this change. I also explored using --recursive on a softlinkfarm for
unpack/install instead of mentioning every package explicitly as this is
also in danger of hitting the limit, but while I have code working I hit
(with our test-bug-740843-versioned-up-down-breaks) a dpkg bug in what
Guillem identified as a problem in the Multi-Arch handling. The solution
for apt would be to number the deb packages so they are sorted
correctly, but that of course misses the point as ideally we (as in apt)
don't want to micro-manage that much anymore meaning so we want to
pretend here that we don't know the right order (as with a different
planner we might really don't know).</p>
<p>In terms of overall progress: Our internal-as-external planner survived
its first big test: It took care of an ~800 packages upgrade of my main
system – that is how much I trust myself – without an issue. In fact the
solution it calculated was very slightly better than what the internal
produced [remember: It is the same code!] probably thanks to the reduced
problem space and the different order in which versions are presented.</p>
<p>I also found a few file descriptor leaks (in EDSP, dpkg calling
debugging and elsewhere) and cleaned up code slightly, so all in all
a quite busy and good week. With an eye on the next that was pure tactic
through: Next weekend is "Strawberry Festival" here which means my
family is having a wine stall there which also has to be assembled first
& soccer EM is running, so especially the earlier two items will eat
time which is why I aimed for a "buffer" last week, so don't get your
hopes up I can keep that production rate for this week. <span class="emoji" data-unicode="1F609" title=";)">😉︎</span></p>
<p>This week I will see about bringing EIPP to production state codewise to
merge it to master. There are still things left which should be done,
but before I do deeper modifications in the actual calling of dpkg
I would like to have EIPP in so that I don't have to "maintain" two
branches (and as it survived the big test, its "ready"). These call
modifications surely will have effects on EIPP as well – currently
a planner can only issue unpack, configure and remove commands, but there
is also the unpack & configure combining --install which should be
exposed (and unpack which is internally called install renamed).</p>
<p>See you next week!</p>Week 6: Haunted by the pasthttps://david.kalnischkies.de/blog/2016/week_6_haunted_by_the_past2016-06-06T07:37:39Z2016-06-06T07:37:39Z<p>This week had me dealing with the errors in our test framework if the
internal is used as an external planner is used instead of the internal
by default. On the upside, I got most of the interesting ones dealt with
or with other words: From 19 down to 10. That doesn't sound like much
(or still much, if you prefer that view), but the remaining are mostly
diffs in the output (at the moment apt reports no progress about EIPP,
in exchange the solver emits some progress directly causing unexpected
lines to appear).</p>
<p>There are three exceptions:</p>
<ol>
<li>
<p>We have a test for forced essential loop-breaking, which means apt by
default refuses to temporarily deinstall essential packages in
pre-depends/conflict loops. See e.g.
<a href="https://bugs.debian.org/748355">#748355</a> if you think that is never going to happen anyway, but
what is likely is that we don't want it to happen, so I have my doubts about
adding a config option to the EIPP protocol just for this. Mhhh.</p>
</li>
<li>
<p>Is differences in the immediate configuration. apt currently
basically has 3 modes: no immediate configuration at all, of essentials
and their dependencies (default) and of all. I played with an annotation
per-package, but discarded it eventually as this should be another
planner strategy thing.</p>
</li>
<li>
<p>What is (pseudo)-essential? As mentioned in 2., apt treats essentials
and their dependencies special. The problem enters the room if we
remember that EIPP doesn't sent the entire universe, but tries to shrink
it, which means entire packages and dependencies could be missing given
the impression of unconnected trees which are in fact connected. I think
here a pseudo-essential marker is needed to give the planner the info if
it wants to use it. The field exists already, I just have to figure out
a good way of implementing it in the planner.</p>
</li>
</ol>
<p>Arriving there took longer than I would have liked to, through. First,
it took me a lot longer than I would like to admit in public to figure
out why downgrades weren't working. I had the protocol designed with
those in mind, the code was expecting them all seemed right even after
the twentiest inspection. Until I figured out that the planner wasn't
even told to install the downgrade as the logic for install was "new ||
upgrade". Adding a "|| downgrade" and voilà…</p>
<p>That was just silly through, the real "problem" was with getting
architecture specific dependencies/provides to be working. Again, the
code was happy to accept them (nothing new, the parsing of such details
is done by good 'old' code). The joy comes from translating apts
single-arch view of the world with provides and implicit dependencies
aplenty back into a multi-arch view to be outputted and feed to the
planner (which takes it and translates it back to single-arch). Long long
ago (2010) I decided that apt would implement Multi-Arch by translating
it to Single-Arch. That has enormous advantages as all apt based tools
already knew how to deal with a single architecture – converting them to
be truly Multi-Arch would have been a huge undertaking I am not sure all
tools would have survived. I still think this was the right decision,
but each time I have to deal with the complexities arising from this
translation I wonder how the world would have been…</p>
<p>The real haunt of the past was something else through: Protocols like
EDSP use a unique numeric identifier to refer to a specific version and
that is cool as APT had such a number around anyhow and uses it often as
an index for arrays. Back then I implemented our internal solver for
EDSP I had a problem through: Where do I store this number so I can
access it later? Obvious! I store it as ID of the package!</p>
<p>Temporarily only of course. In the future I would need a mapping as you
can never be sure that highest ID is also the amount of IDs handed out.
That would need a tiny ABI break through, so it was filled under "for
later" as for now we could be sure about it, so no problem. Forward
a few years and in which I happily forgot about this problem. The tests
happily worked, so I figured I should run it against a real system… and
how it ran: Instant segfault.</p>
<p>Thankfully the ABI situation is slightly better today as we had
a d-pointer now in a strategic place, so I could add the trivial code
needed to introduce a mapping and hide it from the API which helps in
not worrying to much about the interface as it can be easily changed if
needed.</p>
<p>Next week I would like to see the three immediate planners working,
decide on the ForceLoop thing and progress in order. With that done we
would have (simple) feature parity and I could look into what we need,
but I will leave that for next time as that post is way to long already…</p>Week 5: Let there be shortcutshttps://david.kalnischkies.de/blog/2016/week_5_let_there_be_friends2016-05-29T18:57:39Z2016-05-29T18:57:39Z<p>The week started late and Thursday was another holiday here (Fronleichnam,
better known elsewhere as "Corpus Christi") so I was a bit worried for
what I would get done, but it turned out good thanks to a lot of
shortcuts being used by me and
<a href="https://nm.debian.org/public/process/donkult">others</a>.</p>
<p>"First", I looked into #825396 which I had cloned from aptitude.
Locales, downloading and aptitude… doesn't sound much like ordering?
Yes. It does sound like effecting our text interfaces through which is
a huge deal as we have plenty of them. Turns out EDSP (and EIPP) aren't
effected as they use a lower API, so I had to work on this in my "free
time", but okay. At least I could use new funky C++11 stuff – which made
me both requiring at least GCC-5 for compiling and <a href="https://wiki.debian.org/RepositoryFormat?action=diff&rev1=88&rev2=89">document
a requirement</a>
which always existed but was never explicitly said in public.</p>
<p>But back to EIPP: After all this refactoring (merged to git master this
week) and creating the first part (the scenario generation) in the last
weeks my secret goal for this one was to dash out code that eats the
scenario and spits out an order apt would parse and call dpkg according
to it. The reason for this goal was simple: The sooner I can get that
up and running, I can run our testcases against it as well as use it in
"production" to see how well all that stuff works. I wouldn't use it in
production yet, but it seems to work as good as I was expecting it to:
barely.</p>
<p>The code looks also… pretty. The shortcut used here is the C++ friend
keyword to befriend a non-member function so it is allowed to access
internals (protected/private) fields and methods of the befriending
class. You want to avoid such things for obvious reasons usually, but
for such a PoC its good enough.</p>
<p>The goals for next week is to clean up that mess (a bit) and to actually
run it against our testsuite (well, I did 19 failures in 216 testcases,
not too bad, but given that most do not deal with "real" installations
its also not that great) and adept/extend accordingly.</p>
<p>The code e.g. currently completely ignores most of the (very involved)
[pseudo-]essential handling, silently ignores downgrades, gets sent
actions it shouldn't get (purges of rc-only packages) and all these
silly details I will have to work on the next few weeks… So, far from
done, but I am happy for this week. <span class="emoji" data-unicode="1F642" title=":)">🙂︎</span></p>Week 4: I don't speak Perlhttps://david.kalnischkies.de/blog/2016/week_4_I_dont_speak_perl2016-05-24T10:27:39Z2016-05-24T10:27:39Z<p>(as expected, later than usual – only covering last week,not the two days of the new week)</p>
<p>Most of the week was "wasted" on refactoring the code I had written so
far – especially in code I can merge "right away" (not yet done, but
next week) and code strictly related to EIPP.</p>
<p>A major timesink was me trying to come up with a patch for pkg-config to
fix up APTs (and dpkgs) testsuite by using the new DPKG_ROOT facility as
the relevant code is written in Perl and I don't have any experience
with it… lets just say: It was hard, especially with my perfectionism
coming through trying to support spaces and stuff in the path. A bit
more on the rational for this & the patch in all its glory can be found
in the bugreport <a href="https://bugs.debian.org/824774">#824774</a>.</p>
<p>Apart from that bugs and mails were answered, but nothing especially
out of the ordinary so instead of recounting them I refer you to deity@
if you are interested & instead move on to get something done for this
week. <span class="emoji" data-unicode="1F642" title=":)">🙂︎</span></p>
<p>As said last time, the game plan is now that we can generate a scenario
to feed this back into an 'externalized' internal planner and then let
this planner spit out a solution apt would need to eat and act upon. So
lots of fun until midterm…</p>Week 3: The thing not called EIPPhttps://david.kalnischkies.de/blog/2016/week_3_the_thing_not_called_EIPP2016-05-15T18:27:39Z2016-05-15T18:27:39Z<p>IT'S ALIVE! Lacking a better name, I still call this Frankenmonster EIPP
as it shares as intended parts of code and overall protocol design with
its big sister EDSP with parts tacked on and modified as needed…</p>
<p>The week started with 2 EDSP bugs: 1. rather simple and contained: apt
was telling EDSP solvers about packages in 'rc' (removed, but conf files
still present) it only knows about from dpkg's status file. Such
packages aren't installed & can't be installed (if there isn't an other
source available), so giving this version to the solver can allow it to
pick this version for installation, which apt can't do. Hilarity ensues.
A rather uncommon situation through.</p>
<p>The second was more interesting: apt in combination with an external
solver backed by apt-cudf (like aspcud used on buildds for experimental)
refused to work if it had to upgrade a package. Just upgrade…
<a href="https://bugs.debian.org/823906">#823906</a>.</p>
<p>We discussed that a bit, but eventually decided that its a problem in
the EDSP protocol description, APT should verify the protocol assumption
& apt-cudf shouldn't generate Install+Remove requests for upgrades.</p>
<p>The work on this bug also highlighted a inconsistency in the
documentation of the config options for solvers and their strict-pinning
and preferences string, which was corrected as well.</p>
<p>Further, I pushed for a change in the request formulation for
{dist-,}ugrade requests to support all three modes of operandi apt has
by now, but that generated no response so far. I will give that some
more time…</p>
<p>(all of the above is already in git master branch)</p>
<p>But I am here to talk about EIPP: apt can now formulate a request,
generate a scenario and as it would be hard to test otherwise apt can
also call an external planner now instead of the internal one – and one
is already implemented: dump – ripped without changing a bit, just
symlinking it into another directory collects the data just fine thanks
to the very similar protocol syntax (at least, good enough for now).</p>
<p>An EIPP scenario looks completely different to an EDSP scenario on
closer inspection: Very few data points are passed via EIPP – perhaps it
should be more, but for the moment I opted for very few data bits.</p>
<p>The very first change to the initial implementation was hence the
reduction of the amount of versions apt will generate information for
inspired by the first EDSP bug mentioned above: No point in sending
information about versions over which are neither installed nor are
going to, which can eliminate entire packages from the output as well if
nothing of interest forms a dependency on them.</p>
<p>For next week I want to look into how and if I can uplift some of this
avoidance code to EDSP to remove packages from the scenario which can't
be part of the solution, I would like EIPP to avoid mentioning
dependencies which are satisfied the entire time as smaller scenarios
means less data to parse and shuffle around for everyone (beside that
less data is also easier to 'anonymize' if we ever explore "cloud
solutions") and slowly venture into part 2 (of 3) of this project (now
that official code start is around the corner): Looking into having our
internal planner available as an external one similar to how the external
solver 'apt' is our internal solver working over EDSP. That will need
fixating the second part of the protocol (= answer) which currently just
reads "TODO".</p>
<p>This code lives in a private wip branch ATM only, but I hope with the
potential generalizations for EDSP I can move most of it out of them.
I would just like to avoid EIPP ending up in master for the time being
as nothing is fixed – and with apt we have a kind of "if code was
released, you have to support it for years to come" arrangement I would
like to avoid as long as I haven't even decided on the name for this
monster. <span class="emoji" data-unicode="1F609" title=";)">😉︎</span></p>
<p>See you next week.</p>
<p>(Next weeks report will likely be written on monday or even a bit later
as the weekend will be super busy in reallife for me, so don't wonder).</p>Week 2: Family Businesshttps://david.kalnischkies.de/blog/2016/week_2_family_business2016-05-08T20:27:39Z2016-05-08T20:27:39Z<p>That week was a bit slow. Partly do to the holidays involved (Ascension
Thursday), which is actually an official holiday at least in my federal
state (Hesse) of Germany, as that traditionally involves lots of family
meetings with cake and BBQ. Partly as I had some time-consuming errants
to run (producing a flyer is harder than predicted if what you get as
a draft is in a MS Publisher format LibreOffice can thankfully open, but
doesn't completely look like it was intended to… so I had to redo in
a proper program) and partly because I had to wonder how people can be
that incompetent: The family has a 6x urn grave place on the local
graveyard. It has a massive, nice and by now old grave stone which you
could have easily bought a car with more than 50 years ago. Anyway,
a stonemason (the same business the stone was bought from all these
years ago) was tasked with removing old names and placing a new one on
it more than ONE YEAR ago. That job was initially supposed to be done
6 months ago in a few weeks time at most and the stone had to be removed
from the grave so the mason could work on it properly. It ended up
taking a lot longer than initially promised, but good work needs time,
right? That's what I told grandma at least. So, on the plus side: This
week it was finally done and the stone back, but that's the only "good"
thing. Its a wonder there isn't a typo on the stone because the letters
aren't placed in the positions it was ordered and a bunch of other major
fuckups… with a grave & stone this big you have to seriously wonder how
someone can believe it would be a good idea to place letters in a way
that you can't even fit two names properly on the stone… which by itself
would be bad enough, but of course it is not just some random name on
the grave stone… in other words: get your shit together or you have to
go shopping for your own grave pretty soon, god¹ damn!</p>
<p>Relation to GSoC? Talk to your 'costumer' if something isn't working the
way it should – and if it takes longer make at least sure the result is
worth it… but really, beside eating time it also hit the mood hard,
which isn't the pretext for a productive week… so a bunch of small scale
stuff rather than the one big thing I had hoped to get done early. So
not really much of a user-visible change, just mails to bugs hinting of
possible changes (#823534, #823625) related to handling of virtual
packages involved in removing. A bunch of WIP on making EDSP support
'apt upgrade' and further refactoring to get closer to the
2-weeks-upcoming "export problem set" goal; still nothing to show of for
general interest.</p>
<p>Also: Two days I spent on implementing new features for the acquisition
of additional files by apt used by apt-file & appstream at the moment
instead of the GSoC project as apt-file has the problem of needing to
deal with two different locations for Contents files – that will
eventually be solved on the archives side of things, but transitioning
between locations seems like a thing apt should really support for more
than just this case, which hasn't much to do with installation order
planing, but at least the code had to ensure a certain order in index
targets download – so I at least did some ordering code on these days…
<span class="emoji" data-unicode="1F609" title=";)">😉︎</span></p>
<p>So, next week: Still getting apt to produce a textual representation of
the problem the planner has to solve. The parts it shares with EDSP are
easy of course (Dependencies, …), but there are just as obvious
differences in other part: A dependency solver e.g. just cares about
installed or not, while a package can in reality be in fifty shades of
gray between installed and not installed all with potentially slightly
different expectations on further processing.</p>
<p>Anyway, that is all going to make more sense with some code so lets see
where we stand next week.</p>
<p>¹ There is a certain irony in using (originally) religious-flavoured
swearing this close to big (christian) religious days – especially if
you do it while talking about graveyards around Ascension day.</p>