Week 7: Hey, you are leaking!

Remember how I talked last week about how things should be different planners rather than option? Well, while writing I was doubting my self already and only a few hours later I actually did what I said I would probably not do. EIPP has now two options: A boolean Allow-Temporary-Removal-of-Essentials (guess what it does) and a tri-state Immediate-Configuration, which defaults to being unset, so a planner can do whatever it wants to, while yes and no activate or disable it for all packages.

That seems inconsequential, but while these options will not make sense for every planner (which is fine), for "general purpose planners" like our internal one its either this or 6 slightly differently named planners which could get messy…

Messy… remember that I said progress would be printed? Turns out, I was actually wrong, that wasn't progress. Or, well, it was, but not of the planner but of the apt instance itself. You see, the EDSP/EIPP binaries created by me all drop their privileges soon after they are called. They simple don't need root-rights. Its not much in terms of security, but it can prevent mistakes, so that is a plus. And that progress actually was a mistake prevented in normal execution by the dropping, but our testcase environment is special enough that an unprivileged user has rights well above the usual (like writing to the pretend /). So what it tried here was updating the binary caches with the protcol data… not a problem as the next apt run will see these caches as invalid and recreate them (which was the cause for the progress line), but it is a pointless action and easy to prevent. As that worked so well I also made it so that EDSP-like protocols default to privilege dropping as supposedly non of them really needs rights, but dropping is hard to implement… it is configurable through so if one of them needs rights (like dump) it can keep them a bit longer.

So, with our internal solver mostly working as an external one I shifted slowly to working on the surroundings. EDSP-like protocols can now generate and save a (compressed) copy of a scenario – in case an external solver/planner is used its input is saved like as if it would be called via tee and if the internals are used a scenario is generated and put on disk directly. That can be handy for all sorts of things, but mostly for debugging and bug reports. It was always possible to call a special 'dump' solver/planner who streamed its input to a file and produced an EDSP-understandable failure, but while for EDSP this is okay as a solution you don't like can be NACKed and the setup repeated as often as you want to (which is why logging isn't enabled for it) a bug in EIPP means usually a failure in the installation process which has modified the problem space already… so storing the scenario here allows for easy retrieval of this earlier state.

I also started on changing the way apt calls dpkg "for real". The planner is responsible for the order, but I decided that this is still rather "abstract" (e.g. no difference between remove and purge). The first thing I did is rather simple: As apt calls "dpkg --configure -a" at the end of its operation since 'recently' (= last stable release) it could stop explicitly configuring the last batch of packages before that, which tends to be a giant list. So giant, that apt actually has code to check if this all "fits" into the call as commandline have a size limit. On my system it reports ~2 MB (try e.g. with "xargs --show-limits") which seems like enough for everyone™, but that value could be (a lot) smaller and there are old stories of this happening… as with splitting it might fit again, but the chance of exploding also skyrockets. Anyway, the 1.3~exp2 release (in [surprise surprise] experimental) has this change. I also explored using --recursive on a softlinkfarm for unpack/install instead of mentioning every package explicitly as this is also in danger of hitting the limit, but while I have code working I hit (with our test-bug-740843-versioned-up-down-breaks) a dpkg bug in what Guillem identified as a problem in the Multi-Arch handling. The solution for apt would be to number the deb packages so they are sorted correctly, but that of course misses the point as ideally we (as in apt) don't want to micro-manage that much anymore meaning so we want to pretend here that we don't know the right order (as with a different planner we might really don't know).

In terms of overall progress: Our internal-as-external planner survived its first big test: It took care of an ~800 packages upgrade of my main system – that is how much I trust myself – without an issue. In fact the solution it calculated was very slightly better than what the internal produced [remember: It is the same code!] probably thanks to the reduced problem space and the different order in which versions are presented.

I also found a few file descriptor leaks (in EDSP, dpkg calling debugging and elsewhere) and cleaned up code slightly, so all in all a quite busy and good week. With an eye on the next that was pure tactic through: Next weekend is "Strawberry Festival" here which means my family is having a wine stall there which also has to be assembled first & soccer EM is running, so especially the earlier two items will eat time which is why I aimed for a "buffer" last week, so don't get your hopes up I can keep that production rate for this week. 😉

This week I will see about bringing EIPP to production state codewise to merge it to master. There are still things left which should be done, but before I do deeper modifications in the actual calling of dpkg I would like to have EIPP in so that I don't have to "maintain" two branches (and as it survived the big test, its "ready"). These call modifications surely will have effects on EIPP as well – currently a planner can only issue unpack, configure and remove commands, but there is also the unpack & configure combining --install which should be exposed (and unpack which is internally called install renamed).

See you next week!