diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn new file mode 100644 index 0000000000000000000000000000000000000000..6e6c90278be8f3bac6f0f9a892b400aad1a898f4 --- /dev/null +++ b/web/deps/dep17.mdwn @@ -0,0 +1,508 @@ +[[!meta title="DEP-17: Improve situation around aliasing effects from `/usr`-merge"]] + + Title: Improve situation around aliasing effects from `/usr`-merge + DEP: 17 + State: DRAFT + Date: 2023-03-22 + Drivers: Helmut Grohne <helmut@subdivi.de> + URL: https://dep.debian.org/deps/dep17 + Source: https://salsa.debian.org/dep-team/deps/-/blob/master/web/deps/dep17.wdwn + License: CC-BY-4.0 + Abstract: + This document summarizes the problems arising from our current `/usr`-merge + deployment strategy. It goes on to analyze proposed solutions and analyzes + their effects and finally proposes a strategy to be implemented. + +Introduction +============ + +Debian has [chosen](https://lists.debian.org/8311745.KnC49Ya6nT@odyx.org) to implement merged `/usr` by introducing symbolic links such as `/bin` pointing to `usr/bin`. +In the presence of such links, two distinct filenames may refer to the same file on disk. +We say that a filename aliases another when this happens. +The filename that contains a symlink is called the aliased location and the filename that does not is called a canonical location. + +At its core, `dpkg` assumes that every filename uniquely refers to a file on disk. +This assumption is violated when aliasing happens. +As a result, we exercise undefined behavior in `dpkg`. +This is known to cause problems such as unexpected file loss and is currently mitigated by a [file move moratorium](https://lists.debian.org/debian-devel/2021/10/msg00190.html). + +We currently prohibit most situations that may provoke problematic behavior using policy. +This mitigation is not without cost and we want to eliminate it. +Shipping files in their canonical locations tends to simplify packaging. +Once files are moved to their canonical locations, a number of aliasing problems are effectively mitigated. +The goal of this work is to reduce the impact of these matters to the typical package maintainer. +It aims for removing the cognitive load of having to keep in mind which files must be installed to aliased locations and which files must be installed to canonical locations. + +Regardless of what strategy we end up choosing here, we will likely keep some of the temporary changes even in the `forky` release to accommodate external repositories and derivatives. + +Problems +======== + +P1: File loss during canonicalized file move +-------------------------------------------- + +When moving a file from its aliased location to a canonical location in the `data.tar` of a binary package and moving this file from one binary package to another, `dpkg` may unexpectedly delete the file in question during an upgrade procedure. +If the replacing package is unpacked first, the affected file is installed in its canonical location before the replaced package is upgraded or removed. +`dpkg` may then delete the affected file by removing the aliased location - not realizing that it is deleting a file that still is needed. + +This problem was originally observed in [#974552](https://bugs.debian.org/974552) and is the one that motivated the issuance of the moratorium. +Since the moratorium came into effect and file moves have been prevented, no new cases surfaced. +Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in a small two-digit number of cases. +[For instance](https://lists.debian.org/20230426223406.GB1695204@subdivi.de), `/lib/systemd/system/dbus.socket` could have been canonicalized while it has been moved from `dbus` to `dbus-system-bus-common`. +There is an [artificial test `case1.sh` demonstrating the problem](https://lists.debian.org/20230425190728.GA1471384@subdivi.de). + +P2: Missing triggers +-------------------- + +When packages declare a `dpkg` file trigger interest in a location that is subject to aliasing without also declaring interest in the other location, a trigger may not be invoked even though that was expected behavior. +No issue arises when a file trigger is declared on a canonical location and all packages are shipping their files in that canonical location. +However, when the trigger is declared for an aliased location and packages move their files to the canonical location, triggers can be missed. + +This problem is also currently being prevented by the moratorium. +Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in two cases. +The `runit` and `udev` packages declare an interest to aliased locations and would start missing trigger invocations when canonicalizing files in other packages. + +P3: Ineffective diversions +-------------------------- + +When a package uses `dpkg-divert` to displace a file from another package, the diverted location may have become aliased due to the `/usr`-merge. +If a package whose files are being diverted were to canonicalize its files, such a diversion were to become ineffective. +As a result, the content of the affected file were to be dependent on the order of unpacks. + +This problem is also currently being prevented by the moratorium. +Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in a small two-digit number of cases. +[For instance](https://lists.debian.org/20230428080516.GA203171@subdivi.de), `zutils` diverts files from `gzip` below `/bin` and a number of packages such as `molly-guard` divert power management tools such as `/sbin/reboot`. + +Beyond diversions issued by packages, local diversions added by an administrator may also become ineffective. + +P4: Disagreeing alternatives +---------------------------- + +When packages use `update-alternatives`, the alternative location or one of its providers may refer to an aliased location. +As packages move files to their canonical locations, they might want to move their provided alternatives as well. +Just replacing the location in the `update-alternatives --install` invocation is actively harmful in this case as the aliased location would not be removed. +If it were to be removed, a user configuration may inadvertently be deleted unless care is taken to preserve it. + +Similarly, we may want to canonicalize the location of the alternative itself. +If it were just moved, `update-alternatives` would have two seemingly distinct alternatives that conflict with one another. +If such a move is desired, it must be carefully coordinated among all alternative providers. + +Last but not least, the choice of alternative provider is usually referred to using an absolute path. +Therefore, this path is part of the interface and is often scripted via automation tools such as `ansible` or `puppet`. +Changing this path would break such automation. + +These problems affect a [small two-digit amount of cases](https://lists.debian.org/20230428201151.GA2784035@subdivi.de). +They are also mitigated by the moratorium. + +P5: Ineffective `dpkg-statoverride` +----------------------------------- + +When packages move their files to canonical locations, a `dpkg-statoverride` may still refer to the aliased location and thus become ineffective. +This could happen if files were moved without updating the corresponding maintainer scripts accordingly. +Usually, statoverrides are issued in the same package that contains the files being modified. +This affects a [one-digit amount of cases](https://lists.debian.org/20230502135105.GA713645@subdivi.de) and the moratorium is effective as well. + +A statoverride may also be configured as an administrative change. +As files are canonicalized, such overrides become ineffective without any warning. + +P6: Empty directory loss +------------------------ + +Andreas Beckmann discovered that an empty directory may unexpectedly disappear when a package containing an aliased location is being removed as part of a package upgrade or removal. +The first instance of this problem is with [`/usr/lib/modules-load.d` from `systemd`](https://bugs.debian.org/1036920), but it is a generic problem. +The [generic problem](https://lists.debian.org/20230530095300.GA1289743@subdivi.de) affects a [two-digit number of situations](https://lists.debian.org/debian-devel/2023/05/msg00325.html). +This problem is not mitigated by the moratorium and really affects upgrades from `bullseye` to `bookworm` and package removal on `bookworm` installations. +As the moratorium is lifted, empty directories moved from `/` to `/usr` also will go absent. + +P7: Shared multiarch file loss +------------------------------ + +Like directories can be shared between different packages (as in P6), regular files with identical content can be shared between multiple instances of a `Multi-Arch: same` package. +When upgrading one instance such that a location is canonicalized and removing another instance, the canonicalized file may be lost in the transaction if the removal happens after the upgrade. +This scenario has been [reproduced in an artificial case](https://lists.debian.org/20230530095859.GA1300602@subdivi.de). +It is not observed in practice, because it is also mitigated by the moratorium. +Had the moratorium been lifted for the bookworm release, we know that a two-digit number of case would have been affected by this. +In most cases, the files affected by loss are `udev` rules and `hwdb` files contained in shared library packages. + +P8: Boostrapping aspects +------------------------ + +Filesystem bootstrap implementations have diverged due to `/usr`-merge. +`debootstrap` now installs the aliasing symbolic links prior to the initial package extraction. +Whether it performs the merge is dependent on the chosen `--variant`. +Other tools such as `cdebootstrap`, `mmdebstrap` and `multistrap` rely on the `usrmerge` package for merging. +As we canonicalize files in packages, that latter strategy will fail once essential files such as `/bin/sh` or the dynamic linker become canonicalized, because running `usrmerge.postinst` becomes impossible. +Conversely, the former strategy fails if a package such as `base-files` were to actually contain the aliasing symbolic links. +The moratorium also prevents these effects from happening in practice. +There is a [mail with more detail](https://lists.debian.org/20230517093036.GA4104525@subdivi.de). + +P9: Loss of aliasing symlinks +----------------------------- + +As more and more packages release aliased locations, eventually one package is the last package that contains a location referring to a top-level symbolic link. +When upgrading or removing that package, `dpkg` sees that the location is released and deletes it - in effect deleting an important symbolic link. +For instance, `libc6:amd64` is the only package that contains `/lib64`. +Canonicalizing its files would cause `/lib64` to be deleted and make the dynamic linker unavailable. +This is prevented by the moratorium for now. + +P10: `debian-installer` +----------------------- + +As of this writing and in Debian bookworm, the root filesystem used by the Debian installer, which is constructed from `udeb` packages, is not merged. +If packages were to move their files to canonical locations and thereby affect the contents of `udeb` packages, that could make the installer dysfunctional. + +P11: Broken symlinks on moved diversions +---------------------------------------- + +When diverting a location that is provided as a symbolic link, the link may become relative due to [policy 10.5](https://www.debian.org/doc/debian-policy/ch-files.html#symbolic-links). +If the destination of a diversion is located in a different directory than the diverted path, a relative symbolic link may become broken. + +P12: Conflicts do not prevent concurrent unpacks +------------------------------------------------ + +Some proposed mitigations (M7, M18) rely on declared `Conflicts` to prevent aliasing effects from taking effect during concurrent unpacks. +However, concurrent unpack is possible in some situations despite declared `Conflicts`. +It can be [experienced by invoking dpkg directly](https://bugs.debian.org/1058937) and [policy is being clarified](https://bugs.debian.org/1057199). +Since experiencing it with apt seems unlikely (unless mutual conflicts are involved), so it is not clear yet whether this [needs to be mitigated](https://bugs.debian.org/1060700). + +Proposed mitigations +==================== + +Many but not all problems relate to tools contained in the `dpkg` package. +Therefore modifying `dpkg` to improve this situation is a natural thought. +We classify modifications to `dpkg` in three different categories: + * No modification: The relevant changes happen outside the `dpkg` package. + * Implicit changes: The change affects the semantics of `dpkg` without adding to its API. + * Explicit changes: Some part of `dpkg`'s API (such as adding an option or a new `control.tar` member) is changed. + +If changes to `dpkg` are part of the solution, we have to further spread the transition. +`dpkg` usually picks up new `glibc` symbols and therefore gains a `Pre-Depends` on the new `libc6`. +Therefore we cannot assume a fixed `dpkg` for unpacking `libc6`. + +Other than that, a number of mitigations rely on implementation-defined behaviour of `dpkg`. +In particular, the various mitigations that employ diversions, use `dpkg` features in ways that they were not meant to be used (e.g. diverting directories). + +M1: Teaching `dpkg` about aliases +--------------------------------- + +One approach is to explicitly tell `dpkg` about the relevant symbolic links such that it can identify canonical and aliased locations. +This strategy can resolve (depending on how many tools use this information) most problems, except for the bootstrapping aspects (P8). +It is considered an explicit change and can take multiple forms. +An earlier version of this document proposed the addition of an `--add-alias` option for `dpkg` to record aliasing symbolic links after their introduction. +Simon Richter proposed [adding a `control.tar` member to record aliases](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths). +A recurring suggestion is to hard code the aliases used by Debian into `dpkg`. +That latter approach has the downside of affecting unmerged installations such as old releases when using the `--root` option as well as non-Debian users of `dpkg` and is effectively ruled out for that reason. + +In any case, this is a new feature in `dpkg` with a fairly involved implementation as it touches on core data structures. +Since it adds to the API in non-trivial ways, it is not something we can remove anytime soon, so it adds to the permanent maintenance cost of `dpkg`. +It is plausible that this feature interferes badly with other developments in `dpkg` such as filesystem metadata tracking. + +Any package that wants to rely on the new behavior requires a versioned `Pre-Depends` on `dpkg`. +Doing this to `libc6` would introduce a dependency cycle. + +M2: Canonicalize all paths +-------------------------- + +While moving files to their canonical location causes most of the problems, the final state of having all of them moved frees us from the majority of aliasing related problems as well. +Once everything is moved and a new policy prohibits adding files in aliased locations, problems P1, P2, partially P3 (for package authored diversions), partially P5 (for package authored statoverrides), P6, P7 and P9 become irrelevant for the future. +For this reason, moving files also is a mitigation strategy if done exhaustively. +The process of doing this requires a combination of other mitigations in order to not break existing use cases, such as smooth upgrades or bootstrapping. + +M3: Let `dpkg` use the filesystem as source of truth +---------------------------------------------------- + +Problems P1, P6, P7 and P9 are concerned with unexpected deletion of filesystem resources that are subject to aliasing. +As such, a targeted behavior change to the code dealing with deletion of unused filesystem resources in `dpkg` is plausible. +`dpkg` could be updated to perform a new check prior to deleting filesystem objects. +While `dpkg` normally considers its internal database as the sole source of truth, it can be changed to determine the canonicalized location according to the actual filesystem before performing its deletion. +If that resolved location happens to be known in the internal database, a warning can be emitted instead of deleting the file. +A similar mitigation already implemented in `dpkg` converts the attempt to delete a theoretically empty directory that happens to not actually be empty into a warning. +This is a significant deviation from the principles `dpkg` is built upon, but it is a fairly isolated change that is only useful during the move of filesystem resources to their canonical locations. +For systems that do not exercise aliasing, the only downside is a (hopefully minor) degradation in performance of upgrades and removals. +Once all files have been canonicalized (M2), this change can be reverted in `dpkg` given that it has warned about relying on this. +As such, this change does not impose a permanent cost to maintaining `dpkg`. + +It also requires versioned `Pre-Depends` on `dpkg`, which again is impossible for `libc6`. +This mitigation needs to be part of two stable releases to accommodate derivatives and external repositories. + +M4: Protective diversions for aliasing links +-------------------------------------------- + +For each of the aliasing symbolic links, we can introduce a diversion that redirects it to some unimportant location. +Since diversions are not intended to be used with directories, `dpkg` only applies a diversion to the exact filename that is being diverted. +When adding a diversion for one of the aliasing symbolic links, files that are installed below that directory component are unaffected by the diversion. +Any attempt to remove a diverted symbolic link will instead remove the corresponding unimportant location. +In order to avoid a `Pre-Depends` loop, the diversions are created by a dependency-less package (e.g. a new `usrmerge-support` package). +`libc6` as the sole owner of `/lib64` needs a `Pre-Depends` and can do so without introducing a loop. +`base-files` is a prominent owner of many other directories that have become symlinks and also needs such a `Pre-Depends`. +With these in place, P9 is addressed. +The diversions can be removed if the symlinks are installed into some `data.tar` or after two stable releases to accommodate external packages and derivatives. + +M5: Ship symlinks as directories +-------------------------------- + +`base-files` contains `/bin`, `/lib` and `/sbin` as directories and as long as it contains them, `dpkg` will not delete the symbolic links that are now placed there. +If `base-files` were to also ship `/lib64` and all other multilib directories, that would effectively prevent them from being deleted, thus mitigating P9. +Its `postinst` script could still convert them to symbolic links unless already converted. +Unfortunately, `libc6` cannot `Pre-Depends` on `base-files` to ensure the right unpack order, as doing so would create a loop. +We can spread this part to two releases or additionally ship the directories in a dependency-less package (e.g. `usrmerge-support`). + +M6: Divert `dpkg-divert` +------------------------ + +`dpkg-divert` can be wrapped by a script that modifies its behavior using the diversion mechanism on itself. +Whenever a diversion is added for a location that is subject to aliasing, the diversion is duplicated to both affected locations by the wrapper. +Similarly, removal of diversions is also duplicated. +Upon installation of the wrapper, all existing diversions are also duplicated. +If combined with M4, these diversions need to be ignored. +Packages that use diversions and packages that canonicalize files affected by diversions need to issue a `Pre-Depends` on the package that installs this wrapper (e.g. a new `usrmerge-support` package). +The affected packages can be determined in a mechanical way. +The wrapper would be required for at least two stable releases. +This way P3 can be mitigated. + +The wrapper can also duplicate local diversions. +However, for removing the wrapper (and thus removing the aliased diversions), users must update their scripts and automation to canonicalize the locations to be diverted. + +M7: Replacing `Replaces` with `Conflicts` +----------------------------------------- + +When files are moved between packages, these are usually accompanied with `Replaces`. +A concurrent canonicalization may render this `Replaces` ineffective and cause the file loss described in P1. +This scenario cannot happen when the replacing package is unpacked after the replaced package has been upgraded or removed. +As such, changing `Replaces` to `Conflicts` makes this scenario impossible. +The mitigation can be applied in an as-needed manner such that moving files without canonicalizing remains unaffected as does canonicalizing without moving. +The situation can be detected using automated tools and reported mechanically. +If packages had their content canonicalized in bookworm, a low two-digit number of packages would have their `Replaces` changed to `Conflicts`. + +This strategy mostly mitigates P1, but it can become impossible to apply when essential packages are involved (as those must not be temporarily deinstalled) or the upgrade becomes too complex for `apt` due to an excess of `Conflicts`. +These cases can be complemented with the next mitigation M8. +Also consider the limited effectiveness of this mitigation due to P12. + +M8: Protective diversions for moved files +----------------------------------------- + +A package that is at risk of loosing files as in P1 can set up a protective diversion for each affected location in the aliased form. +The replacing `preinst` script has to set up these temporary diversions. +When the replacing `postinst` is run, the replaced package is already upgraded or removed (due to associated `Breaks`) and it can therefore remove the protective diversions. +These diversions only exist during an upgrade, but writing the maintainer scripts can be difficult to get right. +Therefore, M7 should be preferred when applicable. + +M9: Protective diversions for empty directories +----------------------------------------------- + +In a similar vein to M8, empty directories can be saved in a P6 scenario. +If there is one and only one package shipping the empty directory, it can set up a protective diversion in a permanent way. +Note that `dpkg` errors out when creating a diversion for an existing directory as it is not meant to be used that way, but this can be worked around by temporarily moving the directory away. +As long as the diversion exists, the package owning the diversion must ensure that the diverted location actually is a directory in the filesystem. +Otherwise, unpacking other packages may fail. +These diversions probably need to stay for two stable releases. + +A directory can also be empty in multiple packages. +In most such cases, this seems to be accidental and the directory can be deleted from the `data.tar` instead. +In other cases, the empty directory can be migrated to a common package, but this is expected to not be needed in practice. + +M10: Protective diversions for shared files +------------------------------------------- + +In a similar vein to M8 and M9, shared files in `Multi-Arch: same` instances can be saved in a P7 scenario. +The new `preinst` can divert the aliased locations contained in the old version, but it must assign these diversions to some other package (e.g. `usrmerge-support`) in order to become effective for itself. +Likewise the new `postinst` can remove these diversions, because all other instances must have been removed or unpacked by this time. +As with M8, such diversions only exist during an upgrade. + +M11: Ship symbolic links in a package +------------------------------------ + +In order to fix bootstrapping tools, the aliasing symbolic links can be shipped in some `data.tar` of a binary package (e.g. `base-files`). +Doing so requires that all packages participating in filesystem bootstrap have canonicalized their paths first, because we would otherwise get a directory vs symbolic link conflict. +In the presence of such a conflict, the behavior of `dpkg` depends on the order of unpacks. +Instead of avoiding the conflict, all bootstrapping tools can be updated to unpack the symbolic link package before all other packages . +It also requires changing `debootstrap` to no longer create the links prior to extraction as it presently passes `-k` to `tar`, which would result in an unpack failure even without any conflict between the various `data.tar` contents. +With this, P8 is addressed. + +M12: Explicitly duplicate triggers +--------------------------------- + +In order to address the missing trigger invocations from P2, the one-digit number of trigger interests can be manually duplicated. +A relation of triggering packages on the triggered package is not required, because configuration of the triggered package is equivalent to a trigger invocation and from that point on triggers work as expected. +This mitigation also needs to last for two stable releases. + +However, this does not at all mitigate trigger interest in external packages. +External packages declaring an interest in aliased locations need to have this mitigation applied as well. + +M13: Keep alternatives aliased +----------------------------- + +If we keep the alternatives and alternative providers at their aliased locations, we can still move the package contents in the `data.tar` without otherwise impacting the use of alternatives. +The major downside is that we have to eternally remember that some locations in alternatives are expressed in a non-canonical way. +In a sense, we can skip P4 by not canonicalizing alternatives. + +M14: Divert `update-alternatives` +--------------------------------- + +A wrapper to `update-alternatives` can canonicalize all paths and ensure that paths may be referred to by either way. +All packages relying on this behavior must issue a `Depends` on the package introducing the wrapper, thus solving P4. +For the trixie release, both aliased and canonical locations would be accepted such that external repositories and derivatives have a full release cycle to get updated. + +M15: Manually migrate statoverrides +----------------------------------- + +The few packages that install statoverrides can migrate them on their own, mitigating P5. + +This totally leaves local statoverrides and statoverrides from external packages unaddressed. + +M16: Change bootstrap protocol +------------------------------ + +We can consider modifications to the bootstrap protocol to alleviate the problems. +Thus far no concrete proposals to this end have emerged. + +M17: Duplicate empty directories +-------------------------------- + +To avert the loss of empty directories, they could be duplicated to both the canonical and aliased location. +Since both locations are recorded in the `dpkg` database, `dpkg` will not delete them. + +M18: Explicitly duplicate diversions +------------------------------------ + +Diversions of aliased locations can be duplicated to the corresponding canonical location to make them effective in both situations. +If the destination of those diversions equal up to aliasing and a diverted package moves a file from `/` to `/usr`, `dpkg`'s usual protection against removing just unpacked files fails and we experience file loss. +Therefore, the diversion destinations must differ in more than just aliasing. + +If the diverting package needs access to the original implementations, it has two options. +It can either take care to look up both diverted locations or declare `Breaks` for all providers that have not moved their files to `/usr`. + +Packages that are being diverted must take care that they are not unpacked in the presence of a diverter that has not been updated yet. +In principle, this is possible by declaring `Conflicts` for diverters, but due to P12, this approach is not always sufficient. +In particular, when the diverter declares `Breaks` with the provider, this is known to cause file loss. +The alternative is to detect the diversions in the divertee's `preinst` and duplicate them on behalf of the diverting package. +When this is done, the diverting package has to be modified to consider the case where the divertee has duplicated the diversions. +Also the divertee has to handle the case where the diverter has been removed after its `preinst` has been run and delete the duplicated diversion in such a case. +There is an example implementation of this approach for [cryptsetup and cryptsetup-nuke-password](https://bugs.debian.org/1060270#23). + +M19: Swap initial unpack and merge in `debootstrap` +--------------------------------------------------- + +We recognize that effectively, `debootstrap` has changed the bootstrap protocol, but other implementations have not changed. +Adding the aliasing symlinks to any package (M11) would therefore break `debootstrap` and `debootstrap` only. +This can be mitigated by [performing the merge after the initial unpack before the initial configuration](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96). +Doing so partially reverts the change in bootstrap protocol implemented in `debootstrap` such that this change becomes a no-op for `trixie` and beyond. + +M20: Restore empty directories in maintainer scripts +---------------------------------------------------- + +When an empty directory is lost as part of an upgrade, the `postinst` script runs after the unpack phase loosing the directory and can manually restore the directory. +When an empty directory is lost due to a different package being removed, this removal can activate a trigger and similarly invoke a maintainer script of the package owning the empty directory to restore it. + +M21: Add placeholder files +-------------------------- + +Since a directory is kept when it is non-empty, adding any file to it will prevent its loss. + +M22: Also perform the /usr-merge for the Debian installer +--------------------------------------------------------- + +In order to avoid breaking the Debian installer, we can perform the same aliasing that we are doing in the main archive. +This has been [implemented](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39). + +M23: Restore lost files during package configuration +---------------------------------------------------- + +Files lost in an a Multi-Arch upgrade scenario are definitely lost by the time the upgraded `postinst` runs. +Shared files that used to be aliased can be augmented with a hard link in the upgraded package. +This hard link will persist in the installation even though it is not otherwise needed. +The upgraded `postinst` script can check for the existence of possibly lost files and restore them from the other link name. +This approach yields to a window during which affected files actually are missing. +For instance, reloading `udev` at such a time may lead to rules not being applied. + +Comparison +========== + +In the following table, we map mitigations to their properties. +The most fundamental property is which problems they fully (✓) or partially (\*) address. +We also record whether explicit (E), implicit (I) or no ( ) changes are required to `dpkg`. +The order of affected packages is judged as "significantly changed packages + mechanically changed packages". +A mitigation is considered temporary if the relevant changes can be reverted after two stable releases. +A prototype is linked when available. +When a mitigation is incompatible with another, this is noted. +Also indicate which mitigations happen to work when trivially (✓) backported to `bookworm` and which can be made to work with effort (\*). + +| | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | P10 | dpkg | affected | temp | prototype | precludes | backportable | +| --- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --- | ---- | -------- | ---- | ----------------------------------------------- | --------- | ------------ | +| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip/alias) | many | | +| M2 | \* | \* | \* | | \* | \* | \* | \* | \* | | | many | | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | M2 | \* | +| M3 | ✓ | | | | | ✓ | ✓ | | ✓ | | I | 1+0 | ✓ | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | M1,M18 | \* | +| M4 | | | | | | | | | ✓ | | | 2+10 | ✓ | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M1 | \* | +| M5 | | | | | | | | | ✓ | | | 1+0 | \* | | M11 | | +| M6 | | | ✓ | | | | | | | | | 1+30 | ✓ | | M1 | | +| M7 | \* | | | | | | | | | | | 0+10 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | ✓ | +| M8 | ✓ | | | | | | | | | | | 10+0 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | M1 | ✓ | +| M9 | | | | | | ✓ | | | | | | 25+0 | ✓ | | M1,M17 | \* | +| M10 | | | | | | | ✓ | | | | | 30+0 | ✓ | | M1,M23 | ✓ | +| M11 | | | | | | | | ✓ | ✓ | | | 2+0 | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M5,M17 | | +| M12 | | ✓ | | | | | | | | | | 0+2 | ✓ | | | \* | +| M13 | | | | ✓ | | | | | | | | 0+0 | | | M14 | ✓ | +| M14 | | | | ✓ | | | | | | | | 1+0 | ✓ | | M1,M13 | | +| M15 | | | | | ✓ | | | | | | | 5+0 | | | | ✓ | +| M16 | | | | | | | | ✓ | | | | 4+0 | | | M19 | | +| M17 | | | | | | ✓ | | | | | | 25+0 | ✓ | | M1,M2,M9,M11 | ✓ | +| M18 | | | ✓ | | | | | | | | | 30+0 | ✓ | | M1,M3 | ✓ | +| M19 | | | | | | | | ✓ | | | | 1+0 | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | M16 | ✓ | +| M20 | | | | | | ✓ | | | | | | 25+0 | ✓ | [#](https://lists.debian.org/20230815094034.GA3537194@subdivi.de) | | ✓ | +| M21 | | | | | | ✓ | | | | | | 25+0 | ✓ | | | ✓ | +| M22 | | | | | | | | | | ✓ | | 1+0 | | [#](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39) | | ✓ | +| M23 | | | | | | | ✓ | | | | | 30+0 | ✓ | | M1,M10 | ✓ | + +In effect, the most fundamental decision becomes how much change we want in `dpkg`. +On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). +As a middle ground, we can accept a temporary behavior difference (M3) to ease a complete move (M2) with additional mitigations (to be selected). +On the other end, we can work around the problematic behavior (to be selected) while moving the files (M2) without applying any changes to the `dpkg` source code. + +The other fundamental decision becomes how to deal with architecture bootstrap. +On one end is an underspecified idea of changing the bootstrap protocol somehow (M16). +The other end is shipping the aliasing symlinks in some package (M11), but that implies (M2). + +Proposal +======== + +Mechanism +--------- + +Discussion `debian-devel@lists.debian.org` indicates that the project prefers to finish the transition without relying on changes to `dpkg` as primary mechanism. +This amounts to rejecting M1 and selecting M2. +While there is agreement on this, there is disagreement on the reasons for this choice. + +Regarding the bootstrapping matter, a small amount of developers that previously worked on `debootstrap` reached consensus on partially reverting the bootstrap protocol changes added to `debootstrap` (M19). + +Proposed combination of mitigations: + * M2: Canonicalize all paths + * M4: Protective diversions for aliasing links + * M7: Replacing `Replaces` with `Conflicts` + * M8: Protective diversions for moved files + * M10: Protective diversions for shared files + * M11: Ship symbolic links in a package + * M12: Explicitly duplicate triggers + * M13: Keep alternatives aliased (initially, reconsider M14 later) + * M15: Manually migrate statoverrides + * M19: Swap initial unpack and merge in `debootstrap` + * M22: Also perform the /usr-merge for the Debian installer + +This combination resolves the majority of problems. +For P3 (ineffective diversions), we may either choose a central mitigation (M6) or a decentral mitigation (M18). +For P6 (empty directory loss), the majority of instances are being deleted and the remaining ones may choose between M20 (maintainer scripts and triggers) and M21 (placeholder file) and maybe also M9 (protective diversions) on a case-by-case way in cooperation with relevant package maintainers. + +Process +------- + +We continuously [monitor the archive](https://salsa.debian.org/helmutg/dumat) for problematic situations. +When problems are detected, bugs (including RC bugs) are automatically filed (continuous MBF) for the affected packages. +Concurrently, mitigations that are compatible with the moratorium are implemented (e.g. deleting unused empty directories, M12, M19). +`debhelper` is being extended with a new `dh_usrmerge` helper that performs the moving of files in package contents. +This helper can also be enabled using the new `usrmerge` addon or a sufficiently high compatibility level. +A MBF at minor severity informs maintainers about possible problems when canonicalizing paths inside their packages. +Once the automatic bug filing works and buildds are converted to merged chroots, the file move moratorium is lifted. +A small set of essential packages (including `bash`, `dash`, `glibc`, `util-linux`) still must keep some of their files in aliased locations. +Maintainers are asked to always upload changes that move files between packages and changes in canonicalization to `experimental` first and only proceed to `unstable` if they do not receive an automatic bug report within three days. +To speed up the process, patches and NMUs are used to convert packages. +Once most of the transitively essential packages are converted, a concurrent upload of remaining packages is coordinated with the affected maintainers and NMUed. +This ensures that no transitively essential package ships aliased paths and `base-files` contains the aliasing links. diff --git a/web/index.mdwn b/web/index.mdwn index 3bf368d714feb692a3586a8565a7d5ef0506d379..ee84c1b08de8a4e407a3ed59646a702542574afc 100644 --- a/web/index.mdwn +++ b/web/index.mdwn @@ -45,6 +45,7 @@ Status | Title **CANDIDATE** | [*DEP 14: Recommended layout for Git packaging repositories*](deps/dep14) **DRAFT** | [*DEP 15: Reserved namespace for DD-approved non-maintainer changes*](deps/dep15) **DRAFT** | [*DEP 16: Improved confidential voting for General Resolutions*](deps/dep16) +**DRAFT** | [*DEP 17: Improve situation around aliasing effects from `/usr`-merge"*](deps/dep17) """]] Additional information