From 5e940a6d81164278a0348d1ea40a0edda94ca2e4 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Fri, 31 Mar 2023 17:55:20 +0200 Subject: [PATCH 01/25] add dep17: improve support for directory aliasing in dpkg --- web/deps/dep17.mdwn | 114 ++++++++++++++++++++++++++++++++++++++++++++ web/index.mdwn | 1 + 2 files changed, 115 insertions(+) create mode 100644 web/deps/dep17.mdwn diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn new file mode 100644 index 0000000..8f6b2fe --- /dev/null +++ b/web/deps/dep17.mdwn @@ -0,0 +1,114 @@ +[[!meta title="DEP-17: Improve support for directory aliasing in dpkg"]] + + Title: Improve support for directory aliasing in dpkg + DEP: 17 + State: DRAFT + Date: 2023-03-22 + Drivers: Helmut Grohne <helmut@subdivi.de> + URL: https://dep.debian.org/deps/dep17 + Source: https://salsa.debian.org/dep-team/deps/-/blob/master/web/deps/dep17.wdwn + License: CC-BY-4.0 + Abstract: + Adapt dpkg to better deal with situations where multiple different + filenames may refer to the same file due to leading directory components + containing a symbolic link. + +Introduction +============ + +At its core, `dpkg` assumes that every filename uniquely refers to a file on disk. +The situation where two distinct filenames refer to the same file on disk is referred to as aliasing. +Violating this assumption leads to undefined behaviour such as file loss. +The assumption is commonly violated when a leading directory component contains a symbolic link. +A situation where this is known to cause file loss happens when a file is moved from one binary package to another binary package while at the same time changing the filename in a way that retains its final location. +In this situation, `dpkg` may first unpack the new replacing location and then remove the replaced package thus unknowingly remove the aliased file. +Other components such as `dpkg-divert` or `update-alternatives` are likely affected in similar ways. + +The purpose of this DEP is selecting and implementing a change to `dpkg` to improve the way it handles such situations that affect typical Debian installations. + +Naive solution +============== + +In theory, `dpkg` could resolve this automatically. +For every file it touches, it could canonicalize the location using the actual filesystem and check whether any other installed file has the same canonicalized location. +Unfortunately, `dpkg` cannot know which filenames can collide, so it would check every filename in its database. +For canonicalization, it would `stat()` every component of every filename. +This easily amounts to a million or more `stat()` calls on larger installations. +Caching could reduce the impact somewhat, but since Debian introduces aliases during maintainer scripts, it would have to invalidate the cache after maintainer scripts have been run. +The resulting performance would be unacceptable. + +Proposal +======== + +In order to handle aliasing efficiently, `dpkg` gains new options `--add-alias <symlink>` and `--remove-alias <symlink>`. +When creating symbolic links that cause aliasing effects, the creating entity is supposed to inform `dpkg` using an appropriate invocation. +Doing so records the aliasing information in a new mapping inside its administrative directory. +No existing administrative files are modified as a result of this operation. +When `dpkg` operates on paths, it can compute a canonicalized version using a pure function without the need to `stat()` files on disk thus greatly improving performance. +Canonicalized paths are only needed when determining whether a file conflict exists. +In all other cases, original paths continue to be used as symbolic links will be followed by filesystem operations. +The `--add-alias` operation records the target of the symbolic link that must exist prior to invocation. +The `--remove-alias` operation fails if any files are still installed in the aliased location. + +Rejected proposals +================== + +Hardcoding aliases into dpkg +---------------------------- + +It was suggested to include a static aliasing mapping into the `dpkg` source code. +Since `dpkg` is used by multiple projects in different ways (not necessarily Debian-derivatives), this approach would break other consumers. +Also note that Debian's `dpkg` can be used to operate on an installation using different aliases via the `--root` flag. +As such the alias mapping needs to be a property of the installation. + +Modifying package lists in place +-------------------------------- + +`dpkg` could rewrite the extracted `.list` files from `control.tar` and store paths in canonicalized form. +Canonicalization would happen as when a `control.tar` is extracted. +It would also happen either as a one-time conversion during the upgrade of `dpkg` or whenever a `.list` file is read. +Given canonicalized list files, string comparison on files would support conflict detection. +Other pieces to be updated in a similar way include `alternatives`, `diversions`, `statoverride`, and `triggers`. + +This would affect the output of `dpkg -S`, which would then output canonicalized paths. +Packages generated by `dpkg-repack` would have their contents canonicalized as well. + +Managing the aliasing mapping using a control file +-------------------------------------------------- + +It was suggested that the mapping could be managed via a special control file `canonical`. +Given that aliasing is not a common operation, the benefit of handling it declaratively is minor. +Beyond that, aliasing can also happen as an customization issued by an administrator. +Therefore, a command line based approach is preferred. + +Having dpkg move files and create symbolic links +------------------------------------------------ + +When instructed with `--add-alias`, `dpkg` could also create the corresponding symbolic links and move the affected files to their new location. +While that would be convenient, doing so is non-trivial in an atomic way. +Sometimes, the underlying filesystem does not fully conform to POSIX (e.g. `overlayfs`) and such corner cases need to be managed individually. +Since such an implementation already exists outside `dpkg` and its complexity is non-trivial, the moving of files shall remain external. +In case aliases are setup in a bootstrap setting, no moves are necessary. + +Implement aliasing after metadata tracking +------------------------------------------ + +The [metadata tracking](https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking) feature enhances `dpkg` with knowledge about filesystem metadata for installed files. +This includes knowledge of symbolic links, which would help with tracking aliasing. +Unfortunately, progress on this is fairly slow and we think that aliasing support is more urgent. + +Proposal internals +================== + +A new file `aliases` is added to the administrative directory. +Pairs of lines containing link name and destination indicate an alias. +Within this file, no link name or destination may contain another link name. +The `--add-alias` and `--remove-alias` options change this file only and must ensure that the properties are retained. +This leads to a trivial algorithm for canonicalizing paths. +A given path can be scanned for recorded link names as sub path and have them replaced with the recorded destination. +This process is repeated until a scan passes without performing a substitution. +Usually, two scan passes will be sufficient. + +Much of the internal work has been prototyped by [Simon Richter](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) and can be used. +It demonstrates how the `fsys_namenode` can be augmented with a canonicalized path and how `fsys_hash_find_node` can be extended with a new flag to differentiate between lookups considering aliases or exact names. +It differs from what is proposed here in the API to configure aliases and in possibly storing partially canonicalized versions of file names. diff --git a/web/index.mdwn b/web/index.mdwn index 3bf368d..ae02214 100644 --- a/web/index.mdwn +++ b/web/index.mdwn @@ -45,6 +45,7 @@ Status | Title **CANDIDATE** | [*DEP 14: Recommended layout for Git packaging repositories*](deps/dep14) **DRAFT** | [*DEP 15: Reserved namespace for DD-approved non-maintainer changes*](deps/dep15) **DRAFT** | [*DEP 16: Improved confidential voting for General Resolutions*](deps/dep16) +**DRAFT** | [*DEP 17: Improve support for directory aliasing in dpkg*](deps/dep17) """]] Additional information -- GitLab From adc873f00a67cd9581fb8b47385142aea3afdb48 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Fri, 31 Mar 2023 18:59:16 +0200 Subject: [PATCH 02/25] dep17: add option --list-aliases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reported-by: Raphaël Hertzog <hertzog@debian.org> --- web/deps/dep17.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 8f6b2fe..ade39bb 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -40,7 +40,7 @@ The resulting performance would be unacceptable. Proposal ======== -In order to handle aliasing efficiently, `dpkg` gains new options `--add-alias <symlink>` and `--remove-alias <symlink>`. +In order to handle aliasing efficiently, `dpkg` gains new options `--add-alias <symlink>`, `--remove-alias <symlink>` and `--list-aliases`. When creating symbolic links that cause aliasing effects, the creating entity is supposed to inform `dpkg` using an appropriate invocation. Doing so records the aliasing information in a new mapping inside its administrative directory. No existing administrative files are modified as a result of this operation. -- GitLab From 92977e8973c89f8d5cf52a77918b175a271a2649 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Thu, 15 Jun 2023 13:33:24 +0200 Subject: [PATCH 03/25] completely rewrite DEP17 This is a complete rewrite. The initial DEP17 discussion on Debian lists made it clear that there is no consensus for the initial proposition. The discussion went into detail and quantitative analysis of various aspects. Much of that is summarized here. It still mostly is a technical description that tries to keep opinions to a minimum and as such still lacks the actual proposal. --- web/deps/dep17.mdwn | 443 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 353 insertions(+), 90 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index ade39bb..9b9cb1b 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -1,6 +1,6 @@ -[[!meta title="DEP-17: Improve support for directory aliasing in dpkg"]] +[[!meta title="DEP-17: Improve situation around aliasing effects from `/usr`-merge"]] - Title: Improve support for directory aliasing in dpkg + Title: Improve situation around aliasing effects from `/usr`-merge DEP: 17 State: DRAFT Date: 2023-03-22 @@ -9,106 +9,369 @@ Source: https://salsa.debian.org/dep-team/deps/-/blob/master/web/deps/dep17.wdwn License: CC-BY-4.0 Abstract: - Adapt dpkg to better deal with situations where multiple different - filenames may refer to the same file due to leading directory components - containing a symbolic link. + This document summarizes the problems arising from our current `/usr`-merge + deployment strategy. It goes on to analyze proposed solutions and analyzes + their effects and finally proposes a strategy to be implemented. Introduction ============ +Debian has [chosen](https://lists.debian.org/8311745.KnC49Ya6nT@odyx.org) to implement merged `/usr` by introducing symbolic links such as `/bin` pointing to `usr/bin`. +In the presence of such links, two distinct filenames may refer to the same file on disk. +We say that a filename aliases another when this happens. +The filename that contains a symlink is called the aliased location and the filename that does not is called a canonical location. + At its core, `dpkg` assumes that every filename uniquely refers to a file on disk. -The situation where two distinct filenames refer to the same file on disk is referred to as aliasing. -Violating this assumption leads to undefined behaviour such as file loss. -The assumption is commonly violated when a leading directory component contains a symbolic link. -A situation where this is known to cause file loss happens when a file is moved from one binary package to another binary package while at the same time changing the filename in a way that retains its final location. -In this situation, `dpkg` may first unpack the new replacing location and then remove the replaced package thus unknowingly remove the aliased file. -Other components such as `dpkg-divert` or `update-alternatives` are likely affected in similar ways. - -The purpose of this DEP is selecting and implementing a change to `dpkg` to improve the way it handles such situations that affect typical Debian installations. - -Naive solution -============== - -In theory, `dpkg` could resolve this automatically. -For every file it touches, it could canonicalize the location using the actual filesystem and check whether any other installed file has the same canonicalized location. -Unfortunately, `dpkg` cannot know which filenames can collide, so it would check every filename in its database. -For canonicalization, it would `stat()` every component of every filename. -This easily amounts to a million or more `stat()` calls on larger installations. -Caching could reduce the impact somewhat, but since Debian introduces aliases during maintainer scripts, it would have to invalidate the cache after maintainer scripts have been run. -The resulting performance would be unacceptable. +This assumption is violated when aliasing happens. +As a result, we exercise undefined behavior in `dpkg`. +This is known to cause problems such as unexpected file loss and is currently mitigated by a [file move moratorium](https://lists.debian.org/debian-devel/2021/10/msg00190.html). -Proposal +We currently prohibit most situations that may provoke problematic behavior using policy. +This mitigation is not without cost and we want to eliminate it. +Shipping files in their canonical locations tends to simplify packaging. +Once files are moved to their canonical locations, a number of aliasing problems are effectively mitigated. +The goal of this work is to reduce the impact of these matters to the typical package maintainer. +It aims for removing the cognitive load of having to keep in mind which files must be installed to aliased locations and which files must be installed to canonical locations. + +Regardless of what strategy we end up choosing here, we will likely keep some of the temporary changes even in the `forky` release to accommodate external repositories and derivatives. + +Problems ======== -In order to handle aliasing efficiently, `dpkg` gains new options `--add-alias <symlink>`, `--remove-alias <symlink>` and `--list-aliases`. -When creating symbolic links that cause aliasing effects, the creating entity is supposed to inform `dpkg` using an appropriate invocation. -Doing so records the aliasing information in a new mapping inside its administrative directory. -No existing administrative files are modified as a result of this operation. -When `dpkg` operates on paths, it can compute a canonicalized version using a pure function without the need to `stat()` files on disk thus greatly improving performance. -Canonicalized paths are only needed when determining whether a file conflict exists. -In all other cases, original paths continue to be used as symbolic links will be followed by filesystem operations. -The `--add-alias` operation records the target of the symbolic link that must exist prior to invocation. -The `--remove-alias` operation fails if any files are still installed in the aliased location. +P1: File loss during canonicalized file move +-------------------------------------------- + +When moving a file from its aliased location to a canonical location in the `data.tar` of a binary package and moving this file from one binary package to another, `dpkg` may unexpectedly delete the file in question during an upgrade procedure. +If the replacing package is unpacked first, the affected file is installed in its canonical location before the replacing package is upgraded or removed. +`dpkg` may then delete the affected file by removing the aliased location - not realizing that it is deleting a file that still is needed. + +This problem was originally observed in [#974552](https://lists.debian.org/974552) and is the one that motivated the issuance of the moratorium. +Since the moratorium came into effect and file moves have been prevented, no new cases surfaced. +Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in a small two-digit number of cases. +[For instance](https://lists.debian.org/20230426223406.GB1695204@subdivi.de), `/lib/systemd/system/dbus.socket` could have been canonicalized while it has been moved from `dbus` to `dbus-system-bus-common`. +There is an [artificial test `case1.sh` demonstrating the problem](https://lists.debian.org/20230425190728.GA1471384@subdivi.de). + +P2: Missing triggers +-------------------- + +When packages declare a `dpkg` file trigger interest in a location that is subject to aliasing without also declaring interest in the other location, a trigger may not be invoked even though that was expected behavior. +No issue arises when a file trigger is declared on a canonical location and all packages are shipping their files in that canonical location. +However, when the trigger is declared for an aliased location and packages move their files to the canonical location, triggers can be missed. + +This problem is also currently being prevented by the moratorium. +Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in two cases. +The `runit` and `udev` packages declare an interest to aliased locations and would start missing trigger invocations when canonicalizing files in other packages. + +P3: Ineffective diversions +-------------------------- -Rejected proposals -================== +When a package uses `dpkg-divert` to displace a file from another package, the diverted location may have become aliased due to the `/usr`-merge. +If a package whose files are being diverted were to canonicalize its files, such a diversion were to become ineffective. +As a result, the content of the affected file were to be dependent on the order of unpacks. -Hardcoding aliases into dpkg +This problem is also currently being prevented by the moratorium. +Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in a small two-digit number of cases. +[For instance](https://lists.debian.org/20230428080516.GA203171@subdivi.de), `zutils` diverts files from `gzip` below `/bin` and a number of packages such as `molly-guard` divert power management tools such as `/sbin/reboot`. + +Beyond diversions issued by packages, local diversions added by an administrator may also become ineffective. + +P4: Disagreeing alternatives ---------------------------- -It was suggested to include a static aliasing mapping into the `dpkg` source code. -Since `dpkg` is used by multiple projects in different ways (not necessarily Debian-derivatives), this approach would break other consumers. -Also note that Debian's `dpkg` can be used to operate on an installation using different aliases via the `--root` flag. -As such the alias mapping needs to be a property of the installation. +When packages use `update-alternatives`, the alternative location or one of its providers may refer to an aliased location. +As packages move files to their canonical locations, they might want to move their provided alternatives as well. +Just replacing the location in the `update-alternatives --install` invocation is actively harmful in this case as the aliased location would not be removed. +If it were to be removed, a user configuration may inadvertently be deleted unless care is taken to preserve it. + +Similarly, we may want to canonicalize the location of the alternative itself. +If it were just moved, `update-alternatives` would have two seemingly distinct alternatives that conflict with one another. +If such a move is desired, it must be carefully coordinated among all alternative providers. + +Last but not least, the choice of alternative provider is usually referred to using an absolute path. +Therefore, this path is part of the interface and is often scripted via automation tools such as `ansible` or `puppet`. +Changing this path would break such automation. + +These problems affect a [small two-digit amount of cases](https://lists.debian.org/20230428201151.GA2784035@subdivi.de). +They are also mitigated by the moratorium. + +P5: Ineffective `dpkg-statoverride` +----------------------------------- + +When packages move their files to canonical locations, a `dpkg-statoverride` may still refer to the aliased location and thus become ineffective. +This could happen if files were moved without updating the corresponding maintainer scripts accordingly. +Usually, statoverrides are issued in the same package that contains the files being modified. +This affects a [one-digit amount of cases](https://lists.debian.org/20230502135105.GA713645@subdivi.de) and the moratorium is effective as well. + +A statoverride may also be configured as an administrative change. +As files are canonicalized, such overrides become ineffective without any warning. + +P6: Empty directory loss +------------------------ + +Andreas Beckmann discovered that an empty directory may unexpectedly disappear when a package containing an aliased location is being removed as part of a package upgrade or removal. +The first instance of this problem is with [`/usr/lib/modules-load.d` from `systemd`](https://bugs.debian.org/1036920), but it is a generic problem. +The [generic problem](https://lists.debian.org/20230530095300.GA1289743@subdivi.de) affects a [one-digit number of situations](https://lists.debian.org/debian-devel/2023/05/msg00325.html). +This problem is not mitigated by the moratorium and really affects upgrades from `bullseye` to `bookworm` and package removal on `bookworm` installations. + +P7: Shared multiarch file loss +------------------------------ + +Like directories can be shared between different packages (as in P6), regular files with identical content can be shared between multiple instances of a `Multi-Arch: same` package. +When upgrading one instance such that a location is canonicalized and removing another instance, the canonicalized file may be lost in the transaction if the removal happens after the upgrade. +This scenario has been [reproduced in an artificial case](https://lists.debian.org/20230530095859.GA1300602@subdivi.de). +It is not observed in practice, because it is also mitigated by the moratorium. +Had the moratorium been lifted for the bookworm release, we know that a two-digit number of case would have been affected by this. +In most cases, the files affected by loss are `udev` rules and `hwdb` files contained in shared library packages. + +P8: Boostrapping aspects +------------------------ + +Filesystem bootstrap implementations have diverged due to `/usr`-merge. +`debootstrap` now installs the aliasing symbolic links prior to the initial package extraction. +Whether it performs the merge is dependent on the chosen `--variant`. +Other tools such as `cdebootstrap`, `mmdebstrap` and `multistrap` rely on the `usrmerge` package for merging. +As we canonicalize files in packages, that latter strategy will fail once essential files such as `/bin/sh` or the dynamic linker become canonicalized, because running `usrmerge.postinst` becomes impossible. +Conversely, the former strategy fails if a package such as `base-files` were to actually contain the aliasing symbolic links. +The moratorium also prevents these effects from happening in practice. +There is a [mail with more detail](https://lists.debian.org/20230517093036.GA4104525@subdivi.de). + +P9: Loss of aliasing symlinks +----------------------------- + +As more and more packages release aliased locations, eventually one package is the last package that contains a location referring to a top-level symbolic link. +When upgrading or removing that package, `dpkg` sees that the location is released and deletes it - in effect deleting an important symbolic link. +For instance, `libc6:amd64` is the only package that contains `/lib64`. +Canonicalizing its files would cause `/lib64` to be deleted and make the dynamic linker unavailable. +This is prevented by the moratorium for now. + +Proposed mitigations +==================== + +Many but not all problems relate to tools contained in the `dpkg` package. +Therefore modifying `dpkg` to improve this situation is a natural thought. +We classify modifications to `dpkg` in three different categories: + * No modification: The relevant changes happen outside the `dpkg` package. + * Implicit changes: The change affects the semantics of `dpkg` without adding to its API. + * Explicit changes: Some part of `dpkg`'s API (such as adding an option or a new `control.tar` member) is changed. + +If changes to `dpkg` are part of the solution, we have to further spread the transition. +`dpkg` usually picks up new `glibc` symbols and therefore gains a `Pre-Depends` on the new `libc6`. +Therefore we cannot assume a fixed `dpkg` for unpacking `libc6`. + +Other than that, a number of mitigations rely on implementation-defined behaviour of `dpkg`. +In particular, the various mitigations that employ diversions, use `dpkg` features in ways that they were not meant to be used (e.g. diverting directories). + +M1: Teaching `dpkg` about aliases +--------------------------------- + +One approach is to explicitly tell `dpkg` about the relevant symbolic links such that it can identify canonical and aliased locations. +This strategy can resolve (depending on how many tools use this information) most problems, except for the bootstrapping aspects (P8). +It is considered an explicit change and can take multiple forms. +An earlier version of this document proposed the addition of an `--add-alias` option for `dpkg` to record aliasing symbolic links after their introduction. +Simon Richter proposed [adding a `control.tar` member to record aliases](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths). +A recurring suggestion is to hard code the aliases used by Debian into `dpkg`. +That latter approach has the downside of affecting unmerged installations such as old releases when using the `--root` option as well as non-Debian users of `dpkg` and is effectively ruled out for that reason. + +In any case, this is a new feature in `dpkg` with a fairly involved implementation as it touches on core data structures. +Since it adds to the API in non-trivial ways, it is not something we can remove anytime soon, so it adds to the permanent maintenance cost of `dpkg`. +It is plausible that this feature interferes badly with other developments in `dpkg` such as filesystem metadata tracking. + +Any package that wants to rely on the new behavior requires a versioned `Pre-Depends` on `dpkg`. +Doing this to `libc6` would introduce a dependency cycle. + +M2: Canonicalize all paths +-------------------------- + +While moving files to their canonical location causes most of the problems, the final state of having all of them moved frees us from the majority of aliasing related problems as well. +Once everything is moved and a new policy prohibits adding files in aliased locations, problems P1, P2, partially P3 (for package authored diversions), partially P5 (for package authored statoverrides), P6, P7 and P9 become irrelevant for the future. +For this reason, moving files also is a mitigation strategy if done exhaustively. +The process of doing this requires a combination of other mitigations in order to not break existing use cases, such as smooth upgrades or bootstrapping. + +M3: Let `dpkg` use the filesystem as source of truth +---------------------------------------------------- + +Problems P1, P6, P7 and P9 are concerned with unexpected deletion of filesystem resources that are subject to aliasing. +As such, a targeted behavior change to the code dealing with deletion of unused filesystem resources in `dpkg` is plausible. +`dpkg` could be updated to perform a new check prior to deleting filesystem objects. +While `dpkg` normally considers its internal database as the sole source of truth, it can be changed to determine the canonicalized location according to the actual filesystem before performing its deletion. +If that resolved location happens to be known in the internal database, a warning can be emitted instead of deleting the file. +A similar mitigation already implemented in `dpkg` converts the attempt to delete a theoretically empty directory that happens to not actually be empty into a warning. +This is a significant deviation from the principles `dpkg` is built upon, but it is a fairly isolated change that is only useful during the move of filesystem resources to their canonical locations. +For systems that do not exercise aliasing, the only downside is a (hopefully minor) degradation in performance of upgrades and removals. +Once all files have been canonicalized (M2), this change can be reverted in `dpkg` given that it has warned about relying on this. +As such, this change does not impose a permanent cost to maintaining `dpkg`. + +It also requires versioned `Pre-Depends` on `dpkg`, which again is impossible for `libc6`. +This mitigation needs to be part of two stable releases to accommodate derivatives and external repositories. -Modifying package lists in place +M4: Protective diversions for aliasing links +-------------------------------------------- + +For each of the aliasing symbolic links, we can introduce a diversion that redirects it to some unimportant location. +Since diversions are not intended to be used with directories, `dpkg` only applies a diversion to the exact filename that is being diverted. +When adding a diversion for one of the aliasing symbolic links, files that are installed below that directory component are unaffected by the diversion. +Any attempt to remove a diverted symbolic link will instead remove the corresponding unimportant location. +In order to avoid a `Pre-Depends` loop, the diversions are created by a dependency-less package (e.g. a new `usrmerge-support` package). +`libc6` as the sole owner of `/lib64` needs a `Pre-Depends` and can do so without introducing a loop. +`base-files` is a prominent owner of many other directories that have become symlinks and also needs such a `Pre-Depends`. +With these in place, P9 is addressed. +The diversions can be removed if the symlinks are installed into some `data.tar` or after two stable releases to accommodate external packages and derivatives. + +M5: Ship symlinks as directories -------------------------------- -`dpkg` could rewrite the extracted `.list` files from `control.tar` and store paths in canonicalized form. -Canonicalization would happen as when a `control.tar` is extracted. -It would also happen either as a one-time conversion during the upgrade of `dpkg` or whenever a `.list` file is read. -Given canonicalized list files, string comparison on files would support conflict detection. -Other pieces to be updated in a similar way include `alternatives`, `diversions`, `statoverride`, and `triggers`. - -This would affect the output of `dpkg -S`, which would then output canonicalized paths. -Packages generated by `dpkg-repack` would have their contents canonicalized as well. - -Managing the aliasing mapping using a control file --------------------------------------------------- - -It was suggested that the mapping could be managed via a special control file `canonical`. -Given that aliasing is not a common operation, the benefit of handling it declaratively is minor. -Beyond that, aliasing can also happen as an customization issued by an administrator. -Therefore, a command line based approach is preferred. - -Having dpkg move files and create symbolic links ------------------------------------------------- - -When instructed with `--add-alias`, `dpkg` could also create the corresponding symbolic links and move the affected files to their new location. -While that would be convenient, doing so is non-trivial in an atomic way. -Sometimes, the underlying filesystem does not fully conform to POSIX (e.g. `overlayfs`) and such corner cases need to be managed individually. -Since such an implementation already exists outside `dpkg` and its complexity is non-trivial, the moving of files shall remain external. -In case aliases are setup in a bootstrap setting, no moves are necessary. - -Implement aliasing after metadata tracking ------------------------------------------- - -The [metadata tracking](https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking) feature enhances `dpkg` with knowledge about filesystem metadata for installed files. -This includes knowledge of symbolic links, which would help with tracking aliasing. -Unfortunately, progress on this is fairly slow and we think that aliasing support is more urgent. - -Proposal internals -================== - -A new file `aliases` is added to the administrative directory. -Pairs of lines containing link name and destination indicate an alias. -Within this file, no link name or destination may contain another link name. -The `--add-alias` and `--remove-alias` options change this file only and must ensure that the properties are retained. -This leads to a trivial algorithm for canonicalizing paths. -A given path can be scanned for recorded link names as sub path and have them replaced with the recorded destination. -This process is repeated until a scan passes without performing a substitution. -Usually, two scan passes will be sufficient. - -Much of the internal work has been prototyped by [Simon Richter](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) and can be used. -It demonstrates how the `fsys_namenode` can be augmented with a canonicalized path and how `fsys_hash_find_node` can be extended with a new flag to differentiate between lookups considering aliases or exact names. -It differs from what is proposed here in the API to configure aliases and in possibly storing partially canonicalized versions of file names. +`base-files` contains `/bin`, `/lib` and `/sbin` as directories and as long as it contains them, `dpkg` will not delete the symbolic links that are now placed there. +If `base-files` were to also ship `/lib64` and all other multilib directories, that would effectively prevent them from being deleted, thus mitigating P9. +Its `postinst` script could still convert them to symbolic links unless already converted. +Unfortunately, `libc6` cannot `Pre-Depends` on `base-files` to ensure the right unpack order, as doing so would create a loop. +We can spread this part to two releases or additionally ship the directories in a dependency-less package (e.g. `usrmerge-support`). + +M6: Divert `dpkg-divert` +------------------------ + +`dpkg-divert` can be wrapped by a script that modifies its behavior using the diversion mechanism on itself. +Whenever a diversion is added for a location that is subject to aliasing, the diversion is duplicated to both affected locations by the wrapper. +Similarly, removal of diversions is also duplicated. +Upon installation of the wrapper, all existing diversions are also duplicated. +If combined with M4, these diversions need to be ignored. +Packages that use diversions and packages that canonicalize files affected by diversions need to issue a `Pre-Depends` on the package that installs this wrapper (e.g. a new `usrmerge-support` package). +The affected packages can be determined in a mechanical way. +The wrapper would be required for at least two stable releases. +This way P3 can be mitigated. + +The wrapper can also duplicate local diversions. +However, for removing the wrapper (and thus removing the aliased diversions), users must update their scripts and automation to canonicalize the locations to be diverted. + +M7: Replacing `Replaces` with `Conflicts` +----------------------------------------- + +When files are moved between packages, these are usually accompanied with `Replaces`. +A concurrent canonicalization may render this `Replaces` ineffective and cause the file loss described in P1. +This scenario cannot happen when the replacing package is unpacked after the replaced package has been upgraded or removed. +As such, changing `Replaces` to `Conflicts` makes this scenario impossible. +The mitigation can be applied in an as-needed manner such that moving files without canonicalizing remains unaffected as does canonicalizing without moving. +The situation can be detected using automated tools and reported mechanically. +If packages had their content canonicalized in bookworm, a low two-digit number of packages would have their `Replaces` changed to `Conflicts`. + +This strategy mostly mitigates P1, but it can become impossible to apply when essential packages are involved (as those must not be temporarily deinstalled) or the upgrade becomes too complex for `apt` due to an excess of `Conflicts`. +These cases can be complemented with the next mitigation M8. + +M8: Protective diversions for moved files +----------------------------------------- + +A package that is at risk of loosing files as in P1 can set up a protective diversion for each affected location in the aliased form. +The replacing `preinst` script has to set up these temporary diversions. +When the replacing `postinst` is run, the replaced package is already upgraded or removed (due to associated `Breaks`) and it can therefore remove the protective diversions. +These diversions only exist during an upgrade, but writing the maintainer scripts can be difficult to get right. +Therefore, M7 should be preferred when applicable. + +M9: Protective diversions for empty directories +----------------------------------------------- + +In a similar vein to M8, empty directories can be saved in a P6 scenario. +If there is one and only one package shipping the empty directory, it can set up a protective diversion in a permanent way. +Note that `dpkg` errors out when creating a diversion for an existing directory as it is not meant to be used that way, but this can be worked around by temporarily moving the directory away. +As long as the diversion exists, the package owning the diversion must ensure that the diverted location actually is a directory in the filesystem. +Otherwise, unpacking other packages may fail. +These diversions probably need to stay for two stable releases. + +A directory can also be empty in multiple packages. +In most such cases, this seems to be accidental and the directory can be deleted from the `data.tar` instead. +In other cases, the empty directory can be migrated to a common package, but this is expected to not be needed in practice. + +M10: Protective diversions for shared files +------------------------------------------- + +In a similar vein to M8 and M9, shared files in `Multi-Arch: same` instances can be saved in a P7 scenario. +The new `preinst` can divert the aliased locations contained in the old version, but it must assign these diversions to some other package (e.g. `usrmerge-support`) in order to become effective for itself. +Likewise the new `postinst` can remove these diversions, because all other instances must have been removed or unpacked by this time. +As with M8, such diversions only exist during an upgrade. + +M11: Ship symbolic links in a package +------------------------------------ + +In order to fix bootstrapping tools, the aliasing symbolic links can be shipped in some `data.tar` of a binary package (e.g. `base-files`). +Doing so requires that all packages participating in filesystem bootstrap have canonicalized their paths first, because we would otherwise get a directory vs symbolic link conflict. +In the presence of such a conflict, the behavior of `dpkg` depends on the order of unpacks. +Instead of avoiding the conflict, all bootstrapping tools can be updated to unpack the symbolic link package before all other packages . +It also requires changing `debootstrap` to no longer create the links prior to extraction as it presently passes `-k` to `tar`, which would result in an unpack failure even without any conflict between the various `data.tar` contents. +With this, P8 is addressed. + +M12: Explicitly duplicate triggers +--------------------------------- + +In order to address the missing trigger invocations from P2, the one-digit number of trigger interests can be manually duplicated. +A relation of triggering packages on the triggered package is not required, because configuration of the triggered package is equivalent to a trigger invocation and from that point on triggers work as expected. +This mitigation also needs to last for two stable releases. + +However, this does not at all mitigate trigger interest in external packages. +External packages declaring an interest in aliased locations need to have this mitigation applied as well. + +M13: Keep alternatives aliased +----------------------------- + +If we keep the alternatives and alternative providers at their aliased locations, we can still move the package contents in the `data.tar` without otherwise impacting the use of alternatives. +The major downside is that we have to eternally remember that some locations in alternatives are expressed in a non-canonical way. +In a sense, we can skip P4 by not canonicalizing alternatives. + +M14: Divert `update-alternatives` +--------------------------------- + +A wrapper to `update-alternatives` can canonicalize all paths and ensure that paths may be referred to by either way. +All packages relying on this behavior must issue a `Depends` on the package introducing the wrapper, thus solving P4. +For the trixie release, both aliased and canonical locations would be accepted such that external repositories and derivatives have a full release cycle to get updated. + +M15: Manually migrate statoverrides +----------------------------------- + +The few packages that install statoverrides can migrate them on their own, mitigating P5. + +This totally leaves local statoverrides and statoverrides from external packages unaddressed. + +M16: Change bootstrap protocol +------------------------------ + +We can consider modifications to the bootstrap protocol to alleviate the problems. +Thus far no concrete proposals to this end have emerged. + +Comparison +========== + +In the following table, we map mitigations to their properties. +The most fundamental property is which problems they fully (✓) or partially (\*) address. +We also record whether explicit (E), implicit (I) or no (-) changes are required to `dpkg`. +The order of affected packages is judged as "significantly changed packages + mechanically changed packages". +A mitigation is considered temporary if the relevant changes can be reverted after two stable releases. +A prototype is linked when available. +Finally, some mitigations cannot be combined with others. + +| | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | +| --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | +| P1 | ✓ | \* | ✓ | | | | \* | ✓ | | | | | | | | | +| P2 | ✓ | \* | | | | | | | | | | ✓ | | | | | +| P3 | ✓ | \* | | | | ✓ | | | | | | | | | | | +| P4 | ✓ | | | | | | | | | | | | ✓ | ✓ | | | +| P5 | ✓ | \* | | | | | | | | | | | | | ✓ | | +| P6 | ✓ | \* | ✓ | | | | | | ✓ | | | | | | | | +| P7 | ✓ | \* | ✓ | | | | | | | ✓ | | | | | | | +| P8 | | \* | | | | | | | | | ✓ | | | | | | +| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | ✓ | +| dpkg | E | | I | | | | | | | | | | | | | | +| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 4+0 | 4+0 | +| temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | +| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | +| precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1 | M1 | M5 | | M14 | M1,M13 | | | + +In effect, the most fundamental decision becomes how much change we want in `dpkg`. +On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). +As a middle ground, we can accept a temporary behavior difference (M3) to ease a complete move (M2) with additional mitigations (to be selected). +On the other end, we can work around the problematic behavior (to be selected) while moving the files (M2) without applying any changes to the `dpkg` source code. + +The other fundamental decision becomes how to deal with architecture bootstrap. +On one end is an underspecified idea of changing the bootstrap protocol somehow (M16). +The other end is shipping the aliasing symlinks in some package (M11), but that implies (M2). + +Proposal +======== + +This section will have a proposal once consensus has emerged. -- GitLab From 19db6ebdd88a82f6b50be53fdfd5a68b60f94962 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Wed, 21 Jun 2023 07:24:27 +0200 Subject: [PATCH 04/25] dep17: record prototype for M3 --- web/deps/dep17.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 9b9cb1b..0e5cf1c 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -359,7 +359,7 @@ Finally, some mitigations cannot be combined with others. | dpkg | E | | I | | | | | | | | | | | | | | | affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 4+0 | 4+0 | | temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | -| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | +| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1 | M1 | M5 | | M14 | M1,M13 | | | In effect, the most fundamental decision becomes how much change we want in `dpkg`. -- GitLab From 09b961e6f02f28ac4d73a6f657a4002613e19d1f Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Sun, 2 Jul 2023 20:37:52 +0200 Subject: [PATCH 05/25] dep17: M16 addresses P8 rather than P9 --- web/deps/dep17.mdwn | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 0e5cf1c..29aaa29 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -354,8 +354,8 @@ Finally, some mitigations cannot be combined with others. | P5 | ✓ | \* | | | | | | | | | | | | | ✓ | | | P6 | ✓ | \* | ✓ | | | | | | ✓ | | | | | | | | | P7 | ✓ | \* | ✓ | | | | | | | ✓ | | | | | | | -| P8 | | \* | | | | | | | | | ✓ | | | | | | -| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | ✓ | +| P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | +| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | | dpkg | E | | I | | | | | | | | | | | | | | | affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 4+0 | 4+0 | | temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | -- GitLab From 63943aa409a36872e2f35a47c1ce11ccb9e910f2 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Sun, 2 Jul 2023 21:08:25 +0200 Subject: [PATCH 06/25] dep17: note which mitigations are backportable Reported-by: Simon Richter --- web/deps/dep17.mdwn | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 29aaa29..6a80eb8 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -343,7 +343,8 @@ We also record whether explicit (E), implicit (I) or no (-) changes are required The order of affected packages is judged as "significantly changed packages + mechanically changed packages". A mitigation is considered temporary if the relevant changes can be reverted after two stable releases. A prototype is linked when available. -Finally, some mitigations cannot be combined with others. +When a mitigation is incompatible with another, this is noted. +Also indicate which mitigations happen to work when trivially backported to `bookworm`. | | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | | --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | @@ -361,6 +362,7 @@ Finally, some mitigations cannot be combined with others. | temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | | prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1 | M1 | M5 | | M14 | M1,M13 | | | +| backportable | | | | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | | ✓ | | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). -- GitLab From 03f01c8061cd25fa0303ce530937d861227c9925 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Sun, 2 Jul 2023 21:19:18 +0200 Subject: [PATCH 07/25] dep17: more precision on backportability in terms of effort --- web/deps/dep17.mdwn | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 6a80eb8..48ff53f 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -344,7 +344,7 @@ The order of affected packages is judged as "significantly changed packages + me A mitigation is considered temporary if the relevant changes can be reverted after two stable releases. A prototype is linked when available. When a mitigation is incompatible with another, this is noted. -Also indicate which mitigations happen to work when trivially backported to `bookworm`. +Also indicate which mitigations happen to work when trivially (✓) backported to `bookworm` and which can be made to work with effort (\*). | | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | | --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | @@ -362,7 +362,7 @@ Also indicate which mitigations happen to work when trivially backported to `boo | temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | | prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1 | M1 | M5 | | M14 | M1,M13 | | | -| backportable | | | | | | | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | | ✓ | | +| backportable | | \* | \* | \* | | | ✓ | ✓ | \* | ✓ | | \* | ✓ | | ✓ | | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). -- GitLab From 60a2b7f7400d85ed56e8031cded42026306748d9 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Thu, 20 Jul 2023 17:07:32 +0200 Subject: [PATCH 08/25] dep17: record two further mitigations --- web/deps/dep17.mdwn | 49 ++++++++++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 18 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 48ff53f..643022f 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -334,35 +334,48 @@ M16: Change bootstrap protocol We can consider modifications to the bootstrap protocol to alleviate the problems. Thus far no concrete proposals to this end have emerged. +M17: Duplicate empty directories +-------------------------------- + +To avert the loss of empty directories, they could be duplicated to both the canonical and aliased location. +Since both locations are recorded in the `dpkg` database, `dpkg` will not delete them. + +M18: Explicitly duplicate diversions +------------------------------------ + +Diversions of aliased locations can be duplicated to the corresponding canonical location to make them effective in both situations. +Note that the diversion destination may exist on the second diversion. +Therefore, both diversions should be issued as `--no-rename` and the renaming should be performed externally. + Comparison ========== In the following table, we map mitigations to their properties. The most fundamental property is which problems they fully (✓) or partially (\*) address. -We also record whether explicit (E), implicit (I) or no (-) changes are required to `dpkg`. +We also record whether explicit (E), implicit (I) or no ( ) changes are required to `dpkg`. The order of affected packages is judged as "significantly changed packages + mechanically changed packages". A mitigation is considered temporary if the relevant changes can be reverted after two stable releases. A prototype is linked when available. When a mitigation is incompatible with another, this is noted. Also indicate which mitigations happen to work when trivially (✓) backported to `bookworm` and which can be made to work with effort (\*). -| | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | -| --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | -| P1 | ✓ | \* | ✓ | | | | \* | ✓ | | | | | | | | | -| P2 | ✓ | \* | | | | | | | | | | ✓ | | | | | -| P3 | ✓ | \* | | | | ✓ | | | | | | | | | | | -| P4 | ✓ | | | | | | | | | | | | ✓ | ✓ | | | -| P5 | ✓ | \* | | | | | | | | | | | | | ✓ | | -| P6 | ✓ | \* | ✓ | | | | | | ✓ | | | | | | | | -| P7 | ✓ | \* | ✓ | | | | | | | ✓ | | | | | | | -| P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | -| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | -| dpkg | E | | I | | | | | | | | | | | | | | -| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 4+0 | 4+0 | -| temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | -| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | -| precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1 | M1 | M5 | | M14 | M1,M13 | | | -| backportable | | \* | \* | \* | | | ✓ | ✓ | \* | ✓ | | \* | ✓ | | ✓ | | +| | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | M17 | M18 | +| --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | ---- | ---- | +| P1 | ✓ | \* | ✓ | | | | \* | ✓ | | | | | | | | | | | +| P2 | ✓ | \* | | | | | | | | | | ✓ | | | | | | | +| P3 | ✓ | \* | | | | ✓ | | | | | | | | | | | | ✓ | +| P4 | ✓ | | | | | | | | | | | | ✓ | ✓ | | | | | +| P5 | ✓ | \* | | | | | | | | | | | | | ✓ | | | | +| P6 | ✓ | \* | ✓ | | | | | | ✓ | | | | | | | | ✓ | | +| P7 | ✓ | \* | ✓ | | | | | | | ✓ | | | | | | | | | +| P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | | | +| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | | | +| dpkg | E | | I | | | | | | | | | | | | | | | | +| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 4+0 | 4+0 | 10+0 | 30+0 | +| temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | ✓ | ✓ | +| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | | +| precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1,M17 | M1 | M5 | | M14 | M1,M13 | | | M1,M2,M9 | M1,M3 | +| backportable | | \* | \* | \* | | | ✓ | ✓ | \* | ✓ | | \* | ✓ | | ✓ | | ✓ | ✓ | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). -- GitLab From 01794888c54d150c4cd590b41d0b1cfb3041ccb8 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Fri, 21 Jul 2023 16:47:04 +0200 Subject: [PATCH 09/25] dep17: M15 really affects exactly 5 packages fuse, fuse3, ntfs-3g, systemd-cron, yp-tools --- web/deps/dep17.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 643022f..30756d2 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -371,7 +371,7 @@ Also indicate which mitigations happen to work when trivially (✓) backported t | P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | | | | P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | | | | dpkg | E | | I | | | | | | | | | | | | | | | | -| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 4+0 | 4+0 | 10+0 | 30+0 | +| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 5+0 | 4+0 | 10+0 | 30+0 | | temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | ✓ | ✓ | | prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | | | precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1,M17 | M1 | M5 | | M14 | M1,M13 | | | M1,M2,M9 | M1,M3 | -- GitLab From 6ae801d406e6071f3cfd96193409e29939082904 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 09:30:20 +0200 Subject: [PATCH 10/25] dep17: P6: also affects directories moved in upgrade The moratorium currently prevents an empty directory to be moved from / to /usr. Such moves lose directories fairly reliably. --- web/deps/dep17.mdwn | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 30756d2..6461183 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -110,8 +110,9 @@ P6: Empty directory loss Andreas Beckmann discovered that an empty directory may unexpectedly disappear when a package containing an aliased location is being removed as part of a package upgrade or removal. The first instance of this problem is with [`/usr/lib/modules-load.d` from `systemd`](https://bugs.debian.org/1036920), but it is a generic problem. -The [generic problem](https://lists.debian.org/20230530095300.GA1289743@subdivi.de) affects a [one-digit number of situations](https://lists.debian.org/debian-devel/2023/05/msg00325.html). +The [generic problem](https://lists.debian.org/20230530095300.GA1289743@subdivi.de) affects a [two-digit number of situations](https://lists.debian.org/debian-devel/2023/05/msg00325.html). This problem is not mitigated by the moratorium and really affects upgrades from `bullseye` to `bookworm` and package removal on `bookworm` installations. +As the moratorium is lifted, empty directories moved from `/` to `/usr` also will go absent. P7: Shared multiarch file loss ------------------------------ @@ -371,7 +372,7 @@ Also indicate which mitigations happen to work when trivially (✓) backported t | P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | | | | P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | | | | dpkg | E | | I | | | | | | | | | | | | | | | | -| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 10+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 5+0 | 4+0 | 10+0 | 30+0 | +| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 25+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 5+0 | 4+0 | 25+0 | 30+0 | | temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | ✓ | ✓ | | prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | | | precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1,M17 | M1 | M5 | | M14 | M1,M13 | | | M1,M2,M9 | M1,M3 | -- GitLab From b6b1b1a76497361b19a40b068904d832a79ecbc7 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 09:46:41 +0200 Subject: [PATCH 11/25] dep17: add M19: change debootstrap This item has evolved from our discussion on debian-devel@lists.debian.org. --- web/deps/dep17.mdwn | 42 +++++++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 17 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 6461183..72f45c4 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -348,6 +348,14 @@ Diversions of aliased locations can be duplicated to the corresponding canonical Note that the diversion destination may exist on the second diversion. Therefore, both diversions should be issued as `--no-rename` and the renaming should be performed externally. +M19: Swap initial unpack and merge in `debootstrap` +--------------------------------------------------- + +We recognize that effectively, `debootstrap` has changed the bootstrap protocol, but other implementations have not changed. +Adding the aliasing symlinks to any package (M11) would therefore break `debootstrap` and `debootstrap` only. +This can be mitigated by [performing the merge after the initial unpack before the initial configuration](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96). +Doing so partially reverts the change in bootstrap protocol implemented in `debootstrap` such that this change becomes a no-op for `trixie` and beyond. + Comparison ========== @@ -360,23 +368,23 @@ A prototype is linked when available. When a mitigation is incompatible with another, this is noted. Also indicate which mitigations happen to work when trivially (✓) backported to `bookworm` and which can be made to work with effort (\*). -| | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | M17 | M18 | -| --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | ---- | ---- | -| P1 | ✓ | \* | ✓ | | | | \* | ✓ | | | | | | | | | | | -| P2 | ✓ | \* | | | | | | | | | | ✓ | | | | | | | -| P3 | ✓ | \* | | | | ✓ | | | | | | | | | | | | ✓ | -| P4 | ✓ | | | | | | | | | | | | ✓ | ✓ | | | | | -| P5 | ✓ | \* | | | | | | | | | | | | | ✓ | | | | -| P6 | ✓ | \* | ✓ | | | | | | ✓ | | | | | | | | ✓ | | -| P7 | ✓ | \* | ✓ | | | | | | | ✓ | | | | | | | | | -| P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | | | -| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | | | -| dpkg | E | | I | | | | | | | | | | | | | | | | -| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 25+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 5+0 | 4+0 | 25+0 | 30+0 | -| temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | ✓ | ✓ | -| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | | -| precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1,M17 | M1 | M5 | | M14 | M1,M13 | | | M1,M2,M9 | M1,M3 | -| backportable | | \* | \* | \* | | | ✓ | ✓ | \* | ✓ | | \* | ✓ | | ✓ | | ✓ | ✓ | +| | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | M17 | M18 | M19 | +| --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | ---- | ---- | --- | +| P1 | ✓ | \* | ✓ | | | | \* | ✓ | | | | | | | | | | | | +| P2 | ✓ | \* | | | | | | | | | | ✓ | | | | | | | | +| P3 | ✓ | \* | | | | ✓ | | | | | | | | | | | | ✓ | | +| P4 | ✓ | | | | | | | | | | | | ✓ | ✓ | | | | | | +| P5 | ✓ | \* | | | | | | | | | | | | | ✓ | | | | | +| P6 | ✓ | \* | ✓ | | | | | | ✓ | | | | | | | | ✓ | | | +| P7 | ✓ | \* | ✓ | | | | | | | ✓ | | | | | | | | | | +| P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | | | ✓ | +| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | | | | +| dpkg | E | | I | | | | | | | | | | | | | | | | | +| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 25+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 5+0 | 4+0 | 25+0 | 30+0 | 1+0 | +| temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | ✓ | ✓ | | +| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | +| precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1,M17 | M1 | M5 | | M14 | M1,M13 | | M19 | M1,M2,M9 | M1,M3 | M16 | +| backportable | | \* | \* | \* | | | ✓ | ✓ | \* | ✓ | | \* | ✓ | | ✓ | | ✓ | ✓ | ✓ | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). -- GitLab From fee2c7dfde3a08eab1ac44f2dbcc7478044a442b Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 10:23:00 +0200 Subject: [PATCH 12/25] dep17: flip the comparison table As we are adding mitigations, it becomes more wide than tall. --- web/deps/dep17.mdwn | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 72f45c4..8302465 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -368,23 +368,27 @@ A prototype is linked when available. When a mitigation is incompatible with another, this is noted. Also indicate which mitigations happen to work when trivially (✓) backported to `bookworm` and which can be made to work with effort (\*). -| | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | M14 | M15 | M16 | M17 | M18 | M19 | -| --------- | ---- | ---- | --- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | --- | --- | --- | --- | --- | --- | ---- | ---- | --- | -| P1 | ✓ | \* | ✓ | | | | \* | ✓ | | | | | | | | | | | | -| P2 | ✓ | \* | | | | | | | | | | ✓ | | | | | | | | -| P3 | ✓ | \* | | | | ✓ | | | | | | | | | | | | ✓ | | -| P4 | ✓ | | | | | | | | | | | | ✓ | ✓ | | | | | | -| P5 | ✓ | \* | | | | | | | | | | | | | ✓ | | | | | -| P6 | ✓ | \* | ✓ | | | | | | ✓ | | | | | | | | ✓ | | | -| P7 | ✓ | \* | ✓ | | | | | | | ✓ | | | | | | | | | | -| P8 | | \* | | | | | | | | | ✓ | | | | | ✓ | | | ✓ | -| P9 | ✓ | \* | ✓ | ✓ | ✓ | | | | | | ✓ | | | | | | | | | -| dpkg | E | | I | | | | | | | | | | | | | | | | | -| affected | 1+0 | many | 1+0 | 2+10 | 1+0 | 1+30 | 0+10 | 10+0 | 25+0 | 30+0 | 2+0 | 0+2 | 0+0 | 1+0 | 5+0 | 4+0 | 25+0 | 30+0 | 1+0 | -| temporary | | | ✓ | ✓ | \* | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | ✓ | | | ✓ | ✓ | | -| prototype | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | | | | | | | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | -| precludes | many | | M1 | M1 | M11 | M1 | | M1 | M1,M17 | M1 | M5 | | M14 | M1,M13 | | M19 | M1,M2,M9 | M1,M3 | M16 | -| backportable | | \* | \* | \* | | | ✓ | ✓ | \* | ✓ | | \* | ✓ | | ✓ | | ✓ | ✓ | ✓ | +| | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | dpkg | affected | temp | prototype | precludes | backportable | +| --- | -- | -- | -- | -- | -- | -- | -- | -- | -- | ---- | -------- | ---- | ---------------------------------------------- | --------- | ------------ | +| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | many | | +| M2 | \* | \* | \* | | \* | \* | \* | \* | \* | | many | | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | | \* | +| M3 | ✓ | | | | | ✓ | ✓ | | ✓ | I | 1+0 | ✓ | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | M1 | \* | +| M4 | | | | | | | | | ✓ | | 2+10 | ✓ | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M1 | \* | +| M5 | | | | | | | | | ✓ | | 1+0 | \* | | M11 | | +| M6 | | | ✓ | | | | | | | | 1+30 | ✓ | | M1 | | +| M7 | \* | | | | | | | | | | 0+10 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | ✓ | +| M8 | ✓ | | | | | | | | | | 10+0 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | M1 | ✓ | +| M9 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M17 | \* | +| M10 | | | | | | | ✓ | | | | 30+0 | ✓ | | M1 | ✓ | +| M11 | | | | | | | | ✓ | ✓ | | 2+0 | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M5 | | +| M12 | | ✓ | | | | | | | | | 0+2 | ✓ | | | \* | +| M13 | | | | ✓ | | | | | | | 0+0 | | | M14 | ✓ | +| M14 | | | | ✓ | | | | | | | 1+0 | ✓ | | M1,M13 | | +| M15 | | | | | ✓ | | | | | | 5+0 | | | | ✓ | +| M16 | | | | | | | | ✓ | | | 4+0 | | | M19 | | +| M17 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M2,M9 | ✓ | +| M18 | | | ✓ | | | | | | | | 30+0 | ✓ | | M1,M3 | ✓ | +| M19 | | | | | | | | ✓ | | | 1+0 | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | M16 | ✓ | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). -- GitLab From 4aa965b5e115a24d530d94662ae050e1dcede041 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 14:05:01 +0200 Subject: [PATCH 13/25] dep17: improve "precludes" column in comparison table --- web/deps/dep17.mdwn | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 8302465..b84f438 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -368,26 +368,26 @@ A prototype is linked when available. When a mitigation is incompatible with another, this is noted. Also indicate which mitigations happen to work when trivially (✓) backported to `bookworm` and which can be made to work with effort (\*). -| | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | dpkg | affected | temp | prototype | precludes | backportable | -| --- | -- | -- | -- | -- | -- | -- | -- | -- | -- | ---- | -------- | ---- | ---------------------------------------------- | --------- | ------------ | -| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | many | | -| M2 | \* | \* | \* | | \* | \* | \* | \* | \* | | many | | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | | \* | -| M3 | ✓ | | | | | ✓ | ✓ | | ✓ | I | 1+0 | ✓ | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | M1 | \* | -| M4 | | | | | | | | | ✓ | | 2+10 | ✓ | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M1 | \* | -| M5 | | | | | | | | | ✓ | | 1+0 | \* | | M11 | | -| M6 | | | ✓ | | | | | | | | 1+30 | ✓ | | M1 | | -| M7 | \* | | | | | | | | | | 0+10 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | ✓ | -| M8 | ✓ | | | | | | | | | | 10+0 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | M1 | ✓ | -| M9 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M17 | \* | -| M10 | | | | | | | ✓ | | | | 30+0 | ✓ | | M1 | ✓ | -| M11 | | | | | | | | ✓ | ✓ | | 2+0 | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M5 | | -| M12 | | ✓ | | | | | | | | | 0+2 | ✓ | | | \* | -| M13 | | | | ✓ | | | | | | | 0+0 | | | M14 | ✓ | -| M14 | | | | ✓ | | | | | | | 1+0 | ✓ | | M1,M13 | | -| M15 | | | | | ✓ | | | | | | 5+0 | | | | ✓ | -| M16 | | | | | | | | ✓ | | | 4+0 | | | M19 | | -| M17 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M2,M9 | ✓ | -| M18 | | | ✓ | | | | | | | | 30+0 | ✓ | | M1,M3 | ✓ | +| | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | dpkg | affected | temp | prototype | precludes | backportable | +| --- | -- | -- | -- | -- | -- | -- | -- | -- | -- | ---- | -------- | ---- | ----------------------------------------------- | --------- | ------------ | +| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | many | | +| M2 | \* | \* | \* | | \* | \* | \* | \* | \* | | many | | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | M2 | \* | +| M3 | ✓ | | | | | ✓ | ✓ | | ✓ | I | 1+0 | ✓ | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | M1,M18 | \* | +| M4 | | | | | | | | | ✓ | | 2+10 | ✓ | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M1 | \* | +| M5 | | | | | | | | | ✓ | | 1+0 | \* | | M11 | | +| M6 | | | ✓ | | | | | | | | 1+30 | ✓ | | M1 | | +| M7 | \* | | | | | | | | | | 0+10 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | ✓ | +| M8 | ✓ | | | | | | | | | | 10+0 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | M1 | ✓ | +| M9 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M17 | \* | +| M10 | | | | | | | ✓ | | | | 30+0 | ✓ | | M1 | ✓ | +| M11 | | | | | | | | ✓ | ✓ | | 2+0 | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M5,M17 | | +| M12 | | ✓ | | | | | | | | | 0+2 | ✓ | | | \* | +| M13 | | | | ✓ | | | | | | | 0+0 | | | M14 | ✓ | +| M14 | | | | ✓ | | | | | | | 1+0 | ✓ | | M1,M13 | | +| M15 | | | | | ✓ | | | | | | 5+0 | | | | ✓ | +| M16 | | | | | | | | ✓ | | | 4+0 | | | M19 | | +| M17 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M2,M9,M11 | ✓ | +| M18 | | | ✓ | | | | | | | | 30+0 | ✓ | | M1,M3 | ✓ | | M19 | | | | | | | | ✓ | | | 1+0 | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | M16 | ✓ | In effect, the most fundamental decision becomes how much change we want in `dpkg`. -- GitLab From ed0cb7b25ff7272ff5fafa250493c17e218c3603 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 14:05:40 +0200 Subject: [PATCH 14/25] dep17: add two more mitigations for P6 --- web/deps/dep17.mdwn | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index b84f438..7411239 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -356,6 +356,17 @@ Adding the aliasing symlinks to any package (M11) would therefore break `deboots This can be mitigated by [performing the merge after the initial unpack before the initial configuration](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96). Doing so partially reverts the change in bootstrap protocol implemented in `debootstrap` such that this change becomes a no-op for `trixie` and beyond. +M20: Restore empty directories in maintainer scripts +---------------------------------------------------- + +When an empty directory is lost as part of an upgrade, the `postinst` script runs after the unpack phase loosing the directory and can manually restore the directory. +When an empty directory is lost due to a different package being removed, this removal can activate a trigger and similarly invoke a maintainer script of the package owning the empty directory to restore it. + +M21: Add placeholder files +-------------------------- + +Since a directory is kept when it is non-empty, adding any file to it will prevent its loss. + Comparison ========== @@ -389,6 +400,8 @@ Also indicate which mitigations happen to work when trivially (✓) backported t | M17 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M2,M9,M11 | ✓ | | M18 | | | ✓ | | | | | | | | 30+0 | ✓ | | M1,M3 | ✓ | | M19 | | | | | | | | ✓ | | | 1+0 | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | M16 | ✓ | +| M20 | | | | | | ✓ | | | | | 25+0 | ✓ | [#](https://lists.debian.org/20230815094034.GA3537194@subdivi.de) | | ✓ | +| M21 | | | | | | ✓ | | | | | 25+0 | ✓ | | | ✓ | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). -- GitLab From 5a31c37a50814cfdaa63ad34a5008a224065fb0a Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 14:06:02 +0200 Subject: [PATCH 15/25] dep17: add an actual proposal on how to move forward --- web/deps/dep17.mdwn | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 7411239..2ae1a3b 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -415,4 +415,43 @@ The other end is shipping the aliasing symlinks in some package (M11), but that Proposal ======== -This section will have a proposal once consensus has emerged. +Mechanism +--------- + +Discussion `debian-devel@lists.debian.org` indicates that the project prefers to finish the transition without relying on changes to `dpkg` as primary mechanism. +This amounts to rejecting M1 and selecting M2. +While there is agreement on this, there is disagreement on the reasons for this choice. + +Regarding the bootstrapping matter, a small amount of developers that previously worked on `debootstrap` reached consensus on partially reverting the bootstrap protocol changes added to `debootstrap` (M19). + +Proposed combination of mitigations: + * M2: Canonicalize all paths + * M4: Protective diversions for aliasing links + * M7: Replacing `Replaces` with `Conflicts` + * M8: Protective diversions for moved files + * M10: Protective diversions for shared files + * M11: Ship symbolic links in a package + * M12: Explicitly duplicate triggers + * M13: Keep alternatives aliased (initially, reconsider M14 later) + * M15: Manually migrate statoverrides + * M19: Swap initial unpack and merge in `debootstrap` + +This combination resolves the majority of problems. +For P3 (ineffective diversions), we may either choose a central mitigation (M6) or a decentral mitigation (M18). +For P6 (empty directory loss), the majority of instances are being deleted and the remaining ones may choose between M20 (maintainer scripts and triggers) and M21 (placeholder file) and maybe also M9 (protective diversions) on a case-by-case way in cooperation with relevant package maintainers. + +Process +------- + +We continuously [monitor the archive](https://salsa.debian.org/helmutg/dumat) for problematic situations. +When problems are detected, bugs (including RC bugs) are automatically filed (continuous MBF) for the affected packages. +Concurrently, mitigations that are compatible with the moratorium are implemented (e.g. deleting unused empty directories, M12, M19). +`debhelper` is being extended with a new `dh_usrmerge` helper that performs the moving of files in package contents. +This helper can also be enabled using the new `usrmerge` addon or a sufficiently high compatibility level. +A MBF at minor severity informs maintainers about possible problems when canonicalizing paths inside their packages. +Once the automatic bug filing works and buildds are converted to merged chroots, the file move moratorium is lifted. +A small set of essential packages (including `bash`, `dash`, `glibc`, `util-linux`) still must keep some of their files in aliased locations. +Maintainers are asked to always upload changes that move files between packages and changes in canonicalization to `experimental` first and only proceed to `unstable` if they do not receive an automatic bug report within three days. +To speed up the process, patches and NMUs are used to convert packages. +Once most of the transitively essential packages are converted, a concurrent upload of remaining packages is coordinated with the affected maintainers and NMUed. +This ensures that no transitively essential package ships aliased paths and `base-files` contains the aliasing links. -- GitLab From 87a52be6e8a6cfa23d06db89b7023ec82527ce67 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 14:34:37 +0200 Subject: [PATCH 16/25] dep17: fix logic error in P1 Reported-by: Stefano Rivera <stefanor@debian.org> --- web/deps/dep17.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 2ae1a3b..6fb6750 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -42,7 +42,7 @@ P1: File loss during canonicalized file move -------------------------------------------- When moving a file from its aliased location to a canonical location in the `data.tar` of a binary package and moving this file from one binary package to another, `dpkg` may unexpectedly delete the file in question during an upgrade procedure. -If the replacing package is unpacked first, the affected file is installed in its canonical location before the replacing package is upgraded or removed. +If the replacing package is unpacked first, the affected file is installed in its canonical location before the replaced package is upgraded or removed. `dpkg` may then delete the affected file by removing the aliased location - not realizing that it is deleting a file that still is needed. This problem was originally observed in [#974552](https://lists.debian.org/974552) and is the one that motivated the issuance of the moratorium. -- GitLab From 1794d629aea17ea795edd8e550ebeb00d3b6d96f Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Tue, 15 Aug 2023 16:17:00 +0200 Subject: [PATCH 17/25] dep17: fix link to sjr's tree Reported-by: Stefano Rivera <stefanor@debian.org> --- web/deps/dep17.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 6fb6750..53ba64e 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -381,7 +381,7 @@ Also indicate which mitigations happen to work when trivially (✓) backported t | | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | dpkg | affected | temp | prototype | precludes | backportable | | --- | -- | -- | -- | -- | -- | -- | -- | -- | -- | ---- | -------- | ---- | ----------------------------------------------- | --------- | ------------ | -| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths) | many | | +| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip/alias) | many | | | M2 | \* | \* | \* | | \* | \* | \* | \* | \* | | many | | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | M2 | \* | | M3 | ✓ | | | | | ✓ | ✓ | | ✓ | I | 1+0 | ✓ | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | M1,M18 | \* | | M4 | | | | | | | | | ✓ | | 2+10 | ✓ | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M1 | \* | -- GitLab From 169ef57fd39ec954373499c1582f86c768931091 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Wed, 16 Aug 2023 14:43:01 +0200 Subject: [PATCH 18/25] dep17: P10/M22: debian-installer is affected --- web/deps/dep17.mdwn | 59 +++++++++++++++++++++++++++------------------ 1 file changed, 36 insertions(+), 23 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 53ba64e..0aaf7b4 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -145,6 +145,12 @@ For instance, `libc6:amd64` is the only package that contains `/lib64`. Canonicalizing its files would cause `/lib64` to be deleted and make the dynamic linker unavailable. This is prevented by the moratorium for now. +P10: `debian-installer` +----------------------- + +As of this writing and in Debian bookworm, the root filesystem used by the Debian installer, which is constructed from `udeb` packages, is not merged. +If packages were to move their files to canonical locations and thereby affect the contents of `udeb` packages, that could make the installer dysfunctional. + Proposed mitigations ==================== @@ -367,6 +373,11 @@ M21: Add placeholder files Since a directory is kept when it is non-empty, adding any file to it will prevent its loss. +M22: Also perform the /usr-merge for the Debian installer +--------------------------------------------------------- + +In order to avoid breaking the Debian installer, we can perform the same aliasing that we are doing in the main archive. + Comparison ========== @@ -379,29 +390,30 @@ A prototype is linked when available. When a mitigation is incompatible with another, this is noted. Also indicate which mitigations happen to work when trivially (✓) backported to `bookworm` and which can be made to work with effort (\*). -| | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | dpkg | affected | temp | prototype | precludes | backportable | -| --- | -- | -- | -- | -- | -- | -- | -- | -- | -- | ---- | -------- | ---- | ----------------------------------------------- | --------- | ------------ | -| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip/alias) | many | | -| M2 | \* | \* | \* | | \* | \* | \* | \* | \* | | many | | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | M2 | \* | -| M3 | ✓ | | | | | ✓ | ✓ | | ✓ | I | 1+0 | ✓ | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | M1,M18 | \* | -| M4 | | | | | | | | | ✓ | | 2+10 | ✓ | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M1 | \* | -| M5 | | | | | | | | | ✓ | | 1+0 | \* | | M11 | | -| M6 | | | ✓ | | | | | | | | 1+30 | ✓ | | M1 | | -| M7 | \* | | | | | | | | | | 0+10 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | ✓ | -| M8 | ✓ | | | | | | | | | | 10+0 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | M1 | ✓ | -| M9 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M17 | \* | -| M10 | | | | | | | ✓ | | | | 30+0 | ✓ | | M1 | ✓ | -| M11 | | | | | | | | ✓ | ✓ | | 2+0 | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M5,M17 | | -| M12 | | ✓ | | | | | | | | | 0+2 | ✓ | | | \* | -| M13 | | | | ✓ | | | | | | | 0+0 | | | M14 | ✓ | -| M14 | | | | ✓ | | | | | | | 1+0 | ✓ | | M1,M13 | | -| M15 | | | | | ✓ | | | | | | 5+0 | | | | ✓ | -| M16 | | | | | | | | ✓ | | | 4+0 | | | M19 | | -| M17 | | | | | | ✓ | | | | | 25+0 | ✓ | | M1,M2,M9,M11 | ✓ | -| M18 | | | ✓ | | | | | | | | 30+0 | ✓ | | M1,M3 | ✓ | -| M19 | | | | | | | | ✓ | | | 1+0 | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | M16 | ✓ | -| M20 | | | | | | ✓ | | | | | 25+0 | ✓ | [#](https://lists.debian.org/20230815094034.GA3537194@subdivi.de) | | ✓ | -| M21 | | | | | | ✓ | | | | | 25+0 | ✓ | | | ✓ | +| | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | P10 | dpkg | affected | temp | prototype | precludes | backportable | +| --- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --- | ---- | -------- | ---- | ----------------------------------------------- | --------- | ------------ | +| M1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ | | E | 1+0 | | [#](https://salsa.debian.org/sjr/dpkg/-/tree/wip/alias) | many | | +| M2 | \* | \* | \* | | \* | \* | \* | \* | \* | | | many | | [#](https://lists.debian.org/debian-dpkg/2023/05/msg00080.html) | M2 | \* | +| M3 | ✓ | | | | | ✓ | ✓ | | ✓ | | I | 1+0 | ✓ | [#](https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/aliasing-workaround) | M1,M18 | \* | +| M4 | | | | | | | | | ✓ | | | 2+10 | ✓ | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M1 | \* | +| M5 | | | | | | | | | ✓ | | | 1+0 | \* | | M11 | | +| M6 | | | ✓ | | | | | | | | | 1+30 | ✓ | | M1 | | +| M7 | \* | | | | | | | | | | | 0+10 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | ✓ | +| M8 | ✓ | | | | | | | | | | | 10+0 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | M1 | ✓ | +| M9 | | | | | | ✓ | | | | | | 25+0 | ✓ | | M1,M17 | \* | +| M10 | | | | | | | ✓ | | | | | 30+0 | ✓ | | M1 | ✓ | +| M11 | | | | | | | | ✓ | ✓ | | | 2+0 | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M5,M17 | | +| M12 | | ✓ | | | | | | | | | | 0+2 | ✓ | | | \* | +| M13 | | | | ✓ | | | | | | | | 0+0 | | | M14 | ✓ | +| M14 | | | | ✓ | | | | | | | | 1+0 | ✓ | | M1,M13 | | +| M15 | | | | | ✓ | | | | | | | 5+0 | | | | ✓ | +| M16 | | | | | | | | ✓ | | | | 4+0 | | | M19 | | +| M17 | | | | | | ✓ | | | | | | 25+0 | ✓ | | M1,M2,M9,M11 | ✓ | +| M18 | | | ✓ | | | | | | | | | 30+0 | ✓ | | M1,M3 | ✓ | +| M19 | | | | | | | | ✓ | | | | 1+0 | | [#](https://salsa.debian.org/installer-team/debootstrap/-/merge_requests/96) | M16 | ✓ | +| M20 | | | | | | ✓ | | | | | | 25+0 | ✓ | [#](https://lists.debian.org/20230815094034.GA3537194@subdivi.de) | | ✓ | +| M21 | | | | | | ✓ | | | | | | 25+0 | ✓ | | | ✓ | +| M22 | | | | | | | | | | ✓ | | 1+0 | | [#](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39) | | ✓ | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). @@ -435,6 +447,7 @@ Proposed combination of mitigations: * M13: Keep alternatives aliased (initially, reconsider M14 later) * M15: Manually migrate statoverrides * M19: Swap initial unpack and merge in `debootstrap` + * M22: Also perform the /usr-merge for the Debian installer This combination resolves the majority of problems. For P3 (ineffective diversions), we may either choose a central mitigation (M6) or a decentral mitigation (M18). -- GitLab From b35e7cf58c18ebb1743dac8fbda621ff0d9bfafd Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Fri, 17 Nov 2023 07:48:50 +0100 Subject: [PATCH 19/25] DEP17: M22 has been implemented --- web/deps/dep17.mdwn | 1 + 1 file changed, 1 insertion(+) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 0aaf7b4..63f123c 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -377,6 +377,7 @@ M22: Also perform the /usr-merge for the Debian installer --------------------------------------------------------- In order to avoid breaking the Debian installer, we can perform the same aliasing that we are doing in the main archive. +This has been [implemented](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39). Comparison ========== -- GitLab From 03ffcf62b13ba522e5e05a6b9aa30c46df38eaa3 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Fri, 17 Nov 2023 07:49:08 +0100 Subject: [PATCH 20/25] DEP17: another mitigation M23 for P7 --- web/deps/dep17.mdwn | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 63f123c..2ff6898 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -379,6 +379,16 @@ M22: Also perform the /usr-merge for the Debian installer In order to avoid breaking the Debian installer, we can perform the same aliasing that we are doing in the main archive. This has been [implemented](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39). +M23: Restore lost files during package configuration +---------------------------------------------------- + +Files lost in an a Multi-Arch upgrade scenario are definitely lost by the time the upgraded `postinst` runs. +Shared files that used to be aliased can be augmented with a hard link in the upgraded package. +This hard link will persist in the installation even though it is not otherwise needed. +The upgraded `postinst` script can check for the existence of possibly lost files and restore them from the other link name. +This approach yields to a window during which affected files actually are missing. +For instance, reloading `udev` at such a time may lead to rules not being applied. + Comparison ========== @@ -402,7 +412,7 @@ Also indicate which mitigations happen to work when trivially (✓) backported t | M7 | \* | | | | | | | | | | | 0+10 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | | ✓ | | M8 | ✓ | | | | | | | | | | | 10+0 | ✓ | [#](https://lists.debian.org/20230425190728.GA1471384@subdivi.de) | M1 | ✓ | | M9 | | | | | | ✓ | | | | | | 25+0 | ✓ | | M1,M17 | \* | -| M10 | | | | | | | ✓ | | | | | 30+0 | ✓ | | M1 | ✓ | +| M10 | | | | | | | ✓ | | | | | 30+0 | ✓ | | M1,M23 | ✓ | | M11 | | | | | | | | ✓ | ✓ | | | 2+0 | | [#](https://lists.debian.org/20230517093036.GA4104525@subdivi.de) | M5,M17 | | | M12 | | ✓ | | | | | | | | | | 0+2 | ✓ | | | \* | | M13 | | | | ✓ | | | | | | | | 0+0 | | | M14 | ✓ | @@ -415,6 +425,7 @@ Also indicate which mitigations happen to work when trivially (✓) backported t | M20 | | | | | | ✓ | | | | | | 25+0 | ✓ | [#](https://lists.debian.org/20230815094034.GA3537194@subdivi.de) | | ✓ | | M21 | | | | | | ✓ | | | | | | 25+0 | ✓ | | | ✓ | | M22 | | | | | | | | | | ✓ | | 1+0 | | [#](https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39) | | ✓ | +| M23 | | | | | | | ✓ | | | | | 30+0 | ✓ | | M1,M10 | ✓ | In effect, the most fundamental decision becomes how much change we want in `dpkg`. On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1). -- GitLab From 9f12c4359492fccb8fb669e17d8024acea60e927 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Mon, 20 Nov 2023 17:07:54 +0100 Subject: [PATCH 21/25] DEP17: rewrite M18 given that it didn't work before More context at https://bugs.debian.org/1056279. --- web/deps/dep17.mdwn | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 2ff6898..c61e547 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -351,8 +351,13 @@ M18: Explicitly duplicate diversions ------------------------------------ Diversions of aliased locations can be duplicated to the corresponding canonical location to make them effective in both situations. -Note that the diversion destination may exist on the second diversion. -Therefore, both diversions should be issued as `--no-rename` and the renaming should be performed externally. +If the destination of those diversions equal up to aliasing and a diverted package moves a file from `/` to `/usr`, `dpkg`'s usual protection against removing just unpacked files fails and we experience file loss. +Therefore, the diversion destinations must differ in more than just aliasing. +If the diverting package needs access to the original implementations, it can declare `Breaks` for all providers that have not moved their files to `/usr`. +On the flip side, all diverted packages need to declare `Conflicts` for diverters that have not moved to `/usr`. +Consequently, these packages will be upgraded in lock step. +Avoid the use of mutual `Conflicts` as that would require `apt` to temporarily remove one. +If the approach using `Breaks` is chosen, the aliased diversions can be removed in `postinst`. M19: Swap initial unpack and merge in `debootstrap` --------------------------------------------------- -- GitLab From 9c5bb4234c164b25ec00e9c857612ee73e6007d9 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Mon, 20 Nov 2023 17:09:07 +0100 Subject: [PATCH 22/25] DEP17: new diversion problem P11 More context at https://bugs.debian.org/1056279. --- web/deps/dep17.mdwn | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index c61e547..fc626f4 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -151,6 +151,12 @@ P10: `debian-installer` As of this writing and in Debian bookworm, the root filesystem used by the Debian installer, which is constructed from `udeb` packages, is not merged. If packages were to move their files to canonical locations and thereby affect the contents of `udeb` packages, that could make the installer dysfunctional. +P11: Broken symlinks on moved diversions +---------------------------------------- + +When diverting a location that is provided as a symbolic link, the link may become relative due to [policy 10.5](https://www.debian.org/doc/debian-policy/ch-files.html#symbolic-links). +If the destination of a diversion is located in a different directory than the diverted path, a relative symbolic link may become broken. + Proposed mitigations ==================== -- GitLab From 9e82f592a8eff22e64fda43613479976a89df3cd Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Mon, 15 Jan 2024 09:26:52 +0100 Subject: [PATCH 23/25] dep17: fix bug link MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reported-by: AnÃbal Monsalve Salazar <anibal@debian.org> --- web/deps/dep17.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index fc626f4..8ebb5cb 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -45,7 +45,7 @@ When moving a file from its aliased location to a canonical location in the `dat If the replacing package is unpacked first, the affected file is installed in its canonical location before the replaced package is upgraded or removed. `dpkg` may then delete the affected file by removing the aliased location - not realizing that it is deleting a file that still is needed. -This problem was originally observed in [#974552](https://lists.debian.org/974552) and is the one that motivated the issuance of the moratorium. +This problem was originally observed in [#974552](https://bugs.debian.org/974552) and is the one that motivated the issuance of the moratorium. Since the moratorium came into effect and file moves have been prevented, no new cases surfaced. Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in a small two-digit number of cases. [For instance](https://lists.debian.org/20230426223406.GB1695204@subdivi.de), `/lib/systemd/system/dbus.socket` could have been canonicalized while it has been moved from `dbus` to `dbus-system-bus-common`. -- GitLab From 320384f916f51c0d6ff90318e3332061bd94c5e3 Mon Sep 17 00:00:00 2001 From: Helmut Grohne <helmut@subdivi.de> Date: Thu, 18 Jan 2024 11:23:16 +0100 Subject: [PATCH 24/25] dep17: discuss problems around ineffective conflicts --- web/deps/dep17.mdwn | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/web/deps/dep17.mdwn b/web/deps/dep17.mdwn index 8ebb5cb..6e6c902 100644 --- a/web/deps/dep17.mdwn +++ b/web/deps/dep17.mdwn @@ -157,6 +157,14 @@ P11: Broken symlinks on moved diversions When diverting a location that is provided as a symbolic link, the link may become relative due to [policy 10.5](https://www.debian.org/doc/debian-policy/ch-files.html#symbolic-links). If the destination of a diversion is located in a different directory than the diverted path, a relative symbolic link may become broken. +P12: Conflicts do not prevent concurrent unpacks +------------------------------------------------ + +Some proposed mitigations (M7, M18) rely on declared `Conflicts` to prevent aliasing effects from taking effect during concurrent unpacks. +However, concurrent unpack is possible in some situations despite declared `Conflicts`. +It can be [experienced by invoking dpkg directly](https://bugs.debian.org/1058937) and [policy is being clarified](https://bugs.debian.org/1057199). +Since experiencing it with apt seems unlikely (unless mutual conflicts are involved), so it is not clear yet whether this [needs to be mitigated](https://bugs.debian.org/1060700). + Proposed mitigations ==================== @@ -268,6 +276,7 @@ If packages had their content canonicalized in bookworm, a low two-digit number This strategy mostly mitigates P1, but it can become impossible to apply when essential packages are involved (as those must not be temporarily deinstalled) or the upgrade becomes too complex for `apt` due to an excess of `Conflicts`. These cases can be complemented with the next mitigation M8. +Also consider the limited effectiveness of this mitigation due to P12. M8: Protective diversions for moved files ----------------------------------------- @@ -359,11 +368,17 @@ M18: Explicitly duplicate diversions Diversions of aliased locations can be duplicated to the corresponding canonical location to make them effective in both situations. If the destination of those diversions equal up to aliasing and a diverted package moves a file from `/` to `/usr`, `dpkg`'s usual protection against removing just unpacked files fails and we experience file loss. Therefore, the diversion destinations must differ in more than just aliasing. -If the diverting package needs access to the original implementations, it can declare `Breaks` for all providers that have not moved their files to `/usr`. -On the flip side, all diverted packages need to declare `Conflicts` for diverters that have not moved to `/usr`. -Consequently, these packages will be upgraded in lock step. -Avoid the use of mutual `Conflicts` as that would require `apt` to temporarily remove one. -If the approach using `Breaks` is chosen, the aliased diversions can be removed in `postinst`. + +If the diverting package needs access to the original implementations, it has two options. +It can either take care to look up both diverted locations or declare `Breaks` for all providers that have not moved their files to `/usr`. + +Packages that are being diverted must take care that they are not unpacked in the presence of a diverter that has not been updated yet. +In principle, this is possible by declaring `Conflicts` for diverters, but due to P12, this approach is not always sufficient. +In particular, when the diverter declares `Breaks` with the provider, this is known to cause file loss. +The alternative is to detect the diversions in the divertee's `preinst` and duplicate them on behalf of the diverting package. +When this is done, the diverting package has to be modified to consider the case where the divertee has duplicated the diversions. +Also the divertee has to handle the case where the diverter has been removed after its `preinst` has been run and delete the duplicated diversion in such a case. +There is an example implementation of this approach for [cryptsetup and cryptsetup-nuke-password](https://bugs.debian.org/1060270#23). M19: Swap initial unpack and merge in `debootstrap` --------------------------------------------------- -- GitLab From 60e76dc3845d37301a3c93d280396ab83a9016a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20Hertzog?= <hertzog@debian.org> Date: Fri, 8 Mar 2024 07:42:41 +0000 Subject: [PATCH 25/25] Fix title in index page --- web/index.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/index.mdwn b/web/index.mdwn index ae02214..ee84c1b 100644 --- a/web/index.mdwn +++ b/web/index.mdwn @@ -45,7 +45,7 @@ Status | Title **CANDIDATE** | [*DEP 14: Recommended layout for Git packaging repositories*](deps/dep14) **DRAFT** | [*DEP 15: Reserved namespace for DD-approved non-maintainer changes*](deps/dep15) **DRAFT** | [*DEP 16: Improved confidential voting for General Resolutions*](deps/dep16) -**DRAFT** | [*DEP 17: Improve support for directory aliasing in dpkg*](deps/dep17) +**DRAFT** | [*DEP 17: Improve situation around aliasing effects from `/usr`-merge"*](deps/dep17) """]] Additional information -- GitLab