Sockets fail after remove/install of the package (!86) · Merge requests · Libvirt Packaging Team / libvirt

Hi, this isn't as much of a "take this change it will fix all" and more a "start a discussion on this" PR for now.

I know that we have intentionally stopped to stop the .socket units back in:

commit 100e8aa7756b8282ec00f06837adb82bbbc7e153
Author: Guido Günther <agx@sigxcpu.org>
Date:   Thu Apr 2 09:58:59 2020 +0200

    Don't start or restart socket units on package upgrades
    
    Changes get picked up when the corresponding system unit being restartd.
    This avoids problems when socket and service units of the same service get restarted
    together. See #955483 for details.

Among other things - effectively the change above dropped the sockets out of this list in prerm:

# Automatically added by dh_installsystemd/13.3.1ubuntu1
if [ -d /run/systemd/system ] && [ "$1" = remove ]; then
        deb-systemd-invoke stop 'libvirtd-admin.socket' 'libvirtd-ro.socket' 'libvirtd.socket' 'virtlockd-admin.socket' 'virtlockd.service' 'virtlockd.socket' 'virtlogd-admin.socket' 'virtlogd.service' 'virtlogd.socket' >/dev/null || true
fi

But today I realized that this just had things replaced by other issues. What happened is that I was installing/removing/installing libvirt and I was unable to create guests anymore.

libvirt: can't connect to virtlogd: Failed to connect socket to '/run/libvirt/virtlogd-sock': Connection refused

I've found socket status stating this:

systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit
   has been running, no open socket file descriptor left. The socket unit is
   not functional until restarted.

Note: even --purge + re-install does not get this resolved

The proposed change fixes this issue, but it will bring back the old one. Hence this is mostly meant to start a discussion.

OTOH:

do we consider this a bug in dh_installsystemd to report it there?
- if so what do we do until then?
We could identify what minimal change (less than reverting the old change) we'd need and if that is less of an issue ...

I hope this is enough to get the discussion on this going - have a good evening everyone!

I'll close this PR with a full log on current Debian how that looks like (including the recovery with a restart):

$ lxc exec d10-sid bash
root@d10-sid:~# 
root@d10-sid:~# apt install libvirt-daemon-system
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  augeas-lenses dmeventd dnsmasq-base iptables libaugeas0 libdevmapper-event1.02.1 libfuse2 libjansson4 liblvm2cmd2.03 libnetcf1 libnetfilter-conntrack3 libnfnetlink0 libnftables1
  libnftnl11 libvirt-clients libvirt-daemon libvirt-daemon-driver-lxc libvirt-daemon-driver-qemu libvirt-daemon-driver-vbox libvirt-daemon-driver-xen libvirt-daemon-system-systemd
  libvirt0 lvm2 netcat-openbsd nftables thin-provisioning-tools
Suggested packages:
  augeas-doc firewalld augeas-tools fuse libvirt-daemon-driver-storage-gluster libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-zfs numad auditd nfs-common open-iscsi
  pm-utils radvd systemtap zfsutils
The following NEW packages will be installed:
  augeas-lenses dmeventd dnsmasq-base iptables libaugeas0 libdevmapper-event1.02.1 libfuse2 libjansson4 liblvm2cmd2.03 libnetcf1 libnetfilter-conntrack3 libnfnetlink0 libnftables1
  libnftnl11 libvirt-clients libvirt-daemon libvirt-daemon-driver-lxc libvirt-daemon-driver-qemu libvirt-daemon-driver-vbox libvirt-daemon-driver-xen libvirt-daemon-system
  libvirt-daemon-system-systemd libvirt0 lvm2 netcat-openbsd nftables thin-provisioning-tools
0 upgraded, 27 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.2 MB/11.1 MB of archives.
After this operation, 48.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://deb.debian.org/debian sid/main amd64 libjansson4 amd64 2.13.1-1 [39.0 kB]
...
Selecting previously unselected package libvirt0:amd64.
Preparing to unpack .../16-libvirt0_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt0:amd64 (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-clients.
Preparing to unpack .../17-libvirt-clients_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-clients (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-daemon-driver-qemu.
Preparing to unpack .../18-libvirt-daemon-driver-qemu_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon-driver-qemu (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-daemon.
Preparing to unpack .../19-libvirt-daemon_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-daemon-driver-lxc.
Preparing to unpack .../20-libvirt-daemon-driver-lxc_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon-driver-lxc (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-daemon-driver-vbox.
Preparing to unpack .../21-libvirt-daemon-driver-vbox_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon-driver-vbox (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-daemon-driver-xen.
Preparing to unpack .../22-libvirt-daemon-driver-xen_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon-driver-xen (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-daemon-system-systemd.
Preparing to unpack .../23-libvirt-daemon-system-systemd_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon-system-systemd (6.9.0-1+b2) ...
Selecting previously unselected package libvirt-daemon-system.
Preparing to unpack .../24-libvirt-daemon-system_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon-system (6.9.0-1+b2) ...

...
Setting up libvirt-clients (6.9.0-1+b2) ...
...
Setting up libvirt-daemon (6.9.0-1+b2) ...
Setting up libvirt-daemon-driver-vbox (6.9.0-1+b2) ...
Setting up libvirt-daemon-driver-lxc (6.9.0-1+b2) ...
Setting up liblvm2cmd2.03:amd64 (2.03.10-1) ...
Setting up libvirt-daemon-driver-qemu (6.9.0-1+b2) ...
Setting up libvirt-daemon-driver-xen (6.9.0-1+b2) ...
Setting up dmeventd (2:1.02.173-1) ...
dm-event.service is a disabled or a static unit not running, not starting it.
Setting up libvirt-daemon-system (6.9.0-1+b2) ...
Installing new version of config file /etc/apparmor.d/abstractions/libvirt-lxc ...
Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ...
Installing new version of config file /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper ...
Installing new version of config file /etc/apparmor.d/usr.sbin.libvirtd ...
Installing new version of config file /etc/default/libvirt-guests ...
Installing new version of config file /etc/default/virtlockd ...
Installing new version of config file /etc/default/virtlogd ...
Installing new version of config file /etc/libvirt/libvirtd.conf ...
Installing new version of config file /etc/libvirt/qemu.conf ...
Installing new version of config file /etc/libvirt/qemu/networks/default.xml ...
Installing new version of config file /etc/libvirt/virtlogd.conf ...
Installing new version of config file /etc/sasl2/libvirt.conf ...
Job for libvirtd.service failed because the control process exited with error code.
See "systemctl status libvirtd.service" and "journalctl -xe" for details.
virtlockd.service is a disabled or a static unit, not starting it.
virtlogd.service is a disabled or a static unit, not starting it.
Setting up lvm2 (2.03.10-1) ...
Installing new version of config file /etc/lvm/lvm.conf ...
update-initramfs: deferring update (trigger activated)
Processing triggers for dbus (1.12.20-1) ...
Processing triggers for initramfs-tools (0.139) ...
update-initramfs: Generating /boot/initrd.img-5.10.0-1-amd64
Processing triggers for libc-bin (2.31-9) ...
Processing triggers for man-db (2.9.3-2) ...

root@d10-sid:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket
● virtlogd.service - Virtual machine log manager
     Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● virtlogd-admin.socket
             ● virtlogd.socket
       Docs: man:virtlogd(8)
             https://libvirt.org

● virtlogd.socket - Virtual machine log manager socket
     Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
     Active: active (listening) since Wed 2021-01-20 12:39:02 UTC; 11min ago
   Triggers: ● virtlogd.service
     Listen: /run/libvirt/virtlogd-sock (Stream)
     CGroup: /system.slice/virtlogd.socket

Jan 20 12:39:02 d10-sid systemd[1]: Listening on Virtual machine log manager socket.

● virtlogd-admin.socket - Virtual machine log manager socket
     Loaded: loaded (/lib/systemd/system/virtlogd-admin.socket; enabled; vendor preset: enabled)
     Active: inactive (dead)
   Triggers: ● virtlogd.service
     Listen: /run/libvirt/virtlogd-admin-sock (Stream)
root@d10-sid:~# apt remove libvirt-daemon-system
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  augeas-lenses dmeventd dnsmasq-base iptables libaugeas0 libdevmapper-event1.02.1 libfuse2 libjansson4 liblvm2cmd2.03 libnetcf1 libnetfilter-conntrack3 libnfnetlink0 libnftables1
  libnftnl11 libvirt-clients libvirt-daemon libvirt-daemon-driver-lxc libvirt-daemon-driver-qemu libvirt-daemon-driver-vbox libvirt-daemon-driver-xen libvirt-daemon-system-systemd
  libvirt0 lvm2 netcat-openbsd nftables thin-provisioning-tools
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  libvirt-daemon-system
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 510 kB disk space will be freed.
Do you want to continue? [Y/n] Y

(Reading database ... 120110 files and directories currently installed.)
Removing libvirt-daemon-system (6.9.0-1+b2) ...
Warning: Stopping virtlockd.service, but it can still be activated by:
  virtlockd.socket
Warning: Stopping virtlogd.service, but it can still be activated by:
  virtlogd.socket
root@d10-sid:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket
● virtlogd.service
     Loaded: masked (Reason: Unit virtlogd.service is masked.)
     Active: inactive (dead)

● virtlogd.socket
     Loaded: masked (Reason: Unit virtlogd.socket is masked.)
     Active: active (listening) since Wed 2021-01-20 12:39:02 UTC; 12min ago
     CGroup: /system.slice/virtlogd.socket

Jan 20 12:39:02 d10-sid systemd[1]: Listening on Virtual machine log manager socket.
Jan 20 12:51:32 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:51:32 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:51:33 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:51:33 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:51:33 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:51:33 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>

● virtlogd-admin.socket
     Loaded: masked (Reason: Unit virtlogd-admin.socket is masked.)
     Active: inactive (dead)
root@d10-sid:~# apt install libvirt-daemon-system
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  auditd nfs-common open-iscsi pm-utils radvd systemtap zfsutils
The following NEW packages will be installed:
  libvirt-daemon-system
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 164 kB of archives.
After this operation, 510 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian sid/main amd64 libvirt-daemon-system amd64 6.9.0-1+b2 [164 kB]
Fetched 164 kB in 0s (1059 kB/s)             
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libvirt-daemon-system.
(Reading database ... 120061 files and directories currently installed.)
Preparing to unpack .../libvirt-daemon-system_6.9.0-1+b2_amd64.deb ...
Unpacking libvirt-daemon-system (6.9.0-1+b2) ...
Setting up libvirt-daemon-system (6.9.0-1+b2) ...
Failed to restart libvirtd.service: Unit virtlogd.socket is masked.
virtlockd.service is a disabled or a static unit, not starting it.
virtlogd.service is a disabled or a static unit, not starting it.
root@d10-sid:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket
● virtlogd.service - Virtual machine log manager
     Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● virtlogd.socket
             ● virtlogd-admin.socket
       Docs: man:virtlogd(8)
             https://libvirt.org

● virtlogd.socket - Virtual machine log manager socket
     Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
     Active: active (listening) since Wed 2021-01-20 12:39:02 UTC; 13min ago
   Triggers: ● virtlogd.service
     Listen: /run/libvirt/virtlogd-sock (Stream)
     CGroup: /system.slice/virtlogd.socket

Jan 20 12:52:07 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:07 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:07 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:08 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:08 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:08 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:09 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:09 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:09 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>
Jan 20 12:52:09 d10-sid systemd[1]: virtlogd.socket: Socket unit configuration has changed while unit has been running, no open socket file descriptor left. The socket unit is not functi>

● virtlogd-admin.socket - Virtual machine log manager socket
     Loaded: loaded (/lib/systemd/system/virtlogd-admin.socket; enabled; vendor preset: enabled)
     Active: inactive (dead)
   Triggers: ● virtlogd.service
     Listen: /run/libvirt/virtlogd-admin-sock (Stream)
root@d10-sid:~# systemctl restart virtlogd.socket
root@d10-sid:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket
● virtlogd.service - Virtual machine log manager
     Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● virtlogd.socket
             ● virtlogd-admin.socket
       Docs: man:virtlogd(8)
             https://libvirt.org

● virtlogd.socket - Virtual machine log manager socket
     Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
     Active: active (listening) since Wed 2021-01-20 12:53:09 UTC; 1s ago
   Triggers: ● virtlogd.service
     Listen: /run/libvirt/virtlogd-sock (Stream)
     CGroup: /system.slice/virtlogd.socket

Jan 20 12:53:09 d10-sid systemd[1]: Listening on Virtual machine log manager socket.

● virtlogd-admin.socket - Virtual machine log manager socket
     Loaded: loaded (/lib/systemd/system/virtlogd-admin.socket; enabled; vendor preset: enabled)
     Active: inactive (dead)
   Triggers: ● virtlogd.service
     Listen: /run/libvirt/virtlogd-admin-sock (Stream)

Sockets fail after remove/install of the package

Merge request reports