Repeated ssh startup failures related to SSH host keys on first boot

I've been working with an image-specs-based toolchain for a very long while in my customer's case, but it's heavily tweaked: we run a dedicated process during the first boot, dealing with provisioning, self-testing, enrolling against a PKI, etc.

That's why I haven't filed this longstanding issue earlier… but I believe it's been there for many years.

Here's what happens during a first boot, e.g. with Debian 12 on Pi 4:

systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
sshd[795]: sshd: no hostkeys available -- exiting.
systemd[1]: ssh.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: ssh.service: Failed with result 'exit-code'.
systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
systemd[1]: ssh.service: Scheduled restart job, restart counter is at 1.
systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
sshd[799]: sshd: no hostkeys available -- exiting.
systemd[1]: ssh.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: ssh.service: Failed with result 'exit-code'.
systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
systemd[1]: ssh.service: Scheduled restart job, restart counter is at 2.
systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
sshd[800]: sshd: no hostkeys available -- exiting.
systemd[1]: ssh.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: ssh.service: Failed with result 'exit-code'.
systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
systemd[1]: ssh.service: Scheduled restart job, restart counter is at 3.
systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
sshd[801]: sshd: no hostkeys available -- exiting.
systemd[1]: ssh.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: ssh.service: Failed with result 'exit-code'.
systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
systemd[1]: ssh.service: Scheduled restart job, restart counter is at 4.
systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
sshd[802]: sshd: no hostkeys available -- exiting.
systemd[1]: ssh.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: ssh.service: Failed with result 'exit-code'.
systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
systemd[1]: ssh.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
systemd[1]: ssh.service: Start request repeated too quickly.
systemd[1]: ssh.service: Failed with result 'exit-code'.
systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server.
systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
sshd[869]: Server listening on 0.0.0.0 port 22.
sshd[869]: Server listening on :: port 22.

And indeed that can't work any better, as:

• building the image explicitly removes the generated host keys;
• the rpi-generate-ssh-host-keys.service unit calls dpkg-reconfigure -fnoninteractive openssh-server to generate them, but doesn't try to run before the ssh service unit;
• since the ssh service unit has no conditions on keys, it fails in a loop, until they show up.

Now the first instinct might be to try and order units (e.g. via Before=sshd.service). Except the postinst uses deb-systemd-invoke restart on the rescue and on the regular ssh units, which cannot seem to be avoidable. And at that point the systemctl restart call that gets spawned hangs forever.

It looks to me one quite easy way to solve this problem would be to disable the ssh service unit during the build, and get the rpi-generate-ssh-host-keys.service to enable+start it once keys are available? But then it might have some side effects, e.g. maybe some users/consumers of the Debian-generated Raspberry Pi images are already tweaking ssh-related topics, maybe dropping that unit, which might lead to their being left without ssh at all.

At least for the PiRogue Tool Suite environment, I'll address this issue through “downstream” modifications of the Debian-provided image, but I thought I'd make sure this issue is well known first (or finally!).