Fix gitaly getting rejected because golang-1.11 is removed from unstable.

Even with golang-1.13 in buster-backports, it still rejects

gitaly_1.78.0+dfsg-1~~bpo10+1_amd64.deb: Built-Using refers to non-existing source package golang-1.13 (= 1.13.5-1~~bpo10+1)

$ rmadison golang-1.13
golang-1.13 | 1.13.5-1~bpo10+1 | buster-backports       | source, all
golang-1.13 | 1.13.5-1~bpo10+1 | buster-backports-debug | source
golang-1.13 | 1.13.5-1         | testing                | source, all
golang-1.13 | 1.13.5-1         | unstable               | source, all
golang-1.13 | 1.13.5-1         | unstable-debug         | source

@akh-guest

assigned to @akhilvarkey

I'm trying to debug this.

What I understand is that

fasttrack.debian.net is one instance of dak and backports.debian.org is another instance of dak
backports.debian.org has golang-1.13
fasttrack.debian.net does not have golang-1.13

Is the expected behaviour that dak uses the apt configuration to see if golang is available in any of the other sources?

@akshay I think dak is designed with only self contained distros in mind. This may be a feature we need to add, like specify a base distro. So we should be able to say our fasttrack repo needs backports suite from official archive to build and may declare dependencies from there too.

Using rmadison command or via its api may be better than depending on apt sources I think as rmadison is more authoritative and apt sources may be out of sync with official repos.

Has dak import-repository script been considered? It looks like the current solution to this.

$ dak import-repository -h

dak import-repository
  --keyring=/usr/share/keyring/debian-archive-keyring.gpg
  [--key=${fingerprint}]
  [--architectures=a,b,c (default: architectures in origin suite)]
  [--components=main,contrib (default: components in origin suite)]
  [--target-suite=${suite} (default: origin suite name)]
  [--add-overrides]
  [--max-packages=${n} (import at maximum ${n} packages, default: no limit)]
  http://httpredir.debian.org/debian unstable

Things to think about:
 - Import Built-Using sources
   - all / only referenced
 - Remove old packages:
   - by-source: remove source X_v, if no X exists upstream
   - by-version: remove source X_v, if no X_v exists upstream
   (X denotes package name, v version, X_v package at a specific version)
 - Import all or only newest?
 - Expire binary packages?

But it will duplicate the packages, right? All packages from official repos would be added to our repo, right?

There is a --target-suite flag. What if a different (new) suite is used (that is not publicized)? Then users won't see packages from this, right?

Is the concern that of space?

It might be possible to modify this script to only import a few specified packages too.

Well, importing looks like a work around to me. It should ideally query that dak instance to keep the info updated.

But if we can't make querying work, we may have to do the import.

If we have to import, we will probably have to import golang and all golang modules required to build packages going to fasttrack.

The Built-Using field doesn't have a way to refer to a particular archive, does it? In other words, it doesn't look like apt/dak is built to be federated in this manner.

Yes, it is not designed this way. We can try to add this feature, but if it is too hard, we can fall back to the other method of importing packages. We will have to import entire golang packages from backports I think or import it on demand when there is a new package being processed, so try to import that package before we reject.

Along with #17 (closed) this could work with two suites instead of making a hidden suite. I think if we can just limit to importing only the list of packages instead of duplicating the actual packages that would be good. But if that is not possible, lets see this as a feature and provide a mirror for official golang backports from fasttrack :) Disk space is still a concern, but could be added if there is no other option.

There has been a discussion on this in the dak list already - "Setup for built-using" - which says this:

Is this something that could/should be overridden? If so how?

No; dak is very agressive in enforcing that all sources are present. Maybe more than people sometimes want.

In particular it should not rely on references to external archives which might no longer provide source or binary packages that your distribution's overlay still references.

Looking at the source code of dak, I agree too. It will be a mess if we try to insert packages with missing Built-Using.

The mail also includes a reference to how security archive does it:

They just have to be somewhere in the archive. You can import them into a "built-using" suite in the target suite's archive; dak will also copy Built-Using sources from other archives (with some exceptions). (That is what happens on Debian's security archive; see the "Built- Using" section in docs/debian-specific.rst)

Here's the relevant bit from docs/debian-specific.rst

Source packages referred to via Built-Using need to be included in the security archive:

Obtain & verify .dsc

dak import built-using updates/<component> <.dsc...>

If the .dsc is signed by an old key no longer in the keyring, use --ignore-signature. Make extra sure the .dsc is correct.

I think from a security and stability perspective it is better to follow what the security archive does here.

Okay, let's import it then. How do we limit it to importing only the missing packages? Or do we have to import the whole archive?

BTW in this case our repo is an overlay on debian and not a separate independant distro.

I would question the choice of using dak then. https://wiki.debian.org/DebianRepository/Setup says this:

The full package archive similar to the official archive can be created using:

dak

mini-dak

The Private Package Archive (PPA) can be created on a web server with a shell account using:

reprepro

mini-dinstall

Those other options are for simple repos maintained by small number of people and updating is error prone if more than one person uploads. Only dak has gpg based authentication and can scale to allowing large number of people to upload.

I used reprepro earlier and switched to dak for allowing more people to upload. I still use reprepro for golang packages, then that is a single point of failure on me.

Fix gitaly getting rejected because golang-1.11 is removed from unstable.

Child items ...

Activity