While most packages are maintained in git, uploading them to the archive requires an extra dput still. This makes git and the sources in the archive disconnected.
Actual situation
Having git and the archive disconnected has several problems:
The sources in the archive don't necessarily match the git tag for that version.
The developer has to perform extra steps
Expected situation
It would be great if pushing a signed git tag would be sufficient to trigger a build of the binary packages. The "source of truth" would be the git archive. E.g. a
git tag -s <tag> && git push origin <tag>
should be sufficient to release a package into Debian. This would make sure
git clone gives you exactly what's in the archive
There's no way to accidentally have sources that don't match git hence e.g. don't reproduce certain bugs
we'd have all the gitlab-ci etc as qa (we could even only let the tag through if it passes)
Additional information
I left the above pretty vague on purpose since the fact if we want to generate a dsc + tarball and still maintain the "archive" or if all sources should live in a blessed git instance should be discussed. Basically the only requirement would
be that what the developer tags is 1:1 what is used to build the package.
Edited
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Child items
0
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Linked items
0
Link issues together to show that they're related.
Learn more.
Activity
Sort or filter
Newest first
Oldest first
Show all activity
Show comments only
Show history only
Guido Güntherchanged title from Allow to build upload packages via git tag to Allow to upload packages by creating and pushing a signed git tag
changed title from Allow to build upload packages via git tag to Allow to upload packages by creating and pushing a signed git tag
@yadd The way the proposal is written doesn't assume that the current upload structure remains the only option forever. For example pristine-tar can regenerate orig.tar.gz and store .asc, so there could be a way to prepare those as pre-requisites to the signed tag. Similarly, patches could be handled like gbp pq does as a branch on top of the tag, or again the tag could point to the patches-applied, native git log version. That said, a backwards compatible approach is probably more likely to succeed, so I think it will be useful to concentrate on the concrete practical issues raised by e.g. tag2upload rather than blue sky design.
tag2upload was an attempt to do some of this (admittedly still with a source package in the middle), but ran into some challenges around end to end validation at the ftp-master end.
To be clear: I think it would be great if we could pull this off and move away from traditional source packages as the way for developers to upload - but this is at least where some of the challenges are.
tag2upload does not have a source package in the middle, really. It has a source package in parallel, which is obviously necessary for old software which only understands the archive full of .dscs, and humans who prefer to work with .dscs.
But users who use the new git-based methods do not need source packages. In particular, if user A uploads a package with tag2upload, and then user B does "dgit fetch", user B's copy of dgit fetches user A's git objects.
Sadly right now for packages in non-native source format, user B still usually ends up with a copy of the .orig.tar.gz but it would be possible to get rid of that in more cases. A full discussion of this issue doesn't quite fit in this margin...
Sorry, I could have expressed that better - I realize that tag2upload itself is alive. It was the effort to implement a no-setup tag-and-forget-style system using tag2upload that I was thinking of - as far as I can tell that has been stalled (deadlocked?) since those discussions with ftp-master back in 2019. Is that incorrect, do you have an idea for a path forward?
AFAIU tag2upload currently still involves creating a traditional source package to get something into the archive, even if some of that is mostly hidden from the user. My reading of the discussion with ftp-masters is that creating a proxy dsc signed by a proxy service that provides a reference to the original Git testament was the problem. It seems like a way around that would be to e.g. treat a Git repository/tree snapshot/etc as a first class object in Debian (maybe like the git v3 source format, though I think there were other issues there?).
It was the effort to implement a no-setup tag-and-forget-style system using tag2upload that I was thinking of - as far as I can tell that has been stalled (deadlocked?) since those discussions with ftp-master back in 2019. Is that incorrect, do you have an idea for a path forward?
I think the path forward has to be political. You could talk to the DPL, but the DPL's constitutional role doesn't really allow them to overrule and DPLs tend to take a very process-oriented approach. In the past, ftpmaster intransigence has been successfuly overcome with GRs on multiple occasions. I definitely don't have the spoons for such a GR and I am a polarising figure so I ought not to be heavily involved in it anyway. One could also talk to the TC I guess. But I think the best likely route to sorting this out would be for us to persuade the DPL, and the DPL to negotiate firmly with ftpmaster, backed up with the possibility of a DPL-sponsored GR.
If someone wants to drive this process I'm happy to provide references, factual background, technical detail, and so on.
AFAIU tag2upload currently still involves creating a traditional source package to get something into the archive, even if some of that is mostly hidden from the user.
Yes. I think any reasonable solution has to involve producing a source package, since we need to keep supporting existing software that expects to find source code in the archive.
My reading of the discussion with ftp-masters is that creating a proxy dsc signed by a proxy service that provides a reference to the original Git testament was the problem. It seems like a way around that would be to e.g. treat a Git repository/tree snapshot/etc as a first class object in Debian (maybe like the git v3 source format, though I think there were other issues there?).
ftpmaster blocked the git v3 source format too; there were multiple objections. I think the git v3 source format is pretty poor, really. There are good reasons not to want to go in that direction. I could expand on that.
There are no good technical reasons for the objections to the proxy-signed .dsc. (This is not just my opinion; it's widely shared by people who ought to know what they're talking about.) Frankly, I think the objections from some members of ftpmaster arise because: (i) tag2upload treats the archive as a compatibility output thing, making normal git servers and signed git objects more primary; that's not particularly nice if the archive is your baby; (ii) some of the principal objectors dislike me personally.
The archive would still be able to verify the original maintainer signature, so the proxy service wouldn't be able to randomly make stuff up out of whole cloth - the worst it could do is mangle the source package during git to dsc conversion.
But, actually, right now the fact is that most of Debian already treats the dsc as an output artefact, rather than the preferred form for modification. git is de-facto primary for most packages. git to dsc conversion occurs on maintainer machines using uncontrolled and non-reproducible tooling and workflows! As ever, blocking a formal official way to do something (in this case, automatically convert git to dsc) just means that people adopt unofficial local workarounds, which become eventually so usual that everyone thinks it's normal. So tag2upload offers a massive improvement in traceability.
I think that tag2upload ought to be our implementation of this. It exists, is tested, and can already handle a very wide variety of maintainer git workflows. And it truly is just a tag and a push, with a short and simple wrapper script of git-tag(1) and git-push(1), git-debpush(1). As it uses dgit on the backend -- maintainers need not know anything about dgit -- it also means that non-Debian contributors can use dgit clone to obtain the maintainer's entire git history but in a format which is easier for non-Debian contributors to hack on. The technical details have already been thought through and sorted out.
Certain members of the ftpteam object to tag2upload. While their objections are worded in terms of tag2upload in particular, as that's what is on the table, the underlying objections are to any system that satisfies the description at the top of this GitLab issue. Those ftpteam members are not happy with anything that amounts to just a git tag and a git push on the maintainer's end. If we are going to make any progress, we should recognise that the disagreement is about that, not about tag2upload in particular.
Obviously, I agree with everything @spwhitton said. Just to be 100% clear:
tag2upload is a working solution. The code exists right now. It is in src:dgit, where it is tested as part of src:dgit's DEP-8 test suite. So it is known to work. It could be deployed very shortly after we clear the political objections. Indeed I demoed it in a Lighting Talk at the Curitiba Debconf - I did a live upload on stage. (Obviously since I was prevented from deploying the server side, I had to have a stunt copy of the server-side part runnng with my personal DD key on my laptop.)
The objectors characterise their objections as "security concerns". However: as part of my approach to the design and implementation, I got an independent review from a security expert in the Debian keyring-maint team. (Also I have PhD in computer security and decades of real-world computer security experience in my day jobs.)
Stepping back:
tag2upload is the next step in my transition plan to move Debian away from tarballs. The Debian source package format which I did v1 of in 199mumble, and which has been extended since, has IMO been obsoleted by modern version control.
Obviously change in Debian is done very slowly, and I personally have been on the uncomfortable end of many transitions which were done very vigorously and without consent. So I have been trying to make sure that my designs and approaches should keep everyone happy. In particular, my plan involves an extended, possibly indefinite, period of dual running. The goal is that people who don't want to change don't have to. But conversely, no-one should need to mess about with tarballs to contribute to Debian.
Sam Hartman, then DPL, organised a "git packaging requirements gathering session" for Curitiba. To be honest, I felt this was rather late, since I have been gathering requirements and indeed meeting them since dgit was first conceived in Vaumarcus. But of course I was happy to help. As I expected, almost all the requirements anyone came up with in that very-well-attended session fit well into my design. Many of the requirements were already implemented - and, yes, tag2upload is the next big missing piece.