Skip to content

Mass DLA import

Antoine Beaupré requested to merge anarcat/webwml:mass-dla-import2 into master

This is a second attempt at doing a massive catchup for the missing DLAs on the website, as documented in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859122.

This is a reroll of !47 (closed), rebased on master, with the pgp signature fixes (!51 (closed)) and up to date until DLA-1659. This was done with:

cd english/security/
rsync -vPa master.debian.org:/home/debian/lists/debian-lts-announce .
cd debian-lts-announce
xzcat *.xz > ../giant.mbox
cat debian-lts-announce/debian-lts-announce.201710
debian-lts-announce/debian-lts-announce.201902 >> ../giant.mbox
cd ..
mbox2maildir giant.mbox debian-lts-announce.d
for mail in debian-lts-announce.d/new/*; do
  ~/src/parse-dla.pl $mail;
done
# import mailing list failure from local archive
for dla in  DLA-1543-1 DLA-1541-1 DLA-1540-1 DLA-1539-1 DLA-1538-1 DLA-1537-1 DLA-1536-1 DLA-1535-1 DLA-1534-1 DLA-1533-1 DLA-1532-1 DLA-1529-1; do
  ~/src/parse-dla.pl $(notmuch search --output=files subject:$dla )
done

Notice how the october 2017 file is not compressed and needs special handling, and so does the most recent one. The parse-dla.pl script was intentionally modified with patches that come in a different branch (!51 (closed)). Also note that the import corrects the mailing list failures from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913426 using my local archive.

I have used the audit script from !52 (closed) to examine which ones are missing, here's the result, at the time of writing:

checking DLA-1654-1 (2019)....data or .wml file missing.
checking DLA-1130-1 (2017)....data or .wml file missing.
checking DLA-772-1 (2016)....data or .wml file missing.
checking DLA-719-1 (2016)....data or .wml file missing.
checking DLA-706-1 (2016)....data or .wml file missing.
checking DLA-659-1 (2016)....data or .wml file missing.
checking DLA-607-1 (2016)....data or .wml file missing.
checking DLA-580-1 (2016)....data or .wml file missing.
checking DLA-567-1 (2016)....data or .wml file missing.
checking DLA-377-1 (2015)....data or .wml file missing.
checking DLA-267-1 (2015)....data or .wml file missing.
checking DLA-115-2 (2015)....data or .wml file missing.
checking DLA-145-2 (2015)....data or .wml file missing.
checking DLA-0015-1 (2014)....data or .wml file missing.
checking DLA-0014-1 (2014)....data or .wml file missing.
checking DLA-0013-1 (2014)....data or .wml file missing.
checking DLA-0012-1 (2014)....data or .wml file missing.
checking DLA-0011-1 (2014)....data or .wml file missing.
checking DLA-0010-1 (2014)....data or .wml file missing.
checking DLA-0009-1 (2014)....data or .wml file missing.
checking DLA-0008-1 (2014)....data or .wml file missing.
checking DLA-0007-1 (2014)....data or .wml file missing.
checking DLA-0006-1 (2014)....data or .wml file missing.
checking DLA-0005-1 (2014)....data or .wml file missing.
checking DLA-0004-1 (2014)....data or .wml file missing.
checking DLA-0003-1 (2014)....data or .wml file missing.
checking DLA-0002-1 (2014)....data or .wml file missing.
checking DLA-0001-1 (2014)....data or .wml file missing.

Of those:

  • DLA-1654-1 was never published
  • DLA-772-1 is a false positive: it is marked as 2016 in data/DLA/list but was published in 2017
  • the ones before 2018 were already missing, I couldn't find a copy of them in my personal archive, but they might be available in the mailing lists. this can be fixed later.

Many of the manual fixes performed in !47 (closed) were included here as well. Here are how many DLAs missing, per branch:

  • master: 1341
  • mass-dla-import (!47 (closed)): 43
  • mass-dla-import (this MR): 28
Edited by Antoine Beaupré

Merge request reports