TODO.old 14.7 KB
Newer Older
Joerg Jaspert's avatar
Joerg Jaspert committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125

[NB: I use this as a thought record/scribble, not everything on here
     makes sense and/or is actually ever going to get done, so IIWY I
     wouldn't use it as gospel for the future of dak or as a TODO
     list for random hacking.]



  o drop map-unreleased

  o check email only portions of addresses match too, iff the names
  don't, helps with the "James Troup <>"
  vs. "<>" case.

  o ensure .dsc section/prio match .changes section/prio

  o 'dak clean-suites' performance is kind of crap when asked to
     remove a lot of files (e.g. 2k or so).

  o we don't handle the case where an identical orig.tar.gz is
    mentioned in the .changes, but not in unchecked; but should we

  o 'dak ls' could do better sanity checking for -g/-G (e.g. not more
    than one suite, etc.)

  o need to decide on whether we're tying for most errors at once.. if
    so (probably) then make sure code doesn't assume variables exist and
    either way do something about checking error code of check_dsc and
    later functions so we skip later checks if they're bailing.

  o [?, wishlist, distant future] RFC2047-ing should be extended to
    all headers of mails sent out.

  o reject sparc64 binaries in a non '*64*' package.

  o a) we take arguments as parameters that
    we could figure out for ourselves (we're part of the Upload class
    after all), b) we have this 3rd argument which defaults to "any"
    but could in fact be dropped since no one uses it like that.

  o 'dak process-unchecked': doesn't handle bin-only NMUs of stuff
    still in NEW, BYHAND or ACCEPTED (but not the pool) - not a big
    deal, upload can be retried once the source is in the archive, but

  o 'dak security-install' should upload sourceful packages first,
     otherwise with big packages (e.g. X) and esp. when source is !i386,
     half the arches can be uploaded without source, get copied into
     queue/unaccepted and promptly rejected.

  o cnf[Rm::logfile] is misnamed...

<aj> i'd be kinda inclined to go with insisting the .changes file take
   the form ---- BEGIN PGP MESSAGE --- <non -- BEGIN/END lines> --
   BEGIN PGP SIG -- END PGP MESSAGE -- with no lines before or after,
   and rejecting .changes that didn't match that

  o 'dak control-suite' should have a diff mode that accepts diff output!

  o 'dak process-unchecked' crashes if run as a user in -n mode when
    orig.tar.gz is in queue/new...

<elmo_home> [<random>maybe I should reject debian packages with a non-Debian origin or bugs field</>]
<Kamion> [<random>agreed; dunno what origin does but non-Debian bugs fields would be bad]

  o 'dak clean-suites' should make use of select..except select, temporary tables
    etc. rather than looping and calling SQL every time so we can do
    suite removal sanely (see potato-removal document)

  o 'dak rm' will happily include packages in the Cc list that aren't
    being removed...

  o 'dak rm' doesn't remove udebs when removing the source they build from

  o check_dsc_against_db's "delete an entry from files while you're
    not looking" habit is Evil and Bad.

  o 'dak rm' needs to, when not sending bug close mails, promote Cc: to
    To: and send the mail anyways.

  o the lockfile (Archive_Maintenance_In_Progress) should probably be in a conf file

  o 'dak ls' should cross-check the b.source field and if it's not
    null and linked from it != the source given in
    -S/--source-and-binary ignore.

  o 'dak process-unchecked' should do q-d stuff for faster AA [ryan]

  o split the morgue into source and binary so binaries can be purged first!

  o per-architecture priorities for things like different arch'es
    gcc's, silly BSD libftw, palo, etc.

  o use postgres 7.2's built-in stat features to figure out how indices are used etc.

  o 'dak init-archive' shouldn't be using location, it should run down suites instead

  o by default hamstring dak's mail sending so that it won't send
    anything until someone edits a script; it's been used far too
    much to send spam atm :(

  o $ftpdir/indices isn't created by 'dak init-dir' because it's not in dak.conf

  o sanity check depends/recommends/suggests too?  in fact for any
    empty field?

[minor] 'dak process-accepted's copychanges, copydotdak handling
        sucks, the per-suite thing is static for all packages, so work out
        in advance dummy.

[dak ls] # filenames ?
[dak ls] # maintainer, component, install date (source only?), fingerprint?

  o UrgencyLog stuff should minimize it's bombing out(?)
  o Log stuff should open the log file

  o 'dak import-users-from-passwd' should sync debadmin and ftpmaster (?)

  o <drow> Can't read file.:
Ansgar's avatar
Ansgar committed
Joerg Jaspert's avatar
Joerg Jaspert committed
127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162
  You assume that the filenames are relative to accepted/, might want
  to doc or fix that.

  o SecurityQueueBuild doesn't handle the case of foo_3.3woody1 with a
   new .orig.tar.gz followed by a foo_3.3potato1 with the same
   .orig.tar.gz; 'dak process-unchecked' sees it and copes, but the AA
   code doesn't and can't really easily know so the potato AA dir is
   left with no .orig.tar.gz copy.  doh.

  o orig.tar.gz in accepted not handled properly (?)

  o 'dak security-install' doesn't include .orig.tar.gz but it should

  o permissions (paranoia, group write, etc.) configurability and overhaul

  o database.get_location_id should handle the lack of archive_id properly

  o the whole versioncmp thing should be documented

  o aj's bin nmu changes

  o 'dak process-new':
    * priority >> optional
    * arch != {any,all}
    * build-depends wrong (via 'dak compare-suites')
    * suid
    * conflicts
    * notification/stats to admin daily
    o trap 'dak examine-package' exiting
    o distinguish binary only versus others (neuro)

  o cache changes parsed from ordering (careful tho: would be caching
    .changes from world writable incoming, not holding)

  o dak doesn't recognise binonlyNMUs correctly in terms of telling
    who their source is; source-must-exist does, but the info is not
    propagated down.
Joerg Jaspert's avatar
Joerg Jaspert committed
164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183

  o maintainers file needs overrides

    [ change override.maintainer to override.maintainer-from +
      override.maintainer-to and have them reference the maintainers
      table.  Then fix 'dak make-maintainers' to use them and write some scripting
      to handle the Santiago situation. ]

  o Validate Depends (et al.) [it should match  \(\s*(<<|<|<=|=|>=|>|>>)\s*<VERSIONREGEXP>\)]

  o Clean up DONE; archive to tar file every 2 weeks, update tar tvzf INDEX file.

  o testing-updates suite: if binary-only and version << version in
    unstable and source-ver ~= source-ver in testing; then map
    unstable -> testing-updates ?

  o hooks or configurability for debian specific checks (e.g. check_urgency, auto-building support)

  o morgue needs auto-cleaning (?)

  o dak stats: two modes, all included, separate
Joerg Jaspert's avatar
Joerg Jaspert committed
185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247
  o dak stats: add non-US
  o dak stats: add ability to control components, architectures, archives, suites
  o dak stats: add key to expand header



 o What to do with multi-suite uploads?  Presumably hold in unapproved
   and warn?  Or what?  Can't accept just for unstable or reject just
   from stable.

 o Whenever we check for anything in accepted we also need to check in

 o non-sourceful uploads should go straight through if they have
   source in accepted or the archive.

 o security uploads on auric should be pre-approved.


Less Urgent

  o change utils.copy to try rename() first

  o [hard, long term] unchecked -> accepted should go into the db, not
    a suite, but similar.  this would allow dak to get even faster,
    make 'dak ls' more useful, decomplexify specialacceptedautobuild
    and generally be more sane.  may even be helpful to have e.g. new
    in the DB, so that we avoid corner cases like the .orig.tar.gz
    disappearing 'cos the package has been entirely removed but was
    still on stayofexecution when it entered new.

  o Logging [mostly done] (todo: 'dak clean-suites' (hard), .. ?)

  o 'dak process-unchecked': the tar extractor class doesn't need to be redone for each package

  o reverse of source-must-exist; i.e. binary-for-source-must-not-exist
  o REJECT reminders in 'dak clean-queues'.
  o 'dak examine-package' should check for conflicts and warn about them visavis priority [rmurray]
  o store a list of removed/files versions; also compare against them.
    [but be careful about scalability]

  o dak examine-package: print_copyright should be a lot more intelligent
     @ handle copyright.gz
     @ handle copyright.ja and copyright
     @ handle (detect at least) symlinks to another package's doc directory
     @ handle and/or fall back on source files (?)

  o To incorporate from utils:
     @ unreject

  o auto-purge out-of-date stuff from non-free/contrib so that testing and stuff works
  o doogie's binary -> source index
  o jt's web stuff, matt's changelog stuff (overlap)

  o [Hard] Need to merge non-non-US and non-US DBs.

  o Do a checkpc(1)-a-like which sanitizes a config files.
  o fix parse_changes()/build_file_list() to sanity check filenames
  o safety check and/or rename debs so they match what they should be
Joerg Jaspert's avatar
Joerg Jaspert committed
249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284

  o Improve 'dak compare-suites'.
  o Need to optimize all the queries by using EXAMINE and building some INDEXs.
    [postgresql 7.2 will help here]
  o Need to enclose all the setting SQL stuff in transactions (mostly done).
  o Need to finish 'dak init-db' (a way to sync dak.conf and the DB)
  o Need the ability to rebuild all other tables from dists _or_ pools (in the event of disaster) (?)
  o Make the --help and --version options do stuff for all scripts

  o 'dak make-maintainers' can't handle whitespace-only lines (for the moment, this is feature)

  o generic way of saying isabinary and isadsc. (?)

  o s/distribution/suite/g

  o --help doesn't work without /etc/dak/dak.conf (or similar) at
    least existing.

  o rename 'dak compare-suites' (clashes with existing 'dak compare-suites')...

 * Harder:

    o interrupting of stracing 'dak process-unchecked' causes exceptions errors from apt_inst calls
    o dependency checking (esp. stable) (partially done)
    o need to do proper rfc822 escaping of from lines (as opposed to s/\.//g)
    o Revisit linking of binary->source in install() in dak.
    o binary-all should be done on a per-source, per-architecture package
      basis to avoid, e.g. the perl-modules problem.
    o a source-missing-diff check: if the version has a - in it, and it
      is sourceful, it needs orig and diff, e.g. if someone uploads
      esound_0.2.22-6, and it is sourceful, and there is no diff ->
      REJECT (version has a dash, therefore not debian native.)
    o check linking of .tar.gz's to .dsc's.. see proftpd 1.2.1 as an example
    o archive needs md5sum'ed regularly, but takes too long to do all
      in one go; make progressive or weekly.
    o test sig checking stuff in test/ (stupid thing is not modularized due to global abuse)
    o when encountering suspicious things (e.g. file tainting) do something more drastic
Joerg Jaspert's avatar
Joerg Jaspert committed
286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307

 * Easy:

    o suite mapping and component mapping are parsed per changes file,
      they should probably be stored in a dictionary created at startup.
    o don't stat/md5sum files you have entries for in the DB, moron
      boy (Dak.check_source_blah_blah)
    o promote changes["changes"] to mandatory in
      after a month or so (or all .dak files contain in the queue
      contain it).
    o 'dak rm' should behave better with -a and without -b; see
      gcc-defaults removal for an example.
    o Reject on misconfigured kernel-package uploads
    o utils.extract_component_from_section: main/utils -> main/utils, main rather than utils, main
    o Fix 'dak process-unchecked' to warn if run when not in incoming or p-u
    o dak should validate multi-suite uploads; only possible valid one
      is "stable unstable"
    o cron.daily* should change umask (aj sucks)
    o byhand stuff should send notification
    o 'dak rm' should update the stable changelog [joey]
    o update tagdb.dia

 * Bizzarre/uncertain:
Joerg Jaspert's avatar
Joerg Jaspert committed
309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380

    o drop rather dubious currval stuff (?)
    o rationalize os.path.join() usage
    o logging: hostname + pid ?
    o ANAIS should be done in dak (?)
    o Add an 'add' ability to 'dak rm' (? separate prog maybe)
    o Handle the case of 1:1.1 which would overwrite 1.1 (?)
    o maybe drop -r/--regex in 'dak ls', make it the default and
      implement -e/--exact (a la joey's "elmo")
    o dsc files are not checked for existence/perms (only an issue if
      they're in the .dsc, but not the .changes.. possible?)

 * Cleanups & misc:

    o db_access' get_files needs to use exceptions not this None, > 0, < 0 return val BS (?)
    o The untouchable flag doesn't stop new packages being added to ``untouchable'' suites



  o Fix stuff to look in sensible places for libs and config file in debian package (?)


                          --help      manpage
check-archive	          X
check-overrides           X		 X
clean-queues  	          X
clean-suites  	          X		 X
compare-suites	          X
control-overrides         X		 X
control-suite             X		 X
cruft-report    	  X
decode-dot-dak            X
examine-package           X
generate-releases	  X
import-archive            X
import-users-from-passwd  X		 X
init-db	                  X
init-dirs    	          X
ls                        X		 X
make-maintainers          X		 X
make-overrides            X
make-suite-file-list      X
process-accepted          X		 X
process-new               X		 X
process-unchecked         X
queue-report              X
rm	                  X		 X
security-install          X
stats 	                  X
symlink-dists             X


Random useful-at-some-point SQL

UPDATE files SET last_used = '1980-01-01'
  FROM binaries WHERE binaries.architecture = <x>
                  AND binaries.file =;

DELETE FROM bin_associations
                WHERE architecture = <x>
                  AND id = bin_associations.bin);