Skip to content

Migration process

This is about steps we'll have to take immediately before, during, and immediately after, the final migration process. For example, anything that we decide to do manually (as opposed to automation via redlab upload) must be listed here.

Incoming spool

This is a space to dump brainstorming ideas.

We should triage these items and translate them to concrete, ordered steps.

Ordered steps

first upload (13th May)

  • clean git repo gitlab-migration-private completely [aka delete all files inside this git repo] (do not commit this) - this makes sure, that we have have clean state. This needs to be done in the checkout, where redlab download creates the cache.
  • redlab download
  • tag the git repo gitlab-migrate-private
  • push the git repo gitlab-migrate-private
  • GitLab: create the tails group with email notifications disabled
  • GitLab: create the tails/tails project so we can push tails.git
  • Import data into GitLab (with e.g. 3 threads in parallel (different console) redlab upload --resume 1.resume --first_run --parallel 1/3, redlab upload --resume 2.resume --first_run --parallel 2/3, redlab upload --resume 3.resume --first_run --parallel 3/3)
  • Import tails git to Gitlab, so code links work.
  • Do a 2nd pass of redmine upload (create stub issues, close issues, make links between tickets work) (with e.g. 3 threads in parallel (different console) redlab upload --resume 2_1.resume --second_run --parallel 1/3, redlab upload --resume 2_2.resume --second_run --parallel 2/3, redlab upload --resume 2_3.resume --second_run --parallel 3/3)

migration day (19th May)

  • make the public Redmine archive actually public
  • Freeze Redmine: allow us to redlab download data, forbid changes
  • commit changes of gitlab-migration-private especially the gitlab folder
  • switch to sorted branch in gitlab-migration-private
  • delete the redmine folder in git repo gitlab-migration-private completely (do not commit this), tht makes sure, that redlab download will download everything and not use the cache.
  • redlab download
  • tag the git repo gitlab-migrate-private
  • push the git repo gitlab-migrate-private
  • Import/Update data into GitLab (with e.g. 3 threads in parallel (different console)) use --first_run --changed-issues tag1..tag2 to only act on the changed issues.
  • prepare private Redmine archive: ./redlab archive --config config.py --private --url-prefix /code/ --output private --progress
  • prepare public Redmine archive: ./redlab archive --config config.py --url-prefix /code/ --output public --progress
  • upload Redmine public & private archives
  • Wait for immerda to fix (or tell us how to) connect to git+ssh
  • Update the tails.git repository, so 2nd pass works fine.
  • Do a 2nd pass of redmine upload to fix links to commits and issue opening status (redlab upload --second_run)
  • Allow the root account to create groups (https://gitlab.tails.boum.org/admin/users/root/edit)
  • Polish the imported data (during this step, the following items can happen in no particular order):
    • update gitlabracadabra config with redlab gitlabracadabra
    • add the UX:debt label on every issue that's related to #14544. This needs to be done in the same setup that did redlab upload, see #71 (comment 153833)
    • Close Sponsor.* milestones manually
    • Create issue boards for core work #62 (closed)
    • create a role-mirror-to-GitLab user with e-mail: role-mirror-to-GitLab@mail.tails.boum.org
    • create a role-update-website user with e-mail: role-update-website@mail.tails.boum.org
    • create stars for watched issues redlab update --add_star_for_watched_issues #64 (closed)
  • Evaluate the imported issues vs. the specs (below are the items listed as MUST in the specs).
    • Does not require SSH access to the GitLab machine.
    • Preserve meaning of #NNNN id unambiguously.
    • Redirect links to GitLab issues.
    • Preserve issue private/public status. Private issue example, not available in public project. Public issue example, available in a different project.
    • Preserve parent/subtask relationship. Zen: Example issues (1, 2) look good as subtasks are listed and marked items correspond to closed issues. I am not sure if the "enforcing" part of the spec should be taken strictly here, though. hefee: well the first two test instances of GitLab never feedback this as a problem -> I would say, that the implementation is fine. Maybe add this to the Bluprint as an info.
    • Preserve the "Blocks" and "Related to" information. Example issues (1, 2) look OK as they list "Blocked by" and "Blocks" relations.
    • Preserve issue status. open/close status + labels "To Do", "Doing", "Needs Validation", "Duplicate", "Rejected"
    • Preserve "Target version" → Milestone. 96 milestones
    • Preserve the "Feature branch" information. Example issue looks OK.
    • Preserve Category, Affected Tool, Priority, Type of work. hefee: we have labels with "C:" - for Affected tools and Category, "P:" - Priority and "T:" for Type of work
    • Preserve any true "Starter" boolean value. Example search looks good.
    • Preserve "Deliverable for" values for open issues. Example issue. Zen: I see a D:SponsorS_Internal label, but I don't see the "Deliverable for" info on the original issue. hefee: jq . redmine/issues/10601.json {"id": 19, "name": "Deliverable for", "value": "280"} -> 280="SponsorS_Internal" => so that is correct
    • Preserve attachments. Zen: attachments in example issue are pointing to Redmine URL. hefee: that was a discussed and used as best solution.
    • Preserve embedding of attached images. Example issue seems OK, although the image is not on archive (but that's #91 (closed), tracked in an item further below).
    • Convert Textile issue description and comments to Markdown. Zen: Example issue looks good, but I'm not sure what .h1 means. Redmine's Textile and Markdown docs do mention h1., was that maybe a mistake? hefee: properly that was a mistake, yes.
    • Convert public notes to GitLab comments. Example issue looks good.
    • Private comments (notes). Example issue looks good.
    • Preserve issue assignee. Example search looks good.
    • Existing issues assigned to a group must remain trackable, somehow. Sysadmins is an own user that can be used to search for assigned tickets eample search
    • Preserve the value of the Blueprint custom field. Example issue looks good.
    • Preserve metadata changes.
    • Preserve attribution (authorship) of actions.
    • Existing user accounts.
  • Decide whether any identified problem is worth fixing (which might require another pass of redlab upload)
    • Fix image links: ./redlab update --config config.py -v --fix-images-links (#95 (closed))
    • Fix missing attachments in archive (#91 (closed)) (not blocking the migration) (fixed in git, only requires regenerating the archives).
  • Checkpoint: After this point we can enable our permission model: non-root users will not be be able to create issues with arbitrary numbers or do other special things from now on.
  • Run gitlabracadabra to address #33 (closed), #36 (closed), #38 (closed) and #60 (closed)
  • Import Git repositories other than tails.git (after gitlabracadabra has set up the corresponding projects) -- #92 (closed), #93 (closed), #94 (closed).
  • Run gitlabracadabra again (there's some steps it cannot do until Git repositories have been pushed)
  • Checkpoint: it's now OK to allow users to login and push to GitLab
  • Set users' correct email addresses: redlab update --config config.py --user-emails
  • Ask immerda to enable mail sending again (clean mail queue, before asking).
  • Forbid pushing to the now-obsolete location of migrated Git repositories
    • Remove ikiwiki and Weblate permissions for tails.git (merge puppet-git.lizard:gitolite-admin@61_remove-ikiwiki-and-weblate-permissions into master).
    • Remove other obsolete push permissions (#77 (closed))
  • Got ACK from immerda admins, that we can send mails, and be able to sent out password reminders
  • Test user password retrieval and login procedures (unfortunately users have to confirm the user account after passwort reset, we do not find a way to mark the user account as confirmed)
  • Verify that the GitLab container can send email
  • Sync one last time migrated Git repos from their previous location to GitLab.
  • Checkpoint: after that we cannot abort the migration anymore
  • Empty users' To-Do list: redlab update --empty-todo --verbose (#82 (closed))
  • Send an e-mail announcing GitLab and include SSH fingerprints.
    • Inform users that they need to do a second round to confirm their user account
  • Perform or schedule sysadmin steps (they don't block rest of the migration):
    • Configure GitLab remote mirrors in Gitolite (#50 (closed), #57 (closed))
    • Make Ikiwiki remote be GitLab: 61_use-gitlab-as-remote-for-ikiwiki (#61 (closed))
    • Make Weblate gatekeeper pull/push to GitLab: 55-56_weblate-pull-from-gitlab-push-to-gatekeeper (#55 (closed), #56 (closed))
    • Generate Jenkins jobs from GitLab repo: 52_generate-jenkins-jobs-from-gitlab (#52 (closed))
    • Configure a GitLab user for the stale issues reminder (#75 (closed))
    • Query "Needs Validation" label from GitLab issues: 53_query-gitlab-needs-validation (#53 (closed))
    • Allow Ikiwiki to push to GitLab's tails.git (#73 (closed))
    • Allow Weblate Gatekeeper to push to GitLab's tails.git (#74 (closed))
    • Allow infrastructural repositories to be pushed to GitLab (#76 (closed))
  • Fix pushing through SSH (#17716)
  • Enable email notifications (emails_disabled in the Gitlabracadabra configuration)
  • Merge and publish updated doc (tails-team/tails!45)
  • Add rewrite rules for attachments to the static web redirector (#48 (closed))
  • Deploy the static web redirector on www.lizard
    • Include the tails::redmine::redirector class on www.lizard
    • Point DNS for redmine.tails.boum.org to lizard
    • Deal with initial Let's Encrypt vs. Puppet chicken'n'egg problem
  • Enable sign-up (https://gitlab.tails.boum.org/admin/application_settings/general)
  • Checkpoint: GitLab is officially in production
  • Announce to Tails folks that GitLab is now in production: point to the doc created via #69 (closed)
  • Merge and push less critical updated doc & code:
    • GitLab branch of summit.git
    • GitLab branch of accounting.git
    • GitLab branch of fundraising.git
    • GitLab branch of mirrors.git
    • GitLab branch of internal.git
    • GitLab branch of sysadmin.git
    • tails/GitLab branch into tails/master of installer.git
  • Update links to issue boards in wiki/src/contribute/working_together/roles/*.mdwn #62 (closed)
  • close #14544 (there is a subtask still open)
  • Tell maintainers of Debian packages we are upstream for about code having moved:
    • onioncircuits
  • Mirror infrastructural repositories in Gitlab (#49 (closed))

Sysadmin spool that can be done after migration

Edited by intrigeri