Add Backup module

This module would securely and reliably store the user's data.

Several tools that have been mentioned for consideration:

Assumptions:

  • Only handle backup of app data for now, not full system backup, due to urgency.
  • First iteration doesn't need to use snapshots. That can be added later.

Open tasks for first iteration:

  • Add information to apps on how their data needs to be backed up (list of services to be started/stopped, list of directories to be backed up/restored).
  • Decide format/contents of metadata (app name, data type) to be included within backups -> backup manifest and version number for each included app
  • Merge core backup logic from !1300 (closed).
  • When creating a backup, show the list of apps that are installed and support backups. The user can de-select apps to exclude them from the backup.
  • Handle case where no supported apps are installed/enabled.
  • When creating a backup, write the apps backup manifests along with apps version numbers to a json file, and include this file in the backup.
  • When creating a backup, include all of the files and directories listed in the selected apps' manifests. (Implement backups.Packet._process_manifests.)
  • After a backup is created, allow the backup archive to be exported to a removable drive. It can be written to a standard location like "FreedomBox-backups" in the root folder of the removable drive.
  • List backup archive files that are found on removable drives, with the action to restore the files from the archive.
  • Implement restore of files from backup archive.
  • Add functional tests for backup/restore of selected web apps (ikiwiki, mediawiki, repro, dynamicdns, ejabberd).
  • When no supported apps are installed, show the "Create archive" button as disabled, with explanation text below it.
  • Rename "Create archive" -> "New backup".
  • Replace ':' in backup names.
  • Manually test backup/restore for apps that don't have automated tests yet:
    • infinoted
    • matrixsynapse
    • minetest
    • radicale
  • Fix error in backups module setup (No such file or directory: '/var/lib/freedombox/borgbackup')
  • Fix functional tests to only select the app being tested when creating the backup.
  • Also test restoring backups onto a new system.
  • Automatically install applications when their backups are being restored, if they are not already installed.

Later iterations:

  • Add validation of backup info stored in apps' manifests.
  • Implement unit tests for backups module.
  • Use snapshots when available to avoid shutting down services. (#1740)
  • Handle errors during backup and service start/stop. (#1741)
  • Check benefits of using borgmattic vs implementing by ourselves for scheduling automatic backups. (#1742 (closed))
  • Implement and run backup_pre and backup_post, restore_pre and restore_post functions in apps that need them.
  • Export/import of backup archives using browser file download/upload.
  • Export/import of backup archives using network shares.
  • Export/import of backup archives using Tahoe-LAFS. (#1743)
  • Merge create+export backup into a single step ("Create backup on removable disk", etc). Explore if there is a way to eliminate exporting backups step.
  • Explore functional testing for non-web-based apps (infinoted, matrixsynapse, minetest, radicale). (#1744)
  • Increase unit testing coverage of backups module. (#1745 (closed))
  • Backups should not be available to regular users including the 'plinth' user. Otherwise, this would become a way for adversaries to gain any secret from the system if they have compromised Plinth. Adding encryption may be a way to ensure this. This argument is also relevant for exported tarballs. Another safety measure could be for the action script to generate the list of files and directories from application list instead of being given the list. (#1746 (closed))
  • Format of the manifest inside tarball could be a dictionary instead of a list. This would make it easily extensible in the future with additional data. Current list can be a key in that dictionary.
  • Remove asking for a 'label' (or name) for the backup. Assuming a good name is okay. This removes some issues with borg archive naming rules.
  • We need a way to create and access borg repos on other disks and also remotely instead of just in /var/lib/freedombox.
Edited by James Valleroy