Commit 3ca26846 authored by Félix Sipma's avatar Félix Sipma

New upstream version 0.9.4+ds

parent 9d6267b0
Changelog for restic 0.9.4 (2019-01-06)
=======================================
The following sections list the changes in restic 0.9.4 relevant to
restic users. The changes are ordered by importance.
Summary
-------
* Fix #1989: Google Cloud Storage: Respect bandwidth limit
* Fix #2040: Add host name filter shorthand flag for `stats` command
* Fix #2068: Correctly return error loading data
* Fix #2095: Consistently use local time for snapshots times
* Enh #1605: Concurrent restore
* Enh #2089: Increase granularity of the "keep within" retention policy
* Enh #2097: Add key hinting
* Enh #2017: Mount: Enforce FUSE Unix permissions with allow-other
* Enh #2070: Make all commands display timestamps in local time
* Enh #2085: Allow --files-from to be specified multiple times
* Enh #2094: Run command to get password
Details
-------
* Bugfix #1989: Google Cloud Storage: Respect bandwidth limit
The GCS backend did not respect the bandwidth limit configured, a previous commit
accidentally removed support for it.
https://github.com/restic/restic/issues/1989
https://github.com/restic/restic/pull/2100
* Bugfix #2040: Add host name filter shorthand flag for `stats` command
The default value for `--host` flag was set to 'H' (the shorthand version of the flag), this
caused the lookup for the latest snapshot to fail.
Add shorthand flag `-H` for `--host` (with empty default so if these flags are not specified the
latest snapshot will not filter by host name).
Also add shorthand `-H` for `backup` command.
https://github.com/restic/restic/issues/2040
* Bugfix #2068: Correctly return error loading data
In one case during `prune` and `check`, an error loading data from the backend is not returned
properly. This is now corrected.
https://github.com/restic/restic/issues/1999#issuecomment-433737921
https://github.com/restic/restic/pull/2068
* Bugfix #2095: Consistently use local time for snapshots times
By default snapshots created with restic backup were set to local time, but when the --time flag
was used the provided timestamp was parsed as UTC. With this change all snapshots times are set
to local time.
https://github.com/restic/restic/pull/2095
* Enhancement #1605: Concurrent restore
This change significantly improves restore performance, especially when using
high-latency remote repositories like B2.
The implementation now uses several concurrent threads to download and process multiple
remote files concurrently. To further reduce restore time, each remote file is downloaded
using a single repository request.
https://github.com/restic/restic/issues/1605
https://github.com/restic/restic/pull/1719
* Enhancement #2089: Increase granularity of the "keep within" retention policy
The `keep-within` option of the `forget` command now accepts time ranges with an hourly
granularity. For example, running `restic forget --keep-within 3d12h` will keep all the
snapshots made within three days and twelve hours from the time of the latest snapshot.
https://github.com/restic/restic/issues/2089
https://github.com/restic/restic/pull/2090
* Enhancement #2097: Add key hinting
Added a new option `--key-hint` and corresponding environment variable `RESTIC_KEY_HINT`.
The key hint is a key ID to try decrypting first, before other keys in the repository.
This change will benefit repositories with many keys; if the correct key hint is supplied then
restic only needs to check one key. If the key hint is incorrect (the key does not exist, or the
password is incorrect) then restic will check all keys, as usual.
https://github.com/restic/restic/issues/2097
* Enhancement #2017: Mount: Enforce FUSE Unix permissions with allow-other
The fuse mount (`restic mount`) now lets the kernel check the permissions of the files within
snapshots (this is done through the `DefaultPermissions` FUSE option) when the option
`--allow-other` is specified.
To restore the old behavior, we've added the `--no-default-permissions` option. This allows
all users that have access to the mount point to access all files within the snapshots.
https://github.com/restic/restic/pull/2017
* Enhancement #2070: Make all commands display timestamps in local time
Restic used to drop the timezone information from displayed timestamps, it now converts
timestamps to local time before printing them so the times can be easily compared to.
https://github.com/restic/restic/pull/2070
* Enhancement #2085: Allow --files-from to be specified multiple times
Before, restic took only the last file specified with `--files-from` into account, this is now
corrected.
https://github.com/restic/restic/issues/2085
https://github.com/restic/restic/pull/2086
* Enhancement #2094: Run command to get password
We've added the `--password-command` option which allows specifying a command that restic
runs every time the password for the repository is needed, so it can be integrated with a
password manager or keyring. The option can also be set via the environment variable
`$RESTIC_PASSWORD_COMMAND`.
https://github.com/restic/restic/pull/2094
Changelog for restic 0.9.3 (2018-10-13)
=======================================
......@@ -8,7 +136,7 @@ Summary
-------
* Fix #1935: Remove truncated files from cache
* Fix #1978: Do not return an error when the scanner is faster than backup
* Fix #1978: Do not return an error when the scanner is slower than backup
* Enh #1766: Restore: suppress lchown errors when not running as root
* Enh #1909: Reject files/dirs by name first
* Enh #1940: Add directory filter to ls command
......@@ -20,6 +148,7 @@ Summary
* Enh #1920: Vendor dependencies with Go 1.11 Modules
* Enh #1949: Add new command `self-update`
* Enh #1953: Ls: Add JSON output support for restic ls cmd
* Enh #1962: Stream JSON output for ls command
Details
-------
......@@ -32,7 +161,7 @@ Details
https://github.com/restic/restic/issues/1935
* Bugfix #1978: Do not return an error when the scanner is faster than backup
* Bugfix #1978: Do not return an error when the scanner is slower than backup
When restic makes a backup, there's a background task called "scanner" which collects
information on how many files and directories are to be saved, in order to display progress
......@@ -155,6 +284,18 @@ Details
https://github.com/restic/restic/pull/1953
* Enhancement #1962: Stream JSON output for ls command
The `ls` command now supports JSON output with the global `--json` flag, and this change
streams out JSON messages one object at a time rather than en entire array buffered in memory
before encoding. The advantage is it allows large listings to be handled efficiently.
Two message types are printed: snapshots and nodes. A snapshot object will precede node
objects which belong to that snapshot. The `struct_type` field can be used to determine which
kind of message an object is.
https://github.com/restic/restic/pull/1962
Changelog for restic 0.9.2 (2018-08-06)
=======================================
......
......@@ -3,7 +3,7 @@
Introduction
------------
restic is a backup program that is fast, efficient and secure.
restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
For detailed usage and installation instructions check out the `documentation <https://restic.readthedocs.io/en/latest>`__.
......
Bugfix: Do not return an error when the scanner is faster than backup
Bugfix: Do not return an error when the scanner is slower than backup
When restic makes a backup, there's a background task called "scanner" which
collects information on how many files and directories are to be saved, in
......
Enhancement: Stream JSON output for ls command
The `ls` command now supports JSON output with the global `--json`
flag, and this change streams out JSON messages one object at a time
rather than en entire array buffered in memory before encoding. The
advantage is it allows large listings to be handled efficiently.
Two message types are printed: snapshots and nodes. A snapshot
object will precede node objects which belong to that snapshot.
The `struct_type` field can be used to determine which kind of
message an object is.
https://github.com/restic/restic/pull/1962
Enhancement: Concurrent restore
This change significantly improves restore performance, especially
when using high-latency remote repositories like B2.
The implementation now uses several concurrent threads to download and process
multiple remote files concurrently. To further reduce restore time, each remote
file is downloaded using a single repository request.
https://github.com/restic/restic/issues/1605
https://github.com/restic/restic/pull/1719
Bugfix: Google Cloud Storage: Respect bandwidth limit
The GCS backend did not respect the bandwidth limit configured, a previous
commit accidentally removed support for it.
https://github.com/restic/restic/issues/1989
https://github.com/restic/restic/pull/2100
Bugfix: Add host name filter shorthand flag for `stats` command
The default value for `--host` flag was set to 'H' (the shorthand version of
the flag), this caused the lookup for the latest snapshot to fail.
Add shorthand flag `-H` for `--host` (with empty default so if these flags
are not specified the latest snapshot will not filter by host name).
Also add shorthand `-H` for `backup` command.
https://github.com/restic/restic/issues/2040
Enhancement: increase granularity of the "keep within" retention policy
The `keep-within` option of the `forget` command now accepts time ranges with
an hourly granularity. For example, running `restic forget --keep-within 3d12h`
will keep all the snapshots made within three days and twelve hours from the
time of the latest snapshot.
https://github.com/restic/restic/issues/2089
https://github.com/restic/restic/pull/2090
Enhancement: Add key hinting
Added a new option `--key-hint` and corresponding environment variable
`RESTIC_KEY_HINT`. The key hint is a key ID to try decrypting first, before
other keys in the repository.
This change will benefit repositories with many keys; if the correct key hint
is supplied then restic only needs to check one key. If the key hint is
incorrect (the key does not exist, or the password is incorrect) then restic
will check all keys, as usual.
https://github.com/restic/restic/issues/2097
Enhancement: mount: Enforce FUSE Unix permissions with allow-other
The fuse mount (`restic mount`) now lets the kernel check the permissions of
the files within snapshots (this is done through the `DefaultPermissions` FUSE
option) when the option `--allow-other` is specified.
To restore the old behavior, we've added the `--no-default-permissions` option.
This allows all users that have access to the mount point to access all
files within the snapshots.
https://github.com/restic/restic/pull/2017
Bugfix: Correctly return error loading data
In one case during `prune` and `check`, an error loading data from the backend is not returned properly. This is now corrected.
https://github.com/restic/restic/pull/2068
https://github.com/restic/restic/issues/1999#issuecomment-433737921
Enhancement: Make all commands display timestamps in local time
Restic used to drop the timezone information from displayed timestamps, it now
converts timestamps to local time before printing them so the times can be
easily compared to.
https://github.com/restic/restic/pull/2070
Enhancement: Allow --files-from to be specified multiple times
Before, restic took only the last file specified with `--files-from` into
account, this is now corrected.
https://github.com/restic/restic/issues/2085
https://github.com/restic/restic/pull/2086
Enhancement: Run command to get password
We've added the `--password-command` option which allows specifying a command
that restic runs every time the password for the repository is needed, so it
can be integrated with a password manager or keyring. The option can also be
set via the environment variable `$RESTIC_PASSWORD_COMMAND`.
https://github.com/restic/restic/pull/2094
Bugfix: consistently use local time for snapshots times
By default snapshots created with restic backup were set to local time,
but when the --time flag was used the provided timestamp was parsed as
UTC. With this change all snapshots times are set to local time.
https://github.com/restic/restic/pull/2095
......@@ -45,8 +45,12 @@ given as the arguments.
},
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
if backupOptions.Stdin && backupOptions.FilesFrom == "-" {
return errors.Fatal("cannot use both `--stdin` and `--files-from -`")
if backupOptions.Stdin {
for _, filename := range backupOptions.FilesFrom {
if filename == "-" {
return errors.Fatal("cannot use both `--stdin` and `--files-from -`")
}
}
}
var t tomb.Tomb
......@@ -75,7 +79,7 @@ type BackupOptions struct {
StdinFilename string
Tags []string
Host string
FilesFrom string
FilesFrom []string
TimeStamp string
WithAtime bool
}
......@@ -97,11 +101,11 @@ func init() {
f.StringVar(&backupOptions.StdinFilename, "stdin-filename", "stdin", "file name to use when reading from stdin")
f.StringArrayVar(&backupOptions.Tags, "tag", nil, "add a `tag` for the new snapshot (can be specified multiple times)")
f.StringVar(&backupOptions.Host, "host", "H", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
f.StringVarP(&backupOptions.Host, "host", "H", "", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
f.StringVar(&backupOptions.Host, "hostname", "", "set the `hostname` for the snapshot manually")
f.MarkDeprecated("hostname", "use --host")
f.StringVar(&backupOptions.FilesFrom, "files-from", "", "read the files to backup from file (can be combined with file args)")
f.StringArrayVar(&backupOptions.FilesFrom, "files-from", nil, "read the files to backup from file (can be combined with file args/can be specified multiple times)")
f.StringVar(&backupOptions.TimeStamp, "time", "", "time of the backup (ex. '2012-11-01 22:08:41') (default: now)")
f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
}
......@@ -175,12 +179,16 @@ func readLinesFromFile(filename string) ([]string, error) {
// Check returns an error when an invalid combination of options was set.
func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
if opts.FilesFrom == "-" && gopts.password == "" {
return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
if gopts.password == "" {
for _, filename := range opts.FilesFrom {
if filename == "-" {
return errors.Fatal("unable to read password from stdin when data is to be read from stdin, use --password-file or $RESTIC_PASSWORD")
}
}
}
if opts.Stdin {
if opts.FilesFrom != "" {
if len(opts.FilesFrom) > 0 {
return errors.Fatal("--stdin and --files-from cannot be used together")
}
......@@ -302,20 +310,25 @@ func collectTargets(opts BackupOptions, args []string) (targets []string, err er
return nil, nil
}
fromfile, err := readLinesFromFile(opts.FilesFrom)
if err != nil {
return nil, err
}
// expand wildcards
var lines []string
for _, line := range fromfile {
var expanded []string
expanded, err := filepath.Glob(line)
for _, file := range opts.FilesFrom {
fromfile, err := readLinesFromFile(file)
if err != nil {
return nil, errors.WithMessage(err, fmt.Sprintf("pattern: %s", line))
return nil, err
}
// expand wildcards
for _, line := range fromfile {
var expanded []string
expanded, err := filepath.Glob(line)
if err != nil {
return nil, errors.WithMessage(err, fmt.Sprintf("pattern: %s", line))
}
if len(expanded) == 0 {
Warnf("pattern %q does not match any files, skipping\n", line)
}
lines = append(lines, expanded...)
}
lines = append(lines, expanded...)
}
// merge files from files-from into normal args so we can reuse the normal
......@@ -374,7 +387,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
timeStamp := time.Now()
if opts.TimeStamp != "" {
timeStamp, err = time.Parse(TimeFormat, opts.TimeStamp)
timeStamp, err = time.ParseInLocation(TimeFormat, opts.TimeStamp, time.Local)
if err != nil {
return errors.Fatalf("error in time option: %v\n", err)
}
......@@ -382,6 +395,12 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
var t tomb.Tomb
term.Print("open repository\n")
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
p := ui.NewBackup(term, gopts.verbosity)
// use the terminal for stdout/stderr
......@@ -403,12 +422,6 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
t.Go(func() error { return p.Run(t.Context(gopts.ctx)) })
p.V("open repository")
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
p.V("lock repository")
lock, err := lockRepo(repo)
defer unlockRepo(lock)
......
......@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"os"
"path"
"path/filepath"
"github.com/restic/restic/internal/debug"
......@@ -47,12 +48,12 @@ func init() {
flags.StringArrayVar(&dumpOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
}
func splitPath(path string) []string {
d, f := filepath.Split(path)
func splitPath(p string) []string {
d, f := path.Split(p)
if d == "" || d == "/" {
return []string{f}
}
s := splitPath(filepath.Clean(d))
s := splitPath(path.Clean(d))
return append(s, f)
}
......
......@@ -213,7 +213,7 @@ func (s *statefulOutput) PrintObjectNormal(kind, id, nodepath, treeID string, sn
} else {
Printf(" ... path %s\n", nodepath)
}
Printf(" ... in snapshot %s (%s)\n", sn.ID().Str(), sn.Time.Format(TimeFormat))
Printf(" ... in snapshot %s (%s)\n", sn.ID().Str(), sn.Time.Local().Format(TimeFormat))
}
func (s *statefulOutput) PrintObject(kind, id, nodepath, treeID string, sn *restic.Snapshot) {
......
......@@ -59,7 +59,7 @@ func init() {
f.IntVarP(&forgetOptions.Weekly, "keep-weekly", "w", 0, "keep the last `n` weekly snapshots")
f.IntVarP(&forgetOptions.Monthly, "keep-monthly", "m", 0, "keep the last `n` monthly snapshots")
f.IntVarP(&forgetOptions.Yearly, "keep-yearly", "y", 0, "keep the last `n` yearly snapshots")
f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that are older than `duration` (eg. 1y5m7d) relative to the latest snapshot")
f.VarP(&forgetOptions.Within, "keep-within", "", "keep snapshots that are newer than `duration` (eg. 1y5m7d2h) relative to the latest snapshot")
f.Var(&forgetOptions.KeepTags, "keep-tag", "keep snapshots with this `taglist` (can be specified multiple times)")
f.StringVar(&forgetOptions.Host, "host", "", "only consider snapshots with the given `host`")
......
......@@ -59,7 +59,7 @@ func listKeys(ctx context.Context, s *repository.Repository, gopts GlobalOptions
ID: id.Str(),
UserName: k.Username,
HostName: k.Hostname,
Created: k.Created.Format(TimeFormat),
Created: k.Created.Local().Format(TimeFormat),
}
keys = append(keys, key)
......
......@@ -64,10 +64,8 @@ func init() {
type lsSnapshot struct {
*restic.Snapshot
ID *restic.ID `json:"id"`
ShortID string `json:"short_id"`
Nodes []lsNode `json:"nodes"`
StructType string `json:"struct_type"` // "snapshot"
}
......@@ -150,24 +148,22 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
var (
printSnapshot func(sn *restic.Snapshot)
printNode func(path string, node *restic.Node)
printFinish func() error
)
if gopts.JSON {
var lssnapshots []lsSnapshot
enc := json.NewEncoder(gopts.stdout)
printSnapshot = func(sn *restic.Snapshot) {
lss := lsSnapshot{
enc.Encode(lsSnapshot{
Snapshot: sn,
ID: sn.ID(),
ShortID: sn.ID().Str(),
StructType: "snapshot",
}
lssnapshots = append(lssnapshots, lss)
})
}
printNode = func(path string, node *restic.Node) {
lsn := lsNode{
enc.Encode(lsNode{
Name: node.Name,
Type: node.Type,
Path: path,
......@@ -179,25 +175,15 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
AccessTime: node.AccessTime,
ChangeTime: node.ChangeTime,
StructType: "node",
}
s := &lssnapshots[len(lssnapshots)-1]
s.Nodes = append(s.Nodes, lsn)
}
printFinish = func() error {
return json.NewEncoder(gopts.stdout).Encode(lssnapshots)
})
}
} else {
// default output methods
printSnapshot = func(sn *restic.Snapshot) {
Verbosef("snapshot %s of %v filtered by %v at %s):\n", sn.ID().Str(), sn.Paths, dirs, sn.Time)
}
printNode = func(path string, node *restic.Node) {
Printf("%s\n", formatNode(path, node, lsOptions.ListLong))
}
printFinish = func() error {
return nil
}
}
for sn := range FindFilteredSnapshots(ctx, repo, opts.Host, opts.Tags, opts.Paths, args[:1]) {
......@@ -240,5 +226,6 @@ func runLs(opts LsOptions, gopts GlobalOptions, args []string) error {
return err
}
}
return printFinish()
return nil
}
......@@ -53,13 +53,14 @@ For details please see the documentation for time.Format() at:
// MountOptions collects all options for the mount command.
type MountOptions struct {
OwnerRoot bool
AllowRoot bool
AllowOther bool
Host string
Tags restic.TagLists
Paths []string
SnapshotTemplate string
OwnerRoot bool
AllowRoot bool
AllowOther bool
NoDefaultPermissions bool
Host string
Tags restic.TagLists
Paths []string
SnapshotTemplate string
}
var mountOptions MountOptions
......@@ -71,6 +72,7 @@ func init() {
mountFlags.BoolVar(&mountOptions.OwnerRoot, "owner-root", false, "use 'root' as the owner of files and dirs")
mountFlags.BoolVar(&mountOptions.AllowRoot, "allow-root", false, "allow root user to access the data in the mounted directory")
mountFlags.BoolVar(&mountOptions.AllowOther, "allow-other", false, "allow other users to access the data in the mounted directory")
mountFlags.BoolVar(&mountOptions.NoDefaultPermissions, "no-default-permissions", false, "for 'allow-other', ignore Unix permissions and allow users to read all snapshot files")
mountFlags.StringVarP(&mountOptions.Host, "host", "H", "", `only consider snapshots for this host`)
mountFlags.Var(&mountOptions.Tags, "tag", "only consider snapshots which include this `taglist`")
......@@ -118,6 +120,11 @@ func mount(opts MountOptions, gopts GlobalOptions, mountpoint string) error {
if opts.AllowOther {
mountOptions = append(mountOptions, systemFuse.AllowOther())
// let the kernel check permissions unless it is explicitly disabled
if !opts.NoDefaultPermissions {
mountOptions = append(mountOptions, systemFuse.DefaultPermissions())
}
}
c, err := systemFuse.Mount(mountpoint, mountOptions...)
......
package main
import (
"os"
"time"
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/restic"
"github.com/spf13/cobra"
)
var cmdRecover = &cobra.Command{
Use: "recover [flags]",
Short: "Recover data from the repository",
Long: `
The "recover" command build a new snapshot from all directories it can find in
the raw data of the repository. It can be used if, for example, a snapshot has
been removed by accident with "forget".
`,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
return runRecover(globalOptions)
},
}
func init() {
cmdRoot.AddCommand(cmdRecover)
}
func runRecover(gopts GlobalOptions) error {
hostname, err := os.Hostname()
if err != nil {
return err
}
repo, err := OpenRepository(gopts)
if err != nil {
return err
}
lock, err := lockRepo(repo)
defer unlockRepo(lock)
if err != nil {
return err
}
Verbosef("load index files\n")
if err = repo.LoadIndex(gopts.ctx); err != nil {
return err
}
// trees maps a tree ID to whether or not it is referenced by a different
// tree. If it is not referenced, we have a root tree.
trees := make(map[restic.ID]bool)
for blob := range repo.Index().Each(gopts.ctx) {
if blob.Blob.Type != restic.TreeBlob {
continue
}
trees[blob.Blob.ID] = false
}
cur := 0
max := len(trees)
Verbosef("load %d trees\n\n", len(trees))
for id := range trees {
cur++
Verbosef("\rtree (%v/%v)", cur, max)
if !trees[id] {
trees[id] = false