Commit 9d6267b0 authored by Félix Sipma's avatar Félix Sipma

New upstream version 0.9.3+ds

parent eb5088f4
<!--
Welcome! If you have a question or are unsure if you should open an issue,
please use the forum instead!
https://forum.restic.net
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
<!--
Please add the version of restic you're currently using here, this helps us
later to see what has changed in restic when we revisit this issue after some
time.
-->
Describe the issue
------------------
/restic
/.vagrant
/doc/_build
......@@ -6,15 +6,35 @@ matrix:
- os: linux
go: "1.9.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0 RESTIC_BUILD_SOLARIS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.10.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.11.x"
sudo: true
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: osx
go: "1.10.x"
go: "1.11.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/Library/Caches/go-build
- $HOME/gopath/pkg/mod
branches:
only:
......@@ -38,4 +58,4 @@ script:
- go run run_integration_tests.go
after_success:
- bash <(curl -s https://codecov.io/bash) -f all.cov
- test -r all.cov && bash <(curl -s https://codecov.io/bash) -f all.cov
Changelog for restic 0.9.3 (2018-10-13)
=======================================
The following sections list the changes in restic 0.9.3 relevant to
restic users. The changes are ordered by importance.
Summary
-------
* Fix #1935: Remove truncated files from cache
* Fix #1978: Do not return an error when the scanner is faster than backup
* Enh #1766: Restore: suppress lchown errors when not running as root
* Enh #1909: Reject files/dirs by name first
* Enh #1940: Add directory filter to ls command
* Enh #1967: Use `--host` everywhere
* Enh #2028: Display size of cache directories
* Enh #1777: Improve the `find` command
* Enh #1876: Display reason why forget keeps snapshots
* Enh #1891: Accept glob in paths loaded via --files-from
* Enh #1920: Vendor dependencies with Go 1.11 Modules
* Enh #1949: Add new command `self-update`
* Enh #1953: Ls: Add JSON output support for restic ls cmd
Details
-------
* Bugfix #1935: Remove truncated files from cache
When a file in the local cache is truncated, and restic tries to access data beyond the end of the
(cached) file, it used to return an error "EOF". This is now fixed, such truncated files are
removed and the data is fetched directly from the backend.
https://github.com/restic/restic/issues/1935
* Bugfix #1978: Do not return an error when the scanner is faster than backup
When restic makes a backup, there's a background task called "scanner" which collects
information on how many files and directories are to be saved, in order to display progress
information to the user. When the backup finishes faster than the scanner, it is aborted
because the result is not needed any more. This logic contained a bug, where quitting the
scanner process was treated as an error, and caused restic to print an unhelpful error message
("context canceled").
https://github.com/restic/restic/issues/1978
https://github.com/restic/restic/pull/1991
* Enhancement #1766: Restore: suppress lchown errors when not running as root
Like "cp" and "rsync" do, restic now only reports errors for changing the ownership of files
during restore if it is run as root, on non-Windows operating systems. On Windows, the error
is reported as usual.
https://github.com/restic/restic/issues/1766
* Enhancement #1909: Reject files/dirs by name first
The current scanner/archiver code had an architectural limitation: it always ran the
`lstat()` system call on all files and directories before a decision to include/exclude the
file/dir was made. This lead to a lot of unnecessary system calls for items that could have been
rejected by their name or path only.
We've changed the archiver/scanner implementation so that it now first rejects by name/path,
and only runs the system call on the remaining items. This reduces the number of `lstat()`
system calls a lot (depending on the exclude settings).
https://github.com/restic/restic/issues/1909
https://github.com/restic/restic/pull/1912
* Enhancement #1940: Add directory filter to ls command
The ls command can now be filtered by directories, so that only files in the given directories
will be shown. If the --recursive flag is specified, then ls will traverse subfolders and list
their files as well.
It used to be possible to specify multiple snapshots, but that has been replaced by only one
snapshot and the possibility of specifying multiple directories.
Specifying directories constrains the walk, which can significantly speed up the listing.
https://github.com/restic/restic/issues/1940
https://github.com/restic/restic/pull/1941
* Enhancement #1967: Use `--host` everywhere
We now use the flag `--host` for all commands which need a host name, using `--hostname` (e.g.
for `restic backup`) still works, but will print a deprecation warning. Also, add the short
option `-H` where possible.
https://github.com/restic/restic/issues/1967
* Enhancement #2028: Display size of cache directories
The `cache` command now by default shows the size of the individual cache directories. It can be
disabled with `--no-size`.
https://github.com/restic/restic/issues/2028
https://github.com/restic/restic/pull/2033
* Enhancement #1777: Improve the `find` command
We've updated the `find` command to support multiple patterns.
`restic find` is now able to list the snapshots containing a specific tree or blob, or even the
snapshots that contain blobs belonging to a given pack. A list of IDs can be given, as long as they
all have the same type.
The command `find` can also display the pack IDs the blobs belong to, if the `--show-pack-id`
flag is provided.
https://github.com/restic/restic/issues/1777
https://github.com/restic/restic/pull/1780
* Enhancement #1876: Display reason why forget keeps snapshots
We've added a column to the list of snapshots `forget` keeps which details the reasons to keep a
particuliar snapshot. This makes debugging policies for forget much easier. Please remember
to always try things out with `--dry-run`!
https://github.com/restic/restic/pull/1876
* Enhancement #1891: Accept glob in paths loaded via --files-from
Before that, behaviour was different if paths were appended to command line or from a file,
because wild card characters were expanded by shell if appended to command line, but not
expanded if loaded from file.
https://github.com/restic/restic/issues/1891
* Enhancement #1920: Vendor dependencies with Go 1.11 Modules
Until now, we've used `dep` for managing dependencies, we've now switch to using Go modules.
For users this does not change much, only if you want to compile restic without downloading
anything with Go 1.11, then you need to run: `go build -mod=vendor build.go`
https://github.com/restic/restic/pull/1920
* Enhancement #1949: Add new command `self-update`
We have added a new command called `self-update` which downloads the latest released version
of restic from GitHub and replaces the current binary with it. It does not rely on any external
program (so it'll work everywhere), but still verifies the GPG signature using the embedded
GPG public key.
By default, the `self-update` command is hidden behind the `selfupdate` built tag, which is
only set when restic is built using `build.go` (including official releases). The reason for
this is that downstream distributions will then not include the command by default, so users
are encouraged to use the platform-specific distribution mechanism.
https://github.com/restic/restic/pull/1949
* Enhancement #1953: Ls: Add JSON output support for restic ls cmd
We've implemented listing files in the repository with JSON as output, just pass `--json` as an
option to `restic ls`. This makes the output of the command machine readable.
https://github.com/restic/restic/pull/1953
Changelog for restic 0.9.2 (2018-08-06)
=======================================
......
......@@ -46,12 +46,15 @@ Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected!
In addition, you can compile restic with debug support by running
`go run build.go -tags debug` and instructing it to create a debug log by
setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
`go run -mod=vendor build.go -tags debug` and instructing it to create a debug
log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work
For Go < 1.11, you need to remove the `-mod=vendor` option from the build
command.
Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
......@@ -60,9 +63,37 @@ uploading it somewhere or post only the parts that are really relevant.
Development Environment
=======================
In order to compile restic with the `go` tool directly, it needs to be checked
out at the right path within a `GOPATH`. The concept of a `GOPATH` is explained
in ["How to write Go code"](https://golang.org/doc/code.html).
The repository contains several sets of directories with code: `cmd/` and
`internal/` contain the code written for restic, whereas `vendor/` contains
copies of libraries restic depends on. The libraries are managed with the
command `go mod vendor`.
Go >= 1.11
----------
For Go version 1.11 or later, you should clone the repo (without having
`$GOPATH` set) and `cd` into the directory:
$ unset GOPATH
$ git clone https://github.com/restic/restic
$ cd restic
Then use the `go` tool to build restic:
$ go build ./cmd/restic
$ ./restic version
restic 0.9.2-dev (compiled manually) compiled with go1.11 on linux/amd64
You can run all tests with the following command:
$ go test ./...
Go < 1.11
---------
In order to compile restic with Go before 1.11, it needs to be checked out at
the right path within a `GOPATH`. The concept of a `GOPATH` is explained in
["How to write Go code"](https://golang.org/doc/code.html).
If you do not have a directory with Go code yet, executing the following
instructions in your shell will create one for you and check out the restic
......@@ -83,12 +114,7 @@ You can then build restic as follows:
The following commands can be used to run all the tests:
$ go test ./cmd/... ./internal/...
The repository contains two sets of directories with code: `cmd/` and
`internal/` contain the code written for restic, whereas `vendor/` contains
copies of libraries restic depends on. The libraries are managed with the
[`dep`](https://github.com/golang/dep) tool.
$ go test ./...
Providing Patches
=================
......@@ -141,13 +167,14 @@ run
gofmt -w **/*.go
in the project root directory before committing. Installing the script
`fmt-check` from https://github.com/edsrzf/gofmt-git-hook locally as a
pre-commit hook checks formatting before committing automatically, just copy
this script to `.git/hooks/pre-commit`.
in the project root directory before committing. For each Pull Request, the
formatting is tested with `gofmt` for the latest stable version of Go.
Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
locally as a pre-commit hook checks formatting before committing automatically,
just copy this script to `.git/hooks/pre-commit`.
For each pull request, several different systems run the integration tests on
Linux, OS X and Windows. We won't merge any code that does not pass all tests
Linux, macOS and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix
it. If you need help on this, please leave a comment in the pull request, and
we'll be glad to assist. Having a PR with failing integration tests is nothing
......
This diff is collapsed.
# Gopkg.toml example
#
# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
[prune]
unused-packages = true
go-tests = true
......@@ -3,7 +3,7 @@
all: restic
restic:
go run build.go
go run -mod=vendor build.go || go run build.go
clean:
rm -f restic
......
......@@ -7,6 +7,9 @@ branches:
only:
- master
cache:
- '%LocalAppData%\go-build'
init:
- ps: >-
$app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
......@@ -17,8 +20,8 @@ init:
install:
- rmdir c:\go /s /q
- appveyor DownloadFile https://dl.google.com/go/go1.10.windows-amd64.msi
- msiexec /i go1.10.windows-amd64.msi /q
- appveyor DownloadFile https://dl.google.com/go/go1.11.windows-amd64.msi
- msiexec /i go1.11.windows-amd64.msi /q
- go version
- go env
- appveyor DownloadFile http://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
......@@ -26,4 +29,4 @@ install:
- set PATH=bin/;%PATH%
build_script:
- go run run_integration_tests.go
- go run -mod=vendor run_integration_tests.go
This diff is collapsed.
Enhancement: restore: suppress lchown errors when not running as root
Like "cp" and "rsync" do, restic now only reports errors for changing
the ownership of files during restore if it is run as root, on non-Windows
operating systems. On Windows, the error is reported as usual.
https://github.com/restic/restic/issues/1766
Enhancement: Reject files/dirs by name first
The current scanner/archiver code had an architectural limitation: it always
ran the `lstat()` system call on all files and directories before a decision to
include/exclude the file/dir was made. This lead to a lot of unnecessary system
calls for items that could have been rejected by their name or path only.
We've changed the archiver/scanner implementation so that it now first rejects
by name/path, and only runs the system call on the remaining items. This
reduces the number of `lstat()` system calls a lot (depending on the exclude
settings).
https://github.com/restic/restic/issues/1909
https://github.com/restic/restic/pull/1912
Bugfix: Remove truncated files from cache
When a file in the local cache is truncated, and restic tries to access data
beyond the end of the (cached) file, it used to return an error "EOF". This is
now fixed, such truncated files are removed and the data is fetched directly
from the backend.
https://github.com/restic/restic/issues/1935
Enhancement: Add directory filter to ls command
The ls command can now be filtered by directories, so that only files in the
given directories will be shown. If the --recursive flag is specified, then
ls will traverse subfolders and list their files as well.
It used to be possible to specify multiple snapshots, but that has been
replaced by only one snapshot and the possibility of specifying multiple
directories.
Specifying directories constrains the walk, which can significantly speed up
the listing.
https://github.com/restic/restic/issues/1940
https://github.com/restic/restic/pull/1941
Enhancement: Use `--host` everywhere
We now use the flag `--host` for all commands which need a host name, using
`--hostname` (e.g. for `restic backup`) still works, but will print a
deprecation warning. Also, add the short option `-H` where possible.
https://github.com/restic/restic/issues/1967
Bugfix: Do not return an error when the scanner is faster than backup
When restic makes a backup, there's a background task called "scanner" which
collects information on how many files and directories are to be saved, in
order to display progress information to the user. When the backup finishes
faster than the scanner, it is aborted because the result is not needed any
more. This logic contained a bug, where quitting the scanner process was
treated as an error, and caused restic to print an unhelpful error message
("context canceled").
https://github.com/restic/restic/issues/1978
https://github.com/restic/restic/pull/1991
Enhancement: Display size of cache directories
The `cache` command now by default shows the size of the individual cache
directories. It can be disabled with `--no-size`.
https://github.com/restic/restic/issues/2028
https://github.com/restic/restic/pull/2033
Enhancement: Improve the `find` command
We've updated the `find` command to support multiple patterns.
`restic find` is now able to list the snapshots containing a specific tree
or blob, or even the snapshots that contain blobs belonging to a given pack.
A list of IDs can be given, as long as they all have the same type.
The command `find` can also display the pack IDs the blobs belong to, if
the `--show-pack-id` flag is provided.
https://github.com/restic/restic/issues/1777
https://github.com/restic/restic/pull/1780
Enhancement: Display reason why forget keeps snapshots
We've added a column to the list of snapshots `forget` keeps which details the
reasons to keep a particuliar snapshot. This makes debugging policies for
forget much easier. Please remember to always try things out with `--dry-run`!
https://github.com/restic/restic/pull/1876
Enhancement: Accept glob in paths loaded via --files-from
Before that, behaviour was different if paths were appended to command line or
from a file, because wild card characters were expanded by shell if appended to
command line, but not expanded if loaded from file.
https://github.com/restic/restic/issues/1891
Enhancement: Vendor dependencies with Go 1.11 Modules
Until now, we've used `dep` for managing dependencies, we've now switch to
using Go modules. For users this does not change much, only if you want to
compile restic without downloading anything with Go 1.11, then you need to run:
`go build -mod=vendor build.go`
https://github.com/restic/restic/pull/1920
Enhancement: Add new command `self-update`
We have added a new command called `self-update` which downloads the
latest released version of restic from GitHub and replaces the current
binary with it. It does not rely on any external program (so it'll work
everywhere), but still verifies the GPG signature using the embedded GPG
public key.
By default, the `self-update` command is hidden behind the `selfupdate`
built tag, which is only set when restic is built using `build.go` (including
official releases). The reason for this is that downstream distributions will
then not include the command by default, so users are encouraged to use the
platform-specific distribution mechanism.
https://github.com/restic/restic/pull/1949
Enhancement: ls: Add JSON output support for restic ls cmd
We've implemented listing files in the repository with JSON as output, just
pass `--json` as an option to `restic ls`. This makes the output of the command
machine readable.
https://github.com/restic/restic/pull/1953
......@@ -4,8 +4,10 @@ import (
"bufio"
"bytes"
"context"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"strconv"
"strings"
"time"
......@@ -32,13 +34,13 @@ The "backup" command creates a new snapshot and saves the files and directories
given as the arguments.
`,
PreRun: func(cmd *cobra.Command, args []string) {
if backupOptions.Hostname == "" {
if backupOptions.Host == "" {
hostname, err := os.Hostname()
if err != nil {
debug.Log("os.Hostname() returned err: %v", err)
return
}
backupOptions.Hostname = hostname
backupOptions.Host = hostname
}
},
DisableAutoGenTag: true,
......@@ -72,7 +74,7 @@ type BackupOptions struct {
Stdin bool
StdinFilename string
Tags []string
Hostname string
Host string
FilesFrom string
TimeStamp string
WithAtime bool
......@@ -94,7 +96,11 @@ func init() {
f.BoolVar(&backupOptions.Stdin, "stdin", false, "read backup from stdin")
f.StringVar(&backupOptions.StdinFilename, "stdin-filename", "stdin", "file name to use when reading from stdin")
f.StringArrayVar(&backupOptions.Tags, "tag", nil, "add a `tag` for the new snapshot (can be specified multiple times)")
f.StringVar(&backupOptions.Hostname, "hostname", "", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
f.StringVar(&backupOptions.Host, "host", "H", "set the `hostname` for the snapshot manually. To prevent an expensive rescan use the \"parent\" flag")
f.StringVar(&backupOptions.Host, "hostname", "", "set the `hostname` for the snapshot manually")
f.MarkDeprecated("hostname", "use --host")
f.StringVar(&backupOptions.FilesFrom, "files-from", "", "read the files to backup from file (can be combined with file args)")
f.StringVar(&backupOptions.TimeStamp, "time", "", "time of the backup (ex. '2012-11-01 22:08:41') (default: now)")
f.BoolVar(&backupOptions.WithAtime, "with-atime", false, "store the atime for all files and directories")
......@@ -186,18 +192,9 @@ func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
return nil
}
// collectRejectFuncs returns a list of all functions which may reject data
// from being saved in a snapshot
func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, err error) {
// allowed devices
if opts.ExcludeOtherFS && !opts.Stdin {
f, err := rejectByDevice(targets)
if err != nil {
return nil, err
}
fs = append(fs, f)
}
// collectRejectByNameFuncs returns a list of all functions which may reject data
// from being saved in a snapshot based on path only
func collectRejectByNameFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectByNameFunc, err error) {
// exclude restic cache
if repo.Cache != nil {
f, err := rejectResticCache(repo)
......@@ -237,6 +234,21 @@ func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets
return fs, nil
}
// collectRejectFuncs returns a list of all functions which may reject data
// from being saved in a snapshot based on path and file info
func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, err error) {
// allowed devices
if opts.ExcludeOtherFS && !opts.Stdin {
f, err := rejectByDevice(targets)
if err != nil {
return nil, err
}
fs = append(fs, f)
}
return fs, nil
}
// readExcludePatternsFromFiles reads all exclude files and returns the list of
// exclude patterns. For each line, leading and trailing white space is removed
// and comment lines are ignored. For each remaining pattern, environment
......@@ -295,10 +307,21 @@ func collectTargets(opts BackupOptions, args []string) (targets []string, err er
return nil, err
}
// expand wildcards
var lines []string
for _, line := range fromfile {
var expanded []string
expanded, err := filepath.Glob(line)
if err != nil {
return nil, errors.WithMessage(err, fmt.Sprintf("pattern: %s", line))
}
lines = append(lines, expanded...)
}
// merge files from files-from into normal args so we can reuse the normal
// args checks and have the ability to use both files-from and args at the
// same time
args = append(args, fromfile...)
args = append(args, lines...)
if len(args) == 0 && !opts.Stdin {
return nil, errors.Fatal("nothing to backup, please specify target files/dirs")
}
......@@ -327,7 +350,7 @@ func findParentSnapshot(ctx context.Context, repo restic.Repository, opts Backup
// Find last snapshot to set it as parent, if not already set
if !opts.Force && parentID == nil {
id, err := restic.FindLatestSnapshot(ctx, repo, targets, []restic.TagList{}, opts.Hostname)
id, err := restic.FindLatestSnapshot(ctx, repo, targets, []restic.TagList{}, opts.Host)
if err == nil {
parentID = &id
} else if err != restic.ErrNoSnapshotFound {
......@@ -393,7 +416,13 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
return err
}
// rejectFuncs collect functions that can reject items from the backup
// rejectByNameFuncs collect functions that can reject items from the backup based on path only
rejectByNameFuncs, err := collectRejectByNameFuncs(opts, repo, targets)
if err != nil {
return err
}
// rejectFuncs collect functions that can reject items from the backup based on path and file info
rejectFuncs, err := collectRejectFuncs(opts, repo, targets)
if err != nil {
return err
......@@ -414,6 +443,15 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
p.V("using parent snapshot %v\n", parentSnapshotID.Str())
}
selectByNameFilter := func(item string) bool {
for _, reject := range rejectByNameFuncs {
if reject(item) {
return false
}
}
return true
}
selectFilter := func(item string, fi os.FileInfo) bool {
for _, reject := range rejectFuncs {
if reject(item, fi) {
......@@ -436,6 +474,7 @@ func runBackup(opts BackupOptions, gopts GlobalOptions, term *termstatus.Termina
}
sc := archiver.NewScanner(targetFS)
sc.SelectByName = selectByNameFilter
sc.Select = selectFilter