Commit 1824ad6f authored by Félix Sipma's avatar Félix Sipma

Update upstream source from tag 'upstream/0.9.2+ds'

Update to upstream version '0.9.2+ds'
with Debian dir 18dbf5990b3eb014b7ab6594965329aa333b918c
parents 9342e322 eb5088f4
---
name: Bug report
about: Report a problem with restic to help us resolve it and improve
---
<!--
Welcome! - We kindly ask that you:
......@@ -23,10 +28,12 @@ Thanks for understanding, and for contributing to the project!
-->
## Output of `restic version`
Output of `restic version`
--------------------------
## How did you run restic exactly?
How did you run restic exactly?
-------------------------------
<!--
This section should include at least:
......@@ -38,39 +45,46 @@ This section should include at least:
information to diagnose the problem!
-->
## What backend/server/service did you use to store the repository?
What backend/server/service did you use to store the repository?
----------------------------------------------------------------
## Expected behavior
Expected behavior
-----------------
<!--
Describe what you'd like restic to do differently.
-->
## Actual behavior
Actual behavior
---------------
<!--
In this section, please try to concentrate on observations, so only describe
what you observed directly.
-->
## Steps to reproduce the behavior
Steps to reproduce the behavior
-------------------------------
<!--
The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
## Do you have any idea what may have caused this?
Do you have any idea what may have caused this?
-----------------------------------------------
## Do you have an idea how to solve the issue?
Do you have an idea how to solve the issue?
-------------------------------------------
## Did restic help you or made you happy in any way?
Did restic help you or made you happy in any way?
-------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!
......
---
name: Feature request
about: Suggest a new feature or enhancement for restic
---
<!--
Welcome! - We kindly ask that you:
1. Fill out the issue template below - not doing so needs a good reason.
2. Use the forum if you have a question rather than a bug or feature request.
The forum is at: https://forum.restic.net
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
<!--
Please add the version of restic you're currently using here, this helps us
later to see what has changed in restic when we revisit this issue after some
time.
-->
What should restic do differently? Which functionality do you think we should add?
----------------------------------------------------------------------------------
<!--
Please describe the feature you'd like us to add here.
-->
What are you trying to do?
--------------------------
<!--
This section should contain a brief description what you're trying to do, which
would be possible after implementing the new feature.
-->
Did restic help you or made you happy in any way?
-------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!
Sometimes we get tired of reading bug reports all day and a little positive end note does wonders.
Idea by Joey Hess, https://joeyh.name/blog/entry/two_holiday_stories/
-->
<!--
Thank you very much for contributing code or documentation to restic! Please
fill out the following questions to make it easier for us to review your
......@@ -8,19 +10,22 @@ your time and add more commits. If you're done and ready for review, please
check the last box.
-->
### What is the purpose of this change? What does it change?
What is the purpose of this change? What does it change?
--------------------------------------------------------
<!--
Describe the changes here, as detailed as needed.
-->
### Was the change discussed in an issue or in the forum before?
Was the change discussed in an issue or in the forum before?
------------------------------------------------------------
<!--
Link issues and relevant forum posts here.
-->
### Checklist
Checklist
---------
- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
- [ ] I have added tests for all changes in this PR
......
Changelog for restic 0.9.2 (2018-08-06)
=======================================
The following sections list the changes in restic 0.9.2 relevant to
restic users. The changes are ordered by importance.
Summary
-------
* Fix #1854: Allow saving files/dirs on different fs with `--one-file-system`
* Fix #1870: Fix restore with --include
* Fix #1880: Use `--cache-dir` argument for `check` command
* Fix #1893: Return error when exclude file cannot be read
* Fix #1861: Fix case-insensitive search with restic find
* Enh #1906: Add support for B2 application keys
* Enh #874: Add stats command to get information about a repository
* Enh #1772: Add restore --verify to verify restored file content
* Enh #1853: Add JSON output support to `restic key list`
* Enh #1477: S3 backend: accept AWS_SESSION_TOKEN
* Enh #1901: Update the Backblaze B2 library
Details
-------
* Bugfix #1854: Allow saving files/dirs on different fs with `--one-file-system`
Restic now allows saving files/dirs on a different file system in a subdir correctly even when
`--one-file-system` is specified.
The first thing the restic archiver code does is to build a tree of the target
files/directories. If it detects that a parent directory is already included (e.g. `restic
backup /foo /foo/bar/baz`), it'll ignore the latter argument.
Without `--one-file-system`, that's perfectly valid: If `/foo` is to be archived, it will
include `/foo/bar/baz`. But with `--one-file-system`, `/foo/bar/baz` may reside on a
different file system, so it won't be included with `/foo`.
https://github.com/restic/restic/issues/1854
https://github.com/restic/restic/pull/1855
* Bugfix #1870: Fix restore with --include
We fixed a bug which prevented restic to restore files with an include filter.
https://github.com/restic/restic/issues/1870
https://github.com/restic/restic/pull/1900
* Bugfix #1880: Use `--cache-dir` argument for `check` command
`check` command now uses a temporary sub-directory of the specified directory if set using the
`--cache-dir` argument. If not set, the cache directory is created in the default temporary
directory as before. In either case a temporary cache is used to ensure the actual repository is
checked (rather than a local copy).
The `--cache-dir` argument was not used by the `check` command, instead a cache directory was
created in the temporary directory.
https://github.com/restic/restic/issues/1880
* Bugfix #1893: Return error when exclude file cannot be read
A bug was found: when multiple exclude files were passed to restic and one of them could not be
read, an error was printed and restic continued, ignoring even the existing exclude files.
Now, an error message is printed and restic aborts when an exclude file cannot be read.
https://github.com/restic/restic/issues/1893
* Bugfix #1861: Fix case-insensitive search with restic find
We've fixed the behavior for `restic find -i PATTERN`, which was broken in v0.9.1.
https://github.com/restic/restic/pull/1861
* Enhancement #1906: Add support for B2 application keys
Restic can now use so-called "application keys" which can be created in the B2 dashboard and
were only introduced recently. In contrast to the "master key", such keys can be restricted to a
specific bucket and/or path.
https://github.com/restic/restic/issues/1906
https://github.com/restic/restic/pull/1914
* Enhancement #874: Add stats command to get information about a repository
https://github.com/restic/restic/issues/874
https://github.com/restic/restic/pull/1729
* Enhancement #1772: Add restore --verify to verify restored file content
Restore will print error message if restored file content does not match expected SHA256
checksum
https://github.com/restic/restic/pull/1772
* Enhancement #1853: Add JSON output support to `restic key list`
This PR enables users to get the output of `restic key list` in JSON in addition to the existing
table format.
https://github.com/restic/restic/pull/1853
* Enhancement #1477: S3 backend: accept AWS_SESSION_TOKEN
Before, it was not possible to use s3 backend with AWS temporary security credentials(with
AWS_SESSION_TOKEN). This change gives higher priority to credentials.EnvAWS credentials
provider.
https://github.com/restic/restic/issues/1477
https://github.com/restic/restic/pull/1479
https://github.com/restic/restic/pull/1647
* Enhancement #1901: Update the Backblaze B2 library
We've updated the library we're using for accessing the Backblaze B2 service to 0.5.0 to
include support for upcoming so-called "application keys". With this feature, you can create
access credentials for B2 which are restricted to e.g. a single bucket or even a sub-directory
of a bucket.
https://github.com/restic/restic/pull/1901
https://github.com/kurin/blazer
Changelog for restic 0.9.1 (2018-06-10)
=======================================
......
......@@ -164,7 +164,7 @@ history and triaging bugs much easier.
Git commit messages have a very terse summary in the first line of the commit
message, followed by an empty line, followed by a more verbose description or a
List of changed things. For examples, please refer to the excellent [How to
Write a Git Commit Message](http://chris.beams.io/posts/git-commit/).
Write a Git Commit Message](https://chris.beams.io/posts/git-commit/).
If you change/add multiple different things that aren't related at all, try to
make several smaller commits. This is much easier to review. Using `git add -p`
......
This diff is collapsed.
......@@ -19,3 +19,7 @@
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
[prune]
unused-packages = true
go-tests = true
|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage|
|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage| |Reviewed by Hound|
Introduction
------------
......@@ -111,6 +111,14 @@ License
Restic is licensed under `BSD 2-Clause License <https://opensource.org/licenses/BSD-2-Clause>`__. You can find the
complete text in ``LICENSE``.
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by `AppsCode <https://appscode.com>`__!
|AppsCode|
.. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
:target: https://restic.readthedocs.io/en/latest/?badge=latest
.. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
......@@ -123,3 +131,7 @@ complete text in ``LICENSE``.
:target: https://saythanks.io/to/restic
.. |TestCoverage| image:: https://codecov.io/gh/restic/restic/branch/master/graph/badge.svg
:target: https://codecov.io/gh/restic/restic
.. |AppsCode| image:: https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png
:target: https://appscode.com
.. |Reviewed by Hound| image:: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
:target: https://houndci.com
Bugfix: Allow saving files/dirs on different fs with `--one-file-system`
restic now allows saving files/dirs on a different file system in a subdir
correctly even when `--one-file-system` is specified.
The first thing the restic archiver code does is to build a tree of the target
files/directories. If it detects that a parent directory is already included
(e.g. `restic backup /foo /foo/bar/baz`), it'll ignore the latter argument.
Without `--one-file-system`, that's perfectly valid: If `/foo` is to be
archived, it will include `/foo/bar/baz`. But with `--one-file-system`,
`/foo/bar/baz` may reside on a different file system, so it won't be included
with `/foo`.
https://github.com/restic/restic/issues/1854
https://github.com/restic/restic/pull/1855
Bugfix: Fix restore with --include
We fixed a bug which prevented restic to restore files with an include filter.
https://github.com/restic/restic/issues/1870
https://github.com/restic/restic/pull/1900
Bugfix: Use `--cache-dir` argument for `check` command
`check` command now uses a temporary sub-directory of the specified directory
if set using the `--cache-dir` argument. If not set, the cache directory is
created in the default temporary directory as before.
In either case a temporary cache is used to ensure the actual repository is
checked (rather than a local copy).
The `--cache-dir` argument was not used by the `check` command, instead a
cache directory was created in the temporary directory.
https://github.com/restic/restic/issues/1880
Bugfix: Return error when exclude file cannot be read
A bug was found: when multiple exclude files were passed to restic and one of
them could not be read, an error was printed and restic continued, ignoring
even the existing exclude files. Now, an error message is printed and restic
aborts when an exclude file cannot be read.
https://github.com/restic/restic/issues/1893
Enhancement: Add support for B2 application keys
Restic can now use so-called "application keys" which can be created in the B2
dashboard and were only introduced recently. In contrast to the "master key",
such keys can be restricted to a specific bucket and/or path.
https://github.com/restic/restic/issues/1906
https://github.com/restic/restic/pull/1914
Enhancement: Add stats command to get information about a repository
https://github.com/restic/restic/issues/874
https://github.com/restic/restic/pull/1729
Enhancement: Add restore --verify to verify restored file content
Restore will print error message if restored file content does not match
expected SHA256 checksum
https://github.com/restic/restic/pull/1772
Enhancement: Add JSON output support to `restic key list`
This PR enables users to get the output of `restic key list` in JSON in addition
to the existing table format.
https://github.com/restic/restic/pull/1853
\ No newline at end of file
Bugfix: Fix case-insensitive search with restic find
We've fixed the behavior for `restic find -i PATTERN`, which was
broken in v0.9.1.
https://github.com/restic/restic/pull/1861
Enhancement: S3 backend: accept AWS_SESSION_TOKEN
Before, it was not possible to use s3 backend with AWS temporary security credentials(with AWS_SESSION_TOKEN).
This change gives higher priority to credentials.EnvAWS credentials provider.
https://github.com/restic/restic/issues/1477
https://github.com/restic/restic/pull/1479
https://github.com/restic/restic/pull/1647
Enhancement: Update the Backblaze B2 library
We've updated the library we're using for accessing the Backblaze B2 service to
0.5.0 to include support for upcoming so-called "application keys". With this
feature, you can create access credentials for B2 which are restricted to e.g.
a single bucket or even a sub-directory of a bucket.
https://github.com/restic/restic/pull/1901
https://github.com/kurin/blazer
......@@ -190,7 +190,7 @@ func (opts BackupOptions) Check(gopts GlobalOptions, args []string) error {
// from being saved in a snapshot
func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets []string) (fs []RejectFunc, err error) {
// allowed devices
if opts.ExcludeOtherFS {
if opts.ExcludeOtherFS && !opts.Stdin {
f, err := rejectByDevice(targets)
if err != nil {
return nil, err
......@@ -210,7 +210,11 @@ func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets
// add patterns from file
if len(opts.ExcludeFiles) > 0 {
opts.Excludes = append(opts.Excludes, readExcludePatternsFromFiles(opts.ExcludeFiles)...)
excludes, err := readExcludePatternsFromFiles(opts.ExcludeFiles)
if err != nil {
return nil, err
}
opts.Excludes = append(opts.Excludes, excludes...)
}
if len(opts.Excludes) > 0 {
......@@ -234,8 +238,18 @@ func collectRejectFuncs(opts BackupOptions, repo *repository.Repository, targets
}
// readExcludePatternsFromFiles reads all exclude files and returns the list of
// exclude patterns.
func readExcludePatternsFromFiles(excludeFiles []string) []string {
// exclude patterns. For each line, leading and trailing white space is removed
// and comment lines are ignored. For each remaining pattern, environment
// variables are resolved. For adding a literal dollar sign ($), write $$ to
// the file.
func readExcludePatternsFromFiles(excludeFiles []string) ([]string, error) {
getenvOrDollar := func(s string) string {
if s == "$" {
return "$"
}
return os.Getenv(s)
}
var excludes []string
for _, filename := range excludeFiles {
err := func() (err error) {
......@@ -258,17 +272,16 @@ func readExcludePatternsFromFiles(excludeFiles []string) []string {
continue
}
line = os.ExpandEnv(line)
line = os.Expand(line, getenvOrDollar)
excludes = append(excludes, line)
}
return scanner.Err()
}()
if err != nil {
Warnf("error reading exclude patterns: %v:", err)
return nil
return nil, err
}
}
return excludes
return excludes, nil
}
// collectTargets returns a list of target files/dirs from several sources.
......
......@@ -50,7 +50,7 @@ func init() {
f := cmdCheck.Flags()
f.BoolVar(&checkOptions.ReadData, "read-data", false, "read all data blobs")
f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset of data packs")
f.StringVar(&checkOptions.ReadDataSubset, "read-data-subset", "", "read subset n of m data packs (format: `n/m`)")
f.BoolVar(&checkOptions.CheckUnused, "check-unused", false, "find unused blobs")
f.BoolVar(&checkOptions.WithCache, "with-cache", false, "use the cache")
}
......@@ -123,6 +123,7 @@ func newReadProgress(gopts GlobalOptions, todo restic.Stat) *restic.Progress {
//
// * if --with-cache is specified, the default cache is used
// * if the user explicitly requested --no-cache, we don't use any cache
// * if the user provides --cache-dir, we use a cache in a temporary sub-directory of the specified directory and the sub-directory is deleted after the check
// * by default, we use a cache in a temporary directory that is deleted after the check
func prepareCheckCache(opts CheckOptions, gopts *GlobalOptions) (cleanup func()) {
cleanup = func() {}
......@@ -136,8 +137,10 @@ func prepareCheckCache(opts CheckOptions, gopts *GlobalOptions) (cleanup func())
return cleanup
}
cachedir := gopts.CacheDir
// use a cache in a temporary directory
tempdir, err := ioutil.TempDir("", "restic-check-cache-")
tempdir, err := ioutil.TempDir(cachedir, "restic-check-cache-")
if err != nil {
// if an error occurs, don't use any cache
Warnf("unable to create temporary directory for cache during check, disabling cache: %v\n", err)
......
......@@ -198,12 +198,12 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
return false, nil
}
name := node.Name
normalizedNodepath := nodepath
if f.pat.ignoreCase {
name = strings.ToLower(name)
normalizedNodepath = strings.ToLower(nodepath)
}
foundMatch, err := filter.Match(f.pat.pattern, nodepath)
foundMatch, err := filter.Match(f.pat.pattern, normalizedNodepath)
if err != nil {
return false, err
}
......@@ -213,7 +213,7 @@ func (f *Finder) findInSnapshot(ctx context.Context, sn *restic.Snapshot) error
errIfNoMatch error
)
if node.Type == "dir" {
childMayMatch, err := filter.ChildMatch(f.pat.pattern, nodepath)
childMayMatch, err := filter.ChildMatch(f.pat.pattern, normalizedNodepath)
if err != nil {
return false, err
}
......
......@@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
"os"
......@@ -35,33 +36,75 @@ func init() {
flags.StringVarP(&newPasswordFile, "new-password-file", "", "", "the file from which to load a new password")
}
func listKeys(ctx context.Context, s *repository.Repository) error {
tab := NewTable()
tab.Header = fmt.Sprintf(" %-10s %-10s %-10s %s", "ID", "User", "Host", "Created")
tab.RowFormat = "%s%-10s %-10s %-10s %s"
type keyInfo struct {
Current bool `json:"current"`
ID string `json:"id"`
UserName string `json:"userName"`
HostName string `json:"hostName"`
Created string `json:"created"`
}
func (ki keyInfo) CurrentStr() string {
if ki.Current {
return "*"
}
return " "
}
func listKeys(ctx context.Context, s *repository.Repository, gopts GlobalOptions) error {
var (
appendKey func(keyInfo)
printKeys func() error
)
switch gopts.JSON {
case true:
var keys []keyInfo
appendKey = func(key keyInfo) {
keys = append(keys, key)
}
printKeys = func() error {
return json.NewEncoder(gopts.stdout).Encode(keys)
}
default:
tab := NewTable()
tab.Header = fmt.Sprintf(" %-10s %-10s %-10s %s", "ID", "User", "Host", "Created")
tab.RowFormat = "%s%-10s %-10s %-10s %s"
appendKey = func(key keyInfo) {
tab.Rows = append(tab.Rows, []interface{}{key.CurrentStr(), key.ID, key.UserName, key.HostName, key.Created})
}
err := s.List(ctx, restic.KeyFile, func(id restic.ID, size int64) error {
printKeys = func() error {
return tab.Write(globalOptions.stdout)
}
}
if err := s.List(ctx, restic.KeyFile, func(id restic.ID, size int64) error {
k, err := repository.LoadKey(ctx, s, id.String())
if err != nil {
Warnf("LoadKey() failed: %v\n", err)
return nil
}
var current string
if id.String() == s.KeyName() {
current = "*"
} else {
current = " "
key := keyInfo{
Current: id.String() == s.KeyName(),
ID: id.Str(),
UserName: k.Username,
HostName: k.Hostname,
Created: k.Created.Format(TimeFormat),
}
tab.Rows = append(tab.Rows, []interface{}{current, id.Str(),
k.Username, k.Hostname, k.Created.Format(TimeFormat)})
appendKey(key)
return nil
})
if err != nil {
}); err != nil {
return err
}
return tab.Write(globalOptions.stdout)
return printKeys()
}
// testKeyNewPassword is used to set a new password during integration testing.
......@@ -160,7 +203,7 @@ func runKey(gopts GlobalOptions, args []string) error {
return err
}
return listKeys(ctx, repo)
return listKeys(ctx, repo, gopts)
case "add":
lock, err := lockRepo(repo)
defer unlockRepo(lock)
......
// +build !netbsd
// +build !openbsd
// +build !solaris
// +build !windows
......
......@@ -5,6 +5,7 @@ import (
"github.com/restic/restic/internal/errors"
"github.com/restic/restic/internal/filter"
"github.com/restic/restic/internal/restic"
"github.com/restic/restic/internal/restorer"
"github.com/spf13/cobra"
)
......@@ -33,6 +34,7 @@ type RestoreOptions struct {
Host string
Paths []string
Tags restic.TagLists
Verify bool
}
var restoreOptions RestoreOptions
......@@ -48,6 +50,7 @@ func init() {
flags.StringVarP(&restoreOptions.Host, "host", "H", "", `only consider snapshots for this host when the snapshot ID is "latest"`)
flags.Var(&restoreOptions.Tags, "tag", "only consider snapshots which include this `taglist` for snapshot ID \"latest\"")
flags.StringArrayVar(&restoreOptions.Paths, "path", nil, "only consider snapshots which include this (absolute) `path` for snapshot ID \"latest\"")
flags.BoolVar(&restoreOptions.Verify, "verify", false, "verify restored files content")
}
func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
......@@ -104,7 +107,7 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
}
}
res, err := restic.NewRestorer(repo, id)
res, err := restorer.NewRestorer(repo, id)
if err != nil {
Exitf(2, "creating restorer failed: %v\n", err)
}
......@@ -153,6 +156,12 @@ func runRestore(opts RestoreOptions, gopts GlobalOptions, args []string) error {
Verbosef("restoring %s to %s\n", res.Snapshot(), opts.Target)
err = res.RestoreTo(ctx, opts.Target)
if err == nil && opts.Verify {
Verbosef("verifying files in %s\n", opts.Target)
var count int
count, err = res.VerifyFiles(ctx, opts.Target)
Verbosef("finished verifying %d files in %s\n", count, opts.Target)
}
if totalErrors > 0 {