New upstream version 1.3.8+dfsg

# Windows

*.bat text eol=crlf

*.cmd text eol=crlf

# .travis.yml merging

.travis.yml merge=ours

v1.3.8

perf: better decompression speed on large files (+7%) and cold dictionaries (+15%)

perf: slightly better compression ratio at high compression modes

api : finalized advanced API, last stage before "stable" status

api : new --rsyncable mode, by @terrelln

api : support decompression of empty frames into NULL (used to be an error) (#1385)

build: new set of macros to build a minimal size decoder, by @felixhandte

build: fix compilation on MIPS32, reported by @clbr (#1441)

build: fix compilation with multiple -arch flags, by @ryandesign

build: highly upgraded meson build, by @lzutao

build: improved buck support, by @obelisk

build: fix cmake script : can create debug build, by @pitrou

build: Makefile : grep works on both colored consoles and systems without color support

build: fixed zstd-pgo, by @bmwiedemann

cli : support ZSTD_CLEVEL environment variable, by @yijinfb (#1423)

cli : --no-progress flag, preserving final summary (#1371), by @terrelln

cli : ensure destination file is not source file (#1422)

cli : clearer error messages, especially when input file not present

doc : clarified zstd_compression_format.md, by @ulikunitz

misc: fixed zstdgrep, returns 1 on failure, by @lzutao

misc: NEWS renamed as CHANGELOG, in accordance with fboss

v1.3.7

perf: slightly better decompression speed on clang (depending on hardware target)

fix : performance of dictionary compression for small input < 4 KB at levels 9 and 10

build: no longer build backtrace by default in release mode; restrict further automatic mode

build: control backtrace support through build macro BACKTRACE

misc: added man pages for zstdless and zstdgrep, by @samrussell

v1.3.6

perf: much faster dictionary builder, by @jenniferliu

perf: faster dictionary compression on small data when using multiple contexts, by @felixhandte

perf: faster dictionary decompression when using a very large number of dictionaries simultaneously

cli : fix : does no longer overwrite destination when source does not exist (#1082)

cli : new command --adapt, for automatic compression level adaptation

api : fix : block api can be streamed with > 4 GB, reported by @catid

api : reduced ZSTD_DDict size by 2 KB

api : minimum negative compression level is defined, and can be queried using ZSTD_minCLevel().

build: support Haiku target, by @korli

build: Read Legacy format is limited to v0.5+ by default. Can be changed at compile time with macro ZSTD_LEGACY_SUPPORT.

doc : zstd_compression_format.md updated to match wording in IETF RFC 8478

misc: tests/paramgrill, a parameter optimizer, by @GeorgeLu97

v1.3.5

perf: much faster dictionary compression, by @felixhandte

perf: small quality improvement for dictionary generation, by @terrelln

# Code of Conduct

Facebook has adopted a Code of Conduct that we expect project participants to adhere to.

Please read the [full text](https://code.fb.com/codeofconduct/)

so that you can understand what actions will and will not be tolerated.

EXT =

endif

## default: Build lib-release and zstd-release

.PHONY: default

default: lib-release zstd-release

all: allmost examples manual contrib

.PHONY: allmost

allmost: allzstd

	$(MAKE) -C $(ZWRAPDIR) all

allmost: allzstd zlibwrapper

# skip zwrapper, can't build that on alternate architectures without the proper zlib installed

.PHONY: allzstd

	$(MAKE) -C $(PRGDIR) zstd32

	$(MAKE) -C $(TESTDIR) all32

.PHONY: lib lib-release

.PHONY: lib lib-release libzstd.a

lib lib-release :

	@$(MAKE) -C $(ZSTDDIR) $@

	cp $(PRGDIR)/zstd$(EXT) ./zstdmt$(EXT)

.PHONY: zlibwrapper

zlibwrapper:

	$(MAKE) -C $(ZWRAPDIR) test

zlibwrapper: lib

	$(MAKE) -C $(ZWRAPDIR) all

## test: run long-duration tests

.PHONY: test

DEBUGLEVEL ?= 1

test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror

test:

	$(MAKE) -C $(PRGDIR) allVariants MOREFLAGS+="-g -DDEBUGLEVEL=1"

	MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants

	$(MAKE) -C $(TESTDIR) $@

## shortest: same as `make check`

.PHONY: shortest

shortest:

	$(MAKE) -C $(TESTDIR) $@

## check: run basic tests for `zstd` cli

.PHONY: check

check: shortest

## examples: build all examples in `/examples` directory

.PHONY: examples

examples: lib

	CPPFLAGS=-I../lib LDFLAGS=-L../lib $(MAKE) -C examples/ all

## manual: generate API documentation in html format

.PHONY: manual

manual:

	$(MAKE) -C contrib/gen_html $@

## man: generate man page

.PHONY: man

man:

	$(MAKE) -C programs $@

## contrib: build all supported projects in `/contrib` directory

.PHONY: contrib

contrib: lib

	$(MAKE) -C contrib/pzstd all

	$(MAKE) -C contrib/seekable_format/examples all

	$(MAKE) -C contrib/adaptive-compression all

	$(MAKE) -C contrib/largeNbDicts all

.PHONY: cleanTabs

cleanTabs:

	@$(MAKE) -C contrib/pzstd $@ > $(VOID)

	@$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID)

	@$(MAKE) -C contrib/adaptive-compression $@ > $(VOID)

	@$(MAKE) -C contrib/largeNbDicts $@ > $(VOID)

	@$(RM) zstd$(EXT) zstdmt$(EXT) tmp*

	@$(RM) -r lz4

	@echo Cleaning completed

#------------------------------------------------------------------------------

# make install is validated only for Linux, macOS, Hurd and some BSD targets

#------------------------------------------------------------------------------

ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD DragonFly NetBSD MSYS_NT))

ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD DragonFly NetBSD MSYS_NT Haiku))

HOST_OS = POSIX

CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON -DZSTD_LZMA_SUPPORT:BOOL=ON

CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON -DZSTD_LZMA_SUPPORT:BOOL=ON -DCMAKE_BUILD_TYPE=Release

HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0)

EGREP_OPTIONS ?=

ifeq ($HAVE_COLORNEVER, 1)

EGREP_OPTIONS += --color=never

endif

EGREP = egrep $(EGREP_OPTIONS)

# Print a two column output of targets and their description. To add a target description, put a

# comment in the Makefile with the format "## <TARGET>: <DESCRIPTION>".  For example:

#

## list: Print all targets and their descriptions (if provided)

.PHONY: list

list:

	@$(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^$@$$' | xargs

	@TARGETS=$$($(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null \

		| awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \

		| $(EGREP) -v  -e '^[^[:alnum:]]' | sort); \

	{ \

	    printf "Target Name\tDescription\n"; \

	    printf "%0.s-" {1..16}; printf "\t"; printf "%0.s-" {1..40}; printf "\n"; \

	    for target in $$TARGETS; do \

	        line=$$($(EGREP) "^##[[:space:]]+$$target:" $(lastword $(MAKEFILE_LIST))); \

	        description=$$(echo $$line | awk '{i=index($$0,":"); print substr($$0,i+1)}' | xargs); \

	        printf "$$target\t$$description\n"; \

	    done \

	} | column -t -s $$'\t'

.PHONY: install clangtest armtest usan asan uasan

install:

clangtest: clean

	clang -v

	$(MAKE) all CXX=clang-++ CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation"

	$(MAKE) all CXX=clang++ CC=clang MOREFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation"

armtest: clean

	$(MAKE) -C $(TESTDIR) datagen   # use native, faster

gcc7install: apt-add-repo

	APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-7 gcc-7-multilib" $(MAKE) apt-install

gcc8install: apt-add-repo

	APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-8 gcc-8-multilib" $(MAKE) apt-install

gpp6install: apt-add-repo

	APT_PACKAGES="libc6-dev-i386 g++-multilib gcc-6 g++-6 g++-6-multilib" $(MAKE) apt-install

c90build: clean

	$(CC) -v

	CFLAGS="-std=c90" $(MAKE) allmost  # will fail, due to missing support for `long long`

	CFLAGS="-std=c90 -Werror" $(MAKE) allmost  # will fail, due to missing support for `long long`

gnu90build: clean

	$(CC) -v

	CFLAGS="-std=gnu90" $(MAKE) allmost

	CFLAGS="-std=gnu90 -Werror" $(MAKE) allmost

c99build: clean

	$(CC) -v

	CFLAGS="-std=c99" $(MAKE) allmost

	CFLAGS="-std=c99 -Werror" $(MAKE) allmost

gnu99build: clean

	$(CC) -v

	CFLAGS="-std=gnu99" $(MAKE) allmost

	CFLAGS="-std=gnu99 -Werror" $(MAKE) allmost

c11build: clean

	$(CC) -v

	CFLAGS="-std=c11" $(MAKE) allmost

	CFLAGS="-std=c11 -Werror" $(MAKE) allmost

bmix64build: clean

	$(CC) -v

	$(CC) -v

	CFLAGS="-O3 -mbmi -m32 -Werror" $(MAKE) -C $(TESTDIR) test

staticAnalyze: clean

# static analyzer test uses clang's scan-build

# does not analyze zlibWrapper, due to detected issues in zlib source code

staticAnalyze: SCANBUILD ?= scan-build

staticAnalyze:

	$(CC) -v

	CPPFLAGS=-g scan-build --status-bugs -v $(MAKE) all

	CC=$(CC) CPPFLAGS=-g $(SCANBUILD) --status-bugs -v $(MAKE) allzstd examples contrib

endif

Should your project require another programming language,

a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages).

Development branch status : [![Build Status][travisDevBadge]][travisLink]   [![Build status][AppveyorDevBadge]][AppveyorLink]   [![Build status][CircleDevBadge]][CircleLink]

**Development branch status:**

[![Build Status][travisDevBadge]][travisLink]

[![Build status][AppveyorDevBadge]][AppveyorLink]

[![Build status][CircleDevBadge]][CircleLink]

[travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite"

[travisLink]: https://travis-ci.org/facebook/zstd

[CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite"

[CircleLink]: https://circleci.com/gh/facebook/zstd

### Benchmarks

## Benchmarks

For reference, several fast compression algorithms were tested and compared

on a server running Linux Debian (`Linux version 4.14.0-3-amd64`),

For a larger picture including slow modes, [click on this link](doc/images/DCspeed5.png).

### The case for Small Data compression

## The case for Small Data compression

Previous charts provide results applicable to typical file and stream scenarios (several MB). Small data comes with different perspectives.

Hence, deploying one dictionary per type of data will provide the greatest benefits.

Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file.

#### Dictionary compression How To:

### Dictionary compression How To:

1) Create the dictionary

1. Create the dictionary

   `zstd --train FullPathToTrainingSet/* -o dictionaryName`

2) Compress with dictionary

2. Compress with dictionary

   `zstd -D dictionaryName FILE`

3) Decompress with dictionary

3. Decompress with dictionary

   `zstd -D dictionaryName --decompress FILE.zst`

### Build instructions

## Build instructions

#### Makefile

### Makefile

If your system is compatible with standard `make` (or `gmake`),

invoking `make` in root directory will generate `zstd` cli in root directory.

- `make install` : create and install zstd cli, library and man pages

- `make check` : create and run `zstd`, tests its behavior on local platform

#### cmake

### cmake

A `cmake` project generator is provided within `build/cmake`.

It can generate Makefiles or other build scripts

to create `zstd` binary, and `libzstd` dynamic and static libraries.

#### Meson

By default, `CMAKE_BUILD_TYPE` is set to `Release`.

### Meson

A Meson project is provided within [`build/meson`](build/meson). Follow

build instructions in that directory.

A Meson project is provided within `contrib/meson`.

You can also take a look at [`.travis.yml`](.travis.yml) file for an

example about how Meson is used to build this project.

#### Visual Studio (Windows)

Note that default build type is **release**.

### Visual Studio (Windows)

Going into `build` directory, you will find additional possibilities:

- Projects for Visual Studio 2005, 2008 and 2010.

- Automated build scripts for Visual compiler by [@KrzysFR](https://github.com/KrzysFR), in `build/VS_scripts`,

  which will build `zstd` cli and `libzstd` library without any need to open Visual Studio solution.

### Buck

You can build the zstd binary via buck by executing: `buck build programs:zstd` from the root of the repo.

The output binary will be in `buck-out/gen/programs/`.

### Status

## Status

Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases.

Zstandard is considered safe for production environments.

### License

## License

Zstandard is dual-licensed under [BSD](LICENSE) and [GPLv2](COPYING).

### Contributing

## Contributing

The "dev" branch is the one where all contributions are merged before reaching "master".

If you plan to propose a patch, please commit into the "dev" branch, or its own feature branch.