Bees v0.7
This is a long overdue maintenance release collecting some years of
bug fixes. There are no bees metadata format changes in this release.
Highlights:
* Remove 8-CPU thread limit
* Add kernel bugs reference table to docs
* Workarounds for btrfs send and balance issues
* Reduce the number of temporary inodes created
* Use posix_fadvise to optimize page cache usage
* Use private namespace for mounts under systemd
* Assorted bug fixes and small performance improvements
* SIGTERM handler to save crawl state, hash table, and exit
* Higher ref limits per extent on kernels with LOGICAL_INO_V2
Build dependency changes:
* Convert docs to Github Flavored Markdown
* Updates for new compilers including clang
* Remove dependencies on libbtrfs-dev and uuid-dev
* Remove unversioned `libcrucible.so` shared library
Shortlog:
Andrey Brusnik (1):
fs: Change array syntax to pointer syntax
Jiahao XU (7):
Add new options MOUNT_OPTIONS
Modify systemd unit and beesd.in to use private mnt namespace
Further sandbox beesd using systemd.exec options
Update comment in beesd@.service.in
Fix typo when setting default val of MOUNT_OPTIONS in beesd.in
Update default MOUNT_OPTIONS beesd.in
Rm MOUNT_OPTIONS for it is of no use and dangerous
Kai Krakow (9):
Update references to Gentoo
Makefile: Specify version when building from tarball
Makefile: Use the jobserver properly
Makefile: mkdir .depends only when needed
Makefile: Bring back -O3 in a downstream-compatible way
crucible: Try repairing a build failure around swap macro
Makefile: Fix git usage for non-git source archive
bees-context: Remove confusing log message
bees: Avoid unused result with -Werror=unused-result
SeerLite (1):
install.md: Update Arch Linux instructions
Zygo Blaxell (168):
README: split into sections, reformat for github.io
Merge remote-tracking branch 'nilninull/master'
docs: add "what to do when something goes wrong" page
docs: add coredumpctl
src: add bees-version.new.c to .gitignore
hash: reduce hash table extent size to 128KB
scripts: use multiples (not power) of 128K
hash: remove pointless copy
roots: do not allow transid_min to be numeric_limits<uint64_t>::max()
roots: do not accept 18446744073709551615 as max_transid in beescrawl.dat
roots: fix subvol scan rollover on subvols with empty transid range
context: serialize LOGICAL_INO calls
bees: drop unused member m_uuid
context: cache result of home_fd()
roots: simplify BeesRoots::transid_max_nocache
Revert "roots: simplify BeesRoots::transid_max_nocache"
roots: reimplement transid_max_nocache using extent tree root
context: better detection for toxic extents
scripts: put AL16M back to avoid breaking existing scripts
hash: remove preloaded toxic hash blacklist
context: remove limit on the number of references to an extent
fs: support LOGICAL_INO_V2
docs: toxic extents and btrfs send
stats: streamline add_count
fs: remove thread_local storage
fs: if search fails, return empty result set
resolver: don't log hash collision incidents
workarounds: add workaround for btrfs send
docs: reorganize options, add workaround for btrfs send
bees: soft-limit computed thread counts to 8
docs: working with `btrfs send` is kind of a feature
main: single BeesContext instance per process
roots: improve "RO root 6094" message
docs: derive docs/index.md from README.md
README: reintroduce new btrfs-send-compatibility workaround
docs: add instructions for Ubuntu 18.10
docs: use bash "type -p" because dash isn't useful
docs: dash more useful than previously believed
roots: quick fix for task scheduling bug leading to loss of crawl_master
tempfile: drop the fsync()
task: add cancel method
process: ntoa function for signals
time: separate sleep time calculation from sleep_for method
bees: handle SIGTERM and SIGINT, force immediate flush and exit
hash: clean up comments, audit for bugs
build: make libcrucible a static library
docs: tested with GCC 6.3.0
docs: bees can stop now
process: SIGUNUSED is deprecated
bees: make exceptions less prominent in log output
docs: describe expected exceptions and impact of exception handling
docs: add Gotcha for SIGTERM
docs: add some notes about interactions with balance
task: queue and run exactly once per instance
docs: event counter documentation
status: report number of active worker threads in status output
docs: update kernel compatibility page, now recommending 5.0.4
README: highlight DATA CORRUPTION WARNING
docs: update btrfs feature interaction status for flushoncommit and SSD caching layers
docs: tested build with btrfs-progs 4.20.2
bees: don't try to print si_lower and si_upper
BtrfsExtentWalker: use a buffer at least as large as a btrfs metadata page to avoid EOVERFLOW
fs: do not emulate extent-same by clone
lib: fix non-local lambda expression cannot have a capture-default
lib: add cityhash function
hash: prepare for user-selectable hash functions
process: Fix gettid() ambiguity with glibc >= 2.30
context: workaround to prevent LOGICAL_INO and btrfs balance from running concurrently
docs: update known kernel bugs list
bees: initialize context in the correct order
bees: replace uncaught_exception(), deprecated in C++17
docs: use Github Flavored Markdown with table extension
docs: update kernel bug tracking for October 2020
docs: fix table formatting for kernel bugs list
docs: improve send workaround text, add references to backref commits, make grammar more good now
docs: expand the tree mod log issues
docs: btrfs-kernel: 4.20 adds 32-bit single convert bug, tree mod log issue #4
stats: remove nonsense dedup_unique_bytes stat
task: make it build with clang
process: make it build with clang
bees: make it build with clang
bees context: make it build with clang
extentwalker: make it build with clang
clang: fix struct/class declaration/definition mismatches
chatter: make it build with clang
roots: make it build with clang
bees: move usage message out of source file and fix a few inaccuracies
roots: report the search parameters on tree search ioctl error
roots: separate crawl sizes into bytes and items
fs: always use container's actual size not requested size
fs: make operator<() for search ioctl inline
tempfile: remove old comments about fsync and deadlock bugs
context: move prealloc dedupe to a separate Task
fs: don't zero-fill btrfs data containers
string: second argument to stoull is technically a nullptr
lib: introduce Pool, a class for storing reusable anonymous objects
context: move TempFile from TLS to Pool and fix some FdCache issues
tempfile: remove size limit in realign()
context: fix shutdown log messages identifying the wrong thread
include: #undef crc32c
lib: don't rebuild libcrucible unless there is a version change
test: rebuild the tests if libcrucible.a changes
cache: clean up pointer mangling and duplicate code
cache: remove unused #includes
fd: move relative path string to library
lib: namedptr: thread-safe reference counted named object store
src: use correct flags for compiling .c files, fix missing dependencies
fd: deprecate Resource in favor of NamedPtr
fs: add support and workarounds for btrfs fs_info v2
fs: deprecate vector<char>
fs: remove buffer overrun check in get_struct_ptr for non-copying containers
lib: introduce Spanner, a pointer and size delimiting a range
fs: use Spanner to refer to ioctl arg buffer instead of making vector copies
resolve: add bees.h constants for balance and logical_ino serialization
bees: remove si_addr_lsb from siginfo debug message to fix FTBFS
build: include localconf everywhere
lib: fs: stop using libbtrfs-dev helper functions to re-enable buffer length checks
docs: remove libbtrfs-dev as a build-time dependency
docs: btrfs-kernel: add the 5.10 performance regression, the Ctrl-C on balance kernel crash has been fixed
docs: btrfs-kernel: update recommended kernels list, slow backrefs bug has been backported
uuid: drop dependency on uuid.h
docs: drop incomplete build recipe for ubuntu 14.04
docs: note that FIEMAP is also affected by backref performance issue
ntoa: fix bits_ntoa formatting and error handling
ntoa: fix comment disparaging gcc for not implementing C99 compound literals in C++
context: get rid of all instances of pthread_cancel
context: get rid of shared_ptr<BeesContext> in every single cached Fd object
src: bees depends on libcrucible.a
process: SIGCLD is not portable
options: remove default 8 CPU thread limit
pool: use weak_ptr to run destructor earlier
fd: make the close method on IOHandle private
crucible: use '#include "crucible/...' everywhere
test: fd: note when bad cast exception is expected
chatter: add option to remove log level prefix
docs: finally concede that the consensus spelling is "dedupe"
docs: btrfs-kernel: add the extent ref hash bug
task: serialize Task execution when Tasks block due to mutex contention
task: track number of Task objects in program and provide report
task: replace waiting state with run/exec counter
task: handle thread lifecycle more strictly
context: report Task instance count
roots: clean up crawl_master
cache: emit log messages when clearing FD cache
bees: use helper function for readahead
bees: misc comment updates
context: track record extent reference counts
roots: split constructor into separate start method
context: remove unnecessary copies
bees: use a reserved symbol name in BEESLOG
bees: increase StringFile size limit
bees: trace and log improvements during roots and context startup
roots: add a TRACE for transid_max search and crawl_transid thread
docs: update kernel bugs table as of 5.12.3
tracer: annotate both ends of the stack trace
extentwalker: fix the hole position logic
extentwalker: fix missing characters
extentwalker: fix the binary search and add some debug infrastructure
trace: move BeesTrace and BeesNote into their own translation unit
trace: current_exception() is not a replacement for uncaught_exception()
task: set the name of consumer threads so it is not "load_tracker"
context: stop creating new refs when there are too many already
fiemap: don't force flush so we can see the delalloc shenanigans
context: calculate TOTAL RATES correctly
fs: avoid unaligned access when copying btrfs search headers
bees: readahead() in the kernel is posix_fadvise(..., POSIX_FADV_WILLNEED)
hash: use POSIX_FADV_WILLNEED and POSIX_FADV_DONTNEED
docs: add `readahead_` event group
nilninull (1):
FIX: The systemd service file is always installed
rsjaffe (1):
systemd service replace deprecated parameters