Commit 856a0469 authored by Joao Eriberto Mota Filho's avatar Joao Eriberto Mota Filho

Import Debian changes 1.10.2-1

jdupes (1.10.2-1) unstable; urgency=medium

  * New upstream release.
  * Migrated DH level to 11.
  * debian/clean: no longer needed. Removed.
  * debian/control:
      - Bumped Standards-Version to 4.2.0.
      - Changed Vcs-* URLs to salsa.debian.org.
  * debian/copyright:
      - Added rights for Yann Collet.
      - Updated upstream and packaging years.
  * debian/docs:
      - Added OLD_CHANGES and OLD_CONTRIBUTORS.
      - Removed no longer existent TODO.
  * debian/patches/10-disable_test.patch: created to disable a wrong test.
  * debian/tests: created to perform tests.
parents 5b893f1a 7ae2b816
......@@ -20,6 +20,8 @@
/jdupes
/jdupes*.exe
/*.pkg.tar.xz
test_temp
output.log
#
# Backups / patches
......
jdupes 1.10.2
- Add -P/--print option
jdupes 1.10.1
- Fix -I option
jdupes 1.10
- cacheinfo code not included on Windows where it is not used
- Fix -H to work properly on individual files (not just directories)
- Fix memory corruption which causes a crash when using -A option
- Block btrfs dedupe on Linux kernels < 3.0 due to possible data loss bugs
- Removed all references to 'fdupes-jody' and unused TODO file
- Add -C/--chunksize option for tuning I/O chunk size (see README)
- Make more features configurable and exclude them in LOW_MEMORY mode
- Remove HAVE_BTRFS_IOCTL_H deprecated compile-time option
- Remove experimental tree rebalance code
jdupes 1.9
- stderr on Windows is no longer polluted or empty when redirected
......
Installing jdupes
--------------------------------------------------------------------
To install the program, issue the following commands:
-----------------------------------------------------------------------------
To install the program with the default options and flags, just issue the
following commands (note that btrfs support is off by default):
make
su root
make install
This will install the program in /usr/bin. You may change this to a
different location by editing the Makefile. Please refer to the
Makefile for an explanation of compile-time options. If you're having
trouble compiling, please take a look at the Makefile.
This will install the program in /usr/bin. You may change this to a different
location by editing the Makefile. Please refer to the Makefile for an
explanation of compile-time options. If you're having trouble compiling,
please take a look at the Makefile.
Various build options are available and can be turned on at compile
time by setting CFLAGS_EXTRA or by passing it to 'make':
Various build options are available and can be turned on at compile time by
setting CFLAGS_EXTRA or by passing it to 'make':
make CFLAGS_EXTRA=-DYOUR_OPTION
make CFLAGS_EXTRA='-DYOUR_OPTION_ONE -DYOUR_OPTION_TWO'
......@@ -21,38 +22,37 @@ This is a list of options that can be "turned on" this way:
OMIT_GETOPT_LONG Do not use getopt_long() C library call
ON_WINDOWS Modify code to compile with MinGW on Windows
USE_TREE_REBALANCE * Use experimental tree rebalancing code
CONSIDER_IMBALANCE * Change tree rebalance to analyze weights first
* These options may slow down the program somewhat and are off by
default. Do not enable them unless you are experimenting.
Certain options need to be turned on by setting a variable passed to
make instead of using CFLAGS_EXTRA, i.e. 'make DEBUG=1':
Certain options need to be turned on by setting a variable passed to make
instead of using CFLAGS_EXTRA, i.e. 'make DEBUG=1':
DEBUG Turn on algorithm statistic reporting with '-D'
LOUD '-@' for low-level debugging; enables DEBUG
ENABLE_BTRFS Enable '-B/--dedupe' for btrfs deduplication
LOW_MEMORY Build for lower memory usage instead of speed
The LOW_MEMORY option tweaks various knobs in the program to lower
total memory usage. It also disables some features to reduce the size
of certain data structures. The improvements in memory usage are not
very large, but if you're running in a very RAM-limited environment
or have a CPU with very small caches it may be the best choice.
A test directory is included so that you may familiarize yourself with
the way jdupes operates. You may test the program before installing
it by issuing a command such as "./jdupes testdir" or
"./jdupes -r testdir", just to name a couple of examples. Refer to the
documentation for information on valid options.
NO_PERMS Disable permission options and code
NO_HARDLINKS Disable hard linking options and code
NO_SYMLINKS Disable symbolic linking options and code
NO_USER_ORDER Disable -I/-O options and code
NO_UNICODE [Windows only] disable all Unicode support
The LOW_MEMORY option tweaks various knobs in the program to lower total
memory usage. It also disables some features to reduce the size of certain
data structures. The improvements in memory usage are not very large, but if
you're running in a very RAM-limited environment or have a CPU with very small
caches it may be a good choice.
A test directory is included so that you may familiarize yourself with the way
jdupes operates. You may test the program before installing it by issuing a
command such as "./jdupes testdir" or "./jdupes -r testdir", just to name a
couple of examples. See the README for information on valid options.
A comparison shell script is also included. It will run your natively
installed 'jdupes' or 'jdupes' with the directories and extra options
you specify and compare the run times and output a 'diff' of the two
program outputs. Unless the core algorithm or sort behavior is changed,
both programs should produce identical outputs and the 'diff' output
shouldn't appear at all. To use it, type:
installed 'jdupes' or 'jdupes' with the directories and extra options you
specify and compare the run times and output a 'diff' of the two program
outputs. Unless the core algorithm or sort behavior is changed, both programs
should produce identical outputs and the 'diff' output shouldn't appear at
all. To use it, type:
./compare_jdupes.sh [options]
The MIT License (MIT)
Copyright (C) 2015-2017 Jody Lee Bruchon and contributors
Copyright (C) 2015-2018 Jody Lee Bruchon and contributors
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
......
......@@ -85,15 +85,20 @@ ifndef NO_UNICODE
UNICODE=1
COMPILER_OPTIONS += -municode
endif
COMPILER_OPTIONS += -D__USE_MINGW_ANSI_STDIO=1
COMPILER_OPTIONS += -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1
OBJS += win_stat.o
override undefine ENABLE_BTRFS
override undefine HAVE_BTRFS_IOCTL_H
endif
# Remap old BTRFS support option to new name
ifdef HAVE_BTRFS_IOCTL_H
ENABLE_BTRFS=1
# xxHash support
ifdef USE_XXHASH
COMPILER_OPTIONS += -DUSE_HASH_XXHASH64
OBJS += xxhash.o
OBJS_CLEAN += jody_hash.o
else
COMPILER_OPTIONS += -DUSE_HASH_JODYHASH
OBJS += jody_hash.o
OBJS_CLEAN += xxhash.o
endif
# New BTRFS support option
ifdef ENABLE_BTRFS
......@@ -104,7 +109,7 @@ OBJS_CLEAN += act_dedupefiles.o
endif
# Low memory mode
ifdef LOW_MEMORY
COMPILER_OPTIONS += -DLOW_MEMORY -DJODY_HASH_WIDTH=32 -DSMA_PAGE_SIZE=32768
COMPILER_OPTIONS += -DLOW_MEMORY -DJODY_HASH_WIDTH=32 -DSMA_PAGE_SIZE=32768 -DCHUNK_SIZE=16384 -DNO_HARDLINKS -DNO_USER_ORDER
endif
CFLAGS += $(COMPILER_OPTIONS) $(CFLAGS_EXTRA)
......@@ -116,7 +121,7 @@ INSTALL_DATA = $(INSTALL) -m 0644
# to support features not supplied by their vendor. Eg: GNU getopt()
#ADDITIONAL_OBJECTS += getopt.o
OBJS += jdupes.o jody_hash.o jody_paths.o jody_sort.o jody_win_unicode.o string_malloc.o
OBJS += jdupes.o jody_paths.o jody_sort.o jody_win_unicode.o string_malloc.o
OBJS += jody_cacheinfo.o
OBJS += act_deletefiles.o act_linkfiles.o act_printmatches.o act_summarize.o
OBJS += $(ADDITIONAL_OBJECTS)
......@@ -134,6 +139,8 @@ install: jdupes installdirs
$(INSTALL_PROGRAM) $(PROGRAM_NAME) $(DESTDIR)$(BIN_DIR)/$(PROGRAM_NAME)
$(INSTALL_DATA) $(PROGRAM_NAME).1 $(DESTDIR)$(MAN_DIR)/$(PROGRAM_NAME).$(MAN_EXT)
test:
./test.sh
clean:
$(RM) $(OBJS) $(OBJS_CLEAN) $(PROGRAM_NAME) $(PROGRAM_NAME).exe *~ *.gcno *.gcda *.gcov
......
......@@ -73,6 +73,7 @@ Usage: jdupes [options] DIRECTORY...
-1 --one-file-system do not match files on different filesystems/devices
-A --nohidden exclude hidden files from consideration
-B --dedupe Send matches to btrfs for block-level deduplication
-C --chunksize=# override I/O chunk size (min 4096, max 16777216)
-d --delete prompt user for files to preserve and delete all
others; important: under particular circumstances,
data may be lost when using this option together
......@@ -97,6 +98,7 @@ Usage: jdupes [options] DIRECTORY...
mtime (BY=time) or filename (BY=name, the default)
-p --permissions don't consider files with different owner/group or
permission bits as duplicates
-P --print=type print extra info (partial, early, fullhash)
-Q --quick skip byte-for-byte confirmation for quick matching
WARNING: -Q can result in data loss! Be very careful!
-r --recurse for every directory, process its subdirectories too
......@@ -110,7 +112,7 @@ Usage: jdupes [options] DIRECTORY...
-x --xsize=SIZE exclude files of size < SIZE bytes from consideration
--xsize=+SIZE '+' specified before SIZE, exclude size > SIZE
-X --exclude=spec:info exclude files based on specified criteria
specs: dir size+-=
specs: size+-=
Exclusions are cumulative: -X dir:abc -X dir:efg
-z --zeromatch consider zero-length files to be duplicates
-Z --softabort If the user aborts (i.e. CTRL-C) act on matches so far
......@@ -143,6 +145,27 @@ underlying nature of the jdupes algorithm, a lot of matches will be
blocked by this option that probably should not be. This code could use
improvement.
The -C/--chunksize option overrides the size of the I/O "chunk" used for all
file operations. Larger numbers will increase the amount of data read at
once from each file and may improve performance when scanning lots of files
that are larger than the default chunk size by reducing "thrashing" of the
hard disk heads. Smaller numbers may increase algorithm speed depending on
the characteristics of your CPU but will usually increase I/O and system
call overhead as well. The nubmer also directly affects memory usage: I/O
chunk size is used for at least three allocations in the program, so using
a chunk size of 16777216 (16 MiB) will require 48 MiB of RAM. The default
is usually between 32768 and 65536 which results in the fastest raw speed
of the algorithm and generally good all-around performance. Feel free to
experiment with the number on your data set and report your experiences
(preferably with benchmarks and info on your data set.)
Using -P/--print will cause the program to print extra information that
may be useful but will pollute the output in a way that makes scripted
handling difficult. Its current purpose is to reveal more information about
the file matching process by printing match pairs that pass certain steps
of the process prior to full file comparison. This can be useful if you
have two files that are passing early checks but failing after full checks.
Hard and soft (symbolic) linking status symbols and behavior
--------------------------------------------------------------------------
......@@ -305,9 +328,9 @@ Please DO NOT contact Adrian Lopez about issues with jdupes.
Legal Information and Software License
--------------------------------------------------------------------------
jdupes is Copyright (C) 2015-2017 by Jody Bruchon <jody@jodybruchon.com>
Derived from the original 'fdupes' (C) 1999-2017 by Adrian Lopez
Includes other code libraries which are (C) 2015-2017 by Jody Bruchon
jdupes is Copyright (C) 2015-2018 by Jody Bruchon <jody@jodybruchon.com>
Derived from the original 'fdupes' (C) 1999-2018 by Adrian Lopez
Includes other code libraries which are (C) 2015-2018 by Jody Bruchon
The MIT License
......
- Option -R should not have to be separated from the rest,
such that "jdupes -dR testdir", "jdupes -d -R testdir",
"jdupes -Rd testdir", etc., all yield the same results.
- Add option to highlight or identify symlinked files (suggest
using --classify to identify symlinks with @ suffix... when
specified, files containing @ are listed using \@).
- Consider option to match only to files in specific directory.
- Add a way to store stat() info + inital and full hashes for
explicit loading in future runs to speed up repeated calls to
the program.
......@@ -14,6 +14,7 @@
#include <linux/btrfs.h>
#include <sys/ioctl.h>
#include <sys/utsname.h>
#include "act_dedupefiles.h"
/* Message to append to BTRFS warnings based on write permissions */
......@@ -39,6 +40,7 @@ static char *dedupeerrstr(int err) {
extern void dedupefiles(file_t * restrict files)
{
struct utsname utsname;
struct btrfs_ioctl_same_args *same;
char **dupe_filenames; /* maps to same->info indices */
......@@ -51,6 +53,17 @@ extern void dedupefiles(file_t * restrict files)
LOUD(fprintf(stderr, "\nRunning dedupefiles()\n");)
/* Refuse to dedupe on 2.x kernels; they could damage user data */
if (uname(&utsname)) {
fprintf(stderr, "Failed to get kernel version! Aborting.\n");
exit(EXIT_FAILURE);
}
LOUD(fprintf(stderr, "dedupefiles: uname got release '%s'\n", utsname.release));
if (*(utsname.release) == '2' && *(utsname.release + 1) == '.') {
fprintf(stderr, "Refusing to dedupe on a 2.x kernel; data loss could occur. Aborting.\n");
exit(EXIT_FAILURE);
}
/* Find the largest dupe set, alloc space to hold structs for it */
get_max_dupes(files, &max_dupes, &max_files);
/* Kernel dupe count is a uint16_t so exit if the type's limit is exceeded */
......@@ -116,7 +129,7 @@ extern void dedupefiles(file_t * restrict files)
n_dupes = cur_info;
same->logical_offset = 0;
same->length = (unsigned long)files->size;
same->length = (uint64_t)files->size;
same->dest_count = (uint16_t)n_dupes; /* kernel type is __u16 */
fd = open(files->d_name, O_RDONLY);
......
......@@ -8,11 +8,10 @@ ERR=0
# Detect installed program type (fdupes or jdupes)
ORIG_DUPE=false
jdupes -v 2>/dev/null >/dev/null && ORIG_DUPE=jdupes
fdupes-jody -v 2>/dev/null >/dev/null && ORIG_DUPE=fdupes-jody
fdupes -v 2>/dev/null >/dev/null && ORIG_DUPE=fdupes
test ! -z "$WINDIR" && "$WINDIR/jdupes.exe" -v 2>/dev/null >/dev/null && ORIG_DUPE="$WINDIR/jdupes.exe"
if [ ! $ORIG_DUPE -v 2>/dev/null >/dev/null ]
then echo "Cannot run installed jdupes, fdupes-jody, or fdupes"
then echo "Cannot run installed jdupes or fdupes"
exit 1
fi
......
jdupes (1.10.2-1) unstable; urgency=medium
* New upstream release.
* Migrated DH level to 11.
* debian/clean: no longer needed. Removed.
* debian/control:
- Bumped Standards-Version to 4.2.0.
- Changed Vcs-* URLs to salsa.debian.org.
* debian/copyright:
- Added rights for Yann Collet.
- Updated upstream and packaging years.
* debian/docs:
- Added OLD_CHANGES and OLD_CONTRIBUTORS.
- Removed no longer existent TODO.
* debian/patches/10-disable_test.patch: created to disable a wrong test.
* debian/tests: created to perform tests.
-- Joao Eriberto Mota Filho <eriberto@debian.org> Mon, 20 Aug 2018 22:22:01 -0300
jdupes (1.9-1) unstable; urgency=medium
* New upstream release.
......@@ -11,7 +30,7 @@ jdupes (1.8-1) unstable; urgency=medium
* debian/clean: created to remove a file forgotten by the upstream.
* debian/control: bumped Standards-Version to 4.0.0.
* debian/copyright:
- All upstream files are unde MIT license now.
- All upstream files are under MIT license now.
- Updated the upstream copyright years.
* debian/patches/10_fix-segfault.patch: removed. This patch was previously
supplied by the upstream and it is already in 1.8 source code.
......
act_dedupefiles.o
......@@ -2,11 +2,11 @@ Source: jdupes
Section: utils
Priority: optional
Maintainer: Joao Eriberto Mota Filho <eriberto@debian.org>
Build-Depends: debhelper (>= 10)
Standards-Version: 4.1.1
Build-Depends: debhelper (>= 11)
Standards-Version: 4.2.0
Homepage: https://github.com/jbruchon/jdupes
Vcs-Git: https://anonscm.debian.org/git/collab-maint/jdupes.git
Vcs-Browser: https://anonscm.debian.org/git/collab-maint/jdupes.git
Vcs-Browser: https://salsa.debian.org/debian/jdupes
Vcs-Git: https://salsa.debian.org/debian/jdupes.git
Package: jdupes
Architecture: any
......
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: jdupes
Upstream-Contact: Jody Bruchon <jody@jodybruchon.com>
Upstream-Contact: Jody Lee Bruchon <jody@jodybruchon.com>
Source: https://github.com/jbruchon/jdupes
Files: *
Copyright: 1999-2017 Adrian Lopez <adrian2@caribe.net>
2015-2017 Jody Bruchon <jody@jodybruchon.com>
Copyright: 1999-2018 Adrian Lopez <adrian2@caribe.net>
2014-2018 Jody Lee Bruchon <jody@jodybruchon.com>
License: MIT
Comment: jdupes is based in fdupes. Adrian is the fdupes upstream.
Files: xxhash.c
xxhash.h
Copyright: 2012-2016 Yann Collet <cyan@fb.com>
License: BSD-2-Clause
Files: debian/*
Copyright: 2016-2017 Joao Eriberto Mota Filho <eriberto@debian.org>
Copyright: 2016-2018 Joao Eriberto Mota Filho <eriberto@debian.org>
License: MIT
License: MIT
......@@ -31,3 +36,25 @@ License: MIT
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
License: BSD-2-Clause
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE HOLDERS OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
OLD_CHANGES
OLD_CONTRIBUTORS
README
TODO
Description: disable a wrong test.
Author: Joao Eriberto Mota Filho <eriberto@debian.org>
Last-Update: 2018-08-20
--- jdupes-1.10.2.orig/Makefile
+++ jdupes-1.10.2/Makefile
@@ -139,8 +139,8 @@ install: jdupes installdirs
$(INSTALL_PROGRAM) $(PROGRAM_NAME) $(DESTDIR)$(BIN_DIR)/$(PROGRAM_NAME)
$(INSTALL_DATA) $(PROGRAM_NAME).1 $(DESTDIR)$(MAN_DIR)/$(PROGRAM_NAME).$(MAN_EXT)
-test:
- ./test.sh
+#test:
+# ./test.sh
clean:
$(RM) $(OBJS) $(OBJS_CLEAN) $(PROGRAM_NAME) $(PROGRAM_NAME).exe *~ *.gcno *.gcda *.gcov
10-disable_test.patch
Test-Command: jdupes debian/tests/
Depends: @
.TH FDUPES 1
.TH JDUPES 1
.\" NAME should be all caps, SECTION should be 1-8, maybe w/ subsection
.\" other parms are allowed: see man(7), man(1)
.SH NAME
......@@ -32,6 +32,11 @@ issue the btrfs same-extents ioctl to trigger a deduplication on
disk. The program must be built with btrfs support for this option
to be available
.TP
.B -C --chunksize=\fIBYTES\fR
set the I/O chunk size manually; larger values may improve performance
on rotating media by reducing the number of head seeks required, but
also increases memory usage and can reduce performance in some cases
.TP
.B -D --debug
if this feature is compiled in, show debugging statistics and info
at the end of program execution
......@@ -87,6 +92,12 @@ name - sort by filename (default)
don't consider files with different owner/group or permission bits as
duplicates
.TP
.B -P --print=type
print extra information to stdout; valid options are:
early - matches that pass early size/permission/link/etc. checks
partial - files whose partial hashes match
fullhash - files whose full hashes match
.TP
.B -Q --quick
.B [WARNING: RISK OF DATA LOSS, SEE CAVEATS]
skip byte-for-byte verification of duplicate pairs (use hashes only)
......@@ -150,7 +161,7 @@ each link candidate. These arrows are as follows:
.TP
.B ---->
This file was successfully hard linked to the first file in the duplicate
This file was successfully hard linked to the first file in the duplicate
chain
.TP
.B -@@->
......@@ -238,6 +249,21 @@ set for which any amount of data loss is unacceptable. This option is not
included in the help text for the program due to its risky nature.
.B You have been warned!
Using the
.B \-C
or
.BR \-\-chunksize
option to override I/O chunk size can increase performance on rotating
storage media by reducing "head thrashing," reading larger amounts of
data sequentially from each file. This tunable size can have bad side
effects; the default size maximizes algorithmic performance without
regard to the I/O characteristics of any given device and uses a modest
amount of memory, but other values may greatly increase memory usage or
incur a lot more system call overhead. Try several different values to
see how they affect performance for your hardware and data set. This
option does not affect match results in any way, so even if it slows
down the file matching process it will not hurt anything.
.SH REPORTING BUGS
Send all bug reports to jody@jodybruchon.com or use the Issue tracker at
http://github.com/jbruchon/jdupes/issues
......@@ -246,8 +272,5 @@ http://github.com/jbruchon/jdupes/issues
jdupes is a fork of 'fdupes' which is maintained by and contains
extra code copyrighted by Jody Bruchon <jody@jodybruchon.com>
jdupes was once called 'fdupes-jody' but the name was changed at the
request of Adrian Lopez to avoid confusion between the two programs.
Based on 'fdupes' created by Adrian Lopez <adrian2@caribe.net>
This diff is collapsed.
......@@ -8,9 +8,23 @@
extern "C" {
#endif
/* Select hash algorithm */
//#define USE_HASH_JODYHASH /* jodyhash */
//#define USE_HASH_XXHASH64 /* xxHash64 */
/* Failsafes */
#if !defined USE_HASH_JODYHASH && !defined USE_HASH_XXHASH64
#define USE_HASH_JODYHASH
#endif
#if defined USE_HASH_JODYHASH && defined USE_HASH_XXHASH64
#error Multiple USE_HASH options
#endif
/* Detect Windows and modify as needed */
#if defined _WIN32 || defined __CYGWIN__
#define ON_WINDOWS 1
#ifndef ON_WINDOWS
#define ON_WINDOWS 1
#endif
#define NO_SYMLINKS 1
#define NO_PERMS 1
#define NO_SIGACTION 1
......@@ -29,16 +43,30 @@ extern "C" {
#include <sys/types.h>
#include <unistd.h>
#include "string_malloc.h"
#include "jody_hash.h"
#include "jody_sort.h"
#include "version.h"
/* Configure hash function based on choice above */
#if defined USE_HASH_JODYHASH
#define JODY_HASH_NOCOMPAT
#include "jody_hash.h"
#elif defined USE_HASH_XXHASH64
#include "xxhash.h"
#endif
/* Optional btrfs support */
#ifdef ENABLE_BTRFS
#include <sys/ioctl.h>
#include <linux/btrfs.h>
#endif
/* Set hash type (change this if swapping in a different hash function) */
#if defined USE_HASH_JODYHASH
typedef jodyhash_t jdupes_hash_t;
#elif defined USE_HASH_XXHASH64
typedef XXH64_hash_t jdupes_hash_t;
#endif
/* Some types are different on Windows */
#ifdef ON_WINDOWS
typedef uint64_t jdupes_ino_t;
......@@ -83,7 +111,6 @@ extern int err_mode;
/* Low memory option overrides */
#ifdef LOW_MEMORY
#undef USE_TREE_REBALANCE
#ifndef NO_PERMS
#define NO_PERMS 1
#endif
......@@ -150,6 +177,11 @@ extern uint_fast32_t flags;
#define F_HAS_DUPES 0x00000008U
#define F_IS_SYMLINK 0x00000010U
/* Extra print flags */
#define P_PARTIAL 0x00000001U
#define P_EARLYMATCH 0x00000002U
#define P_FULLHASH 0x00000004U
typedef enum {
ORDER_NAME = 0,
ORDER_TIME
......@@ -181,14 +213,12 @@ typedef struct _file {
jdupes_mode_t mode;
off_t size;
jdupes_ino_t inode;
hash_t filehash_partial;
hash_t filehash;
jdupes_hash_t filehash_partial;
jdupes_hash_t filehash;
time_t mtime;
uint32_t flags; /* Status flags */
#ifndef NO_USER_ORDER
unsigned int user_order; /* Order of the originating command-line parameter */
#ifndef NO_PERMS
uid_t uid;
gid_t gid;
#endif
#ifndef NO_HARDLINKS
#ifndef ON_WINDOWS
......@@ -197,17 +227,16 @@ typedef struct _file {
uint32_t nlink; /* link count on Windows is always a DWORD */
#endif
#endif
#ifndef NO_PERMS
uid_t uid;
gid_t gid;
#endif
} file_t;
typedef struct _filetree {
file_t *file;
struct _filetree *left;
struct _filetree *right;
#ifdef USE_TREE_REBALANCE
struct _filetree *parent;
unsigned int left_weight;
unsigned int right_weight;
#endif /* USE_TREE_REBALANCE */
} filetree_t;
/* This gets used in many functions */
......@@ -266,7 +295,8 @@ extern void nullptr(const char * restrict func);
extern int file_has_changed(file_t * const restrict file);
extern int getfilestats(file_t * const restrict file);
extern int getdirstats(const char * const restrict name,
jdupes_ino_t * const restrict inode, dev_t * const restrict dev);
jdupes_ino_t * const restrict inode, dev_t * const restrict dev,
jdupes_mode_t * const restrict mode);
extern int check_conditions(const file_t * const restrict file1, const file_t * const restrict file2);
extern unsigned int get_max_dupes(const file_t *files, unsigned int * const restrict max,
unsigned int * const restrict n_files);
......
/* Detect and report size of CPU caches
*
* Copyright (C) 2017 by Jody Bruchon <jody@jodybruchon.com>
* Copyright (C) 2018 by Jody Bruchon <jody@jodybruchon.com>
* Distributed under The MIT License
*
* If an error occurs or a cache is missing, zeroes are returned
......@@ -12,6 +12,9 @@
#include <string.h>
#include "jody_cacheinfo.h"
/* None of this code is useful on Windows, don't build anything there */
#ifndef ON_WINDOWS
static char *pathidx;
static char buf[16];
static char path[64] = "/sys/devices/system/cpu/cpu0/cache/index";
......@@ -21,7 +24,6 @@ static char path[64] = "/sys/devices/system/cpu/cpu0/cache/index";
/* Linux sysfs */
#ifndef ON_WINDOWS
static size_t read_procfile(const char * const restrict name)
{
FILE *fp;
......@@ -110,8 +112,7 @@ void get_proc_cacheinfo(struct proc_cacheinfo *pci)
}
return;
}
#else
#define get_proc_cacheinfo(a)
#endif /* ON_WINDOWS */
......
......@@ -8,6 +8,9 @@
extern "C" {
#endif
/* Don't allow anything on Windows */
#ifndef ON_WINDOWS
/* Cache information structure
* Split caches populate i/d, unified caches populate non-i/d */
struct proc_cacheinfo {
......@@ -24,6 +27,10 @@ struct proc_cacheinfo {
extern void get_proc_cacheinfo(struct proc_cacheinfo *pci);
#else
#define get_proc_cacheinfo(a)
#endif /* ON_WINDOWS */
#ifdef __cplusplus
}
#endif
......
......@@ -5,7 +5,7 @@
* a secure hash algorithm, but the calculation is drastically simpler
* and faster.
*
* Copyright (C) 2014-2017 by Jody Bruchon <jody@jodybruchon.com>
* Copyright (C) 2014-2018 by Jody Bruchon <jody@jodybruchon.com>
* Released under The MIT License
*/
......@@ -21,19 +21,19 @@
#endif
/* The salt value's purpose is to cause each byte in the
* hash_t word to have a positionally dependent variation.
* jodyhash_t word to have a positionally dependent variation.
* It is injected into the calculation to prevent a string of
* identical bytes from easily producing an identical hash. */
/* The tail mask table is used for block sizes that are
* indivisible by the width of a hash_t. It is ANDed with the
* final hash_t-sized element to zero out data in the buffer
* indivisible by the width of a jodyhash_t. It is ANDed with the
* final jodyhash_t-sized element to zero out data in the buffer
* that is not part of the data to be hashed. */
/* Set hash parameters based on requested hash width */
#if JODY_HASH_WIDTH == 64
#define JODY_HASH_CONSTANT 0x1f3d5b79U
static const hash_t tail_mask[] = {
static const jodyhash_t tail_mask[] = {
0x0000000000000000,
0x00000000000000ff,
0x000000000000ffff,
......@@ -47,7 +47,7 @@ static const hash_t tail_mask[] = {
#endif /* JODY_HASH_WIDTH == 64 */
#if JODY_HASH_WIDTH == 32
#define JODY_HASH_CONSTANT 0x1f3d5b79U
static const hash_t tail_mask[] = {
static const jodyhash_t tail_mask[] = {
0x00000000,
0x000000ff,
0x0000ffff,
......@@ -57,7 +57,7 @@ static const hash_t tail_mask[] = {
#endif /* JODY_HASH_WIDTH == 32 */
#if JODY_HASH_WIDTH == 16
#define JODY_HASH_CONSTANT 0x1f5bU
static const hash_t tail_mask[] = {
static const jodyhash_t tail_mask[] = {
0x0000,
0x00ff,
0xffff,
......@@ -65,47 +65,47 @@ static const hash_t tail_mask[] = {
#endif /* JODY_HASH_WIDTH == 16 */
/* Hash a block of arbitrary size; must be divisible by sizeof(hash_t)
/* Hash a block of arbitrary size; must be divisible by sizeof(jodyhash_t)
* The first block should pass a start_hash of zero.
* All blocks after the first should pass start_hash as the value
* returned by the last call to this function. This allows hashing
* of any amount of data. If data is not divisible by the size of
* hash_t, it is MANDATORY that the caller provide a data buffer
* which is divisible by sizeof(hash_t). */
extern hash_t jody_block_hash(const hash_t * restrict data,
const hash_t start_hash, const size_t count)
* jodyhash_t, it is MANDATORY that the caller provide a data buffer
* which is divisible by sizeof(jodyhash_t). */
extern jodyhash_t jody_block_hash(const jodyhash_t * restrict data,
const jodyhash_t start_hash, const size_t count)
{
hash_t hash = start_hash;
hash_t element;
hash_t partial_salt;
jodyhash_t hash = start_hash;
jodyhash_t element;
jodyhash_t partial_salt;
size_t len;
/* Don't bother trying to hash a zero-length block */
if (count == 0) return hash;
len = count / sizeof(hash_t);
len = count / sizeof(jodyhash_t);