Marc Herbert authored Aug 30, 2019 and Chris Lamb committed Sep 05, 2019

In other words tell the user when GNU ar cannot reproduce the output of
strip-nondeterminism.

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 30, 2019 and Chris Lamb committed Sep 05, 2019

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 29, 2019 and Chris Lamb committed Sep 05, 2019

Note symbol_table.out cannot be produced by GNU ar alone because it can't set
any specified timestamp on the symbol table.

long_member_name.a.out cannot be created by GNU ar alone either for a
completely different, strange and permission-related reason.  Faking a
specified timestamp for archive members is easy pre-archiving with "touch -r".
However this obviously requires 'ar U'. For long filenames, ar U maps
permissions 644 to the string "100644" for some unknown reason. Both ar D and
strip-nondeterminism use the permission string "644" (displayed by diffoscope
as "?rw-r--r--")

This is the script which was used to create these test files:

create_ar_testcase()
{
    ar_base="$1"; shift

    for o in "$@"; do
        objcopy --remove-section=.comment \
        --remove-section=.note.GNU-stack \
        --remove-section=.eh_frame \
        "$o" || true
    done
    touch "$@"

    rm -f "${ar_base}".a.in "${ar_base}".a.out "${ar_base}".a.D
    ar qU "${ar_base}".a.in "$@"

    ar qD "${ar_base}".a.D "$@"
    cp "${ar_base}".a.D "${ar_base}".a
    strip-nondeterminism --normalizers +all -v -T 1423159771 "${ar_base}".a
    mv "${ar_base}".a "${ar_base}".a.out
}

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 28, 2019 and Chris Lamb committed Sep 05, 2019

Long filenames are stored in a special section with a completely
different format. Stop assuming it has regular metadata and stop
corrupting it.

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 28, 2019 and Chris Lamb committed Sep 05, 2019

Don't reset to "644" the pseudo-permissions of the symbols table '/'. Whatever
the placeholder is ("0" with GNU ar 2.31), messing with it diverges with the
output of GNU ar which is already deterministic.

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 28, 2019 and Chris Lamb committed Sep 05, 2019

Signed-off-by: Chris Lamb <lamby@debian.org>

An old version was seemingly merged [locally] by myself. See:

  reproducible-builds/strip-nondeterminism!4 (comment 107844)

Gbp-Dch: ignore

Marc Herbert authored Aug 30, 2019 and Chris Lamb committed Sep 04, 2019

... in verbose mode, when setting it.

In other words tell the user when GNU ar cannot reproduce the output of
strip-nondeterminism.

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 30, 2019 and Chris Lamb committed Sep 04, 2019

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 29, 2019 and Chris Lamb committed Sep 04, 2019

Note symbol_table.out cannot be produced by GNU ar alone because it can't set
any specified timestamp on the symbol table.

long_member_name.a.out cannot be created by GNU ar alone either for a
completely different, strange and permission-related reason.  Faking a
specified timestamp for archive members is easy pre-archiving with "touch -r".
However this obviously requires 'ar U'. For long filenames, ar U maps
permissions 644 to the string "100644" for some unknown reason. Both ar D and
strip-nondeterminism use the permission string "644" (displayed by diffoscope
as "?rw-r--r--")

This is the script which was used to create these test files:

create_ar_testcase()
{
    ar_base="$1"; shift

    for o in "$@"; do
        objcopy --remove-section=.comment \
        --remove-section=.note.GNU-stack \
        --remove-section=.eh_frame \
        "$o" || true
    done
    touch "$@"

    rm -f "${ar_base}".a.in "${ar_base}".a.out "${ar_base}".a.D
    ar qU "${ar_base}".a.in "$@"

    ar qD "${ar_base}".a.D "$@"
    cp "${ar_base}".a.D "${ar_base}".a
    strip-nondeterminism --normalizers +all -v -T 1423159771 "${ar_base}".a
    mv "${ar_base}".a "${ar_base}".a.out
}

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 28, 2019 and Chris Lamb committed Sep 04, 2019

Long filenames are stored in a special section with a completely different
format. Stop assuming it has regular metadata and stop corrupting it.

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 28, 2019 and Chris Lamb committed Sep 04, 2019

Don't reset to "644" the pseudo-permissions of the symbols table '/'. Whatever
the placeholder is ("0" with GNU ar 2.31), messing with it diverges with the
output of GNU ar which is already deterministic.

Signed-off-by: Chris Lamb <lamby@debian.org>

Marc Herbert authored Aug 28, 2019 and Chris Lamb committed Sep 04, 2019

Signed-off-by: Chris Lamb <lamby@debian.org>

Drop accidentally-committed warning emitted on every fixture-based test. Thanks to Marc Herbert for the report.

This reverts part of 145f712b.

Disable the ar normalizer by default so that it can be enabled with "--normalizers=+ar" or similar. (Re: reproducible-builds/strip-nondeterminism#3)

This reverts commit 15b5ca0e.

Add support for enabling and disabling specific normalizers via the command line. (Closes: reproducible-builds/strip-nondeterminism#10)

Support OpenJDK ".jmod" files. Thanks to Emmanuel Bourg for the bug report and implementation idea. (Closes: reproducible-builds/strip-nondeterminism#11, Debian:#933337)

Pass --no-sandbox to if necessary to bypass seccomp-enabled version of file(1) which was causing a huge number of regressions in our testing framework. (re. #932006)

Don't just run the tests but build the Debian package instead using Salsa's centralised scripts so that we get code coverage, Lintian, autopkgtests, etc.

* origin/debian: (137 commits)
  Revert not specifying building the "debian" branch; as we merge master into this branch it will (since the very introduction of the .gitlab-ci.yml file) will have it and will additionally be out of sync.
  Update changelog for 1.2.2-1 release
  Actually replace all instances of libmonkey-patch-perl with libsub-override-perl.
  Update changelog for 1.2.1-1 release
  Replace libmonkey-patch-perl with libsub-override-perl (re. #931730 and reproducible-builds/strip-nondeterminism#8)
  release as 1.2.0-2
  Bump standards version to 4.4.0, no changes needed.
  Update changelog for 1.2.0-1 release
  Update changelog for 1.1.3-1 release
  Add libmonkey-patch-perl as a binary / build dependency.
  Update changelog for 1.1.2-1 release
  Update changelog for 1.1.1-1 release
  Update changelog for 1.1.0-1 release
  Bump Standards-Version to 4.3.0
  Update changelog for 1.0.0-1 release
  Update debian/copyright to drop reference to ar.pm.
  Release version 1.0.0
  Remove javaproperties handler after Emmanuel Bourg's patch was released in openjdk-11 11.0.1+13-3. (Closes: #914289)
  Drop .ar handler; binutils is reproducible. (Closes: #781262, #843811)
  Ignore encrypted .zip files as we can never normalise them. (Closes: #852207)
  ...

Revert not specifying building the "debian" branch; as we merge master into this branch it will (since the very introduction of the .gitlab-ci.yml file) will have it and will additionally be out of sync.

This reverts commit 568676f8.

Revert not specifying building the "debian" branch; as we merge master into this branch it will (since the very introduction of the .gitlab-ci.yml file) will have it and will additionally be out of sync.

This reverts commit 568676f8.

Release version 1.2.2

* tag '1.2.2':
  Release version 1.2.2
  Don't build (release) tags.
  No need to specify not building the "debian" branch; it has no .gitlab-ci.yml.