Felix Lechner authored Jan 10, 2019 and Chris Lamb committed Jan 10, 2019

A recent commit series to split the test preparation from the test runner
causes the following error:

    not a reference at /usr/share/perl5/IO/Async/Channel.pm line 197.

The problem was likely added in commit 6ffd137a

. This now passes (and
also resolves) an explicit reference to the path that identifies each
test.

Signed-off-by: Chris Lamb <lamby@debian.org>

 Apply a patch series from Felix Lechner to split the preparation of tests from the test runner itself. (MR: !122)

Felix Lechner authored Jan 09, 2019 and Chris Lamb committed Jan 10, 2019

Simplifies the logic for process messaging in Test::State and
elsewhere. The key is to issue multiple commands in the worker threads
when changing a test's status. That may not be atomic, but the most
unpleasant consequence should be messages out of sequence. None were
observed so far.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 09, 2019 and Chris Lamb committed Jan 10, 2019

Harnesses are generally better at executing tests than at preparing
test material. Due to the complexity of many Lintian tests, the two
stages are better split.

TAP::Harness may take over the running of the tests. The test
preparation, on the other hand, will continue to run in parallel using
IO::Async::Routing and a setup very much like the existing test
runner. Now it just has its own event loop.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 09, 2019 and Chris Lamb committed Jan 10, 2019

The most portable way to refer to tests is by their paths.  In
addition, the concept of 'suite' for Lintian tests will change
considerably in the near future.  Changes the process label for each
test from ${suite}::${testname} to the relative path to the test
directory.

This changes the output format considerably and may take some time to
get used to. As a positive, the output very much resembles 'prove' or
TAP::Harness and may thus be familiar from other Perl contexts.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 09, 2019 and Chris Lamb committed Jan 10, 2019

Use file paths to test specification directories to identify tests in
the run queue. It is more portable between harnesses and a
prerequisite to using TAP::Harness for the package tests.

Introduces a small change in the output format, but only for the
summary. For skipped tests, the suite is now part of the line for each
test.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 09, 2019 and Chris Lamb committed Jan 09, 2019

This should more or less restore the exit code behavior from before
commit 26805519

.

Signed-off-by: Chris Lamb <lamby@debian.org>

This reverts commit b037521b.

Check for packages that have maintainer scripts that ship support code for legacy versions of Debian. (Closes: #917566)

Apply a patch series from Felix Lechner to add a number of "internal" tests for the test apparatus itself. (MR: !119)

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 09, 2019

Sequence numbers have not been required for some time. The reference
was removed.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 09, 2019

This behavior may change in the future, when 'harness' may become a
suite.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 09, 2019

Adds a small notification when the 'onlyrun' command line option was
used but no tests where selected. The message explains the basic
usage of the 'onlyrun' option.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 09, 2019

Internal harness tests ensure the consistency of test results. They
cannot be deactivated; they run every time.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Oct 18, 2018 and Chris Lamb committed Jan 09, 2019

Currently, four tests ensure the following functionality:

* check-result.t: Match algorithm for Lintian tags works as expected.
* logged-prepare.t: Default values in work directories are correct.
* no-watch-file-in-native.t: No watch file in native template.
* watch-file-in-non-native.t: Watch file in non-native template.

More internal tests may be added in the future. This may also be a
good place to test other critical or fragile components in Lintian
proper.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 09, 2019

Calling 'prove' to run code quality tests makes it too easy to
overlook results when subsequent outputs follow. With TAP::Harness all
tests become part of the same test summary at the bottom (less
scrolling).

Internal harness tests will be added shortly.

The output may be slightly different compared to 'prove'.

Signed-off-by: Chris Lamb <lamby@debian.org>

Mo Zhou authored Jan 09, 2019 and Chris Lamb committed Jan 09, 2019

Signed-off-by: Chris Lamb <lamby@debian.org>

Fix symbols-file-missing-build-depends-package-field false-positives when a package contains more than one library. (Closes: #918473)

Apply a patch series from Felix Lechner to support adaptive rebuilding, speeding up the test test suite speeds up by a factor of 17 (excluding "scripts" tests). (MR: !118)

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

The function that checks the Lintian output took a harness dependent
test state as an argument. It also used a convoluted logic to make
sure that all Test-For tags were seen, and all Test-Against tags were
not. Rewrites the function to return an list of error instead, if
there are any.

Makes the test runner independent from the harness and may later aid
in the experimentation with other harnesses, such as TAP::Harness.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

The log file is the last file written in a test working directory. It
is therefore always the newest. Its young age relative to expected
build product (the Lintian subject) should not trigger a rebuild.
Excludes the log from the calculation.

This is part of adaptive rebuilding for tests.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Uses StagedFileProducer to build tests adaptively.  The idea is that
building test packages takes a lot of time, but is not actually
necessary when only Lintian has changed, while a test specification,
runner and harness all remained the same. On one system the test suite
speeds up by a factor of 17 (when excluding 'scripts'). It should
benefit consumers of the test suite, i.e.  Lintian developers who do
not modify test runner or harness but who run tests repeatedly.

Since the layout of test working directories is still changing, please
run 'rm -f debian/test-out/' should unexpected errors occur. Please
also do so before making mission-critical assessments.

Due to the granularity of file system modification times at 1 second,
the mechanism may not work properly if changes are made, and the test
suite is run, before 1 second has elapsed. This is unlikely to affect
anyone editing Lintian checks by hand.

Implements the adaptive rebuilding strategy for tests.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Uses environment variables that were set up previously and file
modification times for a test specification to calculate and announce
data age, runner age, harness age and Lintian age.

Also computes a rebuilding threshold to be used when building test
packages.

Part of adaptive rebuilding for tests.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Based on HARNESS_EPOCH and file modification times, calculates the age
of a test specification. Used primarily for rebuilding decisions.

The resulting date is also used to fill templates (unless overridden).
Because of this secondary use, the value is further combined with
POLICY_EPOCH to make sure an old date does not trigger a timewarp
warning. Unless a test specification is more recent, the resulting
date is the oldest permitted according to policy. This may become
unnecessary once the disjointed tags proposal (aka tags.d) is
accepted.

Part of adaptive rebuilding for tests.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Changes the file modification time for files generated from templates
to their maximum expected age.  Prevents unnecessary rebuilds when
templates are regenerated even though they have not changed.

Part of adaptive rebuilding for tests.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Provides cleaner separation between test harness and test runner. May
later aid in the experimentation with other test harnesses, such as
TAP::Harness.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

This value is used to re-run Lintian on test tags only when Lintian
has changed. It is part of adaptive rebuilding for tests.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

For the current implementation, RUNNER_EPOCH equals HARNESS_EPOCH.

This is part of adaptive rebuilding for tests.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Uses a facility from the new module ScriptAge to calculate an age for
the harness that includes modules.

Signed-off-by: Chris Lamb <lamby@debian.org>

Felix Lechner authored Jan 08, 2019 and Chris Lamb committed Jan 08, 2019

Provides a way to define and stack file production stages that all
depend on subsets of the same group of files.

After the stages are defined, the processing engine takes an inventory
of all files in a target directory. It excludes some files, like logs,
that should not be considered.

Each stage adds its own products to the list of files to be excluded
before deciding whether to produce them. The decision is based on
relative file modification times, in addition to a systemic rebuilding
threshold. Before rebuilding, each stage asks a lower stage to make
the same determination.

The result is an engine with file production stages that depend on
successively larger sets of files.

Signed-off-by: Chris Lamb <lamby@debian.org>