Closes: #914191

Apparently in some cases `apt-config shell` does not set the variable at
all. Let's be robust for that and avoid an error due to an unset
variable.

Thanks to Antoine Beaupre for debugging this!

Closes: #911963

Introduced in b48ee0f2

Gbp-Dch: Ignore

`[` should not be allowed as a valid character inside the architecture
list. Spotted by Python 3.7's new

    lib/testdesc.py:299: FutureWarning: Possible nested set at position 117

when running tests/testdesc.

Gbp-Dch: Short

Relax the dependency parser to accept "pkgname:archname" test Depends:,
so that depend on packages of foreign architectures.

Closes: #913082

I don't think it is up to autopkgtest to specify this anyways

With the apt-get fallback disabled, autopkgtest should only take packages from
the added suite only when specifically requested. That should also happen for
the autopkgtest itself. The old method (which remains used for the fallback
enabled case) worked fine, except when a source package builds a completely new
binary set. E.g. seen in hyperkitty 1.1.4-8 to 1.2.1-1.

autopkgtest-build-qemu: build VM images with vmdb2

See merge request ci-team/autopkgtest!33

Ubuntu images can't be created ATM due to a bug in Ubuntu's grub2 package:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1786699

However, if I locally hack vmdb2 to not pass the command line option
removed by Ubuntu, Ubuntu images can be created just fine.

Both Debian and Ubuntu images created have been tested with autopkgtest.

Thanks to Simon McVittie for the Debian kernel selection snippet.

Closes: #904972

Fix new pycodestyle warnings

See merge request ci-team/autopkgtest!37

Recent pycodestyle versions add warnings for escape sequences not known
to the current Python version. These currently expand to themselves,
but could mean something different in a future Python version.

In simple cases, use raw strings r'...' for regular expressions,
shell scripts, and other strings likely to contain backslashes that
are intended to be passed through to another interpreter like the
regular expression parser or the shell.

In more complex cases where the string already contains backslashes
(other than in regular expressions), I've continued to use non-raw
strings and just doubled the offending backslashes.

In the test code, I've systematically used r'' for all regular
expressions (even when it doesn't make any difference), since that seems
least likely to introduce mistakes later.

Note that

    assertRegex(foo, 'new\nline and\ttab')
    assertRegex(foo, r'new\nline and\ttab')

both assert that foo contains "new<U+000A>line and<U+0009>tab", where
<U+xxxx> represents the given control character. In the first match,
the regular expression contains a literal U+000A (newline/line feed)
and a literal U+0009 (horizontal tab), which match themselves, whereas
in the second match, the regular expression contains the sequences
backslash, n (which matches a newline) and backslash, t (which matches
a tab).

Signed-off-by: Simon McVittie <smcv@debian.org>

If a new pycodestyle version introduces new warnings, we presumably
want to report on them in all libraries and scripts, but `set -e`
means we bail out early on a failed command. This resulted in, for
example, not inspecting runner/autopkgtest if autopkgtest-virt-qemu
provokes warnings.

Signed-off-by: Simon McVittie <smcv@debian.org>

pycodestyle 2.4.0 warns whenever a line break occurs at a binary
operator; the intention seems to be that projects suppress either
W503 or W504, whichever one they prefer. Recent revisions of PEP 8
recommend breaking long lines before binary operators:

    if (foo
        and bar
        and baz):

but it seems autopkgtest consistently breaks after binary operators:

    if (foo and
        bar and
        baz):

so enforce that one for now.

Signed-off-by: Simon McVittie <smcv@debian.org>

Implement the 'superficial' restriction

See merge request ci-team/autopkgtest!34

Signed-off-by: Simon McVittie <smcv@debian.org>

This feature was previously prototyped under the name 'trivial'.

Superficial tests do not provide significant test coverage, so if they
pass, that does not necessarily mean that the package under test
is actually functional. If a superficial test fails, it will be
treated like any other failing test, but if it succeeds, this is
only a weak indication of success. Continuous integration systems
should treat a package where all non-superficial tests are skipped as
equivalent to a package where all tests are skipped.

For example, debian/tests/build in packages like dbus, libinput and
pango1.0 should be marked as superficial. For packages that have other
tests (like dbus and pango1.0), this has no effect on migration to
testing unless the non-superficial tests happen to be skipped. For
packages where the only test is superficial (like libinput), marking
the test as superficial removes the migration delay bonus on success
(because the test doesn't actually confirm that the library works,
so we have to rely on user testing to detect most regressions), while
retaining the migration delay penalty on failure.

Based on a patch from Rebecca N. Palmer.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #904979

Signed-off-by: Simon McVittie <smcv@debian.org>

Restore ability to boot pre-stretch systemd systems

See merge request ci-team/autopkgtest!35

Related to #896698

lxd-client will go away soon, so suggest lxd instead.

LP: #1788040

autopkgtests are in no way limited to testing programs. Generalize the
requirement that tests must test the installed *packages*, not just
*programs*.

Side issue in #906617.

These are EOL.

Simon McVittie (Collabora) authored Aug 15, 2018 and Simon McVittie committed Aug 16, 2018

The systemctl in Debian jessie didn't support the --no-block option.

Signed-off-by: Simon McVittie <smcv@collabora.com>
Closes: #906227

runner: Exit 8 if every test was skipped or otherwise ignored

See merge request ci-team/autopkgtest!24

Fix test regression (#905677)

See merge request ci-team/autopkgtest!32

Closes: #901804
Signed-off-by: Simon McVittie <smcv@debian.org>

If we are running as root but aspell-doc was already installed, then
dpkg wouldn't print "Unpacking aspell-doc" and the test will fail.

Signed-off-by: Simon McVittie <smcv@debian.org>

Packages that we unpack non-system-wide as non-root don't satisfy
dpkg-checkbuilddeps, so running this test as non-root would result
in a failed build.

Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #905677

runner: Document exit status 14 as possible

See merge request ci-team/autopkgtest!23

We can report an erroneous package and still have skipped tests.

Signed-off-by: Simon McVittie <smcv@debian.org>

Add skip-not-installable restriction

See merge request ci-team/autopkgtest!31

A test might have test dependencies that can't be fulfilled on all
architectures. Therefore, when apt-get installs the dependencies, it will
fail. Don't treat this as a test failure, but instead treat it as if the test
was skipped.

Closes: #905311

If anything in the tests of the sample package emits anything on stderr,
this make the test fail. Since the actual test relies on the exit code
of specific commands, this does not affect the functionality of the
test.

The tested use case was running the LxcRunner tests against a lxc
container created by debci. The Debian lxc template calls debootstrap
with --variant-minbase, so apt-utils is not installed, and that causes a
warning on stderr by apt.

Thanks nthykier for spotting it