Add logging when a command is not found.

Also remove duplicate logging of exit code and command line:
it's already done by processutils.

Change-Id: I0349073cf0f78fb021dcd3e830fedddf2b4a0c92
(cherry picked from commit e4d7c1f4)

Under heavy load, the qemu-img convert method may attempt to create
additional malloc arenas to complete its work. By default this number
is 8 * ncpu * which at about 250 MB of memory which can be consumed
in this mannor should the.

Since this is only something which should realistically occur under
heavy cross-thread load where memory locking prevents a thread from
unlocking a range of memory, (such as what can happen with hypervisors
and VMs inside those hypervisors), then a new arena gets created, and
the memory consumption spikes.

More information is avialable at:
https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html

Change-Id: I8b2b490f2cc0ac5f47c3aaaaa249ce59db6602d6
Story: 2008928
Task: 42529
(cherry picked from commit 76cbb962)

The current value is insufficient for modern images. A similar change to IPA:
https://review.opendev.org/c/openstack/ironic-python-agent/+/778035

Change-Id: Ia1535b375d2cec467011c7456da84b5811ba48dd
(cherry picked from commit df9a2e85)

The base64 module can raise many exceptions, TypeError doesn't even
seem one of them. Capture all errors and provide the exact message.

Change-Id: Id3748e729f8dd31aa6d2d7084bd01088cd9a7ecd
(cherry picked from commit 0704bcdd)

Extended boot records (EBRs) are used in the MSDOS partitioning
scheme to manage extended partitions. EBRs are small partitions
themselves and Ironic will therefore treat them like any other
partition during cleaning. Since EBRs are smaller than a GPT
(33 sectors), dd'ing 33 sectors will fail. This patch proposes
a fix to handle EBRs (and other tiny paritions) by adjusting
the number of blocks writtent to be the partition size at max.

Story: #2008539
Task: #41628
Change-Id: Ie9809b3e4fe2bea6b66ee95c442111942e85d618
(cherry picked from commit 9a60d385)

As discussed during the upstream ironic community meeting on
Monday Dec 14 2020, the lower-constraints job is being removed.

Also disabling dstat in all focal based jobs because of
https://bugs.launchpad.net/ubuntu/+source/dstat/+bug/1866619

Change-Id: Iab279c57676dba8faa87ab7523e577d3b65eb428
(cherry picked from commit 0bbba8e6)

qemu-img convert makes a number of allocation requests, and less
common than the "Resource temporarily unavailable" error is
"Cannot allocate memory". The first seems to be largely raised
upon a thread creation for the image conversion, where the first
appears to be on buffer setup on the original process.

Regardless, we should check for both and retry conversion
automatically.

rhbz# 1892773

Change-Id: If435ed0efe034d4bb99f0281ebc73840c0f38fb3
(cherry picked from commit 9be6afc7)

Update the URL to the upper-constraints file to point to the redirect
rule on releases.openstack.org so that anyone working on this branch
will switch to the correct upper-constraints list automatically when
the requirements repository branches.

Until the requirements repository has as stable/victoria branch, tests will
continue to use the upper-constraints list on master.

Change-Id: Id4769f5ae2029097ec9398cb18035cfefe88df54

Change-Id: I6b0d5e8589f954a1c9521ca55f14d76cfe997622

It forces eventlet on us and can be easily replaced with tenacity.

Change-Id: I0085a3929ec27ab119884e8610b7c15f8dc62d63

Python modules related to coding style checks (listed in blacklist.txt in
openstack/requirements repo) are dropped from lower-constraints.txt
they are not needed during installation.

Change-Id: I3a686b00cb2a6cb7cde4a26001d6d9016aab08eb

This change exposes existing or copies from IPA functions that are
potentially common between ironic, IPA and ironic-lib itself:
1) utils.try_execute
2) disk_utils.udev_settle, partprobe and trigger_device_rescan
3) capabilities parsing and re-combining

Change-Id: I0ab42e082e8003415a1cac857d1c4b2044c5484d

The BMO does auto-pre-formatting on the submitted hints
so we end up in scenarios where we get strings prepended with
comparison operators in cases we never really see otherwise.

Just to be on the safe side, explicitly test this so we don't
accidently break it later.

Change-Id: I046a59753ecb6ec29ddd2564923a800105f56d3f

Instead of using a loop, we use the tenacity library to provide
correct retry feature to the convert_image function.

Change-Id: I0ddef614be55b0f6365a952a906197e7a219ef65

It's tricky to debug authentication when we leave no traces in the
logs and the client only receives a generic error.

Change-Id: I2c248b94938ec37f4b28e0fda4eded51ee48cdc6

Now with a safe version of flake8!

Change-Id: Id524994f6c57124d59d5863129c0879965dd1de2

It's not allowed to return None from WSGI applications, eventlet fails
with TypeError: 'NoneType' object is not iterable.

Use WebOb to build and provide a correct response.

Change-Id: Idf4c6782730549378c03416a118b3b69fa53ca65

As recommended, since version 3.2.0 tox switches pip invocations
to use the module -m pip instead of direct invocation.
We set min version to 3.2.1 [1] to also fix the behavior of
--parallel--safe-build

[1] https://tox.readthedocs.io/en/latest/changelog.html#v3-2-1-2018-08-10

Change-Id: Idc0c845dc499768d8f6ea2859596bc6e391c9f60

The htpasswd tool generates $2y$ prefixed values, which is not common
for bcrypt implementations[1].

This change adds acceptance for $2a$ and $2b$ prefixed values which the
vast majority of bcrypt implementations will produce (including golang
and the bcrypt python library which this feature depends on)

[1] https://en.wikipedia.org/wiki/Bcrypt

Change-Id: I0450657cf3023f47f1894b650d2e400cc4bbcbf4

The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.

Change-Id: If307ca1f689b1ade1428b904bbf122bd6aa5da97

The zeroconf library removed address in version 0.27.0 [1]
This patch should fix potential issues with that change.

[1] https://github.com/jstasiak/python-zeroconf/#changelog

Change-Id: I694ae5587ac8c3aae43d98902f56738773d31284

flake8 new release 3.8.0 added new checks and gate pep8
job start failing. hacking 3.0.1 fix the pinning of flake8 to
avoid bringing in a new version with new checks.

Though it is fixed in latest hacking but 2.0 and 3.0 has cap for
flake8 as <4.0.0 which mean flake8 new version 3.9.0 can also
break the pep8 job if new check are added.

To avoid similar gate break in future, we need to bump the hacking min
version.

- http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014828.html

Change-Id: Ib39f593616e2933ca5c1dbfa231691d6ced60e6c

This middleware is added to ironic-lib so that it can eventually be
used by ironic and ironic-inspector as an alternative to noauth in
standalone environments.

This middleware is passed a path to a file which supports the
Apache htpasswd syntax[1]. This file is read for every request, so no
service restart is required when changes are made.

The only password digest supported is bcrypt, and the ``bcrypt``
python library is used for password checks since it supports ``$2y$``
prefixed bcrypt passwords as generated by the Apache htpasswd utility.

[1] https://httpd.apache.org/docs/current/misc/password_encryptions.html
Needed-By: https://review.opendev.org/727467
Needed-By: https://review.opendev.org/729463

Change-Id: I874783b8ece0eedf27a94dfed5163d0c82f8b9de
Story: 2007656
Task: 39740

Switch to openstackdocstheme 2.2.1 version. Using
this version will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems

Update Sphinx version as well.

Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.

Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: I9dfa47d28d3f76115d6cf6dd86ea04d621d99c65

We increased the default RAM for 3GiB on [1] because we
re-added the linux firmware [2], removing the override
in ironic-lib-base so it will use the value from ironic-base job.

[1] https://review.opendev.org/727169
[2] https://review.opendev.org/726769

Change-Id: I75c9f21ae89b8b2d3ace4af8eee921b14576afa4

We need to list all dependencies in lower-constraints

Change-Id: Ib7ec4c7e5149e445c1d705cbe29bccbf8f0507b4

Change-Id: If571457351240f84ec130a8e1550bf07e1f0507f

This change fixes two potential issues with qemu-img on a busy machine:
1) Retry infrequent "Resource temporarily unavailable"
2) Make address space limits configurable instead of hardcoding to 1Gi

Story: #2007644
Task: #39702
Change-Id: I82a94e68e1aa1f634c7f0877ccd674574d1c9bb0

Now that we are running the Victoria tests that include a
voting py38, we can now add the Python 3.8 metadata to the
package information to reflect that support.

Change-Id: I0f06173f4758094168b9ce6c6253cd0fd3b53bdd
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

* Remove testscenarios, it's an ancient artefact
* Replace the only usage of testtools with a simple check
* Split out doc/requirements.txt
* Clean tox.ini of unnecessary (and probably cargo-culted) stuff
* Added Pygments for syntax highlighting check

Change-Id: I9f6b03606973e1e8123165c46bef53c4ce8e2b92

Full py3 compatible version.
Add all Python3 modules to stdlib list.
Also includes fix to an enum34 dependency bug.

Change-Id: Ib518f6f110a52e891fb1952262e4456181d3d712