When we try to simultaneously attach same volume multiple times
(multiattach), there are multiple attachments created, suppose
attachA and attachB. If attachA marks the volume "reserved" then
attachB can't proceed until the volume is in "in-use" or "available"
state. We retry until we reach any of these states for which we use
the retrying library.

The retrying library defaults to 1 second retry[1] (5 times) which causes
the original failure as the volume takes time to transition between
"reserved" -> "in-use" state. This patch corrects it by adding an
exponential retry time and max exponential retry i.e.

1: 2 ** 1 = 2 seconds
2: 2 ** 2 = 4 seconds
3: 2 ** 3 = 8 seconds
4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)

[1] https://github.com/rholder/retrying/blob/master/retrying.py#L84

Closes-Bug: #1969373

Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea
(cherry picked from commit ba4af147)
(cherry picked from commit c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d)

When we have an image with size > 1 GB, we follow the following steps
to accomodate the image:
1) Detach the volume
2) extend the volume
3) Attach the volume
4) Open the volume device as a file and resume writing the image

Sometimes due to several reasons (mostly network related), the
size of the device file could mismatch with the actual volume size
(or the backend LUN size). This can happen if the extend was performed
(i.e. the control path) but it takes the time to reflect that into
the mapped device (i.e. the data path). This mismatch can cause the
issue "IOError: [Errno 28] No space left on device".
To avoid this scenario, we check if the device size is less than the
volume size, we wait for the extended LUN to show up in mapped device
and then continue the image writing operation.

Closes-Bug: #1959913

Change-Id: I206580f6be615ebc5e15b546b9c23728d4116a5d
(cherry picked from commit f3433ed1)

While fetching connector information the cinder driver of
glance-store is passing 'hostname' as IP address whereas it
actually needs IP address.

This path fetches the IPV4 or IPV6 address from the available
'hostname' and passes it to os-brick.

This worked for other cinder backends which don't rely specifically
on IP address for initializing connection like fibre channel, nfs etc
and even for some iscsi backends since, in some environments,
the hostname is same as the ip address. This is not the case always
and we should pass the correct IPv4/IPv6 address of the host to
avoid this issue.

Closes-Bug: #1955668

Change-Id: Ic79815972e654a8bfe2775f57c68cfa0bf115e2f
(cherry picked from commit 8b4d97a6)

Update the URL to the upper-constraints file to point to the redirect
rule on releases.openstack.org so that anyone working on this branch
will switch to the correct upper-constraints list automatically when
the requirements repository branches.

Until the requirements repository has as stable/xena branch, tests will
continue to use the upper-constraints list on master.

Change-Id: Ia07bd6bd0d32f202270c9c0d7d4523aa352e6a05

Change-Id: I31375cdafa0b111535457bb5605ee2b31e2b30bf

Change-Id: I595207658e812556d726121c1fe4400668c819ec

Swift driver reraises the client exception causing glance-api
to return 500. Lets raise g_s StoreFull correctly instead.

Change-Id: I5be151afe242a75142e74d488c4787a35929e189
Closes-bug: #1926404

We implemented cinder's new attachment API support with patch[1].
It was needed to add multiattach volume handling added with this
patch.
There is no special configuration change or user interference needed.
If a volume is of a multiattach type, then it will be handled as
a multiattach volume.

[1] https://review.opendev.org/c/openstack/glance_store/+/782200

Implements: blueprint attachment-api-and-multiattach-support
Closes-Bug: #1904546

Change-Id: Iffb825492a20fd877476acad05f817b399072f01

Change-Id: I1de76a70e07bbf2bd70900b4d93104a1b0fda74b

There's an issue when cinder nfs is configured as glance backend and
the image is > 1GB i.e. extend operation is performed on volume.
Currently glance writes the qcow2 header in the raw volume and
cinder nfs driver used to work on format autodetection which is
changed to manual detection here[1]
This fixes the extend volume problem for raw volumes, however, if
cinder nfs is configured to create qcow2 volumes then we will run into
the same problem.
This patch blocks creating images when the nfs volume is qcow2.

Refactoring work:
1) attachment_delete call is removed in nfs related code as it was
already done in the finally block
2) handle_exceptions decorator is removed from volume delete method
as it raises BackendException and incase when volume is not found
(possibly deleted manually), the image deletion fails so we don't
want to raise any error in this case

[1] https://review.opendev.org/c/openstack/cinder/+/761152

Closes-Bug: #1901138
Change-Id: I8ce6f36f1cb4b0ed6bcc5f3869fab3bb64fe3390

... because v2 API was deprecated during Pike and was already removed
during Xena[1].

[1] e05b261af7dcd24096b229860df65dff1d385910

Change-Id: I0e6d72ce82ce89aaaa8c56862bb8266d0351cd86

Cinder introduced new attachment API flow in microversion 3.27
(also attachment_complete added in mv 3.44 and support for passing
mode added in mv 3.54) which provides a clean interface to
interact with cinder for attachments and is also required for
multiattach volume support (Related future work).
Nova uses it since a long time and is proven to be stable,
this patch implements the same for glance.
The create volume and delete volume calls are also moved to
cinder_utils file to use the generic exception handler
and keep similar code together for consistency.

Partially Implements: blueprint attachment-api-and-multiattach-support

Change-Id: I2758ed1d5b8e0981faa3eff6f83e1ce5975a01d2

When processing the image data from a client, the previous and small
chunk size had a significant overhead. Let's be aligned with other chunk
size related parameters such as DEFAULT_LARGE_OBJECT_MIN_CHUNK_SIZE
(=5MiB) and DEFAULT_LARGE_OBJECT_CHUNK_SIZE (=10MiB).

The performance difference should be tangible especially with a
singlepart upload. And the upload time can be shortened for example as
follows:

100 MiB:   36.5s -> 4.0s
200 MiB: 2m16.8s -> 6.1s
300 MiB: 4m50.4s -> 9.1s

Closes-Bug: 1934849
Change-Id: Icecac80dd9e4e9f7ffa76bb7ca63d8d112036b70

add two new config options for cinder store

- cinder_store_user_domain_name
- cinder_store_project_domain_name

that allow to set the internal user and project to Keystone domains
other that the 'Default' one.

Closes-Bug: #1930299
Change-Id: I1d6c07b6c0e7e6a4da9adabaa026f024b64bb029

With changing the SOAP library backing oslo.vmware [1], the cookiejar of
a session must be accessed differently. Therefore, oslo.vmware
introduced a property on the client to abstract this change away. This
commit uses the new place to access the attribute.

[1]
https://specs.openstack.org/openstack/oslo-specs/specs/victoria/oslo-vmware-soap-library-switch.html

Change-Id: Iec6c4be5c61710f2a7a0056168d0c31ade987b1f

Currently we only pass the configured multipath option when getting
the connector properties and do not pass it when creating the
connector object which in turn does the multipathing in os-brick.
So we are only passing multipathing option to cinder for exporting &
mapping the volume to paths but not while creating the connector object
and hence doing multipathing partially.
This patch passes that option and fixes the stated issue.

Related-Bug: #1863983

Co-Authored-By: Gorka Eguileor <geguileo@redhat.com>

Change-Id: I82979c47ec9ec1dd6cc864d3827ab219177251f8

This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for xena.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: Ibd1cce5992992b474bf667cd18b14586c28d4e5f

Add file to the reno documentation build to show release notes for
stable/wallaby.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/wallaby.

Sem-Ver: feature
Change-Id: I642302a357abe16562a63cd233c1526211970279

In SingleTenant authv3 context, connection manager does not evaluate
swift_store_endpoint and always takes endpoint from catalog.

The change ensures CONF.glance_store.swift_store_endpoint
will take over catalog value also in that case.

Closes-Bug: #1885651
Change-Id: Ib18ff19cd539e0117909f849672036b8c9e5f049

Change-Id: Ie278638466c3848c18cc353d966d2275370eaca0

This will let us see if a proposed patch against glance_store will
break glance's functional job. We already run a tempest job against
glance_store so we should be loosely covered there already.

Depends-On: https://review.opendev.org/c/openstack/glance/+/776369
Change-Id: Icf96eb49d560f2deb543232435094b34934b9406

Currently when configuring multiple cinder stores,
glance validates the volume types configured in glance-api.conf
during service startup.
This check however has a dependency on cinder API service to be
up and running.
During certain deployment scenarios, it is not always mandatory
that cinder is already running when we are starting glance hence
this check will fail the glance service hence the deployment
This patch removes the startup exceptions and instead logs a warning.
Also the check is done during volume create so handling for NotFound
exception from cinder is also handled.

Change-Id: I468523b947ad6fcff4b074d2ed1cb7a9e52b15ca
Closes-Bug: #1915163

v2 API of cinder is already deprecated, and the cinder community is now
planning to remove it.
This change makes cinder store use v3 API instead, which is the current
latest version actively maintainced.

Closes-Bug: #1915602
Change-Id: I8f374eb1b26ec408c52c6279da79cfe470d59a1c

Change-Id: I6c232e80ed5975e156f6ddf34e9a8944ac35f2d6

md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.

md5 is allowed when in a non-security context.  There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.

In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.

Some downstream python versions already support this parameter.  To
support these versions, a new encapsulation of md5() has been added to
oslo_utils.  See https://review.opendev.org/#/c/750031/

This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.

It looks like the uses of the md5 are primarily for checksums and
generation of etags.

With this patch, all the unit and functional tests appear to pass
on a FIPS enabled system.

Change-Id: I0603ba217d6dc19f5c9f73c60c7b365efd28d30b
Depends-On: https://review.opendev.org/#/c/760160

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: Icd238d1acd2e9e5968399743225525faed2c28d3

This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for wallaby.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: I09f7eccd31f4c865cd514b824345a0a0de1d01e3

Add file to the reno documentation build to show release notes for
stable/victoria.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.

Change-Id: I3468449a517edfac288de349a4079c666a263a05
Sem-Ver: feature

The old context references (user, tenant) errors out during functional
testing on glance side[1][2] with the following error:

2020-09-07 07:29:34,125 ERROR [glance.async_.taskflow_executor] Failed to
execute task 6ee97c27-b87b-42f9-86af-151ef9a7ecd4: Property 'tenant' has
moved to 'project_id' in version '2.6' and will be removed in version '3.0'

(similar error for user/user_id)

This patch updates the same.

[1] https://review.opendev.org/#/c/750144/
[2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fdc/750144/2/check/openstack-tox-functional-py36/fdc43f4/testr_results.html

Closes-Bug: #1894735

Change-Id: I279e442d7d00a6ebca0679cbf46a295fa07b6306

This message is not actually an error inside the system for most of the
time.  The primary reason behind this is that it is possible that there
are VMs using this image and therefore it can't be deleted, the operator
has no actionable thing to do about this.

For those who are monitoring ERROR level messages and alerting on them,
this is noisy and not actionable, drop this error message to warning.

Change-Id: If5447c3f5b4f678943273c2bdb2e44cd25d5b226