When we have an image with size > 1 GB, we follow the following steps
to accomodate the image:
1) Detach the volume
2) extend the volume
3) Attach the volume
4) Open the volume device as a file and resume writing the image

Sometimes due to several reasons (mostly network related), the
size of the device file could mismatch with the actual volume size
(or the backend LUN size). This can happen if the extend was performed
(i.e. the control path) but it takes the time to reflect that into
the mapped device (i.e. the data path). This mismatch can cause the
issue "IOError: [Errno 28] No space left on device".
To avoid this scenario, we check if the device size is less than the
volume size, we wait for the extended LUN to show up in mapped device
and then continue the image writing operation.

Closes-Bug: #1959913

Change-Id: I206580f6be615ebc5e15b546b9c23728d4116a5d
(cherry picked from commit f3433ed1)
(cherry picked from commit ea5139be)

While fetching connector information the cinder driver of
glance-store is passing 'hostname' as IP address whereas it
actually needs IP address.

This path fetches the IPV4 or IPV6 address from the available
'hostname' and passes it to os-brick.

This worked for other cinder backends which don't rely specifically
on IP address for initializing connection like fibre channel, nfs etc
and even for some iscsi backends since, in some environments,
the hostname is same as the ip address. This is not the case always
and we should pass the correct IPv4/IPv6 address of the host to
avoid this issue.

Closes-Bug: #1955668

Change-Id: Ic79815972e654a8bfe2775f57c68cfa0bf115e2f
(cherry picked from commit 8b4d97a6)
(cherry picked from commit 83d9c5dc)
Conflicts:
	glance_store/_drivers/cinder.py
	glance_store/tests/unit/test_cinder_store.py
	glance_store/tests/unit/test_multistore_cinder.py

Change-Id: I1de76a70e07bbf2bd70900b4d93104a1b0fda74b
(cherry picked from commit b42ffa40)

Currently we only pass the configured multipath option when getting
the connector properties and do not pass it when creating the
connector object which in turn does the multipathing in os-brick.
So we are only passing multipathing option to cinder for exporting &
mapping the volume to paths but not while creating the connector object
and hence doing multipathing partially.
This patch passes that option and fixes the stated issue.

Related-Bug: #1863983

Co-Authored-By: Gorka Eguileor <geguileo@redhat.com>

Change-Id: I82979c47ec9ec1dd6cc864d3827ab219177251f8
(cherry picked from commit 04e5ead7)

Update the URL to the upper-constraints file to point to the redirect
rule on releases.openstack.org so that anyone working on this branch
will switch to the correct upper-constraints list automatically when
the requirements repository branches.

Until the requirements repository has as stable/wallaby branch, tests will
continue to use the upper-constraints list on master.

Change-Id: If0ff588f0b97c870209bd7b5063e6ddc1e4be1c5

Change-Id: Idb0ed63d97eec0fc4560d4d9dd02bfb8e27c2f3c

Change-Id: Ie278638466c3848c18cc353d966d2275370eaca0

This will let us see if a proposed patch against glance_store will
break glance's functional job. We already run a tempest job against
glance_store so we should be loosely covered there already.

Depends-On: https://review.opendev.org/c/openstack/glance/+/776369
Change-Id: Icf96eb49d560f2deb543232435094b34934b9406

Currently when configuring multiple cinder stores,
glance validates the volume types configured in glance-api.conf
during service startup.
This check however has a dependency on cinder API service to be
up and running.
During certain deployment scenarios, it is not always mandatory
that cinder is already running when we are starting glance hence
this check will fail the glance service hence the deployment
This patch removes the startup exceptions and instead logs a warning.
Also the check is done during volume create so handling for NotFound
exception from cinder is also handled.

Change-Id: I468523b947ad6fcff4b074d2ed1cb7a9e52b15ca
Closes-Bug: #1915163

v2 API of cinder is already deprecated, and the cinder community is now
planning to remove it.
This change makes cinder store use v3 API instead, which is the current
latest version actively maintainced.

Closes-Bug: #1915602
Change-Id: I8f374eb1b26ec408c52c6279da79cfe470d59a1c

Change-Id: I6c232e80ed5975e156f6ddf34e9a8944ac35f2d6

md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.

md5 is allowed when in a non-security context.  There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.

In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.

Some downstream python versions already support this parameter.  To
support these versions, a new encapsulation of md5() has been added to
oslo_utils.  See https://review.opendev.org/#/c/750031/

This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.

It looks like the uses of the md5 are primarily for checksums and
generation of etags.

With this patch, all the unit and functional tests appear to pass
on a FIPS enabled system.

Change-Id: I0603ba217d6dc19f5c9f73c60c7b365efd28d30b
Depends-On: https://review.opendev.org/#/c/760160

For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: Icd238d1acd2e9e5968399743225525faed2c28d3

This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for wallaby.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: I09f7eccd31f4c865cd514b824345a0a0de1d01e3

Add file to the reno documentation build to show release notes for
stable/victoria.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.

Change-Id: I3468449a517edfac288de349a4079c666a263a05
Sem-Ver: feature

The old context references (user, tenant) errors out during functional
testing on glance side[1][2] with the following error:

2020-09-07 07:29:34,125 ERROR [glance.async_.taskflow_executor] Failed to
execute task 6ee97c27-b87b-42f9-86af-151ef9a7ecd4: Property 'tenant' has
moved to 'project_id' in version '2.6' and will be removed in version '3.0'

(similar error for user/user_id)

This patch updates the same.

[1] https://review.opendev.org/#/c/750144/
[2] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fdc/750144/2/check/openstack-tox-functional-py36/fdc43f4/testr_results.html

Closes-Bug: #1894735

Change-Id: I279e442d7d00a6ebca0679cbf46a295fa07b6306

This message is not actually an error inside the system for most of the
time.  The primary reason behind this is that it is possible that there
are VMs using this image and therefore it can't be deleted, the operator
has no actionable thing to do about this.

For those who are monitoring ERROR level messages and alerting on them,
this is noisy and not actionable, drop this error message to warning.

Change-Id: If5447c3f5b4f678943273c2bdb2e44cd25d5b226

Since the resize decisions are packed tight into a helper, it
is trivial to test the boundary conditions alone. This adds a test
to do that, including the previously-uncovered condition of the
8G resize ceiling.

Change-Id: I04b4e2f220cf93275f0b14240cbbfff3da94e4d1

This brings over the small-footprint FakeData generator from glance and
refactors the rbd resize tests to use it instead of a static string.

Change-Id: I842578ca933294e4340ba080be488999e7910727

Inside cinder, default types are returned as dictionary but the
API call return it as a object.
Correcting name reference for the same from dictionary (['name'])
to object (.name)

Change-Id: Iaf7f1174f80624a0e1f47bea3c0c0b36d2345557

This patch adds support for configuring multiple cinder stores.
The following changes are part of this patch:
1) location URL format: previously cinder://<volume-id>, the format is
now changed to cinder://<store-name>/<volume-id>
2) Check for configured volume types: During initialization of service,
all volume types configured in the cinder store(s) will be checked if
they exist in deployment or not, the store with the wrong type will be
disabled. if cinder store is configured using traditional/old way
(i.e. using 'stores' config option) then this validation will not be performed.
3) Update legacy images: This change is partially depedent on the glance
changes[1] which trigger updating the URL for legacy images and the
cinder store checks the configured stores volume type against the existing
image-volume's type and update it respectively.

[1] https://review.opendev.org/#/c/742373/

Implements: blueprint multiple-cinder-backend-support
Change-Id: I0ba36d1095886fd1d96926af37a3a65a178c004f

Add new configuration option ``rbd_thin_provisioning`` and
``filesystem_thin_provisioning`` to rbd and filesystem
store to enable or not sparse upload.

A sparse file means that we do not actually write null byte
sequences but only the data itself at a given offset, the
"holes" which can appear will automatically be interpreted by
the storage backend as null bytes, and do not really consume
your storage.

Change-Id: I129e30f490e3920e9093c2b793f89b70ce310a50
Co-Authored-By: Grégoire Unbekandt <gregoire.unbekandt@gmail.com>
Partially Implements: blueprint handle-sparse-image

Change the RBD store to resize the image by up to 8GiB
at the time to not resize on every write.

Trim the image in Ceph after all data has been written
to the actual size in case we overshot the resize.

Partial-Bug: #1792710
Related-to: spec-lite-Ceph-Store-Optimization

Change-Id: I7f0bffda222b663d4316c5d6c03fdbd0d3337035

At the moment if you need to configure cinder store of glance in your
environment then you need to mannualy copy rootwrap related files
from glance_store/etc/glance to /etc/glance folder.

Added those rootwrap.* related files as data_files in setup.cfg
which will copies those to /etc/glance on installation.

Change-Id: I9217f4a1379687241d23fe34deecd1f482cbb49c
Closes-Bug: #1890844

This is to add missing white space between words in log message.

Change-Id: Ie166d3d29b10452238f6d3b325bb8c2f36b5fae1

As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).

Set bionic nodeset for py36 job and focal nodeset for rest all.

[1] https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-fo$

Change-Id: Ide3cefdcf5e99f9593890a8f9bd4c8520af12d29
Story: #2007865
Task: #40185

It replaces legacy-tempest-dsvm-full-ceph-plugin-src-glance_store
which is defined inside openstack-zuul-jobs.
It is really a variant of the devstack-plugin-ceph-tempest job.

Remove .zuul.yaml from the ignored file list as in other jobs.
Change-Id: Id9bc9172560421e3ebcbb3842da367ffd9a6bfd4

Importing pkg_resources causes the app to scan the entire import path
for all distributions, not just those providing entry points. The
scanner in stevedore will have a cache of the entry point data, making
it significantly faster.

This also more accurately reflects the way oslo.config loads option
plugins.

Change-Id: Ibbac3240a99684de0dea2f690f4276a3d24e3fe1
Signed-off-by: Doug Hellmann <doug@doughellmann.com>