We updated these policies when we introduces system scope and default
roles, but the policy names accidentally changed, which makes the policy
files render with an alias because oslo.policy thinks the names are
changing.

Conflicts:
      keystone/common/policies/application_credential.py in later
      releases the deprecated parameters were moved from the
      DocumentedRuleDefault object to the DeprecatedRule object, which
      is a non-functional change.

Change-Id: I1121f1abe769ee83ffc285103a95ee95540ce727
(cherry picked from commit 60e898c4)
(cherry picked from commit 7b28f1b3b47d0e4a22afe99c64d047016a772da5)
(cherry picked from commit 14d2f594)

There was a trailing s in two of these policies and it caused the policy
names to mismatch, which causes confusion with the rendered policy files
and potentially causes uses with deprecation logic.

Change-Id: I54021986d17c57d7733d53caa4032c2767eaf25e
(cherry picked from commit 82da8824)
(cherry picked from commit 6dff22b5baa1a19842aca435782fe1f9789f72cc)
(cherry picked from commit a57ae85c)

This cause the sample generated policy file to alias the old name with
the new policy name, which isn't needed since we're not renaming these
policies at all and it was likely a typo.

Conflicts:
      keystone/common/policies/identity_provider.py
      In later releases the deprecation parameters were moved up to the
      deprecated options and not in the DocumentedRule defaults.

Change-Id: Idfd9adbbe800bbc21814d94002a2b61524cce28a
(cherry picked from commit c10d5c88)
(cherry picked from commit bdd8f82f)

This change hides the AccountLocked exception from being returned
to the end user to hide sensitive information that a potential
malicious person could gain insight from.

The notification handler catches the AccountLocked exception as
before, but after sending the audit notification, it instead
bubbles up Unauthorized rather than AccountLocked.

Co-Authored-By: Samuel de Medeiros Queiroz <samueldmq@gmail.com>

Change-Id: Id51241989b22c52810391f3e8e1cadbf8613d873
Related-Bug: #1688137
(cherry picked from commit ac2631ae)

Keystone's update_user() method in the SQL driver processes a lot of
information about how to update users. This includes evaluating password
logic and authentication attempts for PSI-DSS. This logic is evaluated
after keystone pulls the user record from SQL and before it exits the
context manager, which performs the write.

When multiple clients are all updating the same user reference, it's
more likely they will see an HTTP 500 because of race conditions exiting
the context manager. The HTTP 500 is due to stale data when updating
password expiration for old passwords, which happens when setting a new
password for a user.

This commit attempts to handle that case more gracefully than throwing a
500 by detecting StaleDataErrors from sqlalchemy and retrying.  The
identity sql backend will retry the request for clients that have
stale data change from underneath them.

Change-Id: I75590c20e90170ed862f46f0de7d61c7810b5c90
Closes-Bug: 1885753
(cherry picked from commit ceae3566)
(cherry picked from commit f47e635b)

python-ldap3.0 or later running on python3 uses str or bytes
data type according to what fields are returned.
local_id may be a bytes data type.
To handle it properly, mapping[key] needs to be examined for
identifying its data type and what python version is used.

Closes-Bug: #1901654
Change-Id: Iac097235fd31e166028c169d14ec0937c663c21c
(cherry picked from commit f7df9fba)

The application credential policies use the `rule:owner` policy to allow
users to manage their own credentials. The policy engine pulled the
user_id attribute from the request path instead of the actual
application credential. This allowed for users to exploit the
enforcement and view or delete application credentials they don't own.

This commit attempts to resolve the issue by updating the flask
parameters before they're translated to policy arguments and target
data, prior to policy enforcement.

Change-Id: I903d20fa41270499ca1c39d296120dd97cef5405
Closes-Bug: 1901207
(cherry picked from commit 2d7bf10a)

Update the URL to the upper-constraints file to point to the redirect
rule on releases.openstack.org so that anyone working on this branch
will switch to the correct upper-constraints list automatically when
the requirements repository branches.

Until the requirements repository has as stable/victoria branch, tests will
continue to use the upper-constraints list on master.

Change-Id: I97a59aeda28e193f34f3c4d003ff7b90b9582e0f

Lower-constraints is not a requirement of the OpenStack Python PTI
[0] and there currently is a discussion on the mailing list [1]
about dropping the test, with the oslo team already having done
so [2].

The new dependency resolver in pip fails due to incompatible
dependency versions in our lower-constraints file, meaning that
we were never providing any real guarantees with it.

To unblock the CI, I am disabling lower-constraints job for now,
with the option to reenable it in case we fix the constraints,
and based on the outcome of the mailing list discussions and
consensus.

[0]. https://governance.openstack.org/tc/reference/pti/python.html
[1]. http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019672.html
[2]. http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019659.html

Change-Id: I47e747058891596e7717848a0b0bc10f0a235b2f
(cherry picked from commit d6610594)

This patch ensures to delete the system role assignments from
all the assignment tables in keystone after deleting the role
user has over the system.

This also make sure of deleting stale role assignments before
deleting role for the deployments that are already in this state.

Closes-Bug: #1878938

Change-Id: I4df19c45c870ff3fb78578ca1fb7dd0d35da3c82
(cherry picked from commit c1dcbb05)
(cherry picked from commit b83170a3)

Keystone's paging implementation contains a memory leak. The issue is
noticeable if you integrate keystone with an LDAP server that supports
paging and set `keystone.conf [ldap] page_size` to a low integer
(e.g., 5).

Keystone's LDAP backend uses `python-ldap` to interact with LDAP
servers. For paged requests, it uses `search_ext()`, which is an
asynchronous API [0]. The server responds with a message ID, which the
client uses to retrieve all data for the request. In keystone's case,
the `search_ext()` method is invoked with a page control that tells
the server to deliver responses in increments according to the page
size configured with `keystone.conf [ldap] page_size`. So long as the
client has the original connection used to fetch the message ID, it
can request the rest of the information associated to the request.

Keystone's paging implementation loops continuously for paged
requests. It takes the message ID it gets from `search_ext()` and
calls `result3()`, asking the server for the data associated with that
specific message. Keystone continues to do this until the server sends
an indicator that it has no more data relevant to the query (via a
cookie). The `search_ext()` and `result3()` methods must use the same
LDAP connection.

Given the above information, keystone uses context managers to provide
connections. This is relevant when deploying connection pools, where
certain connections are re-used from a pool. Keystone relies on Python
context managers to handle connections, which is pretty typical
use-case for context managers. Connection managers allow us to do the
following (assuming pseudocode):

  with self.get_connection as conn:
      response = conn.search_s()
      return format(response)

The above snippet assumes the `get_connection` method provides a
connection object and a callable that implements `search_s`. Upon
exiting the `with` statement, the connection is disconnected, or put
back into the pool, or whatever the implementation of the context
manager decides to do. Most connections in the LDAP backend are
handled in this fashion.

Unfortunately, the LDAP driver is somewhat oblivious to paging, it's
control implementation, or the fact that it uses an asynchronous API.
Instead, the driver leaves it up to the handler objects it uses for
connections to determine if the request should be controlled via
paging. This is an anti-pattern since the backend establishes the
connection for the request but doesn't ensure that connection is
safely handled for asynchronous APIs.

This forces the `search_ext()` and `result3()` implementations in the
PooledLDAPHandler to know how to handle connections and context
managers, since it needs to ensure the same connection is used for
paged requests. The current code tried to clean up the context
manager responsible for connections after the results are collected
from the server using the message ID. I believe it does this because
it needs to get a new connection for each message in the paged
results, even though it already operates from within a connection
established via a context manager and the PooledLDAPHandler almost
always returns the same connection object from the pool. The code
tries to use a weak reference to create a callback that tears down the
context manager when nothing else references it. At a high-level, the
idea is similar to the following pseudocode:

  with self.get_connection as conn:
      while True:
	ldap_data = []
	context_manager = self.get_connection()
	connection = context_manager.__enter__()
	message_id = connection.search_ext()
	results = connection.result3(message_id)
	ldap_data.append(results)
	context_manager.__exit__()

I wasn't able to see the callback get invoked or work as described in
comments, resulting in memory bloat, especially with low page sizes
which results in more requests. A weak reference invokes the callback
when the weak reference is called, but there are no other references
to the original object [1]. In our case, I don't think we invoke that
path because we don't actually do anything with the weak reference. We
assume it's going to run the callback when the object is garbage
collected.

This commit attempts to address this issue by using the concept of a
finalizer [2], which was designed for similar cases. It also attempts
to hide the cleanup implementation in the AsynchronousMessage object,
so that callers don't have to worry about making sure they invoke the
finalizer.

An alternative approach would be to push more of the paging logic and
implementation up into the LDAP driver. This would make it easier to
put the entire asynchronous API flow for paging into a `with`
statement and relying on the normal behavior of context managers to
clean up accordingly. This approach would remove the manual cleanup
invocation, regardless of using weak references or finalizer objects.
However, this approach would likely require a non-trivial amount of
design work to refactor the entire LDAP backend. The LDAP backend has
other issues that would complicate the re-design process:

  - Handlers and connection are generalized to mean the same thing
  - Method names don't follow a convention
  - Domain-specific language from python-ldap bleeds into keystone's
    implementation (e.g., get_all, _ldap_get_all, add_member) at
    different points in the backend (e.g., UserApi (BaseLdap), GroupApi
    (BaseLdap), KeystoneLDAPHandler, PooledLDAPHandler,
    PythonLDAPHandler)
  - Backend contains dead code from when keystone supported writeable
    LDAP backends
  - Responsibility for connections and connection handling is spread
    across objects (BaseLdap, LDAPHandler)
  - Handlers will invoke methods differently based on configuration at
    runtime, which is a sign that the relationship between the driver,
    handlers, and connection objects isn't truely polymorphic

While keeping the logic for properly handling context managers and
connections in the Handlers might not be ideal, it is a relatively
minimal fix in comparison to a re-design or backend refactor. These
issues can be considered during a refactor of the LDAP backend if or
when the community decides to re-design the LDAP backend.

[0] https://www.python-ldap.org/en/python-ldap-3.3.0/reference/ldap.html#ldap.LDAPObject.search_ext
[1] https://docs.python.org/3/library/weakref.html#weakref.ref
[2] https://docs.python.org/3/library/weakref.html#finalizer-objects

Closes-Bug: 1896125
Change-Id: Ia45a45ff852d0d4e3a713dae07a46d4ff8d370f3

Change-Id: Id36dce6f6c686b76df904dec705a1c5fe5b483be

As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).

Fixing:
- bug#1886298
Bump the lower constraints for required deps which added python3.8 support
in their later version.

Story: #2007865
Task: #40190

Closes-Bug: #1886298

[1] https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal

Change-Id: I5712f29beee2bd7d8ba857c0ce2cd2287646d6b0

l-c job template moved the l-c jobs running on Focal
and currently fails on many constraints.

Let's keep running l-c job on bionic as it was before and we
can move it to Focal once issues are identified and fixed.

- Fixing the hacking tests which are behaving differently between
< 3.8.0 (until Ubuntu Bionic) and 3.8.2 (Ubuntu Focal).

Squashing below review also
- https://review.opendev.org/#/c/750786/

Co-Author: Lance Bragstad <lbragstad@gmail.com>

Change-Id: If733e9824d87d8c73797f753e4daf95489bed9c2

This makes it easier for operators to troubleshoot connection issues to
Memcached.

Related-Bug: 1332058

Change-Id: I6e67363822480314b93608bb1eae3514f1480f6d

Although, Keystone doesn't use the pysaml2 signature on [0]
Would be nice to bump the pysaml2 version for, at least, 5.0.0[1] in
order to have the the CVE fix included[2].

[0]https://opendev.org/openstack/keystone/src/branch/master/keystone/federation/idp.py#L440-L521
[1] https://github.com/IdentityPython/pysaml2/releases/tag/v5.0.0
[2] https://github.com/advisories/GHSA-qf7v-8hj3-4xw7

Change-Id: I1d3776f7f1feb6485feecb140703f23027ca3a6f

In Ubuntu Bionic (18.04) mysql 5.7 version used to create
the user implicitly when using using the GRANT.

Ubuntu Focal (20.04) has mysql 8.0 and with mysql 8.0 there
is no implicit user creation with GRANT. We need to
create the user first before using GRANT command.

This patch updates tools/test-setup.sh so that keystone supports
ubuntu focal.

Story: #2007865
Task: #40190

Change-Id: I86d10729cfc7c02f12df611b56f6e263969dfe4b
Closes-Bug: #1885825

The update registered limit updates the specified registered
limit. It will be wrong to describe it as "Update registered
limits". It should be singular. Same for updating a project
limit. This patch fixes the same.

Change-Id: Ie28f0661bd4402ebb8f9de37fff4c36b925c3b04

This patch closes the review comments of [1].

[1]https://review.opendev.org/#/c/745752/

Change-Id: I06b02b2ebfed35d4e82c5fc35ce8eb0bb20b2fc5

msgpack v1.0 changed its data format [1] and during a rolling upgrade, attempts
to unpack cached tokens with old data format with the new default raw=False
result in the following error:

  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in
  position 3: invalid start byte

This passes raw=True to support backward-compat with the old format
until we are guaranteed to have msgpack >= 1.0 in the N-1 release of
a rolling upgrade.

Closes-Bug: #1891244

[1]
https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10

Change-Id: I6c61df6c097fef698c659c79402c4381ec7f3586

In the new version of PyMySql the error > 1000, will be
operational error [1], which is failing keystone migration
test cases [2] for backend mysql because we raise dberror
[3] which does not handle operational error.

PyMySQL hasn't been raised to 0.10.0 in the upper-constraints
yet, so this patch isn't going to be able to install it.
We can't raise the u-c since the current keystone jobs are
failing with it.

This patch overrides the test cases for backend SQL and skips
the same. This is so to make sure that failed test cases are
skipped because Once the upper-constraints are updated to
0.10.0 for PyMySql and merged, will revert the skip and
handle for 0.10.0.

[1]https://github.com/PyMySQL/PyMySQL/commit/c3e5a63514c57d1f4c9d5e7bf4b7e10b0608b0e1
[2]https://da7bb9864083b9045f13-6176f3344d2541229da3be8329641f28.ssl.cf5.rackcdn.com/741837/2/check/cross-keystone-py36/d1a2e73/testr_results.html
[3]https://github.com/openstack/keystone/blob/033e7aff870f2ccd4dec607e9c47efff630ece29/keystone/tests/unit/test_sql_upgrade.py#L1867

Related-Bug: #1890325
Change-Id: I207bb816affcb3e2725321de9a90a40c027a9f87

This Patch fixes the 'middleware' spelling.

Change-Id: I6659ca49db86e5c20ecf80e4c4fff93328616eb6

This patch fixes spelling "project" in test_sql_upgrade.py file.

Change-Id: I8b1a8dbea5fb17707e59fae8605cc615f1b51f2c

If LDAP returns a UUID as an octet string the LDAP driver will fail to
convert it to something meaningful. The error usually looks something
like:

  ID attribute objectGUID not found in LDAP object

Microsoft AD's `objectGUID` parameter is stored and transmitted as an
octet string [0]. If you attempt to use the `objectGUID` to generate
user or group IDs, you'll get an HTTP 404 because keystone can't decode
it properly. This is unfortunate because `objectGUID` are a fixed
length, UUID format, and ideal for generating IDs in keystone. As
opposed to using the object's CN, which is variable length, and can
generate hashes that are larger than keystone's database table limit for
user IDs.

[0] https://docs.microsoft.com/en-us/windows/win32/ad/reading-an-objectampaposs-objectguid-and-creating-a-string-representation-of-the-guid

Change-Id: Id80b17bdff015e10340e636102576b7435bd564f
Closes-Bug: 1889936