This change addresses several issues in the creation and use of EC2/S3
credentials with keystone tokens.

1. Disable altering credential owner attributes or metadata

Without this patch, an authenticated user can create an EC2 credential
for themself for a project they have a role on, then update the
credential to target a user and project completely unrelated to them. In
the worst case, this could be the admin user and a project the admin
user has a role assignment on. A token granted for an altered credential
like this would allow the user to masquerade as the victim user. This
patch ensures that when updating a credential, the new form of the
credential is one the acting user has access to: if the system admin
user is changing the credential, the new user ID or project ID could be
anything, but regular users may only change the credential to be one
that they still own.

Relatedly, when a user uses an application credential or a trust to
create an EC2 credential, keystone automatically adds the trust ID or
application credential ID as metadata in the EC2 access blob so that it
knows how the token can be scoped when it is used. Without this patch, a
user who has created a credential in this way can update the access blob
to remove or alter this metadata and escalate their privileges to be
fully authorized for the trustor's, application credential creator's, or
OAuth1 access token authorizor's privileges on the project. This patch
fixes the issue by simply disallowing updates to keystone-controlled
metadata in the credential.

2. Respect token roles when creating EC2 credentials

Without this patch, a trustee, an application credential user, or an
OAuth1 access token holder could create an EC2 credential or an
application credential using any roles the trustor, application
credential creator, or access token authorizor had on the project,
regardless of whether the creator had delegated only a limited subset of
roles. This was because the trust_id attribute of the EC2 access blob
was ignored, and no metadata for the application credential or access
token was recorded either. This change ensures that the access
delegation resource is recorded in the metadata of the EC2 credential
when created and passed to the token provider when used for
authentication so that the token provider can look up the correct roles
for the request.

Conflicts (six removal in  e2d83ae9, pep8 fixes in e2d83ae9):
      keystone/api/credentials.py
      keystone/tests/unit/test_v3_application_credential.py
      keystone/tests/unit/test_v3_credential.py

Conflicts due to flask reorg:
	keystone/api/_shared/EC2_S3_Resource.py
	keystone/api/credentials.py
	keystone/api/users.py
	keystone/tests/unit/test_v3_credential.py

Moved the test_update_credential_non_owner unit test to
CredentialSelfServiceTestCase since in this branch the default policies
are not affected by #1872733.

NOTE: the application credential functional changes, along with its
tests were removed from the stable/pike backport as stable/pike does not
support application credentials.

Change-Id: I39d0d705839fbe31ac518ac9a82959e108cb7c1d
Closes-bug: #1872733
Closes-bug: #1872755
Closes-bug: #1872735
(cherry picked from commit 37e9907a)
(cherry picked from commit 2f2736eb)
(cherry picked from commit 27caafe3daa552663719954f2cd6713dd4493178)
(cherry picked from commit bfba75fc3c5c8f119f74dbf31347e008824a2134)
(cherry picked from commit 53d1ccb8a1bdbb5aa0efaacf9739b1a6f436e191)
(cherry picked from commit 6db1bb09a048dfb7f337484698a9a19fdbbe9546)

EC2 token requests contain a signature that signs the entire request,
including the access timestamp. While the signature is checked, the
timestamp is not, and so these signed requests remain valid
indefinitely, leaving the token API vulnerable to replay attacks. This
change introduces a configurable TTL for signed token requests and
ensures that the timestamp is actually validated against it.

The check will work for either an AWS Signature v1/v2 'Timestamp'
parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
parameter[2].

Although this technically adds a new feature and the default value of
the feature changes behavior, this change is required to protect
credential holders and therefore must be backported to all supported
branches.

[1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
[2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

Conflicts due to six removal in e2d83ae9:
	keystone/api/_shared/EC2_S3_Resource.py
	keystone/tests/unit/test_contrib_ec2_core.py

Conflicts due to v2.0 API testing in stable/queens. The v2.0 tests were
removed in Rocky but in earlier releases we tested similar functionality
between v3 and v2.0. This conflict was resolved by porting the timestamp
to the v2.0 API test:
	keystone/tests/unit/test_contrib_ec2_core.py

Conflicts due to flask reorg:
	keystone/api/_shared/EC2_S3_Resource.py

Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
Closes-bug: #1872737
(cherry picked from commit ab89ea74)
(cherry picked from commit 8d5becbe)
(cherry picked from commit e3f65d6f)
(cherry picked from commit 1ef38285)
(cherry picked from commit 35f09e2b7c00e03cd1d52a2337b51be38dd79480)
(cherry picked from commit d6f1006dd0ca8f7999f32133d348fcec8f2299d3)

Neutron-grenade have a dependency on stestr 1.0.0 or later in stable/ocata.
Unfortunately, we didn't add stestr to openstack/requirements till stable/pike.
There's been an attempts to fix the problem in stable/ocata [1][2]. But it
appears we won't be able to retrofit it because of the dependency conflicts.
However, we still want to land critical security patches in stable/pike and
since stable/pike is already in Extended Maintenance, it's best to just remove
the neutron-grenade job.

[1] https://review.opendev.org/#/c/718202/
[2] https://review.opendev.org/#/c/726983/

Change-Id: Icb32bda29b3c12d54269f4f257db5658538a5620

Without this patch, when an OAuth1 request token is authorized with a
limited set of roles, the roles for the access token are ignored when
the user uses it to request a keystone token. This means that user of an
access token can use it to escallate their role assignments beyond what
was authorized by the creator. This patch fixes the issue by ensuring
the token model accounts for an OAuth1-scoped token and correctly
populating the roles for it.

Modified to work with older test helper function:

  keystone/tests/unit/test_v3_oauth1.py

Conflicts:

  keystone/models/token_model.py

  The keystone token model was refactored in the Rocky release. This
  commit only backports the test so that we have test coverage against
  the bug and proves there wasn't a regression in Queens. As such, the
  code changes to token_model.py (where the bug was introduced) are not
  applicable to Pike.

  releasenotes/notes/bug-1873290-ff7f8e4cee15b75a.yaml

  Removed the release note since there isn't anything to signal to
  operators regarding a vulnerability. We're only adding test coverage
  to prove that stable/queens isn't vulnerable.

Change-Id: I02f9836fbd4d7e629653977fc341476cfd89859e
Closes-bug: #1873290
(cherry picked from commit 6c73690f)
(cherry picked from commit ba89d277)
(cherry picked from commit 5ff52dbaa2082991d229d8557a8e4b65256d6c53)
(cherry picked from commit 2483a578a80a916d9f5acd672d85830385b236e2)
(cherry picked from commit 10bc689a6796f85c44d19e0c18f0e37b0a87474c)
(cherry picked from commit d590441ce6897a7a169db7262eb17bcd5d90bcd2)

Import the legacy-tempest-dsvm-ldap-domain-specific-driver job[1] into the
keystone repo and convert it to be Zuulv3 native.

This cherry-pick removes the explicit use of python3 since it doesn't
quite work on this old of a branch.

[1] https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/playbooks/legacy/tempest-dsvm-ldap-domain-specific-driver

Change-Id: Ie0b9f13d6fb06b776d6a58d5d1087c20df8a7cda
Needed-by: https://review.opendev.org/687444
(cherry picked from commit 52ab0cf5)
(cherry picked from commit 0f6c6061)
(cherry picked from commit 429923fb)
(cherry picked from commit 2f715506)
(cherry picked from commit 8b8a8f2c54820e13403746784843df017812896f)

The openSUSE 42.3 distribution is eol, remove this experimental job so
that the job can be removed from Zuul.

Note that master has job for newer openSUSE running.

Change-Id: Icd25b8695816f4d89a7aa00a9b9e9387866438f3

Bandit is a linter, let's cap the version in a stable branch
so that new versions of bandit do not introduce pepe8 failures.

This is capped to version 1.4, same version as listed in
upper-constraints file in requirements repo for stable/pike branch.

Change-Id: Idd2b6254bb4fcf4b168cd35a1a27a9da2ca2280d

There's a regression[0] in bandit 1.6.0 which causes bandit to stop
respecting excluded directories, and our tests throw a bunch of
violations. Blacklist this version, but allow newer versions as there is
already a pull request[1] to fix it, and I expect it will be included in
the next release.

[0] https://github.com/PyCQA/bandit/issues/488
[1] https://github.com/PyCQA/bandit/pull/489

Change-Id: Ie4dbfb3f54e4aac00e0537d5760b7a8fc81b35a2
(cherry picked from commit ebac8330)

This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.

Without this change, when an admin tries to delete an LDAP-backed
domain, it fails due to the foreign key relationship in the users table.
Previously, we were assuming that LDAP users existed solely in the LDAP
directory, but this is not true with shadow users. This patch fixes the
logic to delete the shadow users upon domain deletion.

Altered cherry-pick: use injected API provider since provider_api
mechanism doesn't exist in pike.

Change-Id: I12a08001e3aa08e4db9438cae425ad1a0a8070f7
Closes-bug: #1801873
(cherry picked from commit 1b16725d)

This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Ifbe3d26c4bf37831fefd3737e7c8e1d90d6f6db7

The publishing credentials for this job are misconfigured in Zuul and
result in the whole post pipeline failing, which causes tarballs not to
be updated on tarballs.openstack.org[1]. Remove the misconfigured job to
get the post pipeline working again.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2019-03-11

Change-Id: I4d94a433ba32bce7ee926cdde487eeec980c6b8b
(cherry picked from commit f0c2e798)

keystone was not able to find any users while
the LDAP user name attribute was configured to
"samaccountname", but could find users when
reconfigured to use "sAMAccountName". LDAP is
not supposed to be case-sensitive, so either
should work.

This patch addresses the above problem by making
both the attributes into lower case. Also updated
the ldap_result example supporting python3.

Change-Id: I51813ac41489baed04f3cadbccd748e03025313e
Closes-Bug: #1753585
(cherry picked from commit 816b472a)

The issue occurs if a user has a group that
does not map to a project in OpenStack. At
which point an exception is raised and the
websso login blows up with a 500 message.
This is because of the exception being raised
when the group name not matches thus replacing
that with a log.

Change-Id: Ia7321705db118af28f3dc6e01d5b18e8650aa633
Closes-Bug: #1789450
(cherry picked from commit ee46f735)

This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.

Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.

Because this patch is generated by a script, there may be some
cosmetic changes to the layout of the YAML file(s) as the contents are
normalized.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: I549ec25953d8a5870c0aad211ccb9ca8418ba151
Story: #2002586
Task: #24304

The GET /v3/OS-FEDERATION/projects and GET /v3/OS-FEDERATION/domains
APIs were introduced to handle tokens from federated users, but now
that GET /v3/auth/projects and GET /v3/auth/domains know how to handle
federated tokens, they're just duplicate APIs.

In the past we deprecated these federated auth APIs, but they still
used separate code paths from GET /v3/auth/projects and GET
/v3/auth/domains. The two code paths are true duplication in that they
don't expect to differ over time and should provide the same user
experience.

Instead of running the risk that comes with two code paths that do the
same thing, we should consolidate them.

Conflicts:
  keystone/federation/controllers.py due to the fact that pre-Queens
  code used a different dependency framework. This was reworked in
  the Queens release, causing a conflict with this patch since it
  touches the same code.

Co-Authored-By: Kristi Nikolla <kristi@nikolla.me>

Closes-Bug: 1779205
Change-Id: Ib906c42e1dd2c2408ccd2e256ffd876af02af3fe
(cherry picked from commit df5d7557)

The JSON Schema validation implementation of nullable(), which makes
values possible to be null was not adding None to the enum if it exists.
This causes validation to fail on ``None`` especially in the case of
keystone's boolean parameter_type implementation. ``nullable()`` now
adds ``None`` to the enum if the enum exists.

Closes-Bug: #1763824
Change-Id: I176fa90df63049661413c445554dba9b7d87272a
(cherry picked from commit 78adf4b4)

Since the users table has a foreign key to the projects table[1], users
must be deleted before the domain can be deleted. However, the
notification emitted from the domain deletion comes too late, and
keystone runs into a foreign key reference error before it can delete
the users. This patch addresses the problem by adding a new internal
notification to alert the identity manager that users should be deleted.
This uses a new notification rather than the existing notification
because the existing one is used to alert listeners that the domain
deletion has been fully completed, whereas this one must happen in the
middle of the domain delete process.

The callback must also only try to delete SQL users. The LDAP driver
doesn't support deleting users, and we can't assume other drivers
support it either. Moreover, the foreign key reference is only a problem
for SQL users anyway.

Because our backend unit tests run with SQLite and foreign keys do not
work properly, we can't properly expose this bug in our unit tests, but
there is an accompanying tempest test[2][3] to validate this fix.

[1] https://github.com/openstack/keystone/blob/2bd88d3/keystone/common/sql/expand_repo/versions/014_expand_add_domain_id_to_user_table.py#L140-L141
[2] https://review.openstack.org/#/c/509610
[3] https://review.openstack.org/#/c/509947

Change-Id: If5bdb6f5eef80b50b000aed5188ce7da4dfd1083
Closes-bug: #1718747
(cherry picked from commit 62ee18b3)

Because we have policy in code, we should be able to use the
oslo.policy CLI scripts to produce sample policy files and render
complete policies based on overrides on disk. This was broken
because keystone wasn't removing unexpected commandline arguments
before passing them to oslo.config to parse. This prevented
people from generating complete policy files like they would for
horizon.

This commit exposes a get_enforcer() that substitutes an empty list
in place of arguments passed in through the system. This makes it
so that oslo.config doesn't choke when processing configuration
values.

Change-Id: I22583258eac5b3a64208355d18ccfa62dba1871d
Closes-Bug: 1740951
(cherry picked from commit 85c957c5)

For compliance with the Project Testing Interface as described in:

https://governance.openstack.org/tc/reference/project-testing-interface.html

Refer to:

http://lists.openstack.org/pipermail/openstack-dev/2017-November/124815.html

Change-Id: Ic05a55fb14741bb0802c217c81b0035b6e1f7de2
(cherry picked from commit 07bba320)

This change adds a note (*New in Pike*) to the section for running
db_sync --check to clarify which release this feature was added. This
is similar to other sections in this document that specify the release
that a feature was added in and when it is available.

Change-Id: I7d778fa249997bb24e4563d6960910b93637bbf2
Closes-Bug: #1734549
(cherry picked from commit 756d2814)

All users and groups are required to have a name. With this fix,
Keystone will ignore users and groups that do have only white
spaces as value for the LDAP attribute which Keystone has been
configured to use for that entity's name.

Change-Id: Id539e1b7e1cea8b05cd9bb753707e1fc98244d29
Closes-Bug: #1727726
(cherry picked from commit 789573a0)

LDAP servers have sizelimit configuration to limit the number of
user/group objects that can be returned for an LDAP query. This
change catches the size limit exceeded exception when users/groups
returned from ldap search query exceeds the configured limit and
responds with an appropriate error message instead of default
500 error message.

Change-Id: I9949bb7d458b4b037616c701e0e4d362bfa36473
Closes-Bug: #1712415
(cherry picked from commit f776fc18)

It looks the document will land on latest release page
(https://docs.openstack.org/keystone/latest), but it declares
for documenting the OpenStack Ocata release, which seems outdated.

Change-Id: I88ecd028eb31b9eaf68ddf743e949de8302736ac
(cherry picked from commit ad1b677c)

This change breaks the get-started-identity, keystone-users,
and keystone-openrc pages into multiple copies for each distribution
release in the keystone install guides. Before this change, sphinx
would not render the links correctly and these pages would be skipped
in both the RDO and Ubuntu guides.

To avoid unnecessary duplication of the same text, the files were
moved into common/ and are simply included into the newly created
files for each release.

Change-Id: I7b478504f1e68034ea993c0214705045da86bb6e
Closes-Bug: #1716899
Closes-Bug: #1716792
(cherry picked from commit 04515333)

The installation guide previously listed this as a step but
it is no longer needed since `admin_token_auth` is no longer in the
paste file by default

Change-Id: If36bafbecd657c3acd769f11636a1d844be0ac66
Closes-Bug: 1716797
(cherry picked from commit 5194a369)

Some notes were updated due to the docs migration but they were only
relevant for the ocata or newton release. This commit prevents those
notes from being rendered in the pike release notes document.

Change-Id: Id705bad286c75870746ef56c2c9c44a1cdf049a7