Functions get_port_group() and get_port_groups() were failing
with KeyError while getting to access non-existent Port_Group
table with OVS 2.9.

This patch adds additional check if Port Group is supported in the
env where the code runs.

Change-Id: If3ce6ab1d1318a6d5142989b3bf45ecce97762e3
Closes-Bug: #1844491
(cherry picked from commit 4c42313b)

When Load Balancer and its member has FIP assigned
and environment is configured to use DVR the member
FIP needs to be centralized. It is current core OVN
limitation, that should be solved in [1].

This patch adds this mechanism to OVN Client and
OVN Octavia provider driver.

It covers cases:

1) FIP association on port that is a member of
   some LB - make it centralized.
2) FIP association on LB VIP - find a members
   FIPs and centralized them.
3) Add a member to LB that has FIP already
   configured - checks if a member has FIP
   and centralize it.
4) The reverse of each of the above cases.

In addition I needed to extend OVN LB member external_id
entry to add an information about member subnet_id
in order to easly track member port from mechanism OVN
driver.
That means I needed also to support both old and new
conventions. This patch adds also this code.

Old convention:
member_`member_id`_`ip_address`:`port`

New convention:
member_`member_id`_`ip_address`:`port`_`subnet_id`

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1793897

Related-Bug: #1860662

Change-Id: I254f0ac28f7585b699a8238e01ffb37dd70282ef

Right now neutron-server bumps the nb_cfg parameter in NB_Global
table which needs to be propagated by northd to SB_Global,
processed by agents, and write it back into SB_Global.
This requires processing by neutron-server but unfortunatelly
the server checks straight away and many times the value read
is behind the expected value.

All this results in frequent false positives showing dead agents
when they are not.

This patch is relaxing the checks by allowing a difference of 1
between the read and expected values.

Change-Id: Ic850665bde77c6891691e831317a0ec320ed197b
Backported-From: https://review.opendev.org/#/c/703612/


Closes-Bug: 1860436
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>

Sometimes there is a race condition on creation of VIP port
or deleting it that ends with exception and blocks LB stack
creation / deletion.

Because for now we don't have runnig master branch
for OVN provider driver this change will be applied
first on stable/train and then cherry-picked.

Change-Id: I2aaae7c407caba7a57e2ca2d4ed524f3bb63953f
Closes-Bug: #1860141

For now while updating FIP  check if port or logical_ip
has changed and only then we deleted the NAT entry.

Unfortunately each time when FIP update occurs the
method _create_or_update_floatingip() is used. It first deletes
LSP pointed by FIP and adds it again along with new NAT entries.
Based on author comment this actions are required.

So if we don't update FIP with logical_ip or new port_id,
like update a description, the NAT entries gets duplicated.

Since all is wrapped withing a transaction and to not wait for
proper fix (this code need sa refactor based on commments with NAT
external_id column) I think thats safe just to delete the NAT entry
in such situation like described above.

(cherry picked from commit 45ae9dfb7d5acacc72fcf9f071a9db1beb0ca972)

Change-Id: Iea532e2a02b7992305d1b90aa040e064901c340c
Related-Bug: #1859977

The command 'openstack overcloud container image upload' is a
deprecated command and has issues with latest releases.
This patch replaces the above command with 'openstack tripleo
container image prepare' which properly uploads container
images to local repository.

Change-Id: Ibe0dba6d76397a650d83238e4a6abc9f73e41465

In just added functional test there is a bug in dict
keys comparison. In python2 the dict keys are dumped to list that
is unordered. This makes random gate failures.

In py3 its dumped into dict_keys object that doesn't need
to be sorted.

Change-Id: I3aa7927b2eb0e9dd34935107835aa677fbcd1924

In some corner cases while updating a member, OVN
Provider Driver sends a malformed status update
to Octavia that breaks the update operation and
resources stuck in PENDING_UPDATE state.

In OVN while resource is administratively disabled
we suffix its ID with string ':D'.
This patch filters this string from resources ID
before sending Octavia a status update.

Because for now we don't have running master branch
for OVN provider driver this change will be applied
first on stable/train and then cherry-picked.

Change-Id: Ib2ce04625faddd6ed263678bad2f4eb10929a520
Closes-Bug: #1860140

It looks like sometimes, the device_id for ports created
by Neutron DHCP Agent can be in the form of:

- dhcp-$hostuuid-$networkid
- 'reserved_dhcp_port' (DEVICE_ID_RESERVED_DHCP_PORT)

Current code is only taking the first form into account when
skipping Neutron DHCP Agent ports. This patch is changing it
to include both forms.

Closes-Bug: #1848521
Co-Authored-By: Lucas Alvares Gomes <lucasagomes@gmail.com>
Change-Id: Ifbfc551ac68dcc5d3d39a155f7642f2f2d9272c4
Signed-off-by: Daniel Alvarez <dalvarez@redhat.com>
(cherry picked from commit 15bf8f26)

In case of subnet not found all the related actions could
not be taken. This patch prevent this situation to happen.
This could be possible that LB is going to be deleted after
the subnet (for example remove of the LoadBalancer in Error
state).

The same could happend with Logical Switch. Sometimes the
member network is delated in meantime and then we tries to
the members. It means we would need to remove LB to LS and
LB to LR associations, but it could not happend because of
missing LS. In those terms now we just pass the flow
and remove the LS to LR counters in external_ids.

Change-Id: I44f6de4c9a2b1c6054417dbb7f647a6af8339660
Closes-Bug: 1857009

Every time when a request comes from Octavia API and
is passed down to provider driver Octavia creates new
instance of Provider driver. This is good approach
while an API layer don't want to wait for previous API
actions to be end first, if centralized driver approach
will be used.

On different angle we have OvnProviderDriver and its
architecutre that was not created to work in dynamic
threads environment, but has centralized design.
Actual architecture looks like:

OvnProviderDriver object starts OvnPrividerHelper object
that manipulates OVN Databases. In OvnProviderHelper there is
a Queue.Queue object that stores actions to be executed
on databases with proper order. In addition OvnProviderHelper
starts dedicated thread that picks up actions from Queue
and execute those on OVN NorthBound database.

That solution does not scale while using it with Octavia
approach. There is a problem with thread leaking, because
in OvnProviderHelper we don't stop runner thread and we thread
it as daemon.

In addition on each API call the OVN Driver and dependent
components objects are created, this is big overhead.

To mitigate those architectural issues there is a possibility
to cache created OVNProviderHelper object and use one
instance of it for one Octavia API worker process.

This change must be cherry-picked to stable branches.

As next steps OVNProvider driver should be decomposed to:
   * Provider Agent as described in [1]
   * Provider Driver without Queue mechanism

[1] https://docs.openstack.org/octavia/latest/contributor/guides/providers.html#provider-agents

Change-Id: Ifaed77bbe6c36a63c18e568ec1d6ec5450dc8e03
Closes-Bug: 1856550

This change introduces a way of providing multiple listeners and pools
on the same Load Balancer, but with different L4 protocols.

In case there are more than one unique protocol defined in Octavia pools
or Octavia listeners the OVN driver now adds another Load_Balancer row
per protocol, but with the same name. The selection of which OVN
Load_Balancer should be used is done by a protocol key.

This change also adds possibility to expose two listeners on the same
port, but with different L4 protocol. The common use-case is providing
a DNS service that listens both on UDP and TCP sockets.

Partial-Bug: #1789157
Change-Id: I2ee13b75713e7538d1b4f70207a90a95315eb914

Previously this test needed OVN 2.12 release.
As it is shipped lets enable the test.

The test test_internal_dns.InternalDNSTest is configured
to use hardcoded domain_name value [0]. So in order
to have it passing we need to change our gates configuration
accordingly.

In addition we need to configure parameter ovn.dns_servers for
gate, because otherwise dns_server: 127.0.0.53 will be passed
for VM while using Ubuntu 18.04 on gates [1].

OPTION:  60 ( 12) Vendor class identifier   udhcp 1.23.2
---------------------------------------------------------------------------

  TIME: 2019-09-30 10:14:12.519
    IP: 10.1.0.1 (fa:16:3e:23:d4:4f) > 10.1.0.3 (fa:16:3e:82:52:f4)
    OP: 2 (BOOTPREPLY)
 HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
   XID: c9c04b3a
  SECS: 0
 FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 10.1.0.3
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: fa:16:3e:82:52:f4:00:00:00:00:00:00:00:00:00:00
 SNAME: .
 FNAME: .
OPTION:  53 (  1) DHCP message type         5 (DHCPACK)
OPTION: 121 ( 14) Classless Static Route    20a9fea9fe0a0100  .......
                                            02000a010001     ......
OPTION:   6 (  4) DNS server                127.0.0.53
OPTION:  15 ( 19) Domainname                openstackgate.local
OPTION:  51 (  4) IP address leasetime      43200 (12h)
OPTION:  26 (  2) Interface MTU             1342
OPTION:   1 (  4) Subnet mask               255.255.255.240
OPTION:   3 (  4) Routers                   10.1.0.1
OPTION:  54 (  4) Server identifier         10.1.0.1
OPTION:   0 (  0) pad
OPTION:   0 (  0) pad
OPTION:   0 (  0) pad
OPTION:   0 (255) pad

[0] https://github.com/openstack/neutron-tempest-plugin/blob/af04e1db7d6e5d7a2170a16a168f29bdbea42fe5/neutron_tempest_plugin/scenario/test_internal_dns.py#L75
[1] https://github.com/openstack/networking-ovn/blob/0337b2e582fa96c8937794bc5a72ec744e979ad1/networking_ovn/common/ovn_client.py#L1612
Change-Id: I50ae263d49bb512a33c0202107d97970bcbaf6ae
(cherry picked from commit fd50c883)

Previously [0] passed parameter was taken from the port domain.
In fact it is wrong, based on what is done in ml2/ovs it should be
taken from neutron configuration [1].

[0] https://review.opendev.org/#/c/660423
[1] https://github.com/openstack/neutron/blob/b0a93df47637958fc3fc172f15e8593d6e9b181a/neutron/agent/linux/dhcp.py#L427

Change-Id: I13f9213c558fc95d97052fb8b98246f1ef555120
(cherry picked from commit 85de4e8e)

This patch adds Octavia CI jobs for OVN.
Notes:

- OVN currently doesn't support L7. L7 tempest tests are disabled.
- OVN currently doesn't support health check for LB members. Health monitor
  tests are disabled.
- Skipped ipv6 tests because mixing IPv4 and IPv6 members
  is not allowed in OVN.
- OVN currently doesn't support flavors. Related tests are disabled.

Co-Authored-By: Reedip Banerjee <rbanerje@redhat.com>
Co-Authored-By: Carlos Goncalves <cgoncalves@redhat.com>
Co-Authored-By: Maciej Józefczyk <mjozefcz@redhat.com>

Change-Id: If28ec1ce8179afb47a4ef11ee953f3d7db98408a

Add file to the reno documentation build to show release notes for
stable/train.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.

Change-Id: I5807327737707a7c1057aa5b384c212d96faf7e0
Sem-Ver: feature

Prior to this patch, the "unknown" address was being set the logical
switch ports solely based on whether the port security was enabled or
disabled. That's not how it's intended to work.

With this patch the "unknown" address is only set to the normal logical
switch ports, those which types are "router", "localnet" or "localport"
won't be affected.

The maintenance task was updated to correct this behavior for existing
ports (the maintenance was suppose to be removed in the T cycle, this
patch bumps it to U cycle instead due to this change).

Closes-Bug: #1838535
Related-Bug: #1815270
Change-Id: I3c01bd7d1685c8a7e13a55e545e98baf19e9a0f9
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
(cherry picked from commit 6a89dbfe)

This is a tweak to changes done to fix bug 1834637. Specifically,
this change addresses scaling. The previous gerrit change had
modifications to all OVN sub-ports performed as a single
transaction. That did not account for race-condition on neutron
DB queries, which leads to timeouts under heavy loads.

Another cleanup done by this change is to fold an additional
update on neutron db into ovn trunk driver. That saves
update from doing another database transaction, thus making
it faster. The no longer needed function in mech_driver was
called _update_subport_host_if_needed

By breaking the iteration into multiple transactions, the
change in time is marginal:

  Service-level agreement
  NeutronTrunks :: neutron.create_trunk

  from 34.2 sec to 35.6 for 50%ile
  from 35.6 sec to 36.1 for 95%ile

This patch doesn't go to master as it's frozen now because of
the code being merged to Neutron. The patch will be sent to master
Neutron once code migration is finished.

Change-Id: I7de3ac2a7cf8869ead8ab5fbb34a9861a96d3a0c
Closes-Bug: #1834637
Co-authored-by: Maciej Józefczyk <mjozefcz@redhat.com>

The following issues that are now exposed by hacking version 2.0.0

./networking_ovn/tests/unit/common/test_hash_ring_manager.py:92:38:
F602 dictionary key variable another_host_node repeated with different
values
 hash_dict_after_rebalance = {another_host_node: 'fake-uuid',

Also fixing

./networking_ovn/ovsdb/ovsdb_monitor.py:359:13: F841 local variable
'e' is assigned to but never used

which is already fixed in the neutron repo, so this is only an
issue in networking-ovn repo.

Also, this gerrit addresses an issue with functional tests where
ovsdbapp expects OVN schema to be located at '/usr/local/share/ovn/'.
So we are hard coding OVN_SRCDIR as /usr/local/share/openvswitch
to unblock the Zuul gate. The real fix is to make ovsdbapp smarter
about that directory, and then revert that change in tox.ini. This
particular issue is related to change
https://review.opendev.org/#/c/698293/ . Since functional tests are
not run for changes in tox.ini [1] so that fell through thecracks.

[1]: https://github.com/openstack/networking-ovn/blob/0d603a7e105d2cbe8ca21a1ee752e1f3c81879e0/zuul.d/legacy-networking-ovn-jobs.yaml#L10

Depends-On: https://review.opendev.org/699083/


Change-Id: I9396d8af17d281eef15777e70a7d8fc0791f20b2
Signed-off-by: Flavio Fernandes <flaviof@redhat.com>
Co-authored-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Co-authored-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
Co-authored-by: Terry Wilson <twilson@redhat.com>
(cherry picked from commit b869f30e0b082d789c6668ec1a3ee69f2142798e)

A change in TripleO [1] changed behavior and forbids update when backend is
changed. Unless ForceNeutronDriverUpdate is set. This patch sets the
parameter when migrating from ML2/OVS to OVN. More information at [2].

[1] https://opendev.org/openstack/tripleo-common/commit/379f886119727fe440e384234db53f48730e27ad
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1769880

Change-Id: Ifea66cbabc12f22fdbe62b4bed5bff785869bcce

Downstream CI ML2/OVS->ML2/OVN migration job fails on
prepare-migration : Set image tag (tripleo deployment) task due to
change in container images names from 'Docker*' to 'Container*' in
containers-default-parameters.yaml

Change-Id: I8b7f03a9afb94418ff3f2e9a148f5cfb92b966f1

Some of networking-ovn functional tests inherit from ovsdbapp base
class. ovsdbapp uses environment variables to define directories
containing database schemas. This patch passes environment variable to
tox if it's defined. If it's not defined, an empty string is used and
code later uses its own defaults.

This patch doesn't go to master as it's frozen now because of the code
being merged to Neutron. The patch will be sent to master Neutron once
code migration is finished.

Change-Id: I5ccfc625352505c5b53f18e051e4d18598d5e286
Closes-bug: #1855903