The InstancePCIRequest.request_id is used to correlate allocated
PciDevice objects with the InstancePCIRequest object triggered the PCI
allocation. For neutron port based PCI requests the
IstancePCIRequest.request_id was already set to a generated UUID by
nova. But for Flavor based request the request_id was kept None. The
placement PCI scheduling code depends on the request_id to be a unique
identifier of the request. So this patch starts filling the request_id
for flavor based requests as well.

This change showed than in some places nova still uses the request_id ==
None condition to distinguish between flavor based and neutron based
requests. This logic is now adapted to use the newer and better
InstancePCIRequest.source based approach. Also we took the opportunity
to move the logic of querying PCI devices allocated to an instance to the
Instance ovo.

This change fills the request_id for newly created flavor based
InstancePCIRequest ovos. But the change in logic to use the
InstancePCIRequest.source property instead of the request_id == None
condition works even if the request_id is None for already existing
InstancePCIRequest objects. So this patch does not include a data
migration logic to fill request_id for existing objects.

blueprint: pci-device-tracking-in-placement
Change-Id: I53e03ff7a0221db682b043fb6d5adba3f5c9fdbe

This patch continues cleaning up the "whitelist" terminology by
renaming exception.PciConfigInvalidWhitelist

blueprint: pci-device-tracking-in-placement
Change-Id: I054e17090818f01f7e7621ad6144fd7908346da2

A later patch in the pci-device-tracking-in-placement work
will extend the existing [pci]passthrough_whitelist config syntax.
So we take the opportunity here to deprecate the old non inclusive
passthrough_whitelist name and introduce a better one.

All the usage of CONF.pci.passthrough_whitelist is now changed over to
the new device_spec config. Also the in tree documentation is updated
accordinly.

However the nova code still has a bunch of references to the
"whitelist" terminology. That will be handled in subsequent patches.

blueprint: pci-device-tracking-in-placement
Change-Id: I843032e113642416114f169069eebf6a56ed78dd

We saw in the field that the pci_devices table can end up in
inconsistent state after a compute node HW failure and re-deployment.
There could be dependent devices where the parent PF is in available
state while the children VFs are in unavailable state. (Before the HW
fault the PF was allocated hence the VFs was marked unavailable).

In this state this PF is still schedulable but during the
PCI claim the handling of dependent devices in the PCI tracker fill fail
with the error: "Attempt to consume PCI device XXX from empty pool".

The reason of the failure is that when the PF is claimed, all the
children VFs are marked unavailable. But if the VF is already
unavailable such step fails.

One way the deployer might try to recover from this state is to remove
the VFs from the hypervisor and restart the compute agent. The compute
startup already has a logic to delete PCI devices that are unused and
not reported by the hypervisor. However this logic only removed devices
in 'available' state and ignored devices in 'unavailable' state.

If a device is unused and the hypervisor is not reporting the device any
more then it is safe to delete that device from the PCI tracker. So this
patch extends the logic to allow deleting 'unavailable' devices. There
is a small window when dependent PCI device is in 'unclaimable' state.
From cleanup perspective this is an analogous state. So it is also
added to the cleanup logic.

Related-Bug: #1969496
Change-Id: If9ab424cc7375a1f0d41b03f01c4a823216b3eb8

Addresses a long-standing TODO.

Change-Id: I57a1c4ca2ab23ca991344e73fb4ab5fbb2922723
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

The 'nova.pci' module is poorly understood and difficult to debug.
Start adding type hints to make this a little easier to parse and catch
dumb errors. Some code needs to be reworked to make 'mypy' happy, but
it's mostly just type annotations.

Note that because of how the 'nova.objects' module works, we need to
delay interpolation by using forward references, or expressing the type
as string literals to be resolved later [1].

[1] https://www.python.org/dev/peps/pep-0484/#forward-references

Change-Id: I2a609606806c6cabdf95d53339335f61743fc5b0
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>

This change extend the vnic type to PCI request dev type mapping to
support the vDPA vnic type.

This change extends the PCI stats module to filter out VDPA 'dev_type'
pools if its not explicitly requested.

This change explicitly filters out the vDPA dev_type from the pci alias
schema since that is not supported.

Blueprint: libvirt-vdpa-support
Change-Id: I91dd7993395f693c7d26c1caa44fa365f5cbec12

For the upcoming `socket` affinity, PCI stats needs to know the
host's NUMA topology in order to calculate the socket affinity of PCI
devices (based on their 'numa_node'). In this patch, the PCI manager
starts using its compute_node parameter to give a NUMA topology to PCI
stats.

PCI stats needs to track the NUMA topology at the object (not the
class) level, so all the '_filter_*()' classmethods that, in a
subsequent patch, will either use 'self.numa_topology' directly or
call a method that does, are moved to the object level. They were only
ever called via self anyways.

Implements: blueprint pci-socket-affinity
Change-Id: I1d270cc3e88a74097eefe3b887106222ae06fe1c

To implement the `socket` PCI NUMA affinity policy, we'll need to
track the host NUMA topology in the PCI stats code. To achieve this,
PCI stats will need to know the compute node it's running on. Prepare
for this by replacing the node_id parameter with compute_node. Node_id
was previously optional, but that looks to have been only to
facilitate testing, as that's the only place where it was not passed
it. We use compute_node (instead of just making node_id mandatory)
because it allows for an optimization later on wherein the PCI manager
does not need to pull the ComputeNode object from the database
needlessly.

Implements: blueprint pci-socket-affinity
Change-Id: Idc839312d1449e9327ee7e3793d53ed080a44d0c

Nova and QEMU[1] supports PCI devices with a PCI address that has 16 bit
domain. However there are hypervisors that reports PCI addresses with
32 bit domain. While today we cannot assign these to guests this should
not prevent the nova-compute service to start.

This patch changes the PCI manager to ignore such PCI devices.

Please note that this patch does not change fact that nova does not
allow specifying PCI addresses with 32bit domain in the
[pci]/passthrough_whitelist configuration. Such configuration is still
rejected at nova-compute service startup.

Closes-Bug: #1897528

[1] https://github.com/qemu/qemu/blob/f2a1cf9180f63e88bb38ff21c169da97c3f2bad5/hw/core/qdev-properties.c#L993

Change-Id: I59a0746b864610b6a314078cf5661d3d2b84b1d4