1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
|
===============
Resource Limits
===============
Nova supports configuring limits on individual resources including CPU, memory,
disk and network. These limits can be used to enforce basic Quality-of-Service
(QoS) policies on such resources.
.. note::
Hypervisor-enforced resource limits are distinct from API-enforced user and
project quotas. For information on the latter, refer to :doc:`quotas`.
.. warning::
This feature is poorly tested and poorly maintained. It may no longer work
as expected. Where possible, consider using the QoS policies provided by
other services, such as
:cinder-doc:`Cinder </admin/blockstorage-basic-volume-qos.html>` and
:neutron-doc:`Neutron </admin/config-qos.html>`.
Configuring resource limits
---------------------------
Resource quota enforcement support is specific to the virt driver in use on
compute hosts.
libvirt
~~~~~~~
The libvirt driver supports CPU, disk and VIF limits. Unfortunately all of
these work quite differently, as discussed below.
CPU limits
^^^^^^^^^^
Libvirt enforces CPU limits in terms of *shares* and *quotas*, configured
via :nova:extra-spec:`quota:cpu_shares` and :nova:extra-spec:`quota:cpu_period`
/ :nova:extra-spec:`quota:cpu_quota`, respectively. Both are implemented using
the `cgroups v1 cpu controller`__.
CPU shares are a proportional weighted share of total CPU resources relative to
other instances. It does not limit CPU usage if CPUs are not busy. There is no
unit and the value is purely relative to other instances, so an instance
configured with value of 2048 will get twice as much CPU time as a VM
configured with the value 1024. For example, to configure a CPU share of 1024
for a flavor:
.. code-block:: console
$ openstack flavor set $FLAVOR --property quota:cpu_shares=1024
The CPU quotas require both a period and quota. The CPU period specifies the
enforcement interval in microseconds, while the CPU quota specifies the maximum
allowed bandwidth in microseconds that the each vCPU of the instance can
consume. The CPU period must be in the range 1000 (1mS) to 1,000,000 (1s) or 0
(disabled). The CPU quota must be in the range 1000 (1mS) to 2^64 or 0
(disabled). Where the CPU quota exceeds the CPU period, this means the guest
vCPU process is able to consume multiple pCPUs worth of bandwidth. For example,
to limit each guest vCPU to 1 pCPU worth of runtime per period:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:cpu_period=1000 \
--property quota:cpu_quota=1000
To limit each guest vCPU to 2 pCPUs worth of runtime per period:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:cpu_period=1000 \
--property quota:cpu_quota=2000
Finally, to limit each guest vCPU to 0.5 pCPUs worth of runtime per period:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:cpu_period=1000 \
--property quota:cpu_quota=500
.. note::
Smaller periods will ensure a consistent latency response at the expense of
burst capacity.
CPU shares and CPU quotas can work hand-in-hand. For example, if two instances
were configured with :nova:extra-spec:`quota:cpu_shares`\ =1024 and
:nova:extra-spec:`quota:cpu_period`\ =100000 (100mS) for both, then configuring
both with a :nova:extra-spec:`quota:cpu_quota`\ =75000 (75mS) will result in
them sharing a host CPU equally, with both getting exactly 50mS of CPU time.
If instead only one instance gets :nova:extra-spec:`quota:cpu_quota`\ =75000
(75mS) while the other gets :nova:extra-spec:`quota:cpu_quota`\ =25000 (25mS),
then the first will get 3/4 of the time per period.
.. __: https://man7.org/linux/man-pages/man7/cgroups.7.html
Memory Limits
^^^^^^^^^^^^^
The libvirt driver does not support memory limits.
Disk I/O Limits
^^^^^^^^^^^^^^^
Libvirt enforces disk limits through maximum disk read, write and total bytes
per second, using the :nova:extra-spec:`quota:disk_read_bytes_sec`,
:nova:extra-spec:`quota:disk_write_bytes_sec` and
:nova:extra-spec:`quota:disk_total_bytes_sec` extra specs, respectively. It can
also enforce disk limits through maximum disk read, write and total I/O
operations per second, using the :nova:extra-spec:`quota:disk_read_iops_sec`,
:nova:extra-spec:`quota:disk_write_iops_sec` and
:nova:extra-spec:`quota:disk_total_iops_sec` extra specs, respectively. For
example, to set a maximum disk write of 10 MB/sec for a flavor:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:disk_write_bytes_sec=10485760
Network bandwidth limits
^^^^^^^^^^^^^^^^^^^^^^^^
.. warning::
These limits are enforced via libvirt and will only work where the network
is connect to the instance using a tap interface. It will not work for
things like :doc:`SR-IOV VFs <pci-passthrough>`.
:neutron-doc:`Neutron's QoS policies </admin/config-qos.html>` should be
preferred wherever possible.
Libvirt enforces network bandwidth limits through inbound and outbound average,
using the :nova:extra-spec:`quota:vif_inbound_average` and
:nova:extra-spec:`quota:vif_outbound_average` extra specs, respectively.
In addition, optional *peak* values, which specifies the maximum rate at which
a bridge can send data (kB/s), and *burst* values, which specifies the amount
of bytes that can be burst at peak speed (kilobytes), can be specified for both
inbound and outbound traffic, using the
:nova:extra-spec:`quota:vif_inbound_peak` /
:nova:extra-spec:`quota:vif_outbound_peak` and
:nova:extra-spec:`quota:vif_inbound_burst` /
:nova:extra-spec:`quota:vif_outbound_burst` extra specs, respectively.
For example, to configure **outbound** traffic to an average of 262 Mbit/s
(32768 kB/s), a peak of 524 Mbit/s, and burst of 65536 kilobytes:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:vif_outbound_average=32768 \
--property quota:vif_outbound_peak=65536 \
--property quota:vif_outbound_burst=65536
.. note::
The speed limit values in above example are specified in kilobytes/second,
whle the burst value is in kilobytes.
VMWare
~~~~~~
In contrast to libvirt, the VMWare virt driver enforces resource limits using
consistent terminology, specifically through relative allocation levels, hard
upper limits and minimum reservations configured via, for example, the
:nova:extra-spec:`quota:cpu_shares_level` /
:nova:extra-spec:`quota:cpu_shares_share`, :nova:extra-spec:`quota:cpu_limit`,
and :nova:extra-spec:`quota:cpu_reservation` extra specs, respectively.
Allocation levels can be specified using one of ``high``, ``normal``, ``low``,
or ``custom``. When ``custom`` is specified, the number of shares must be
specified using e.g. :nova:extra-spec:`quota:cpu_shares_share`. There is no
unit and the values are relative to other instances on the host. The upper
limits and reservations, by comparison, are measure in resource-specific units,
such as MHz for CPUs and will ensure that the instance never used more than or
gets less than the specified amount of the resource.
CPU limits
^^^^^^^^^^
CPU limits are configured via the :nova:extra-spec:`quota:cpu_shares_level` /
:nova:extra-spec:`quota:cpu_shares_share`, :nova:extra-spec:`quota:cpu_limit`,
and :nova:extra-spec:`quota:cpu_reservation` extra specs.
For example, to configure a CPU allocation level of ``custom`` with 1024
shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:cpu_shares_level=custom \
--quota:cpu_shares_share=1024
To configure a minimum CPU allocation of 1024 MHz and a maximum of 2048 MHz:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:cpu_reservation=1024 \
--quota:cpu_limit=2048
Memory limits
^^^^^^^^^^^^^
Memory limits are configured via the
:nova:extra-spec:`quota:memory_shares_level` /
:nova:extra-spec:`quota:memory_shares_share`,
:nova:extra-spec:`quota:memory_limit`, and
:nova:extra-spec:`quota:memory_reservation` extra specs.
For example, to configure a memory allocation level of ``custom`` with 1024
shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:memory_shares_level=custom \
--quota:memory_shares_share=1024
To configure a minimum memory allocation of 1024 MB and a maximum of 2048 MB:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:memory_reservation=1024 \
--quota:memory_limit=2048
Disk I/O limits
^^^^^^^^^^^^^^^
Disk I/O limits are configured via the
:nova:extra-spec:`quota:disk_io_shares_level` /
:nova:extra-spec:`quota:disk_io_shares_share`,
:nova:extra-spec:`quota:disk_io_limit`, and
:nova:extra-spec:`quota:disk_io_reservation` extra specs.
For example, to configure a disk I/O allocation level of ``custom`` with 1024
shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:disk_io_shares_level=custom \
--quota:disk_io_shares_share=1024
To configure a minimum disk I/O allocation of 1024 MB and a maximum of 2048 MB:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:disk_io_reservation=1024 \
--quota:disk_io_limit=2048
Network bandwidth limits
^^^^^^^^^^^^^^^^^^^^^^^^
Network bandwidth limits are configured via the
:nova:extra-spec:`quota:vif_shares_level` /
:nova:extra-spec:`quota:vif_shares_share`,
:nova:extra-spec:`quota:vif_limit`, and
:nova:extra-spec:`quota:vif_reservation` extra specs.
For example, to configure a network bandwidth allocation level of ``custom``
with 1024 shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:vif_shares_level=custom \
--quota:vif_shares_share=1024
To configure a minimum bandwidth allocation of 1024 Mbits/sec and a maximum of
2048 Mbits/sec:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:vif_reservation=1024 \
--quota:vif_limit=2048
Hyper-V
~~~~~~~
CPU limits
^^^^^^^^^^
The Hyper-V driver does not support CPU limits.
Memory limits
^^^^^^^^^^^^^
The Hyper-V driver does not support memory limits.
Disk I/O limits
^^^^^^^^^^^^^^^
Hyper-V enforces disk limits through maximum total bytes and total I/O
operations per second, using the :nova:extra-spec:`quota:disk_total_bytes_sec`
and :nova:extra-spec:`quota:disk_total_iops_sec` extra specs, respectively. For
example, to set a maximum disk read/write of 10 MB/sec for a flavor:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:disk_total_bytes_sec=10485760
Network bandwidth limits
^^^^^^^^^^^^^^^^^^^^^^^^
The Hyper-V driver does not support network bandwidth limits.
|
