1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
|
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Block Device Mapping in Nova
============================
Nova has a concept of block devices that can be exposed to cloud instances.
There are several types of block devices an instance can have (we will go into
more details about this later in this document), and which ones are available
depends on a particular deployment and the usage limitations set for tenants
and users. Block device mapping is a way to organize and keep data about all of
the block devices an instance has.
When we talk about block device mapping, we usually refer to one of two things
1. API/CLI structure and syntax for specifying block devices for an instance
boot request
2. The data structure internal to Nova that is used for recording and keeping,
which is ultimately persisted in the block_device_mapping table. However,
Nova internally has several "slightly" different formats for representing
the same data. All of them are documented in the code and or presented by
a distinct set of classes, but not knowing that they exist might trip up
people reading the code. So in addition to BlockDeviceMapping [1]_ objects
that mirror the database schema, we have:
2.1 The API format - this is the set of raw key-value pairs received from
the API client, and is almost immediately transformed into the object;
however, some validations are done using this format. We will refer to this
format as the 'API BDMs' from now on.
2.2 The virt driver format - this is the format defined by the classes in
:mod: `nova.virt.block_device`. This format is used and expected by the code
in the various virt drivers. These classes, in addition to exposing a
different format (mimicking the Python dict interface), also provide a place
to bundle some functionality common to certain types of block devices (for
example attaching volumes which has to interact with both Cinder and the
virt driver code). We will refer to this format as 'Driver BDMs' from now
on.
For more details on this please refer to the :doc:`Driver BDM Data
Structures <../reference/block-device-structs>` refernce document.
.. note::
The maximum limit on the number of disk devices allowed to attach to
a single server is configurable with the option
:oslo.config:option:`compute.max_disk_devices_to_attach`.
API BDM data format and its history
-----------------------------------
In the early days of Nova, block device mapping general structure closely
mirrored that of the EC2 API. During the Havana release of Nova, block device
handling code, and in turn the block device mapping structure, had work done on
improving the generality and usefulness. These improvements included exposing
additional details and features in the API. In order to facilitate this, a new
extension was added to the v2 API called `BlockDeviceMappingV2Boot` [2]_, that
added an additional `block_device_mapping_v2` field to the instance boot API
request.
Block device mapping v1 (aka legacy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This was the original format that supported only cinder volumes (similar to how
EC2 block devices support only EBS volumes). Every entry was keyed by device
name (we will discuss why this was problematic in its own section later on
this page), and would accept only:
* UUID of the Cinder volume or snapshot
* Type field - used only to distinguish between volumes and Cinder volume
snapshots
* Optional size field
* Optional `delete_on_termination` flag
While all of Nova internal code only uses and stores the new data structure, we
still need to handle API requests that use the legacy format. This is handled
by the Nova API service on every request. As we will see later, since block
device mapping information can also be stored in the image metadata in Glance,
this is another place where we need to handle the v1 format. The code to handle
legacy conversions is part of the :mod: `nova.block_device` module.
Intermezzo - problem with device names
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Using device names as the primary per-instance identifier, and exposing them in
the API, is problematic for Nova mostly because several hypervisors Nova
supports with its drivers can't guarantee that the device names the guest OS
assigns are the ones the user requested from Nova. Exposing such a detail
in the public API of Nova is obviously not ideal, but it needed to stay for
backwards compatibility. It is also required for some (slightly obscure)
features around overloading a block device in a Glance image when booting an
instance [3]_.
The plan for fixing this was to allow users to not specify the device name of a
block device, and Nova will determine it (with the help of the virt driver), so
that it can still be discovered through the API and used when necessary, like
for the features mentioned above (and preferably only then).
Another use for specifying the device name was to allow the "boot from volume"
functionality, by specifying a device name that matches the root device name
for the instance (usually `/dev/vda`).
Currently (mid Liberty) users are discouraged from specifying device names
for all calls requiring or allowing block device mapping, except when trying to
override the image block device mapping on instance boot, and it will likely
remain like that in the future. Libvirt device driver will outright override
any device names passed with it's own values.
Block device mapping v2
^^^^^^^^^^^^^^^^^^^^^^^
New format was introduced in an attempt to solve issues with the original
block device mapping format discussed above, and also to allow for more
flexibility and addition of features that were not possible with the simple
format we had.
New block device mapping is a list of dictionaries containing the following
fields (in addition to the ones that were already there):
* source_type - this can have one of the following values:
* `image`
* `volume`
* `snapshot`
* `blank`
* destination_type - this can have one of the following values:
* `local`
* `volume`
* guest_format - Tells Nova how/if to format the device prior to attaching,
should be only used with blank local images. Denotes a swap disk if the value
is `swap`.
* device_name - See the previous section for a more in depth explanation of
this - currently best left empty (not specified that is), unless the user
wants to override the existing device specified in the image metadata.
In case of Libvirt, even when passed in with the purpose of overriding the
existing image metadata, final set of device names for the instance may still
get changed by the driver.
* disk_bus and device_type - low level details that some hypervisors (currently
only libvirt) may support. Some example disk_bus values can be: `ide`, `usb`,
`virtio`, `scsi`, while device_type may be `disk`, `cdrom`, `floppy`, `lun`.
This is not an exhaustive list as it depends on the virtualization driver,
and may change as more support is added. Leaving these empty is the most
common thing to do.
* boot_index - Defines the order in which a hypervisor will try devices when
attempting to boot the guest from storage. Each device which is capable of
being used as boot device should be given a unique boot index, starting from
0 in ascending order. Some hypervisors may not support booting from multiple
devices, so will only consider the device with boot index of 0. Some
hypervisors will support booting from multiple devices, but only if they are
of different types - eg a disk and CD-ROM. Setting a negative value or None
indicates that the device should not be used for booting. The simplest
usage is to set it to 0 for the boot device and leave it as None for any
other devices.
* volume_type - Added in microversion 2.67 to the servers create API to
support specifying volume type when booting instances. When we snapshot a
volume-backed server, the block_device_mapping_v2 image metadata will
include the volume_type from the BDM record so if the user then creates
another server from that snapshot, the volume that nova creates from that
snapshot will use the same volume_type. If a user wishes to change that
volume type in the image metadata, they can do so via the image API.
Valid source / destination combinations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Combination of the ``source_type`` and ``destination_type`` will define the
kind of block device the entry is referring to. The following
combinations are supported:
* `image` -> `local` - this is only currently reserved for the entry
referring to the Glance image that the instance is being booted with
(it should also be marked as a boot device). It is also worth noting
that an API request that specifies this, also has to provide the
same Glance uuid as the `image_ref` parameter to the boot request
(this is done for backwards compatibility and may be changed in the
future). This functionality might be extended to specify additional
Glance images to be attached to an instance after boot (similar to
kernel/ramdisk images) but this functionality is not supported by
any of the current drivers.
* `volume` -> `volume` - this is just a Cinder volume to be attached to the
instance. It can be marked as a boot device.
* `snapshot` -> `volume` - this works exactly as passing `type=snap` does.
It would create a volume from a Cinder volume snapshot and attach that
volume to the instance. Can be marked bootable.
* `image` -> `volume` - As one would imagine, this would download a Glance
image to a cinder volume and attach it to an instance. Can also be marked
as bootable. This is really only a shortcut for creating a volume out of
an image before booting an instance with the newly created volume.
* `blank` -> `volume` - Creates a blank Cinder volume and attaches it. This
will also require the volume size to be set.
* `blank` -> `local` - Depending on the guest_format field (see below),
this will either mean an ephemeral blank disk on hypervisor local
storage, or a swap disk (instances can have only one of those).
Nova will not allow mixing of BDMv1 and BDMv2 in a single request, and
will do basic validation to make sure that the requested block device
mapping is valid before accepting a boot request.
.. [1] In addition to the BlockDeviceMapping Nova object, we also have the
BlockDeviceDict class in :mod: `nova.block_device` module. This class
handles transforming and validating the API BDM format.
.. [2] This work predates API microversions and thus the only way to add it was
by means of an API extension.
.. [3] This is a feature that the EC2 API offers as well and has been in Nova
for a long time, although it has been broken in several releases. More info
can be found on `this bug <https://launchpad.net/bugs/1370250>`
FAQs
----
1. Is it possible to configure nova to automatically use cinder to back all
root disks with volumes?
No, there is nothing automatic within nova that converts a
non-:term:`boot-from-volume <Boot From Volume>` request to convert the
image to a root volume. Several ideas have been discussed over time which
are captured in the spec for `volume-backed flavors`_. However, if you wish
to force users to always create volume-backed servers, you can configure
the API service by setting :oslo.config:option:`max_local_block_devices`
to 0. This will result in any non-boot-from-volume server create request to
fail with a 400 response.
.. _volume-backed flavors: https://review.opendev.org/511965/
|
