1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
|
.. _cleaning:
=============
Node Cleaning
=============
Overview
========
When hardware is recycled from one workload to another, Ironic performs
cleaning on the node to ensure it's ready for another workload. This ensures
the tenant will get a consistent baremetal node deployed every time.
Ironic implements cleaning by collecting a list of steps to perform on a node
from each Power, Deploy, and Management driver assigned to the node. These
steps are then arranged by priority and executed on the node when it is moved
to cleaning state, if cleaning is enabled.
Typically, nodes move to cleaning state when moving from active -> available.
Nodes also traverse cleaning when going from manageable -> available. For a
full understanding of all state transitions into cleaning, please see
:ref:`states`.
Ironic added support for cleaning nodes in the Kilo release.
Enabling Cleaning
=================
To enable cleaning, ensure your ironic.conf is set as follows: ::
[conductor]
clean_nodes=true
This will enable the default set of steps, based on your hardware and Ironic
drivers. If you're using an agent_* driver, this includes, by default, erasing
all of the previous tenant's data.
.. _InbandvsOutOfBandCleaning:
In-Band vs Out-of-Band
======================
Ironic uses two main methods to perform actions on a node: in-band and
out-of-band. Ironic supports using both methods to clean a node.
In-Band
-------
In-band steps are performed by Ironic making API calls to a ramdisk running
on the node using a Deploy driver. Currently, only the ironic-python-agent
ramdisk used with an agent_* driver supports in-band cleaning. By default,
ironic-python-agent ships with a minimal cleaning configuration, only erasing
disks. However, with this ramdisk, you can add your own cleaning steps and/or
override default cleaning steps with a custom Hardware Manager.
There is currently no support for in-band cleaning using the Ironic pxe
ramdisk.
Out-of-Band
-----------
Out-of-band are actions performed by your management controller, such as IPMI,
iLO, or DRAC. Out-of-band steps will be performed by Ironic using a Power or
Management driver. Which steps are performed depends on the driver and hardware.
For Out-of-Band cleaning operations supported by iLO drivers, refer to
:ref:`ilo_node_cleaning`.
FAQ
===
How are cleaning steps ordered?
-------------------------------
Cleaning steps are ordered by integer priority, where a larger integer is a
higher priority. In case of a conflict between priorities across drivers,
the following resolution order is used: Power, Management, Deploy.
How do I skip a cleaning step?
------------------------------
Cleaning steps with a priority of 0 or None are skipped.
How do I change the priority of a cleaning step?
------------------------------------------------
Most out-of-band cleaning steps have an explicit configuration option for
priority.
Changing the priority of an in-band (ironic-python-agent) cleaning step
currently requires use of a custom HardwareManager. The only exception is
erase_devices, which can have its priority set in ironic.conf. For instance,
to disable erase_devices, you'd use the following config::
[agent]
agent_erase_devices_priority=0
To enable/disable the in-band disk erase using ``agent_ilo`` driver, use the
following config::
[ilo]
clean_priority_erase_devices=0
What cleaning step is running?
------------------------------
To check what cleaning step the node is performing or attempted to perform and
failed, either query the node endpoint for the node or run ``ironic node-show
$node_ident`` and look in the `internal_driver_info` field. The `clean_steps`
field will contain a list of all remaining steps with their priority, and the
first one listed is the step currently in progress or that the node failed
before going into cleanfail state.
Should I disable cleaning?
--------------------------
Cleaning is recommended for Ironic deployments, however, there are some
tradeoffs to having it enabled. For instance, Ironic cannot deploy a new
instance to a node that is currently cleaning, and cleaning can be a time
consuming process. To mitigate this, we suggest using disks with support for
cryptographic ATA Security Erase, as typically the erase_devices step in the
deploy driver takes the longest time to complete of all cleaning steps.
Why can't I power on/off a node while it's cleaning?
----------------------------------------------------
During cleaning, nodes may be performing actions that shouldn't be
interrupted, such as BIOS or Firmware updates. As a result, operators are
forbidden from changing power state via the Ironic API while a node is
cleaning.
Troubleshooting
===============
If cleaning fails on a node, the node will be put into cleanfail state and
placed in maintenance mode, to prevent Ironic from taking actions on the
node.
Nodes in cleanfail will not be powered off, as the node might be in a state
such that powering it off could damage the node or remove useful information
about the nature of the cleaning failure.
A cleanfail node can be moved to manageable state, where they cannot be
scheduled by Nova and you can safely attempt to fix the node. To move a node
from cleanfail to manageable: ``ironic node-set-provision-state manage``.
You can now take actions on the node, such as replacing a bad disk drive.
Strategies for determining why a cleaning step failed include checking the
Ironic conductor logs, viewing logs on the still-running ironic-python-agent
(if an in-band step failed), or performing general hardware troubleshooting on
the node.
When the node is repaired, you can move the node back to available state, to
allow it to be scheduled by Nova.
::
# First, move it out of maintenance mode
ironic node-set-maintenance $node_ident false
# Now, make the node available for scheduling by Nova
ironic node-set-provision-state $node_ident provide
The node will begin cleaning from the start, and move to available state
when complete.
|
