CXMAN-255: Retry config reset up to 3 times

We can't really guarantee that the CDB clear has occurred unless
the IPMI command returns properly, so, we better try it a couple
times.

This could cause a few unnecessary SPI erases in some cases, but
it's better than potentially reporting success when it hasn't
actually been reset at all.

1 files changed, 11 insertions, 9 deletions

diff --git a/cxmanage_api/node.py b/cxmanage_api/node.py
index b96e440..f71a380 100644
--- a/cxmanage_api/node.py
+++ b/cxmanage_api/node.py
@@ -939,25 +939,27 @@ updating the EEPROM
 communication.
 
         """
-        try:
 
-            # Reset CDB
+        # Clear CDB. Retry it up to 3 times.
+        for _ in range(2):
+            try:
+                self.bmc.reset_firmware()
+                break
+            except IpmiError as error:
+                if str(error) != "Error resetting firmware to factory default":
+                    raise
+                time.sleep(5) # pausing between retries seems to help a little
+        else:
             self.bmc.reset_firmware()
 
-        except IpmiError as err:
-            if ('Error resetting firmware to factory default' != str(err)):
-                raise
-
+        # Reset ubootenv
         try:
-
-            # Reset ubootenv
             fwinfo = self.get_firmware_info()
 
             running_part = self._get_partition(fwinfo, "UBOOTENV", "FIRST")
             factory_part = self._get_partition(fwinfo, "UBOOTENV", "SECOND")
             image = self._download_image(factory_part)
             self._upload_image(image, running_part)
-
         except NoPartitionError:
             pass  # Only one partition? Don't mess with it!