docs(psci): add design proposal for OS-initiated mode

Change-Id: Ia3662e08d98d01a93951309835816969e1602624
Signed-off-by: Wing Li <wingers@google.com>

2 files changed, 717 insertions, 0 deletions

diff --git a/docs/design_documents/index.rst b/docs/design_documents/index.rst
index 3d82e69cb..d20fc5809 100644
--- a/docs/design_documents/index.rst
+++ b/docs/design_documents/index.rst
@@ -10,6 +10,7 @@ Design Documents
    measured_boot_poc
    drtm_poc
    rss
+   psci_osi_mode
 
 --------------
 
diff --git a/docs/design_documents/psci_osi_mode.rst b/docs/design_documents/psci_osi_mode.rst
new file mode 100644
index 000000000..3296e270c
--- /dev/null
+++ b/docs/design_documents/psci_osi_mode.rst
@@ -0,0 +1,716 @@
+PSCI OS-initiated mode
+======================
+
+:Author: Maulik Shah & Wing Li
+:Organization: Qualcomm Innovation Center, Inc. & Google LLC
+:Contact: Maulik Shah <quic_mkshah@quicinc.com> & Wing Li <wingers@google.com>
+:Status: RFC
+
+.. contents:: Table of Contents
+
+Introduction
+------------
+
+Power state coordination
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+A power domain topology is a logical hierarchy of power domains in a system that
+arises from the physical dependencies between power domains.
+
+Local power states describe power states for an individual node, and composite
+power states describe the combined power states for an individual node and its
+parent node(s).
+
+Entry into low-power states for a topology node above the core level requires
+coordinating its children nodes. For example, in a system with a power domain
+that encompasses a shared cache, and a separate power domain for each core that
+uses the shared cache, the core power domains must be powered down before the
+shared cache power domain can be powered down.
+
+PSCI supports two modes of power state coordination: platform-coordinated and
+OS-initiated.
+
+Platform-coordinated
+~~~~~~~~~~~~~~~~~~~~
+
+Platform-coordinated mode is the default mode of power state coordination, and
+is currently the only supported mode in TF-A.
+
+In platform-coordinated mode, the platform is responsible for coordinating power
+states, and chooses the deepest power state for a topology node that can be
+tolerated by its children.
+
+OS-initiated
+~~~~~~~~~~~~
+
+OS-initiated mode is optional.
+
+In OS-initiated mode, the calling OS is responsible for coordinating power
+states, and may request for a topology node to enter a low-power state when
+its last child enters the low-power state.
+
+Motivation
+----------
+
+There are two reasons why OS-initiated mode might be a more suitable option than
+platform-coordinated mode for a platform.
+
+Scalability
+^^^^^^^^^^^
+
+In platform-coordinated mode, each core independently selects their own local
+power states, and doesn't account for composite power states that are shared
+between cores.
+
+In OS-initiated mode, the OS has knowledge of the next wakeup event for each
+core, and can have more precise control over the entry, exit, and wakeup
+latencies when deciding if a composite power state (e.g. for a cluster) is
+appropriate. This is especially important for multi-cluster SMP systems and
+heterogeneous systems like big.LITTLE, where different processor types can have
+different power efficiencies.
+
+Simplicity
+^^^^^^^^^^
+
+In platform-coordinated mode, the OS doesn't have visibility when the last core
+at a power level enters a low-power state. If the OS wants to perform last man
+activity (e.g. powering off a shared resource when it is no longer needed), it
+would have to communicate with an API side channel to know when it can do so.
+This could result in a design smell where the platform is using
+platform-coordinated mode when it should be using OS-initiated mode instead.
+
+In OS-initiated mode, the OS can perform last man activity if it selects a
+composite power state when the last core enters a low-power state. This
+eliminates the need for a side channel, and uses the well documented API between
+the OS and the platform.
+
+Current vendor implementations and workarounds
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* STMicroelectronics
+
+  * For their ARM32 platforms, they're using OS-initiated mode implemented in
+    OP-TEE.
+  * For their future ARM64 platforms, they are interested in using OS-initiated
+    mode in TF-A.
+
+* Qualcomm
+
+  * For their mobile platforms, they're using OS-initiated mode implemented in
+    their own custom secure monitor firmware.
+  * For their Chrome OS platforms, they're using platform-coordinated mode in
+    TF-A with custom driver logic to perform last man activity.
+
+* Google
+
+  * They're using platform-coordinated mode in TF-A with custom driver logic to
+    perform last man activity.
+
+Both Qualcomm and Google would like to be able to use OS-initiated mode in TF-A
+in order to simplify custom driver logic.
+
+Requirements
+------------
+
+PSCI_FEATURES
+^^^^^^^^^^^^^
+
+PSCI_FEATURES is for checking whether or not a PSCI function is implemented and
+what its properties are.
+
+.. c:macro:: PSCI_FEATURES
+
+   :param func_id: 0x8400_000A.
+   :param psci_func_id: the function ID of a PSCI function.
+   :retval NOT_SUPPORTED: if the function is not implemented.
+   :retval feature flags associated with the function: if the function is
+       implemented.
+
+CPU_SUSPEND feature flags
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Reserved, bits[31:2]
+* Power state parameter format, bit[1]
+
+  * A value of 0 indicates the original format is used.
+  * A value of 1 indicates the extended format is used.
+
+* OS-initiated mode, bit[0]
+
+  * A value of 0 indicates OS-initiated mode is not supported.
+  * A value of 1 indicates OS-initiated mode is supported.
+
+See sections 5.1.14 and 5.15 of the PSCI spec (DEN0022D.b) for more details.
+
+PSCI_SET_SUSPEND_MODE
+^^^^^^^^^^^^^^^^^^^^^
+
+PSCI_SET_SUSPEND_MODE is for switching between the two different modes of power
+state coordination.
+
+.. c:macro:: PSCI_SET_SUSPEND_MODE
+
+   :param func_id: 0x8400_000F.
+   :param mode: 0 indicates platform-coordinated mode, 1 indicates OS-initiated
+       mode.
+   :retval SUCCESS: if the request is successful.
+   :retval NOT_SUPPORTED: if OS-initiated mode is not supported.
+   :retval INVALID_PARAMETERS: if the requested mode is not a valid value (0 or
+       1).
+   :retval DENIED: if the cores are not in the correct state.
+
+Switching from platform-coordinated to OS-initiated is only allowed if the
+following conditions are met:
+
+* All cores are in one of the following states:
+
+  * Running.
+  * Off, through a call to CPU_OFF or not yet booted.
+  * Suspended, through a call to CPU_DEFAULT_SUSPEND.
+
+* None of the cores has called CPU_SUSPEND since the last change of mode or
+  boot.
+
+Switching from OS-initiated to platform-coordinated is only allowed if all cores
+other than the calling core are off, either through a call to CPU_OFF or not yet
+booted.
+
+If these conditions are not met, the PSCI implementation must return DENIED.
+
+See sections 5.1.19 and 5.20 of the PSCI spec (DEN0022D.b) for more details.
+
+CPU_SUSPEND
+^^^^^^^^^^^
+
+CPU_SUSPEND is for moving a topology node into a low-power state.
+
+.. c:macro:: CPU_SUSPEND
+
+   :param func_id: 0xC400_0001.
+   :param power_state: the requested low-power state to enter.
+   :param entry_point_address: the address at which the core must resume
+       execution following wakeup from a powerdown state.
+   :param context_id: this field specifies a pointer to the saved context that
+       must be restored on a core following wakeup from a powerdown state.
+   :retval SUCCESS: if the request is successful.
+   :retval INVALID_PARAMETERS: in OS-initiated mode, this error is returned when
+       a low-power state is requested for a topology node above the core level,
+       and at least one of the node's children is in a local low-power state
+       that is incompatible with the request.
+   :retval INVALID_ADDRESS: if the entry_point_address argument is invalid.
+   :retval DENIED: only in OS-initiated mode; this error is returned when a
+       low-power state is requested for a topology node above the core level,
+       and at least one of the node's children is running, i.e. not in a
+       low-power state.
+
+In platform-coordinated mode, the PSCI implementation coordinates requests from
+all cores to determine the deepest power state to enter.
+
+In OS-initiated mode, the calling OS is making an explicit request for a
+specific power state, as opposed to expressing a vote. The PSCI implementation
+must comply with the request, unless the request is not consistent with the
+implementation's view of the system's state, in which case, the implementation
+must return INVALID_PARAMETERS or DENIED.
+
+See sections 5.1.2 and 5.4 of the PSCI spec (DEN0022D.b) for more details.
+
+Power state formats
+~~~~~~~~~~~~~~~~~~~
+
+Original format
+
+* Power Level, bits[25:24]
+
+  * The requested level in the power domain topology to enter a low-power
+    state.
+
+* State Type, bit[16]
+
+  * A value of 0 indicates a standby or retention state.
+  * A value of 1 indicates a powerdown state.
+
+* State ID, bits[15:0]
+
+  * Field to specify the requested composite power state.
+  * The state ID encodings must uniquely describe every possible composite
+    power state.
+  * In OS-initiated mode, the state ID encoding must allow expressing the
+    power level at which the calling core is the last to enter a powerdown
+    state.
+
+Extended format
+
+* State Type, bit[30]
+* State ID, bits[27:0]
+
+Races in OS-initiated mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In OS-initiated mode, there are race windows where the OS's view and
+implementation's view of the system's state differ. It is possible for the OS to
+make requests that are invalid given the implementation's view of the system's
+state. For example, the OS might request a powerdown state for a node from one
+core, while at the same time, the implementation observes that another core in
+that node is powering up.
+
+To address potential race conditions in power state requests:
+
+* The calling OS must specify in each CPU_SUSPEND request the deepest power
+  level for which it sees the calling core as the last running core (last man).
+  This is required even if the OS doesn't want the node at that power level to
+  enter a low-power state.
+* The implementation must validate that the requested power states in the
+  CPU_SUSPEND request are consistent with the system's state, and that the
+  calling core is the last core running at the requested power level, or deny
+  the request otherwise.
+
+See sections 4.2.3.2, 6.2, and 6.3 of the PSCI spec (DEN0022D.b) for more
+details.
+
+Caveats
+-------
+
+CPU_OFF
+^^^^^^^
+
+CPU_OFF is always platform-coordinated, regardless of whether the power state
+coordination mode for suspend is platform-coordinated or OS-initiated. If all
+cores in a topology node call CPU_OFF, the last core will power down the node.
+
+In OS-initiated mode, if a subset of the cores in a topology node has called
+CPU_OFF, the last running core may call CPU_SUSPEND to request a powerdown state
+at or above that node's power level.
+
+See section 5.5.2 of the PSCI spec (DEN0022D.b) for more details.
+
+Implementation
+--------------
+
+Current implementation of platform-coordinated mode
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Platform-coordinated is currently the only supported power state coordination
+mode in TF-A.
+
+The functions of interest in the ``psci_cpu_suspend`` call stack are as follows:
+
+* ``psci_validate_power_state``
+
+  * This function calls a platform specific ``validate_power_state`` handler,
+    which takes the ``power_state`` parameter, and updates the ``state_info``
+    object with the requested states for each power level.
+
+* ``psci_find_target_suspend_lvl``
+
+  * This function takes the ``state_info`` object containing the requested power
+    states for each power level, and returns the deepest power level that was
+    requested to enter a low power state, i.e. the target power level.
+
+* ``psci_do_state_coordination``
+
+  * This function takes the target power level and the ``state_info`` object
+    containing the requested power states for each power level, and updates the
+    ``state_info`` object with the coordinated target power state for each
+    level.
+
+* ``pwr_domain_suspend``
+
+  * This is a platform specific handler that takes the ``state_info`` object
+    containing the target power states for each power level, and transitions
+    each power level to the specified power state.
+
+Proposed implementation of OS-initiated mode
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To add support for OS-initiated mode, the following changes are proposed:
+
+* Add a boolean build option ``PSCI_OS_INIT_MODE`` for a platform to enable
+  optional support for PSCI OS-initiated mode. This build option defaults to 0.
+
+.. note::
+
+   If ``PSCI_OS_INIT_MODE=0``, the following changes will not be compiled into
+   the build.
+
+* Update ``psci_features`` to return 1 in bit[0] to indicate support for
+  OS-initiated mode for CPU_SUSPEND.
+* Define a ``suspend_mode`` enum: ``PLAT_COORD`` and ``OS_INIT``.
+* Define a ``psci_suspend_mode`` global variable with a default value of
+  ``PLAT_COORD``.
+* Implement a new function handler ``psci_set_suspend_mode`` for
+  PSCI_SET_SUSPEND_MODE.
+* Since ``psci_validate_power_state`` calls a platform specific
+  ``validate_power_state`` handler, the platform implementation should populate
+  the ``state_info`` object based on the state ID from the given ``power_state``
+  parameter.
+* ``psci_find_target_suspend_lvl`` remains unchanged.
+* Implement a new function ``psci_validate_state_coordination`` that ensures the
+  request satisfies the following conditions, and denies any requests
+  that don't:
+
+  * The requested power states for each power level are consistent with the
+    system's state
+  * The calling core is the last core running at the requested power level
+
+  This function differs from ``psci_do_state_coordination`` in that:
+
+  * The ``psci_req_local_pwr_states`` map is not modified if the request were to
+    be denied
+  * The ``state_info`` argument is never modified since it contains the power
+    states requested by the calling OS
+
+* Update ``psci_cpu_suspend_start`` to do the following:
+
+  * If ``PSCI_SUSPEND_MODE`` is ``PLAT_COORD``, call
+    ``psci_do_state_coordination``.
+  * If ``PSCI_SUSPEND_MODE`` is ``OS_INIT``, call
+    ``psci_validate_state_coordination``. If validation fails, propagate the
+    error up the call stack.
+
+* Update the return type of the platform specific ``pwr_domain_suspend``
+  handler from ``void`` to ``int``, to allow the platform to optionally perform
+  validations based on hardware states.
+
+.. image:: ../resources/diagrams/psci-osi-mode.png
+
+Testing
+-------
+
+The proposed patches can be found at
+https://review.trustedfirmware.org/q/topic:psci-osi.
+
+Testing on FVP and Google platforms
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The proposed patches add a new CPU Suspend in OSI mode test suite to TF-A Tests.
+This has been enabled and verified on the FVP_Base_RevC-2xAEMvA platform and
+Google platforms, and excluded from all other platforms via the build option
+``PLAT_TESTS_SKIP_LIST``.
+
+Testing on STM32MP15
+^^^^^^^^^^^^^^^^^^^^
+
+The proposed patches have been tested and verified on the STM32MP15 platform,
+which has a single cluster with 2 CPUs, by Gabriel Fernandez
+<gabriel.fernandez@st.com> from STMicroelectronics with this device tree
+configuration:
+
+.. code-block:: devicetree
+
+   cpus {
+           #address-cells = <1>;
+           #size-cells = <0>;
+
+           cpu0: cpu@0 {
+                   device_type = "cpu";
+                   compatible = "arm,cortex-a7";
+                   reg = <0>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD0>;
+                   power-domain-names = "psci";
+           };
+           cpu1: cpu@1 {
+                   device_type = "cpu";
+                   compatible = "arm,cortex-a7";
+                   reg = <1>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD1>;
+                   power-domain-names = "psci";
+           };
+
+           idle-states {
+                   cpu_retention: cpu-retention {
+                           compatible = "arm,idle-state";
+                           arm,psci-suspend-param = <0x00000001>;
+                           entry-latency-us = <130>;
+                           exit-latency-us = <620>;
+                           min-residency-us = <700>;
+                           local-timer-stop;
+                   };
+           };
+
+           domain-idle-states {
+                   CLUSTER_STOP: core-power-domain {
+                           compatible = "domain-idle-state";
+                           arm,psci-suspend-param = <0x01000001>;
+                           entry-latency-us = <230>;
+                           exit-latency-us = <720>;
+                           min-residency-us = <2000>;
+                           local-timer-stop;
+                   };
+           };
+   };
+
+   psci {
+           compatible = "arm,psci-1.0";
+           method = "smc";
+
+           CPU_PD0: power-domain-cpu0 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&pd_core>;
+                   domain-idle-states = <&cpu_retention>;
+           };
+
+           CPU_PD1: power-domain-cpu1 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&pd_core>;
+                   domain-idle-states = <&cpu_retention>;
+           };
+
+           pd_core: power-domain-cluster {
+                   #power-domain-cells = <0>;
+                   domain-idle-states = <&CLUSTER_STOP>;
+           };
+   };
+
+Testing on Qualcomm SC7280
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The proposed patches have been tested and verified on the SC7280 platform by
+Maulik Shah <quic_mkshah@quicinc.com> from Qualcomm with this device tree
+configuration:
+
+.. code-block:: devicetree
+
+   cpus {
+           #address-cells = <2>;
+           #size-cells = <0>;
+
+           CPU0: cpu@0 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x0>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD0>;
+                   power-domain-names = "psci";
+           };
+
+           CPU1: cpu@100 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x100>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD1>;
+                   power-domain-names = "psci";
+           };
+
+           CPU2: cpu@200 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x200>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD2>;
+                   power-domain-names = "psci";
+           };
+
+           CPU3: cpu@300 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x300>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD3>;
+                   power-domain-names = "psci";
+           }
+
+           CPU4: cpu@400 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x400>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD4>;
+                   power-domain-names = "psci";
+           };
+
+           CPU5: cpu@500 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x500>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD5>;
+                   power-domain-names = "psci";
+           };
+
+           CPU6: cpu@600 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x600>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD6>;
+                   power-domain-names = "psci";
+           };
+
+           CPU7: cpu@700 {
+                   device_type = "cpu";
+                   compatible = "arm,kryo";
+                   reg = <0x0 0x700>;
+                   enable-method = "psci";
+                   power-domains = <&CPU_PD7>;
+                   power-domain-names = "psci";
+           };
+
+           idle-states {
+                   entry-method = "psci";
+
+                   LITTLE_CPU_SLEEP_0: cpu-sleep-0-0 {
+                           compatible = "arm,idle-state";
+                           idle-state-name = "little-power-down";
+                           arm,psci-suspend-param = <0x40000003>;
+                           entry-latency-us = <549>;
+                           exit-latency-us = <901>;
+                           min-residency-us = <1774>;
+                           local-timer-stop;
+                   };
+
+                   LITTLE_CPU_SLEEP_1: cpu-sleep-0-1 {
+                           compatible = "arm,idle-state";
+                           idle-state-name = "little-rail-power-down";
+                           arm,psci-suspend-param = <0x40000004>;
+                           entry-latency-us = <702>;
+                           exit-latency-us = <915>;
+                           min-residency-us = <4001>;
+                           local-timer-stop;
+                   };
+
+                   BIG_CPU_SLEEP_0: cpu-sleep-1-0 {
+                           compatible = "arm,idle-state";
+                           idle-state-name = "big-power-down";
+                           arm,psci-suspend-param = <0x40000003>;
+                           entry-latency-us = <523>;
+                           exit-latency-us = <1244>;
+                           min-residency-us = <2207>;
+                           local-timer-stop;
+                   };
+
+                   BIG_CPU_SLEEP_1: cpu-sleep-1-1 {
+                           compatible = "arm,idle-state";
+                           idle-state-name = "big-rail-power-down";
+                           arm,psci-suspend-param = <0x40000004>;
+                           entry-latency-us = <526>;
+                           exit-latency-us = <1854>;
+                           min-residency-us = <5555>;
+                           local-timer-stop;
+                   };
+           };
+
+           domain-idle-states {
+                   CLUSTER_SLEEP_0: cluster-sleep-0 {
+                           compatible = "arm,idle-state";
+                           idle-state-name = "cluster-power-down";
+                           arm,psci-suspend-param = <0x40003444>;
+                           entry-latency-us = <3263>;
+                           exit-latency-us = <6562>;
+                           min-residency-us = <9926>;
+                           local-timer-stop;
+                   };
+           };
+   };
+
+   psci {
+           compatible = "arm,psci-1.0";
+           method = "smc";
+
+           CPU_PD0: cpu0 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>;
+           };
+
+           CPU_PD1: cpu1 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>;
+           };
+
+           CPU_PD2: cpu2 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>;
+           };
+
+           CPU_PD3: cpu3 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&LITTLE_CPU_SLEEP_0 &LITTLE_CPU_SLEEP_1>;
+           };
+
+           CPU_PD4: cpu4 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>;
+           };
+
+           CPU_PD5: cpu5 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>;
+           };
+
+           CPU_PD6: cpu6 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>;
+           };
+
+           CPU_PD7: cpu7 {
+                   #power-domain-cells = <0>;
+                   power-domains = <&CLUSTER_PD>;
+                   domain-idle-states = <&BIG_CPU_SLEEP_0 &BIG_CPU_SLEEP_1>;
+           };
+
+           CLUSTER_PD: cpu-cluster0 {
+                   #power-domain-cells = <0>;
+                   domain-idle-states = <&CLUSTER_SLEEP_0>;
+           };
+   };
+
+Comparisons on Qualcomm SC7280
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+CPUIdle states
+~~~~~~~~~~~~~~
+
+* 8 CPUs, 1 L3 cache
+* Platform-coordinated mode
+
+  * CPUIdle states
+
+    * State0 - WFI
+    * State1 - Core collapse
+    * State2 - Rail collapse
+    * State3 - L3 cache off and system resources voted off
+
+* OS-initiated mode
+
+  * CPUIdle states
+
+    * State0 - WFI
+    * State1 - Core collapse
+    * State2 - Rail collapse
+
+  * Cluster domain idle state
+
+    * State3 - L3 cache off and system resources voted off
+
+.. image:: ../resources/diagrams/psci-flattened-vs-hierarchical-idle-states.png
+
+Results
+~~~~~~~
+
+* The following stats have been captured with fixed CPU frequencies from the use
+  case of 10 seconds of device idle with the display turned on and Wi-Fi and
+  modem turned off.
+* Count refers to the number of times a CPU or cluster entered power collapse.
+* Residency refers to the time in seconds a CPU or cluster stayed in power
+  collapse.
+* The results are an average of 3 iterations of actual counts and residencies.
+
+.. image:: ../resources/diagrams/psci-pc-mode-vs-osi-mode.png
+
+OS-initiated mode was able to scale better than platform-coordinated mode for
+multiple CPUs. The count and residency results for state3 (i.e. a cluster domain
+idle state) in OS-initiated mode for multiple CPUs were much closer to the
+results for a single CPU than in platform-coordinated mode.
+
+--------------
+
+*Copyright (c) 2023, Arm Limited and Contributors. All rights reserved.*