Update the Loop docs (#47895)

* first review of loops page

* round two

* round three

* puts 'not deprecating' message in a note

* fixes note syntax

* clarifies loop_control sections

* loops require list input

* incorporates bcoca and sivel feedback

* fixes links

* more bcoca feedback

* adds examples to loop update guidelines

* adds explicit code-blocks

* remove stray copypasta

1 files changed, 186 insertions, 139 deletions

diff --git a/docs/docsite/rst/user_guide/playbooks_loops.rst b/docs/docsite/rst/user_guide/playbooks_loops.rst
index d924d50695..ea91d611dc 100644
--- a/docs/docsite/rst/user_guide/playbooks_loops.rst
+++ b/docs/docsite/rst/user_guide/playbooks_loops.rst
@@ -1,21 +1,59 @@
 .. _playbooks_loops:
 
+*****
 Loops
-=====
+*****
 
-Often you'll want to do many things in one task, such as create a lot of users, install a lot of packages, or
-repeat a polling step until a certain result is reached.
+Sometimes you want to repeat a task multiple times. In computer programming, this is called a loop. Common Ansible loops include changing ownership on several files and/or directories with the :ref:`file module <file_module>`, creating multiple users with the :ref:`user module <user_module>`, and
+repeating a polling step until a certain result is reached. Ansible offers two keywords for creating loops: ``loop`` and ``with_<lookup>``.
 
-This chapter is all about how to use loops in playbooks.
+.. note::
+   * We added ``loop`` in Ansible 2.5. It is not yet a full replacement for ``with_<lookup>``, but we recommend it for most use cases.
+   * We have not deprecated the use of ``with_<lookup>`` - that syntax will still be valid for the foreseeable future.
+   * We are looking to improve ``loop`` syntax - watch this page and the `changelog <https://github.com/ansible/ansible/tree/devel/changelogs>`_ for updates.
 
-.. contents:: Topics
+.. contents::
+   :local:
+
+Comparing ``loop`` and ``with_*``
+=================================
+
+* The ``with_<lookup>`` keywords rely on :ref:`lookup_plugins` - even  ``items`` is a lookup.
+* The ``loop`` keyword is equivalent to ``with_list``, and is the best choice for simple loops.
+* The ``loop`` keyword will not accept a string as input, see :ref:`query_vs_lookup`.
+* Generally speaking, any use of ``with_*`` covered in :ref:`migrating_to_loop` can be updated to use ``loop``.
+* Be careful when changing ``with_items`` to ``loop``, as ``with_items`` performed implicit single-level flattening. You may need to use ``flatten(1)`` with ``loop`` to match the exact outcome. For example, to get the same output as:
+
+.. code-block:: yaml
+
+  with_items:
+    - 1
+    - [2,3]
+    - 4
+
+you would need::
+
+  loop: [1, [2,3] ,4] | flatten(1)
+
+* Any ``with_*`` statement that requires using ``lookup`` within a loop should not be converted to use the ``loop`` keyword. For example, instead of doing:
+
+.. code-block:: yaml
+
+  loop: "{{ lookup('fileglob', '*.txt', wantlist=True) }}"
+
+it's cleaner to keep::
+
+  with_fileglob: '*.txt'
 
 .. _standard_loops:
 
-Standard Loops
-``````````````
+Standard loops
+==============
+
+Iterating over a simple list
+----------------------------
 
-To save some typing, repeated tasks can be written in short-hand like so::
+Repeated tasks can be written as standard loops over a simple list of strings. You can define the list directly in the task::
 
     - name: add several users
       user:
@@ -26,42 +64,43 @@ To save some typing, repeated tasks can be written in short-hand like so::
          - testuser1
          - testuser2
 
-If you have defined a YAML list in a variables file, or the 'vars' section, you can also do::
+You can define the list in a variables file, or in the 'vars' section of your play, then refer to the name of the list in the task::
 
     loop: "{{ somelist }}"
 
-The above would be the equivalent of::
+Either of these examples would be the equivalent of::
 
     - name: add user testuser1
       user:
         name: "testuser1"
         state: present
         groups: "wheel"
+
     - name: add user testuser2
       user:
         name: "testuser2"
         state: present
         groups: "wheel"
 
-.. note:: Before 2.5 Ansible mainly used the ``with_<lookup>`` keywords to create loops, the `loop` keyword is basically analogous to ``with_list``.
-
-
-Some plugins like, the yum and apt modules can take lists directly to their options, this is more optimal than looping over the task.
-See each action's documentation for details, for now here is an example::
+You can pass a list directly to a parameter for some plugins. Most of the packaging modules, like :ref:`yum_module` and :ref:`apt_module`, have this capability. When available, passing the list to a parameter is better than looping over the task. For example::
 
    - name: optimal yum
      yum:
-       name: "{{list_of_packages}}"
+       name: "{{  list_of_packages  }}"
        state: present
 
-   - name: non optimal yum, not only slower but might cause issues with interdependencies
+   - name: non-optimal yum, slower and may cause issues with interdependencies
      yum:
-       name: "{{item}}"
+       name: "{{  item  }}"
        state: present
-     loop: "{{list_of_packages}}"
+     loop: "{{  list_of_packages  }}"
+
+Check the :ref:`module documentation <modules_by_category>` to see if you can pass a list to any particular module's parameter(s).
 
-Note that the types of items you iterate over do not have to be simple lists of strings.
-If you have a list of hashes, you can reference subkeys using things like::
+Iterating over a list of hashes
+-------------------------------
+
+If you have a list of hashes, you can reference subkeys in a loop. For example::
 
     - name: add several users
       user:
@@ -72,8 +111,11 @@ If you have a list of hashes, you can reference subkeys using things like::
         - { name: 'testuser1', groups: 'wheel' }
         - { name: 'testuser2', groups: 'root' }
 
-Also be aware that when combining :doc:`playbooks_conditionals` with a loop, the ``when:`` statement is processed separately for each item.
-See :ref:`the_when_statement` for an example.
+When combining :ref:`playbooks_conditionals` with a loop, the ``when:`` statement is processed separately for each item.
+See :ref:`the_when_statement` for examples.
+
+Iterating over a dictionary
+---------------------------
 
 To loop over a dict, use the ``dict2items`` :ref:`dict_filter`::
 
@@ -90,79 +132,18 @@ To loop over a dict, use the ``dict2items`` :ref:`dict_filter`::
 
 Here, we don't want to set empty tags, so we create a dictionary containing only non-empty tags.
 
+Registering variables with a loop
+=================================
 
-.. _complex_loops:
-
-Complex loops
-`````````````
-
-Sometimes you need more than what a simple list provides, you can use Jinja2 expressions to create complex lists:
-For example, using the 'nested' lookup, you can combine lists::
-
-    - name: give users access to multiple databases
-      mysql_user:
-        name: "{{ item[0] }}"
-        priv: "{{ item[1] }}.*:ALL"
-        append_privs: yes
-        password: "foo"
-      loop: "{{ ['alice', 'bob'] |product(['clientdb', 'employeedb', 'providerdb'])|list }}"
-
-.. note:: ``with_`` loops are actually a combination of things ``with_`` + ``lookup()``, even ``items`` is a lookup. ``loop`` can be used in the same way as shown above.
-
-
-Using lookup vs query with loop
-```````````````````````````````
-
-In Ansible 2.5 a new jinja2 function was introduced named :ref:`query`, that offers several benefits over ``lookup`` when using the new ``loop`` keyword.
-
-This is better described in the lookup documentation. However, ``query`` provides a simpler interface and a more predictable output from lookup plugins, ensuring better compatibility with ``loop``.
+You can register the output of a loop as a variable. For example::
 
-In certain situations the ``lookup`` function may not return a list which ``loop`` requires.
+   - shell: "echo {{ item }}"
+     loop:
+       - "one"
+       - "two"
+     register: echo
 
-The following invocations are equivalent, using ``wantlist=True`` with ``lookup`` to ensure a return type of a list::
-
-    loop: "{{ query('inventory_hostnames', 'all') }}"
-
-    loop: "{{ lookup('inventory_hostnames', 'all', wantlist=True) }}"
-
-
-.. _do_until_loops:
-
-Do-Until Loops
-``````````````
-
-.. versionadded:: 1.4
-
-Sometimes you would want to retry a task until a certain condition is met.  Here's an example::
-
-    - shell: /usr/bin/foo
-      register: result
-      until: result.stdout.find("all systems go") != -1
-      retries: 5
-      delay: 10
-
-The above example runs the shell module iteratively until the module's result has "all systems go" in its stdout or the task has
-been retried for 5 times with a delay of 10 seconds. The default value for "retries" is 3 and "delay" is 5.
-
-The task returns the results returned by the last task run. The results of individual retries can be viewed by -vv option.
-The registered variable will also have a new key "attempts" which will have the number of the retries for the task.
-
-.. note:: If the ``until`` parameter isn't defined, the value for the ``retries`` parameter is forced to 1.
-
-Using register with a loop
-``````````````````````````
-
-After using ``register`` with a loop, the data structure placed in the variable will contain a ``results`` attribute that is a list of all responses from the module.
-
-Here is an example of using ``register`` with ``loop``::
-
-    - shell: "echo {{ item }}"
-      loop:
-        - "one"
-        - "two"
-      register: echo
-
-This differs from the data structure returned when using ``register`` without a loop::
+When you use ``register`` with a loop, the data structure placed in the variable will contain a ``results`` attribute that is a list of all responses from the module. This differs from the data structure returned when using ``register`` without a loop::
 
     {
         "changed": true,
@@ -218,13 +199,52 @@ During iteration, the result of the current item will be placed in the variable:
       register: echo
       changed_when: echo.stdout != "one"
 
+.. _complex_loops:
 
+Complex loops
+=============
 
-Looping over the inventory
-``````````````````````````
+Iterating over nested lists
+---------------------------
 
-If you wish to loop over the inventory, or just a subset of it, there are multiple ways.
-One can use a regular ``loop`` with the ``ansible_play_batch`` or ``groups`` variables, like this::
+You can use Jinja2 expressions to iterate over complex lists. For example, a loop can combine nested lists::
+
+    - name: give users access to multiple databases
+      mysql_user:
+        name: "{{ item[0] }}"
+        priv: "{{ item[1] }}.*:ALL"
+        append_privs: yes
+        password: "foo"
+      loop: "{{ ['alice', 'bob'] |product(['clientdb', 'employeedb', 'providerdb'])|list }}"
+
+
+.. _do_until_loops:
+
+Retrying a task until a condition is met
+----------------------------------------
+
+.. versionadded:: 1.4
+
+You can use the ``until`` keyword to retry a task until a certain condition is met.  Here's an example::
+
+    - shell: /usr/bin/foo
+      register: result
+      until: result.stdout.find("all systems go") != -1
+      retries: 5
+      delay: 10
+
+This task runs up to 5 times with a delay of 10 seconds between each attempt. If the result of any attempt has "all systems go" in its stdout, the task succeeds. The default value for "retries" is 3 and "delay" is 5.
+
+To see the results of individual retries, run the play with ``-vv``.
+
+When you run a task with ``until`` and register the result as a variable, the registered variable will include a key called "attempts", which records the number of the retries for the task.
+
+.. note:: You must set the ``until`` parameter if you want a task to retry. If ``until`` is not defined, the value for the ``retries`` parameter is forced to 1.
+
+Looping over inventory
+----------------------
+
+To loop over your inventory, or just a subset of it, you can use a regular ``loop`` with the ``ansible_play_batch`` or ``groups`` variables::
 
     # show all the hosts in the inventory
     - debug:
@@ -248,41 +268,37 @@ There is also a specific lookup plugin ``inventory_hostnames`` that can be used
         msg: "{{ item }}"
       loop: "{{ query('inventory_hostnames', 'all:!www') }}"
 
-More information on the patterns can be found on :doc:`intro_patterns`
+More information on the patterns can be found on :ref:`intro_patterns`
 
-.. _loop_control:
+.. _query_vs_lookup:
 
-Loop Control
-````````````
+Ensuring list input for ``loop``: ``query`` vs. ``lookup``
+==========================================================
 
-.. versionadded:: 2.1
+The ``loop`` keyword requires a list as input, but the ``lookup`` keyword returns a string of comma-separated values by default. Ansible 2.5 introduced a new Jinja2 function named :ref:`query` that always returns a list, offering a simpler interface and more predictable output from lookup plugins when using the ``loop`` keyword.
 
-In 2.0 you are again able to use loops and task includes (but not playbook includes). This adds the ability to loop over the set of tasks in one shot.
-Ansible by default sets the loop variable ``item`` for each loop, which causes these nested loops to overwrite the value of ``item`` from the "outer" loops.
-As of Ansible 2.1, the ``loop_control`` option can be used to specify the name of the variable to be used for the loop::
+You can force ``lookup`` to return a list to ``loop`` by using ``wantlist=True``, or you can use ``query`` instead.
 
-    # main.yml
-    - include_tasks: inner.yml
-      loop:
-        - 1
-        - 2
-        - 3
-      loop_control:
-        loop_var: outer_item
+These examples do the same thing::
 
-    # inner.yml
-    - debug:
-        msg: "outer item={{ outer_item }} inner item={{ item }}"
-      loop:
-        - a
-        - b
-        - c
+    loop: "{{ query('inventory_hostnames', 'all') }}"
 
-.. note:: If Ansible detects that the current loop is using a variable which has already been defined, it will raise an error to fail the task.
+    loop: "{{ lookup('inventory_hostnames', 'all', wantlist=True) }}"
+
+
+.. _loop_control:
 
+Adding controls to loops
+========================
+.. versionadded:: 2.1
+
+The ``loop_control`` keyword lets you manage your loops in useful ways.
+
+Limiting loop output with ``label``
+-----------------------------------
 .. versionadded:: 2.2
 
-When using complex data structures for looping the display might get a bit too "busy", this is where the ``label`` directive comes to help::
+When looping over complex data structures, the console output of your task can be enormous. To limit the displayed output, use the ``label`` directive with ``loop_control``::
 
     - name: create servers
       digital_ocean:
@@ -299,11 +315,13 @@ When using complex data structures for looping the display might get a bit too "
       loop_control:
         label: "{{ item.name }}"
 
-This will now display just the ``label`` field instead of the whole structure per ``item``, it defaults to ``{{ item }}`` to display things as usual.
+The output of this task will display just the ``name`` field for each ``item`` instead of the entire contents of the multi-line ``{{ item }}`` variable.
 
+Pausing within a loop
+---------------------
 .. versionadded:: 2.2
 
-Another option to loop control is ``pause``, which allows you to control the time (in seconds) between execution of items in a task loop.::
+To control the time (in seconds) between the execution of each item in a task loop, use the ``pause`` directive with ``loop_control``::
 
     # main.yml
     - name: create servers, pause 3s before creating next
@@ -316,19 +334,47 @@ Another option to loop control is ``pause``, which allows you to control the tim
       loop_control:
         pause: 3
 
+Tracking progress through a loop with ``index_var``
+---------------------------------------------------
 .. versionadded:: 2.5
 
-If you need to keep track of where you are in a loop, you can use the ``index_var`` option to loop control to specify a variable name to contain the current loop index.::
+To keep track of where you are in a loop, use the ``index_var`` directive with ``loop_control``. This directive specifies a variable name to contain the current loop index::
+
+  - name: count our fruit
+    debug:
+      msg: "{{ item }} with index {{ my_idx }}"
+    loop:
+      - apple
+      - banana
+      - pear
+    loop_control:
+      index_var: my_idx
 
-    - name: count our fruit
-      debug:
-        msg: "{{ item }} with index {{ my_idx }}"
+Defining inner and outer variable names with ``loop_var``
+---------------------------------------------------------
+.. versionadded:: 2.1
+
+You can nest two looping tasks using ``include_tasks``. However, by default Ansible sets the loop variable ``item`` for each loop. This means the inner, nested loop will overwrite the value of ``item`` from the outer loop.
+You can specify the name of the variable for each loop using ``loop_var`` with ``loop_control``::
+
+    # main.yml
+    - include_tasks: inner.yml
       loop:
-        - apple
-        - banana
-        - pear
+        - 1
+        - 2
+        - 3
       loop_control:
-        index_var: my_idx
+        loop_var: outer_item
+
+    # inner.yml
+    - debug:
+        msg: "outer item={{ outer_item }} inner item={{ item }}"
+      loop:
+        - a
+        - b
+        - c
+
+.. note:: If Ansible detects that the current loop is using a variable which has already been defined, it will raise an error to fail the task.
 
 .. versionadded:: 2.8
 
@@ -354,23 +400,24 @@ Variable                    Description
       loop_control:
         extended: yes
 
+.. _migrating_to_loop:
+
 Migrating from with_X to loop
-`````````````````````````````
+=============================
 
 .. include:: shared_snippets/with2loop.txt
 
-
 .. seealso::
 
-   :doc:`playbooks`
+   :ref:`about_playbooks`
        An introduction to playbooks
-   :doc:`playbooks_reuse_roles`
+   :ref:`playbooks_reuse_roles`
        Playbook organization by roles
-   :doc:`playbooks_best_practices`
+   :ref:`playbooks_best_practices`
        Best practices in playbooks
-   :doc:`playbooks_conditionals`
+   :ref:`playbooks_conditionals`
        Conditional statements in playbooks
-   :doc:`playbooks_variables`
+   :ref:`playbooks_variables`
        All about variables
    `User Mailing List <https://groups.google.com/group/ansible-devel>`_
        Have a question?  Stop by the google group!