1 files changed, 97 insertions, 0 deletions
diff --git a/hacking/shippable/README.md b/hacking/shippable/README.md
new file mode 100644
index 0000000000..1f48d65d80
--- /dev/null
+++ b/hacking/shippable/README.md
@@ -0,0 +1,97 @@
+# Shippable Scripts
+
+## Scripts
+
+This directory contains the following scripts:
+
+- download.py - Download results from Shippable.
+- incidental.py - Report on incidental code coverage using data from Shippable.
+- run.py - Start new runs on Shippable.
+
+## Incidental Code Coverage
+
+### Background
+
+Incidental testing and code coverage occurs when a test covers one or more portions of code as an unintentional side-effect of testing another portion of code.
+
+For example, the ``yum`` integration test intentionally tests the ``yum`` Ansible module.
+However, in doing so it also uses, and unintentionally tests the ``file`` module as well.
+
+As part of the process of migrating modules and plugins into collections, integration tests were identified that provided exclusive incidental code coverage.
+That is, tests which would be migrated out of the repository which covered code which would not be covered by any remaining tests.
+
+These integration test targets were preserved as incidental tests with the ``incidental_`` prefix prior to migration.
+The plugins necessary to support these tests were also preserved in the ``test/support/`` directory.
+
+The long-term goal for these incidental tests is to replace them with tests that intentionally cover the relevant code.
+As additional intentional tests are added, the exclusive coverage provided by incidental tests will decline, permitting them to be removed without loss of test coverage.
+
+### Reducing Incidental Coverage
+
+Reducing incidental test coverage, and eventually removing incidental tests involves the following process:
+
+1. Run the entire test suite with code coverage enabled. 
+   This is done automatically each day on Shippable.
+2. Download code coverage data from Shippable for local analysis. 
+   Example:
+   ```shell
+   # download results to ansible/ansible directory under cwd
+   # substitute the correct run number for the Shippable coverage run you want to download
+   hacking/shippable/download.py https://app.shippable.com/github/ansible/ansible/runs/162160 --all -v
+   ```
+3. Analyze code coverage data to see which portions of the code are covered by each test.
+   Example:
+   ```shell script
+   # make sure ansible-test is in $PATH
+   source hacking/env-setup
+   # run the script using whichever directory results were downloaded into
+   hacking/shippable/incidental.py ansible/ansible/162160
+   ```
+4. Create new intentional tests, or extend existing ones, to cover code that is currently covered by incidental tests.
+   Reports are created by default in a ``test/results/.tmp/incidental/{hash}/reports/`` directory.
+   The ``{hash}`` value is based on the input files used to generate the report.
+
+Over time, as the above process is repeated, exclusive incidental code coverage will decline.
+When incidental tests no longer provide exclusive coverage they can be removed.
+
+> CAUTION: Only one incidental test should be removed at a time, as doing so may cause another test to gain exclusive incidental coverage.
+
+### Reading Incidental Coverage Reports
+
+Each line of code covered will be included in a report.
+The left column contains the line number where the source occurs.
+If the coverage is for Python code a comment on the right side will indicate the coverage arcs involved.
+
+Below is an example of a report:
+
+```
+Target: incidental_win_psexec
+GitHub: https://github.com/ansible/ansible/blob/6994ef0b554a816f02e0771cb14341a421f7cead/test/integration/targets/incidental_win_psexec
+
+Source: lib/ansible/executor/task_executor.py (2 arcs, 3/1141 lines):
+GitHub: https://github.com/ansible/ansible/blob/6994ef0b554a816f02e0771cb14341a421f7cead/lib/ansible/executor/task_executor.py
+
+ 705                  if 'rc' in result and result['rc'] not in [0, "0"]:  ### (here) -> 706
+ 706                      result['failed'] = True  ### 705 -> (here)  ### (here) -> 711
+
+ 711              if self._task.until:  ### 706 -> (here)
+```
+
+The report indicates the test target responsible for the coverage and provides a link to the source on GitHub using the appropriate commit to match the code coverage.
+
+Each file covered in the report indicates the lines affected, and in the case of Python code, arcs.
+A link to the source file on GitHub using the appropriate commit is also included.
+
+The left column includes the line number for the source code found to the right.
+In the case of Python files, the rightmost comment indicates the coverage arcs involved.
+
+``### (here) -> 706`` for source line 705 indicates that execution flowed from line 705 to line 706.
+Multiple outbound line numbers can be present.
+
+``### 706 -> (here)`` for source line 711 indicates that execution flowed from line 706 to line 711.
+Multiple inbound line numbers can be present.
+
+In both cases ``(here)`` is simply a reference to the current source line.
+
+Arcs are only available for Python code.
+PowerShell code only reports covered line numbers.