new connection plugin aws_ssm (#49652)

* new connection plugin aws_ssm

Return code may be at the end of the last command output line.

Marking regex.

Ensure command status code is on it's owm line - last 3 lines are not part of command stdout.

* Adding timeout parameter - aws_ssm_timeout

Default 10 second timeout (https://docs.ansible.com/ansible/2.4/intro_configuration.html#timeout) is marginal.
This avoids changing this default and allowing the SSM timeout to be controlled via inventory.
This change wraps commands so commands which may never return do timeout.

* Added integration tests

Added AWS SSM Executor, target and config functions

Fixed more code for integration tests

Improved execution

Added S3 bucket name

Fixed pylint

Reverted lib changes

Reverted few more changes

Improved support for integration test execution

added ansible role for aws_ssm_integration_test setup and teardown and modifiled runme.sh

Reset to 17fa565 commit

inventory file location changed

change inventory file location

deleted meta and handlers folder as it is not required

deleted main.yml inside vars, removed extra space from tasks/main.yml, Added appropriate tags for ec2 and delete test folder as it is not required

deleted main.yml inside vars, removed extra space from tasks/main.yml, Added appropriate tags for ec2 and delete test folder as it is not required

modified task/main.yml

added region variable and fixed pattern for using variable

modified policy for IAM role

moved to first line of scrip set -eux

Updated Session Manager plugin installation

edited custme policy

Included tags for Session Manager plugin installation

Added README.md

Upddated README and added support for ssm-plugin for Amazon-Linux

Added Windows Integration test support

Improved user data for Linux

Added random value generation for the role and policy,delete vars_to_delete.yml

upadte README in vars

fixed typo

update policy

Updated IAM policy file

update playbook

Updated playbook to include ssm-agent userdata

modified jing2 template

modified jing2 template

modified jing2 template and fixed role deletion

fixed role name issue while deleting and task name

Updated playbook to include wait_connection for ec2

Corrected Synatx changes and updated ssm-plugin debian file

Changed region variable to us-east-1

Removed vars file and updated  to /tmp dir

fixed typo

Improved setup

Fixed boto3 dependency

Fixed missing tag

Added boto as dependency as well

Improved execution workflow

Trying other way of defining tags

Fixed undefined var

Changed AMI ID to Amazon Linux

Improved Tags

Ok, created different directory for WIndows test execution

Fixed IAM Role Name for Windows

Fixed inventory not found

Improved integration test execution

Fixed Windows Inventory path

Fixed wrong Windows AMI ID

Fixes issue for windows test execution

* Don't attempt to terminate sessions without a session id.

* Added Unit test cases file for AWS SSM Connection plugin

updated test file with close

updated unit test file with start_session

updated test files

* Eliminate AWS CLI dependency for terminal session.

* Removing unused code, cleanup logic.

Reduce mark length - 52^26 should be plenty
Be explicit about subprocess.Popen options
Simplify if/else for mark end
_stdin_readline is not used now

* updated test file

Added exec command and fixed close session unit tests

updated test files

Improved ansible ssm test command

updated file for lint checks

updated for pylint checks

New Unit_testcases for pre-signed URL file

removing additonal spaces and white spaces

remaning error changes

fixed changes

fixed spaces issues

python 2.7 version and whitespaces

python 2.7 version and whitespaces

python 2.7 skip if

space issue with 16:1

Unit test cases for windows and linux

Unit test cases for windows and linux with issues fixed issues

Unit test cases for windows and linux with issues fixed issues1

* Added support for S3 Pre-signed URLs

* Updated documentation and comments

* Documentation and curl dependency removal for controller machine

Fixing lint errors and removing requirements.

* Adding support for Windows remote EC2 instances.

* Added Encoding fixes

* Updating author section and adding obvious requirement for the SSM agent.

* Refactor stdout post processing
Attempt to get real return code on error (test using ansible -m raw -a 'cmd /c exit 99'.
Fixes problem at terminal width (ansible windows -i ./hosts.yml -m setup).

* Refactor back to a single module.

* Fixed fetch file for windows

* ssm usage examples for linux and windows

* Update aws_ssm.py

Service state corrected.

* Strip line continuation when at terminal width - otherwise replace.
Strip ANSI control sequences only for Windows.

Test playbook:

---
- name: test
  hosts: windows
  gather_facts: False
  vars:
    small: 'abc'
  tasks:
  - name:
    set_fact:
      large: "{{ lookup('password', '/dev/null length=2000 chars=ascii_letters,digits,hexdigits,punctuation') }}"
  - name: small fixed
    raw: echo '{{ small }}'
    register: small_result
    changed_when: False
  - name: check
    assert:
      that:
        - "(item | length) == (small | length)"
        - "item == small"
      msg: "'{{ item | length }} must equal '{{ small | length }}' and '{{ item }}' must match '{{ small }}'"
    with_items:
      - "{{ small_result.stdout_lines[0] }}"
  - name: large random
    raw: echo '{{ large }}'
    register: large_result
    changed_when: False
  - name: check
    assert:
      that:
        - "(item | length) == (large | length)"
        - "item == large"
      msg: "'{{ item | length }} must equal '{{ large | length }}' and '{{ item }}' must match '{{ large }}'"
    with_items:
      - "{{ large_result.stdout_lines[0] }}"
  - name: gather facts
    setup:

* Correct module parameter names.

* Updated Windows Executable variable.

Updated Windows Executable variable to "ansible_shell_type".

Fixing Examples with raw declaration

Updated the plugin timeout variable.

* Fix to work with dynamic inventory plug-in

* IntegrationTest template updates

* Removing unsupported flag for integration tests.
Fixing unit test.

* Adding shippable group.

* SSM Usage examples with dynamic inventory plugin

* Fixing yamllint errors.

* Fixed Integration tests

* Fixed Integration tests

* Updates for python3.

Removing python3 restriction.

* Remove python3 restriction.
Change block from retry to always

* Fixed Integration tests with Python 3

* Fixed shellcheck

* Fix for Windows which could pick up end mark prematurely
Move debug to _wrap_command and use a single return point

Single-quotes not needed around linux marks
Fix typo in comment

End mark to new command.

* Unit test cases now works on Python2 and Python3

* Skip tests on Python 2.6

* Fix for wait_for_connection module for windows.

* Updated changes as per review comments

* Fixing broken pipe error seen with session-manager-plugin version 1.1.17.0.
Eliminating sleep as this looks to be fixed in session-manager-plugin version 1.1.17.0.

* Adding back delays for Windows with session-manager-plugin 1.1.17.0.

* Updating Windows AMI ID for integration Test

* Upgrading windows ssm agent to the latest

* Adding boilerplate code.

* Windows ami and integration test updates

* Revert "Windows ami and integration test updates"

This reverts commit cd6ca3579b7cda584bd9c065f9c0835bddb23627.

* Updating windows ami for Integration tests

* Integration test suite updates and fixes.

* Updates and fixes

* Eliminate duplicate processing for exit code on failed command.

* Add powershell wrap.

* Refactor windows post_process.

* AMI Lookup, aliases, OSC filter, test suite updates

Co-authored-by: Gaurav Ashtikar <gau1991@gmail.com>
Co-authored-by: Deepak Choudhary <40276333+deepsvc@users.noreply.github.com>
Co-authored-by: Hanumanth <46720371+hanumantharaomvl@users.noreply.github.com>
Co-authored-by: KUMAR MAYANK <mayank@flux7.com>

1 files changed, 156 insertions, 0 deletions

diff --git a/test/integration/targets/connection_aws_ssm/aws_ssm_integration_test_setup/tasks/main.yml b/test/integration/targets/connection_aws_ssm/aws_ssm_integration_test_setup/tasks/main.yml
new file mode 100644
index 0000000000..dae7e27747
--- /dev/null
+++ b/test/integration/targets/connection_aws_ssm/aws_ssm_integration_test_setup/tasks/main.yml
@@ -0,0 +1,156 @@
+---
+## Task file for setup/teardown AWS resources for aws_ssm integration testing
+- block:
+    - name: set up aws connection info
+      set_fact:
+        aws_connection_info: &aws_connection_info
+          aws_access_key: "{{aws_access_key}}"
+          aws_secret_key: "{{aws_secret_key}}"
+          security_token: "{{security_token}}"
+          region: "{{aws_region}}"
+      no_log: yes
+
+    - name: AMI Lookup
+      ec2_ami_info:
+        owners: 'amazon'
+        filters:
+          name: '{{ item }}'
+        <<: *aws_connection_info  
+      register: ec2_amis
+      loop:
+        - '{{ linux_ami_name }}'
+        - '{{ windows_ami_name }}'
+
+    - name: Set facts with latest AMIs
+      vars:
+        latest_linux_ami: '{{ ec2_amis.results[0].images | sort(attribute="creation_date") | last }}'
+        latest_windows_ami: '{{ ec2_amis.results[1].images | sort(attribute="creation_date") | last }}'
+      set_fact:
+        linux_ami_id: '{{ latest_linux_ami.image_id }}'
+        windows_ami_id: '{{ latest_windows_ami.image_id }}'
+
+    - name: Install Session Manager Plugin for Debian/Ubuntu
+      include_tasks: debian.yml
+      when: ansible_distribution == "Ubuntu" or ansible_distribution == "Debian"
+      register: install_plugin_debian
+
+    - name: Install Session Manager Plugin for RedHat/Amazon
+      include_tasks: redhat.yml
+      when: ansible_distribution == "CentOS" or ansible_distribution == "RedHat" or ansible_distribution == "Amazon"
+      register: install_plugin_redhat
+
+    - name: Fail if the plugin was not installed
+      fail:
+        msg: The distribution does not contain the required Session Manager Plugin
+      when:
+        - install_plugin_debian is skipped
+        - install_plugin_redhat is skipped
+
+    - name: Install Boto3
+      pip:
+        name: boto3
+
+    - name: Install Boto
+      pip:
+        name: boto
+
+    - name: Ensure IAM instance role exists
+      iam_role:
+        name: "ansible-test-{{resource_prefix}}-aws-ssm-role"
+        assume_role_policy_document: "{{ lookup('file','ec2-trust-policy.json') }}"
+        state: present
+        create_instance_profile: yes
+        managed_policy:
+        - AmazonEC2RoleforSSM
+        <<: *aws_connection_info
+      register: role_output
+
+    - name: Create S3 bucket
+      s3_bucket:
+        name: "{{resource_prefix}}-aws-ssm-s3"
+        <<: *aws_connection_info
+      register: s3_output
+
+    - name: Wait for IAM Role getting created
+      pause:
+        seconds: 10
+
+    - name: Create Linux EC2 instance
+      ec2:
+        instance_type: "{{instance_type}}"
+        image: "{{linux_ami_id}}"
+        wait: "yes"
+        count: 1
+        instance_profile_name: "{{role_output.iam_role.role_name}}"
+        instance_tags:
+          Name: "{{resource_prefix}}-integration-test-aws-ssm-linux"
+        user_data: |
+                    #!/bin/sh
+                    sudo systemctl start amazon-ssm-agent
+        state: present
+        <<: *aws_connection_info
+      register: linux_output
+
+    - name: Create Windows EC2 instance
+      ec2:
+        instance_type: "{{instance_type}}"
+        image: "{{windows_ami_id}}"
+        wait: "yes"
+        count: 1
+        instance_profile_name: "{{role_output.iam_role.role_name}}"
+        instance_tags:
+          Name: "{{resource_prefix}}-integration-test-aws-ssm-windows"
+        user_data: |
+                    <powershell>
+                    Invoke-WebRequest -Uri "https://amazon-ssm-us-east-1.s3.amazonaws.com/latest/windows_amd64/AmazonSSMAgentSetup.exe" -OutFile "C:\AmazonSSMAgentSetup.exe"
+                    Start-Process -FilePath C:\AmazonSSMAgentSetup.exe -ArgumentList "/S","/v","/qn" -Wait
+                    Restart-Service AmazonSSMAgent
+                    </powershell>
+        state: present
+        <<: *aws_connection_info
+      register: windows_output
+
+    - name: Wait for EC2 to be available
+      wait_for_connection:
+        delay: 300
+
+    - name: Create Inventory file for Linux host
+      template:
+        dest: "{{playbook_dir}}/inventory-linux.aws_ssm"
+        src: inventory-linux.aws_ssm.j2
+
+    - name: Create Inventory file for Windows host
+      template:
+        dest: "{{playbook_dir}}/inventory-windows.aws_ssm"
+        src: inventory-windows.aws_ssm.j2
+
+    - name: Create AWS Keys Environement
+      template:
+        dest: "{{playbook_dir}}/aws-env-vars.sh"
+        src: aws-env-vars.j2
+      no_log: yes
+
+  always:
+    - name: Create EC2 Linux vars_to_delete.yml
+      template:
+        dest: "{{playbook_dir}}/ec2_linux_vars_to_delete.yml"
+        src: ec2_linux_vars_to_delete.yml.j2
+      ignore_errors: yes
+
+    - name: Create EC2 Windows vars_to_delete.yml
+      template:
+        dest: "{{playbook_dir}}/ec2_windows_vars_to_delete.yml"
+        src: ec2_windows_vars_to_delete.yml.j2
+      ignore_errors: yes
+
+    - name: Create S3 vars_to_delete.yml
+      template:
+        dest: "{{playbook_dir}}/s3_vars_to_delete.yml"
+        src: s3_vars_to_delete.yml.j2
+      ignore_errors: yes
+
+    - name: Create IAM Role vars_to_delete.yml
+      template:
+        dest: "{{playbook_dir}}/iam_role_vars_to_delete.yml"
+        src: iam_role_vars_to_delete.yml.j2
+      ignore_errors: yes