Ansible Amazon Base Repository

Ansible Amazon Base Repository is an MVP monorepo to rapidly develop scalable, reliable, high-quality components for Amazon Linux instance configuration management.

Setting Up Ansible Environment

Configuring Python VENV and Ansible

Note: This document assumes that you are working on Mac

Create a new virtual environment with pyenv

$ pyenv virtualenv miniconda3-latest ansible

Activate your new python virtual environment

$ pyenv activate ansible

Install poetry

$ conda install poetry

Install dependencies

$ poetry install

Check ansible

$ ansible --version

Editor Configuration

Code editors are major software development productivity tools. VSCode is a game changer.

VSCode should be configured for typical Python development with the following extensions:

VSCode should be configured to associate most of the files in the repository with Ansible. Check that your ansible files are have Ansible set in the "Select Language Mode". In VSCode the difference between YAML and Ansible Language Mode is night and day. Example .vscode/settings.json

{ "files.associations": { "kitchen*": "yaml", "*.yml": "ansible"  }, }

Python should be setup with the following:

"editor.renderWhitespace": "all", "editor.rulers": [ 80, 100, 120    ], "[python]": { "editor.tabSize": 4, "editor.insertSpaces": true, "editor.formatOnSave": true    }, "[yaml]": { "editor.insertSpaces": true, "editor.tabSize": 2, "editor.autoIndent": "none", "editor.quickSuggestions": { "other": true, "comments": false, "strings": true        }, "editor.formatOnPaste": true    }, "python.linting.enabled": true, "python.linting.pylintEnabled": true, "python.linting.banditEnabled": true, "python.linting.banditArgs": [ "-x", "./tests", "-r", ], "python.linting.pylintArgs": [ "--disable=C0301,C0111,E0402,W0702,W0108,W0703"    ], "python.linting.pycodestyleEnabled": true, "python.linting.pycodestyleArgs": [ "--ignore E501"    ], "python.formatting.provider": "black", "python.languageServer": "Pylance", "python.envFile": "/Users/current.user/.vspyenv", "python.testing.pytestArgs": [ "-s", "-vvvv"    ], "pythonTestExplorer.testFramework": "pytest", "markdownlint.config": { "MD013": { "line_length": 120, "tables": false, "code_blocks": false        }, "MD025": false, "MD033": false, "MD036": false, "MD041": false    }, "testExplorer.hideEmptyLog": false,

Setting Up VirtualBox Environment

Access to local VMs running Amazon Linux helps rapidly and safely iterate on Ansible code.

Install VirtualBox and Vagrant

VirtualBox can be installed with Homebrew. However, every once in a while the latest build of VirtualBox has a broken functionality. At the time of writing, VirtualBox 6.1.28 has a broken Host Network Manager. Good build of VirtualBox is 6.1.26. Install Virtual Box from the link.

Vagrant is a HashiCorp Ruby project to provide VirtualBox abstraction. Use Homebrew to install vagrant

$ brew install vagrant

Configuring Test Kitchen

Test Kitchen is a Ruby project to automate Infrastructure as Code development life-cyle.

Install RVM

$ gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB

$ \curl -sSL https://get.rvm.io | bash -s stable --ruby

Add RVM to your profile

# Add RVM[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm"

Install test-kitchen gems

$ gem install test-kitchen kitchen-ansible kitchen-ec2

Downloading Amazon Linux v2 Vagrant Box

Amazon Linux v2 is a feature-rich Linux distribution maintained by Amazon. Amazon page about Amazon Linux Images lists various formats available, including Amazon Linux v2 virtualbox.

Download Amazon Linux v2 vagrant box from HashiCorp Vagrant Cloud.

Import vagrant box:

$ vagrant box add amazon2 <downloaded box>
$ vagrant box list

Build an Optimized Amazon Linux 2 Vagrant Box

Default Amazon Linux 2 vagrant box does not come with Ansible installed. Any time we run kitchen with the default Amazon Linux 2 box, kitchen will spend time installing Ansible. In order to save development time, we will build a box that includes Ansible and Docker.

Run kitchen converge with kitchen.box.yml

$ KITCHEN_YAML=kitchen.box.yml kitchen converge box

List running VirtualBox VMs and make note of the full name of the kitchen-ansible-amazon-base-box-amazon VM

$ VBoxManage list vms
"kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e" {9cf5ed90-d3a0-4e95-b742-6c9249c0cf34}

Run vagrant to export kitchen-ansible-amazon-base-box-amazon

$ vagrant package --base kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Attempting graceful shutdown of VM...
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Clearing any previously set forwarded ports...
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Exporting VM...
==> kitchen-ansible-amazon-base-box-amazon-cb9dedd7-fd27-4344-b026-bd3a2b7a340e: Compressing package to: /Users/current.user/work/ansible-amazon-base/package.box

Import package.box as amazon2-ansible

$ vagrant box add amazon2-ansible package.box
$ vagrant box list
amazon2         (virtualbox, 0)
amazon2-ansible (virtualbox, 0)

Remove kitchen instance

KITCHEN_YAML=kitchen.box.yml kitchen destroy box

Remove package.box from the local folder

Environment Variable Overwrites for Kitchen

Kitchen will use amazon2-ansible box by default. If you are using a different box, you can set KITCHEN_ANSIBLE_BOX environment variable.

Running Test Kitchen

Change directory to playbooks and run

$ kitchen list

To create a VirtualBox VM and apply an Ansible playbook run

$ kitchen converge docker

To destroy a created VM run

$ kitchen destroy docker

To connect to a VM run

$ kitchen ssh docker

Running Integration Tests

Integration tests are written in TestInfra. To run tests

$ kitchen verify docker

Running Ansible Playbooks on AWS Instances

In order to run Ansible Playbooks on AWS instances, first configure ANSIBLE_ROLES_PATH to point to the roles directory of the repository, e.g.

$ export ANSIBLE_ROLES_PATH=/Users/current.user/work/ansible-amazon-base/role

Also configure ~/.ansible.cfg to format Ansible output as a more readable YAML

[defaults]stdout_callback = yaml

Choose a playbook and run Ansible

$ ansible-playbook -v -u ec2-user --private-key ~/.ssh/<instance>.pem -i <instance_ip>, playbooks/gst/gst_jupyter.yml

Example Project: Build a Local Docker/Containerd Server VM

Why bother with Docker Desktop when you can build your own Containerd Server?

Check "Host Network Manager" in the File menu of your VirtualBox. Add an interface and make note of the subnet.

Edit kitchen.yml cedocker suit and set private_network to a static IP of your choice.

- name: cedocker    provisioner: name: ansible_playbook      playbook: ./playbooks/docker/docker.yml    driver: vm_hostname: cedocker.local      network: - ['private_network', {ip: '192.168.98.121'}]

Run kitchen converge

$ kitchen converge cedocker
...
       PLAY RECAP *********************************************************************
       localhost                  : ok=5    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

       Downloading files from <cedocker-amazon>
       Finished converging <cedocker-amazon> (0m5.54s).
-----> Test Kitchen is finished. (1m15.18s)

SSH into your Containerd VM. Change vagrant's user password from 'vagrant' to something secure. Add your SSH key to ~/.ssh/authorized_keys. Change permissions on /var/run/docker.sock.

$ ssh [email protected]
[email protected]'s password:
Last login: Mon Nov 15 05:47:31 2021 from 10.0.2.2

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

This system is built by the Bento project by Chef Software
More information can be found at https://github.com/chef/bento

[vagrant@cedocker ~]$ passwd
Changing password for user vagrant.
Changing password for vagrant.
(current) UNIX password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[vagrant@cedocker ~]$ vi ~/.ssh/authorized_keys
[vagrant@cedocker ~]$ sudo chmod a+rw /var/run/docker.sock
exit
logout
Connection to 192.168.98.121 closed.

Create a new Docker context:

$ docker context create cedocker --docker "host=ssh://[email protected]"
cedocker
Successfully created context "cedocker"
$ docker context use cedocker
$ docker context ls
NAME         DESCRIPTION                               DOCKER ENDPOINT                KUBERNETES ENDPOINT   ORCHESTRATOR
cedocker *                                             ssh://[email protected]
default      Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                          swarm

Start using your new Containerd server:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.10
 API version:       1.41
 Go version:        go1.17.2
 Git commit:        b485636f4b
 Built:             Fri Oct 15 14:45:13 2021
 OS/Arch:           darwin/amd64
 Context:           cedocker
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.14
  Git commit:       b0f5bc3
  Built:            Tue Sep 28 19:56:28 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0
  GitCommit:        84113eef6fc27af1b01b3181f31bbaf708715301
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Ansible References

GitHub - aia/ansible-amazon-base at pythonawesome.com
MVP monorepo to rapidly develop scalable, reliable, high-quality components for Amazon Linux instance configuration management - GitHub - aia/ansible-amazon-base at pythonawesome.com
GitHubaia