#+TITLE: Nomad Local Development
#+SETUPFILE: ~/s3olmax/org/conf/setup.config
#+FILETAGS: :nomad:iac:ansible:vault:consul:vagrant

Set up and manage a local Nomad Cluster using Consul, Vault, Traefik and Vagrant.

# #+CAPTION: project
# #+attr_html: :width 200px
# [[file:images/helm-plus-metabase.png][file:./images/*.png]]

* Useful Commands
#+begin_src bash
$ vagrant up --provider=libvirt
$ vagrant ssh node-1 -c "sudo ifconfig"

# Consul
$ vagrant ssh node-1 -c "consul catalog nodes"
$ vagrant ssh node-1 -c "consul catalog services -node=node-1 -tags"
# Vault
$ curl http://192.168.0.111:8200/v1/sys/health | python -m json.tool   # Check Vault 'unsealed'
$ vault kv put secret/aws/s3 aws_access_key_id=somekeyid               # Create new secret
$ curl -X PUT -H "Content-Type: application/json" -d '{"key":"****"}' \ 
 "http://192.168.0.111:8200/v1/sys/unseal"                             # Unseal
# Nomad
$ NOMAD_ADDR="http://192.168.122.111:4646" nomad server members -detailed
$ nomad node-status
$ nomad job run <path/to/jobspec.nomad>

$ nomad job status <Job_ID>
$ nomad job stop <Job_ID>"

$ docker exec -i -t ansible_controller bash
$ rsync -avz -e "ssh -i $HOME/.vagrant.d/insecure_private_key" \ 
 example/ root@192.168.0.111:/root/example
#+end_src

* Get Started
- When being launched the =entrypoint= will pull all =ansible-galaxy= roles as
  defined in the =ansible/requirements.yml= file.
** 1. Environment Setup
| # | Action               | Command     |
| 1 | Clone the repository | =host:~/ ❯ git clone <nomad-cluster>= |
| 2 | Edit =Vagrantfile= and change the IP addresses accordingly with your own local network | Also ensure match with =ansible/inventory/group_vars/service_traefik/traefik.yml= |
| 3 | Spin up Vagrant machines | =host:<nomad-cluster> ❯ vagrant up --provider=libvirt= |
| 4 | Build the container for the Ansible controller | =host:<nomad-cluster> ❯ docker-compose up --build= |
| 5 | Access Ansible Controller | =host:<nomad-cluster> ❯ docker exec -i -t ansible_controller bash= |
| 6 | Create siimlink to ansible-vault script | =$ ln -s ./ansible/scripts/vault-pass-client.py vault-pass-client.py= |

** 2. Stack Deployment
| # | Action               | Command     |
| 1 | Change the IP addresses of the =hosts= file accordingly with what you set in the =Vagrantfile= and local network | =$ vagrant ssh node-1 -c "sudo ifconfig"= |
| 3 | Run the individual playbooks a first time (use any password as Vault password) to deploy Consul+Dnsmasq, and Vault | =(ansible_controller) ~$ ansible-playbook ansible/consul.yml + vault.yml= |
| 4 | Unseal Vault: head to http://192.168.0.111:8200/ and follow the process (remember to store the master key and the root token) | http://192.168.0.111:8200/ |
| 5 | Provide a Vault token to Nomad (remember to store the new password) | In 'General Instructions' follow the 'Integrate Ansible-Vault' section and insert encrypted token into =ansible/inventory/group_vars/service_nomad.yml= |
| 6 | Run the playbook a second time to configure Nomad with the Vault token | =ansible-playbook --vault-id dev@vault-pass-client.py ansible/site.yml= |

*** Main IP Addresses
| Service  | IP                        | Hostname     |
| Consul   | http://192.168.0.111:8500 | http://consul.service.lab.consul:8500 |
| Vault    | http://192.168.0.111:8200 | http://active.vault.service.lab.consul:8200 |
| Nomad    | http://192.168.0.111:4646 | http://nomad-servers.service.lab.consul:4646 |
| Traefik  | http://192.168.0.111:8081 | |

** 3. New Applications Deployment
| # | Action               | Command     |
| 1 | Find the Nomad job description | =/home/vagrant/example/APPNAME.nomad= |
| 2 | Inside the Nomad server run the =levant= deployment script | =(vagrant@node-1):~/example$ bash ./deploy-dev.sh= |

** Confirm services
#+begin_src bash
$ curl http://192.168.122.111/0/
$ curl -H Host:vault.service.consul http://192.168.122.111/ui/
#+end_src

** Environment teardown
| # | Action               | Command     |
| 1 | Stop the Vagrant machines    | =host:<nomad-cluster> ❯ vagrant halt/destroy= |

* General Instructions
** Integrate Ansible-Vault
- The variable =nomad_vault_token= needs to be created via =Ansible-Vault= 
- In the Ansible-Controller set up =gpg= and =pass=
#+begin_src bash
# 'pass' needs a pregenerated 'gpg' key
(ansible_controller)~$ gpg --list-key       # will be empty
(ansible_controller)~$ gpg --full-generate-key   # the key will have a full name

# Create password-store and ansible-vault encryption password
(ansible_controller)~$ pass init "<gpg_full_key_name>"
(ansible_controller)~$ pass insert dev   # 'dev' is the vault-id label

# Create inline secret to be used with ansible-vault
# Ensure keyname 'dev' matches in ansible.cfg and ansible/script/pass-keyring-client.py l.131 !!
# 130 if not keyname:
# 131        keyname = "dev"  <-- HERE
$ ansible-vault encrypt_string --vault-id dev@vault-pass-client.py '[VAULT_ROOT_TOKEN]' --name 'nomad_vault_token'
#+end_src

#+begin_src bash
(ansible_controller)~$ ansible-playbook --vault-id dev@ansible/scripts/vault-pass-client.py ansible/nomad.yml
(ansible_controller)~$ ansible-playbook --vault-id dev@prompt ansible/nomad.yml
# Confirm Nomad
http://192.168.0.111:4646
#+end_src

** Troubleshoot Vagrant
- *Error while activating network: Call to virNetworkCreate failed: internal*
  *error: Network is already in use*
  #+begin_src bash
# This is a libvirt/KVM network problem

# 1. Identify default interface
$ sudo virsh net-list
$ sudo virsh net-dumpxml vagrant-libvirt
$ sudo virsh net-edit vagrant-libvirt

# 2. Delete Interface and vagrant virtual network manually
# You can also check libvirt networks in VMM > Edit > Connection Details > Virtual Networks
$ sudo virsh net-destroy vagrant-libvirt
  #+end_src
  
* Author

Olaf Marangone
Contact: olmighty99@gmail.com

* Credits
- Based on the amazing work of [[https://github.com/marco-lancini][Marco Lancini]] - [[https://github.com/marco-lancini/offensive-infrastructure][GitHub]]
- It is further based on the following blog post series [[https://www.marcolancini.it/2019/blog-offensive-infrastructure-hashistack/][Part 2]].