Ansible is the recommended tool for experiment automation. This guide will go over the basics of using Ansible for Merge experiments, the general Ansible docs can be found here.
Ansible allows you to automate a collection of experiment nodes from a centralized file called a playbook.
We recommend executing playbooks from an XDC attached to your experiment materialization.
The following topology will be used in this example. It's the simple two node topology depicted in the diagram above.
A playbook is a sequence of plays that can be anything from provisioning users,
copying files, configuring software to running generic shell commands and
scripts. Here is an example of a playbook that installs and runs the
network performance tool across two nodes. Here we assume the nodes are called
b and that the experiment network IP addresses of these nodes
are 10.0.0.1 and 10.0.0.2 respectively.
Ansible requires a file called an inventory to tell it how to reach the nodes
it's automating. This file is typically called
hosts. Here is an example for
our two node topology.
Names in square brackets indicate groups, the names below the square brackets
are the members of a group. The
all group was used in the first sequence of
plays above, and then individual nodes were used for the follow-on play
You should be able to retrieve a very basic inventory from the xdc via the
xdc ansible inventory (or store it by redirecting the output to a
Placing the inventory at location
/etc/ansible/hosts causes ansible to use it
by default (without requiring a command line option for inventory). You can also
alter this location in your
Ansible configuration is kept in a file called
ansible.cfg. This file can be
/etc/ansible/ansible.cfg, or it can be placed in the local directory
where you execute a playbook from. There are a few options that are very useful
to set for executing from an XDC.
To execute the
playbook.yml above we simply type
After executing the playbook, you should have a results file in your local directory like the following
If you are running a larger experiment, or if you need to optimize your ansible runs, you can change some ansible configuration options to try and reduce ansible's memory load on the XDC, such as caching your hosts' facts in a json file.
OpenSSH command update
It is a relatively small optimization, but you can ask ssh to try your ssh keys before any other authentication types.
Timing analysis callbacks
If you want to try to do some analysis on your execution times or other ansible functions, you can turn on callbacks in the configuration file and get a additional information. See ansible documentation: callbacks:profile_tasks.
You can see what callback plugins are available on your host via
ansible-doc -t callback -l. profile_tasks, role_tasks, and timer are the
plugins you're looking for.
Callbacks may also take some time, so feel free to turn them back off once you are no longer in need of timing analysis.
Host grouping and role assignment
Host grouping is another effective optimization technique that can be utilized from your playbook via inventory group and utilization of roles.
An example of a play running roles against a group of servers:
This kind of role grouping is useful to keep ansible working on the same set of hosts through several sets of changes.
Additionally, if you have a set of tasks or a role where there is no ordering
necessary between hosts, you can specify an alternative "strategy". Strategies
in the ansible documentation, but available strategy
their descriptions are available via the command
ansible-doc -t strategy -l
ansible-doc -t strategy <strategy name>. You may wish to explore the
You can run long-running tasks "in the background" using ansible's asynchronous task handler. This is a bit of a complex topic, so it is suggested to read the available documentation in its entirety before attempting asynchronous tasks, and it is very important to keep in mind that future tasks depending on asynchronous tasks must wait for them or risk play failure.
Many people use a tool called pip to install python packages, and ansible
includes a built-in pip package management module, however, using pip
as a package manager is several times slower than using the system-level package
manager (that is,
apt). If you have python package dependencies,
try searching the system package managers for those dependencies to see if the
version available to the system is acceptable for your use. If so, you can
install them using the system-level packager and reduce play wait time for pip.
Grouping the system packager installs into existing system packager plays also
reduces ansible play run times.
Note: Pypi is currently limiting pip's access to xmlrpc calls (it is disallowed). There is a chance that pip installation will be unavailable or limited in the future.