Experiment Development Containers (XDCs)

Overview

Users interact with materialized experiments through experiment development containers (XDC). Users can create XDCs and attach them to materialized experiments through either the mrg CLI or the web GUI, and can connect to XDCs from their workstations using either a Jupyter web interface or through a standard SSH command line client. Users commonly use XDCs as a single, centralized point of command and control for experiment automation and orchestration.

This walkthrough will show the process of creating an XDC, attaching it to a materialization, and connecting to it through a standard SSH configuration.

XDC basics

At a high level an XDC provides a typical Linux environment through which the user can connect to and thus control their materialization nodes over a secure network. XDCs run on the Merge portal and act as an intermediary between your workstation and the nodes of an experiment materialization.

The following summarizes the basic characteristics of the Merge XDC:

• Any user can create an XDC either via the web GUI or mrg CLI.
• Each XDC is associated with a single project. That project may be a user’s personal project, or it may be a project with multiple additional members.
• Each XDC runs in an environment that is accessible to each member of the project. Each project member has access to the XDC, even if they did not create the XDC themselves.
• Each XDC runs an Ubuntu 20.04 container with standard Ubuntu tools and binaries.
• XDCs are accessible both online via a Jupyter web interface and via SSH.
• XDC have remote permanent storage mounted so data remains after an XDC is destroyed.
  • /home/$USER - All home directories of project members
  • /project - A shared project directory that all project members can access

Creating an XDC via the CLI

First, create an XDC with mrg

mrg new xdc x0.murphy

This creates a new XDC x0 associated with the personal project of the user murphy.

Attaching an XDC to a materialization

Assume the user has a materialization named world.hello.murphy. We now attach the XDC to this materialization

mrg xdc attach x0.murphy world.hello.murphy

Reaching an XDC via SSH

First, add the SSH jump host to the SSH configuration for your account on your local workstation, (~/.ssh/config on typical installations) filling in <username> with your MergeTB username and possibly updating Hostname based on the address of your portal:

Host mergejump
    Hostname jump.redstartb.net
	Port 2022
	User <username>
	IdentityFile ~/.ssh/merge_key
	ServerAliveInterval 30

Now, use SSH to reach your XDC, using the -J option to specify the name of the jump host:

# ssh -i ~/.ssh/merge_key -J <jumphost> <username>@<xdc>-<project>
ssh -i ~/.ssh/merge_key -J mergejump murphy@x0-murphy

Detaching an XDC from a materialization

Assume now the user wants to attach x0 to a new materialization named planet.hello.murphy. We must detach from the first materialization before attaching to the second:

mrg xdc detach x0.murphy
mrg xdc attach x0.murphy planet.hello.murphy