Experiment Development Containers (XDCs)

Overview

Users interact with materialized experiments through experiment development containers (XDC). Users can create XDCs and attach them to materialized experiments through either the mrg CLI or the web GUI, and can connect to XDCs from their workstations using either a Jupyter web interface or through a standard SSH command line client. Users commonly use XDCs as a single, centralized point of command and control for experiment automation and orchestration.

This walk through will show the process of creating an XDC, attaching it to a materialization, and connecting to it through a standard SSH configuration.

XDC basics

At a high level an XDC provides a typical Linux environment through which the user can connect to and thus control their materialization nodes over a secure network. XDCs run on the Merge portal and act as an intermediary between your workstation and the nodes of an experiment materialization.

The following summarizes the basic characteristics of the Merge XDC:

• Any user can create an XDC either via the web GUI or mrg CLI.
• Each XDC is associated with a single project. That project may be a user’s personal project, or it may be a project with multiple additional members.
• Each XDC runs in an environment that is accessible to each member of the project. Each project member has access to the XDC, even if they did not create the XDC themselves.
• Each XDC runs an Ubuntu 20.04 container with standard Ubuntu tools and binaries.
• XDCs are accessible both online via a Jupyter web interface and via SSH.
• XDC have remote permanent storage mounted so data remains after an XDC is destroyed.
  • /home/$USER - All home directories of project members
  • /project - A shared project directory that all project members can access

Creating an XDC via the CLI

First, create an XDC with mrg

mrg new xdc x0.murphy

This creates a new XDC x0 associated with the personal project of the user murphy.

Attaching an XDC to a materialization

Assume the user has a materialization named world.hello.murphy. We now attach the XDC to this materialization

mrg xdc attach x0.murphy world.hello.murphy

Reaching an XDC via SSH

First, add the SSH jump host to the SSH configuration for your account on your local workstation, (~/.ssh/config on typical installations) filling in <username> with your MergeTB username and possibly updating Hostname based on the address of your portal:

Host mergejump
    Hostname jump.mod.deterlab.net
	Port 2022
	User <username>
	IdentityFile ~/.ssh/merge_key
	ServerAliveInterval 30

Now, use SSH to reach your XDC, using the -J option to specify the name of the jump host and your destination XDC hostname

• From the perspective of the jump host, the XDC hostname will be a combination of the XDC name and the project name: <xdc>-<project>.
• If your local username and your MergeTB username differ, you must explicitly set the xdc username to your MergeTB username. To do this, prepend the username to the xdc name like so: <username>@<xdc>-<project>. Otherwise, you can get away with just using: <xdc>-<project>.

Putting it all together, you’ll want to invoke SSH in this general pattern:

ssh -i ~/.ssh/merge_key -J <jumphost> [<username>@]<xdc>-<project>

Here is the command in the context of our example, where your local username does match the one you use for MergeTB:

ssh -i ~/.ssh/merge_key -J mergejump x0-murphy

And here is the command in the context of our example, where your local username does not match the one you use for MergeTB:

ssh -i ~/.ssh/merge_key -J mergejump murphy@x0-murphy

SSH configuration customizations

You can add an additional stanza to your SSH configuration for your account on your local workstation to capture this recipe for a particular XDC.

Below is an example of a general pattern for such a stanza:

Host <some_prefix_to_avoid_collision_with_other_hostnames>-<xdc>-<project>
        ProxyJump mergejump
        Hostname <xdc>-<project>
        User <username>
        IdentityFile ~/.ssh/merge_key
        ServerAliveInterval 30

In our example, it would look like this:

Host my-favorite-xdc-x0-murphy
        ProxyJump mergejump
        Hostname x0-murphy
        User murphy
        IdentityFile ~/.ssh/merge_key
        ServerAliveInterval 30

Then all you’d have to do to login via SSH would be:

ssh my-favorite-xdc-x0-murphy

This recipe is provided as an example of one way to make connecting via SSH less error prone and more ergonomic for repeated use.

It is by no means the only way to do this. For example, one could make more sophisticated use of wildcards etc in the SSH config markup to setup a single stanza that would apply to any XDCs created for a specific MergeTB project.

Finally, there might be situations where one should not use this particular recipe at all. You will have to gauge this on a case-by-case basis.

Please refer to the SSH docs and any tips/guides online for general SSH best practices and conventions.

SSH configuration for old OpenSSH versions

Older versions of OpenSSH do not support the ProxyJump option. This is the case, for example, for the FreeBSD version on users.isi.deterlab.net.

On these systems, we must construct the ProxyCommand option manually in our ~/.ssh/config as follows:

Host <xdc>-<project>
	User <username>
	IdentityFile ~/.ssh/merge_key
	ServerAliveInterval 30
    ProxyCommand ssh jump.mod.deterlab.net -p 2022 -i ~/.ssh/merge_key -W %h:%p

i.e., for user murphy with XDC named x0:

Host x0-murphy
    User murphy
	IdentityFile ~/.ssh/merge_key
	ServerAliveInterval 30
    ProxyCommand ssh jump.mod.deterlab.net -p 2022 -i ~/.ssh/merge_key -W %h:%p

The XDC can then be reached simply via:

ssh <xdc>-<project>

e.g.,

ssh x0-murphy

Note that you can add multiple XDCs to the top-level Host as needed. For example, if murphy created a new XDC x1 in the project ddos, this could be configured as:

Host x0-murphy x1-ddos
    User murphy
	IdentityFile ~/.ssh/merge_key
	ServerAliveInterval 30
    ProxyCommand ssh jump.mod.deterlab.net -p 2022 -i ~/.ssh/merge_key -W %h:%p

It is also possible to simply add the wildcard host *-*, which will match any XDC patterns, but will of course match hostanmes that might not be XDCs, so use at your own discretion.

Reaching an XDC via JupyterLab

XDCs are also accessible via a JupyterLab web interface. Each XDC can be reached via HTTP(s) using a unique URL. To find the URL, simply run mrg list xdc:

$ mrg list xdc
Name.Project    Attached    Reference                           Creator    Memory    CPU        Image    URL
------------    --------    ---------                           -------    ------    ---        -----    ---
x0.murphy                   x0-murphy.xdc.mod.deterlab.net      murphy     0GB       0 cores             https://x0-murphy.xdc.mod.deterlab.net/jupyter/lab?token=83879b5f-7e74-4efd-a818-25a7460825ca

and then copy/paste the URL into your web browser.

From the JupyterLab page, click the ‘Terminal’ icon to create a new shell session in the XDC. By default, you will be logged in as root using an sh shell. To get a more familiar environment, login to your username via su as follows:

# su - <username>

e.g.,

# su - murphy
murphy@x0:~$

https://x0-murphy.xdc.mod.deterlab.net/jupyter/lab?token=83879b5f-7e74-4efd-a818-25a7460825ca

Reaching your materialization from an XDC

When an XDC is attached to a materialization a network tunnel is created between the XDC and the materialization network. (As shown above in XDC basics).

The XDC is also configured to resolve names from your experiment into infranet addresses. As discussed in the Node Resolution and Naming documentation, each node in your experiment is known by names on both the experiment network and on the infranet. The experiment network is your network - it is the topology defined by you in your experiment model. The infranet is a command and control network used by the testbed to provision and configure your experiment nodes, and is the only network reachable from an XDC.

To resolve names into infranet addresses from an XDC, simply use their short names – i.e., the names you put in you model:

ssh a

Detaching an XDC from a materialization

Assume now the user wants to attach x0 to a new materialization named planet.hello.murphy. We must detach from the first materialization before attaching to the second:

mrg xdc detach x0.murphy
mrg xdc attach x0.murphy planet.hello.murphy