Experiment Development Containers (XDCs)
Attention
This guide assumes you are using the reference portal atredstartb.net
. If you are using a
different portal, substitute addresses in this document that reference redstartb.net
accordingly.
Consult your project leader if you are not sure of the portal address.Overview
Users interact with materialized experiments through experiment development containers (XDC).
Users can create XDCs and attach them to materialized experiments through either the mrg
CLI or
the web GUI, and can connect to XDCs from their workstations using either a Jupyter web interface or
through a standard SSH command line client. Users commonly use XDCs as a single, centralized point
of command and control for experiment automation and orchestration.
This walk through will show the process of creating an XDC, attaching it to a materialization, and connecting to it through a standard SSH configuration.
XDC basics
At a high level an XDC provides a typical Linux environment through which the user can connect to and thus control their materialization nodes over a secure network. XDCs run on the Merge portal and act as an intermediary between your workstation and the nodes of an experiment materialization.
The following summarizes the basic characteristics of the Merge XDC:
- Any user can create an XDC either via the web GUI or
mrg
CLI. - Each XDC is associated with a single project. That project may be a user’s personal project, or it may be a project with multiple additional members.
- Each XDC runs in an environment that is accessible to each member of the project. Each project member has access to the XDC, even if they did not create the XDC themselves.
- Each XDC runs an Ubuntu 20.04 container with standard Ubuntu tools and binaries.
- XDCs are accessible both online via a Jupyter web interface and via SSH.
- XDC have remote permanent storage mounted so data remains after an XDC is destroyed.
/home/$USER
- All home directories of project members/project
- A shared project directory that all project members can access
Creating an XDC via the CLI
First, create an XDC with mrg
mrg new xdc x0.murphy
This creates a new XDC x0
associated with the personal project of the user murphy
.
Attaching an XDC to a materialization
Assume the user has a materialization named world.hello.murphy
. We now attach the XDC
to this materialization
mrg xdc attach x0.murphy world.hello.murphy
Tip
Remember that XDCs are created in the context of a specific project (which could be a user’s personal project). An XDC created in the namespace of one project cannot be attached to a materialization in a different project, even if the user is a member of both projects.Reaching an XDC via SSH
First, add the SSH jump host to the SSH configuration for your account on your local workstation,
(~/.ssh/config
on typical installations) filling in <username>
with your MergeTB username and
possibly updating Hostname
based on the address of your portal:
Host mergejump
Hostname jump.redstartb.net
Port 2022
User <username>
IdentityFile ~/.ssh/merge_key
ServerAliveInterval 30
Tip
The identity file~/.ssh/merge_key
is populated automatically after you successfully login
via the CLI (mrg login <username>
) on your local workstationNow, use SSH to reach your XDC, using the -J
option to specify the name of the jump host and your destination XDC hostname
- From the perspective of the jump host, the XDC hostname will be a combination of the XDC name and the project name:
<xdc>-<project>
. - If your local username and your MergeTB username differ, you must explicitly set the xdc username to your MergeTB username. To do this, prepend the username to the xdc name like so:
<username>@<xdc>-<project>
. Otherwise, you can get away with just using:<xdc>-<project>
.
Putting it all together, you’ll want to invoke SSH in this general pattern:
ssh -i ~/.ssh/merge_key -J <jumphost> [<username>@]<xdc>-<project>
Here is the command in the context of our example, where your local username does match the one you use for MergeTB:
ssh -i ~/.ssh/merge_key -J mergejump x0-murphy
And here is the command in the context of our example, where your local username does not match the one you use for MergeTB:
ssh -i ~/.ssh/merge_key -J mergejump murphy@x0-murphy
Attention
The hostname used to reach your XDC via SSH is dash delimited (e.g., x0-murphy
) rather than dot delimited (x0.murphy
).
Also, do not attempt to use the FQDN of the XDC as it appears in the web GUI display of XDCs (x0-murphy.redstartb.net
in our example).
Tip
You can add an additional stanza to your SSH configuration for your account on your local workstation to capture this recipe for a particular XDC.
Below is an example of a general pattern for such a stanza:
Host <some_prefix_to_avoid_collision_with_other_hostnames>-<xdc>-<project>
ProxyJump mergejump
Hostname <xdc>-<project>
User <username>
IdentityFile ~/.ssh/merge_key
ServerAliveInterval 30
In our example, it would look like this:
Host my-favorite-xdc-x0-murphy
ProxyJump mergejump
Hostname x0-murphy
User murphy
IdentityFile ~/.ssh/merge_key
ServerAliveInterval 30
Then all you’d have to do to login via SSH would be:
ssh my-favorite-xdc-x0-murphy
This recipe is provided as an example of one way to make connecting via SSH less error prone and more ergonomic for repeated use.
It is by no means the only way to do this. For example, one could make more sophisticated use of wildcards etc in the SSH config markup to setup a single stanza that would apply to any XDCs created for a specific MergeTB project.
Finally, there might be situations where one should not use this particular recipe at all. You will have to gauge this on a case-by-case basis.
Please refer to the SSH docs and any tips/guides online for general SSH best practices and conventions.
Detaching an XDC from a materialization
Assume now the user wants to attach x0
to a new materialization named planet.hello.murphy
. We must
detach from the first materialization before attaching to the second:
mrg xdc detach x0.murphy
mrg xdc attach x0.murphy planet.hello.murphy