Hello World (Web Interface)
Welcome to your guided tour of networked system experimentation in Merge. Start by following the “Getting Started” instructions to create your account and have it approved.
This guide assumes you are using the reference portal at
mod.deterlab.net. If you are using a
different portal, substitute addresses in this document that reference
Consult your project leader if you are not sure of the portal address.
Now let’s dive in.
- This guide assumes the username
murphy. Adjust for your username accordingly.
- The URLs shown in the screenshots are from a test installation and will not match the URLs in your browser.
- The web interface is known as “Launch” and will be referred to using this name.
Create Experiment Walk-through
This document walks you though the steps to go from nothing to a fully active experiment on the testbed. It then details how to cease the experiment and free all resources used for others. Please take a moment to read the Overview of the Merge Testbed Platform and Experimentation Concepts. Here are the basic steps (don’t worry if you do not understand the terminology, it will be explained).
- Create an experiment in a project.
- Design the experiment network model.
- Push this model to an experiment.
- Realize this model. (Reserve resources.)
- Materialize this realization. (Create the actual nodes and network topologies.)
- Do work.
- Dematerialize the experiment.
- Relinquish the realization. (Free the experiment resources.)
Login to the Portal
Go to the launch page for your portal: https://launch.mod.deterlab.net. Login to the site using the
password created above. On successful login, you’ll be greeted with the list of Projects you have access to. Initially this will just be your personal project. Your personal project name is the same as your username. It’s created when your account is activated.
NoteTo join existing projects, ask the project administrators to add you to their projects. Once added, you’ll have access to everything in these projects and be able to collaborate with other project members.
Launch displays many elements in a table. These tables (usually) will have the following features.
- A search bar at the top lets you filter the elements. Useful for large tables.
- Up/Down arrows directly after column headings sorts the table by that column.
- Table rows end with an action menu which pops up when the
kabab(the three vertical ) is clicked. These actions act on the element in that row.
- We suggest clicking the kebab each time you see a new table to get a sense of what actions are available.
Create an Experiment
To get started we’ll create an experiment.
Every experiment is a part of a project. Here we are using murphy’s personal project
murphy for the home of our
Experiments tab on the left.
New Experiment button. A dialog will spawn in which you can fill out the details of the new experiment. Details of the fields are:
Project- this is a drop down list of projects you have access to. At the moment you only have access to your personal project. As you join other projects, they will be added here.
Creator- your account. This cannot be changed.
Name- the name of the experiment. We are using
Description- an optional field to describe the experiment.
Submit, the experiment is created. You will see your new experiment listed.
Now that the experiment exists, we can view the details by clicking the name of the project,
hello, in the
Experiments tab. This will take us to a page the experiment’s details page.
In addition to the information we’ve already seen, this display shows us two new things.
- The address for a Merge-hosted Git repository for our experiment https://git.mod.deterlab.net/murphy/hello
- The experiment models associated with this experiment and any live realizations based on the model revisions.
The Git repository is created just for your experiment. It’s a normal git repository so you can store experiment related data there. The Portal also uses it to store the model and all model revisions for your experiment. So let’s create a new model and push it to our new
Compiling and Pushing an Experiment Model
The experiment model expressed as a python file. We create the file then push it to the experiment Git repository. You can push it using standard git tools if you’d like, but Launch can do this for you.
An AsideThe experiment models are stored in the experiment Git repository and must have a specific name and path. The file must be named
model.pyand be in the root directory of the repository. Launch hides these details but they are good to know. After we create out first model we suggest that you
git clonethe experiment repository and examine its contents. Assuming this experiment, the command would be
git clone https://git.mod.deterlab.net/murphy/hello.
Launch has a built-in model editor. Click the
Model Editor tab to get to it.
If you have an existing model, you could
Browse to it or drop it in the editor. We’re starting from scratch though, so click
Start from scratch to load an empty file. Cut and paste the following into the editor.
from mergexp import * # Create a network topology object. net = Network('hello-world') # Create three nodes. a,b,c = [net.node(name) for name in ['a', 'b', 'c']] # Create a link connecting the three nodes. link = net.connect([a,b,c]) # Give IP addresses to the nodes on the link just created. link[a].socket.addrs = ip4('10.0.0.1/24') link[b].socket.addrs = ip4('10.0.0.2/24') link[c].socket.addrs = ip4('10.0.0.3/24') # Make this file a runnable experiment based on our three node topology. experiment(net)
Note that on the upper left of the editor frame, we now have more options. In addition to standard file operations like Clear, Upload, Download, etc, there are two model specific buttons.
compile- this button sends the current file in the editor to the Portal for compilation. If there is an error in the model, the GUI will display the error. If the compilation is successful the GUI displays the model topology under the editor.
push- this button pushes the current file in the editor to the experiment repository. The portal will see that a new model has been pushed, and compile the model in the background. The compiled model is then added as a
Revisionto the experiment.
WarningCompilations do not complete successfully unless the model file ends with the experiment statement. Compiling a model without that statement will result in an error.
WarningOnce a model is pushed to the experiment it is committed to the experiment repository. This is true even if the model has an error and does not compile. So it is a good idea to always compile the model and confirm it compiles correctly and the model topology is what you expect before you push the model. before ! If a bad model is pushed, it is not the end of the world. You will just have a revision of the model that it not useful as it cannot be realized.
Here’s an example compilation failure. (I changed the
Network to an
M, making the symbol invalid.)
And here’s a successful compilation. Note the success message and the model topology.
Once you confirm that the model compiles, push the model to your experiment by pushing the button.
This dialog currently has no editable entries. It is purely informative. It shows the experiment the model is being pushed to along with the branch and tag the model is under in the repository. Now push the model by clicking
Above the editor you will see a confirmation message that the model was successfully pushed. (Or an error if it was not).
You can click on the experiment name to be taken to the experiment details page or the commit hash of the revision to be taken to the revision details page. Click the experiment name
We see that our model revision has been added to the experiment at the bottom of the page. The revision compiled successfully, but has not been realized. This is the next step to create a working experiment.
Realizing the Experiment
Realization is the act of finding a suitable set of resources for your experiment, and allocating those resources for your experiment’s exclusive use. The specific resources are reserved just for your experiment. The realization is a reservation on all the hardware needed to run the actual experiment. To free the resources for use by others, you need to
relinquish the realization. But now we will realize the single experiment revision in our experiment.
Go to the
Experiments tab and click on the name of the single experiment,
hello. Click on the
Revision hash we want to realize. There is only one as we’ve only pushed a single revision. In the images this is
aa7e9133. For you it will be something else. This takes you to the revision details page.
Click on the revision tabs to see the model in different formats. (Source code, experiment Representation (XIR), Graphviz topology, etc). Click on the topology tools to get a sense of what’s available in the topology display. When you’re ready to realize the model, click the
Realize Revision button.
Give the realization a name,
world. And click
Submit. This will start the realization process. If the process started successfully, a “realization successful” pop up will be displayed.
Assessing Realization Status
Realization, like most everything on a merge portal, is an asynchronous process. (Realization for this model should take less than a second.) To check the status of your realization, click the
Realizations tab. If the realization is complete you are greeted with a single successful realization.
If the realization failed, the
Succeeded entry in the table would be a
Failed button. Clicking the button will show you why the realization failed. A failed realization is usually due to the testbed not currently having the resources requested available at the time of realization.
You can see the realization details by clicking on the first entry in the row, the name,
The realization details are
Name- the hostname of the node as given the model
Experiment Address- the address on the experiment network (if given in the model)
Intranet Address- the address on the command and control network used by Merge
Kind- the kind of node. Physical or virtual
Resource- the name of the machine in the facility being used
Image- the OS the node is running
Facility- which facility the node exists in
Materializing the Realization
Materialization is the process of taking the resources reserved by your realization and making them exist in the world. Nodes (bare metal nodes and/or virtual machines) in the experiment are turned on, imaged with the correct operating system and configured according to experiment specifications. The network to implement the experiment topology is created (along with an command and control “infrastructure” network that the portal uses for node management).
To materialize your realization, click the kabab menu and choose
Materialize. This will kick off the materialization process. A pop-up will let you know the process has started. (The
Relinquish command will delete the realization and free its resources for use to others.)
WarningThere is currently no way to confirm that the materialization was successful or the status of the materialization process at the moment. This is under active development.
To see the materialization details click the
Now click the single materialization shown,
The materialization information is very similar to the realization information. Note though that this table has a kebab action menu, indicating that actions may be taken on individual nodes in the materialization. Also note though that the actions listed (
Reboot) are not clickable as these actions are still under development. Once complete, you will be able to reboot or open a web console on the node.
Interacting with the Experiment
Create an XDC
The entry point to your experiment is an eXperiment Development Container (XDC). This container runs inside the Portal and serves as a home base for experiment development, configuration, and analysis. The container exists in a project space and can “attach” to any materialization in the same project that it is in. The attach process creates a tunnel from the XDC to the “infrapod” inside your experiment. From there, you can connect and interact with nodes in your experiment.
XDCs tab on the left then the
New XDC button. Fill out the dialog specifying the name of the XDC and the project under which it resides. (The projects drop down menu is populated with projects you have access to. At the moment, that is the single
murphy project.) Click
Submit to create the XDC. The XDCs table will reload and show the new XDC
NoteThe XDC will take a few moments to spin up. While it is spinning up the field
FQDNwill be shown as
N/A. Once the XDC is active the field will show the fully qualified name of the XDC. The XDC listing will not automatically refresh to show this. You’ll need to refresh the page yourself.
The XDC details are as follows:
name- the short name of the XDC. The long name, which you’ll see references to elsewhere is
<xdc name>.<project name>. In our case this is
Project- which project the XDC is in
Attached- the materialization to which the XDC is attached
URL- This is a link to the Jupyter Lab instance running on the XDC. (Note that the link has an embedded token in it. Do not share this link with anyone you do not want to have access to the XDC.)
FDQN- the fully qualified domain name of the XDC
Creator- who created the XDC. In this case, it’s you.
Memory Limit- the maximum amount of memory in GB the XDC can use
CPU Limit- maximum limit of CPU time given to the XDC
Image- the container image
Attached- the materialization this XDC is attached to
The data in some of these fields are not currently given to Launch, so they may be empty or have zero values.
Login to the XDC
Now that the XDC is running, you can log into it. When an XDC is created user accounts are added to it for all members of the project in which the XDC exists. The merge generated users ssh keys are placed in these account. There are two ways to login to an XDC: 1) open a console in the XDC’s jupyter instance or 2) login via
ssh. Logging in via ssh is covered here in the XDC documentation) so we’ll cover the first way.
XDCs tab, then click the
Jupyter link. This will spawn a new window with a Juptyer lab instance.
Terminal button. This will open a terminal on the XDC with you logged in as root. Use
sudo to login as yourself
sudo su - murphy. You are now logged into the XDC. The merge command line utility,
mrg is installed on all XDCs. You can use that to build a secure tunnel between the XDC and your experiment. This is covered in the next section.
The command line utility,
mrg, needs to be configured on first use in any XDC. (The configuration is stored in your home directory and your home directory is mounted on all XDCs so you only need to configure
mrg one time.)
mrg talks to the Merge API to do work, so it needs to know where the API endpoint is. For the DeterLab testbed, this is
grpc.mod.deterlab.net. To configure
mrg use the
mrg config set command:
mrg config set server grpc.mod.deterlab.net. You can run
mrg config get to confirm the configuration.
Once the server is set, you need to login to
mrg can authenticate you as a user to the Merge API. This just like logging into Launch. Run
mrg login murphy. You will be prompted for your password.
Attaching to the Experiment
Attaching to an experiment is the process of creating a secure tunnel between an XDC and a materialization’s network. Once attached, an XDC can connect directly to any node in the materialization. To attach the XDC to a materialization, click the
XDCs tab. Choose the row of XDC to attach. In our case there is only one, the
xdc.murphy XDC. Click the action menu and choose
This spawns a dialog. The XDC name is the one you chose. Now choose the materialization to attach to. The
Materialization drop-down menu is populated with all the materializations you have access to in that project. There is only one at the moment,
world.hello.murphy so that is chosen for you automatically. Click
The attach process takes a few moments. The XDC list will refresh, but before the attach process completes. So wait a moment then refresh the
XDCs page. After this you can see the
Attached column shows that your XDC
xdc.murphy is attached to the
Configuring and using the Experiment Materializations
Please see the documentation on Experiment Automation.
Once you’re done using the your experiment, you should free the resources for others. Merge is smart about experiment clean up and will free and relinquish all resources when an experiment is deleted. We go through the steps here though for informational purposes and in case you ever need to partially cleanup an experiment. Knowing these steps are useful.
To detach an XDC, navigate to the
XDCs tab. Find the XDC to detach (in our case there is only one). Choose
Detach from the action menu.
Note that the
Attached column is now empty. (If it is not, refresh the page.)
Dematerialize the Experiment
Dematerializing an experiment does the opposite of materialization. The nodes are reimaged and the experiment networks are torn down. Note that this does not free the resources for use to others. The realization is resource reservation. We’ll relinquish the resources in the next step.
Materializations tab on the left. Find the row that corresponds to the materialization you want to dematerialize. In our case there is only one,
world.hello.murphy. Click the kabab menu at the end of the row and choose
Relinquish the Resources
Now we’ll free the resources you’ve reserved with your realization.
Realizations tab. Find the realization you want to relinquish. In our case there is only one. Click the kabab menu on the far right of the row and choose
Relinquish. The table will refresh and that entry is gone. The realization is relinquished.
(If you want to re-realize, just browse to the
hello.murphy experiment page, choose the model revision, and click the
Realize Revision button.)
Support tab in Launch has links to a few ways to get support. For immediate help the best way to contact us is via Chat. Bug reports or feature requests can be given to the Merge Testbed Gitlab repository issues system.