Hello World (Web Interface)

Create Experiment Walk-through

This document walks you though the steps to go from nothing to a fully active experiment on the testbed. It then details how to cease the experiment and free all resources used for others. Please take a moment to read the Overview of the Merge Testbed Platform and Experimentation Concepts. Here are the basic steps (don’t worry if you do not understand the terminology, it will be explained).

1. Create an experiment in a project.
2. Design the experiment network model.
3. Push this model to an experiment.
4. Realize this model. (Reserve resources.)
5. Materialize this realization. (Create the actual nodes and network topologies.)
6. Do work.
7. Dematerialize the experiment.
8. Relinquish the realization. (Free the experiment resources.)

Login to the Portal

Go to the launch page for your portal: https://launch.mod.deterlab.net. Login to the site using the username and password created above. On successful login, you’ll be greeted with a list of news items.

To access the list of Projects you have access to, click Projects in the left side menu bar. Initially this will just be your personal project. Your personal project name is the same as your username. It’s created when your account is activated.

Create an Experiment

To get started we’ll create an experiment.

Every experiment is a part of a project. Here we are using murphy’s personal project murphy for the home of our hello experiment.

Click the Experiments tab on the left.

Click the New Experiment button. A dialog will spawn in which you can fill out the details of the new experiment. Details of the fields are:

• Project - this is a drop down list of projects you have access to. At the moment you only have access to your personal project. As you join other projects, they will be added here.
• Creator - your account. This cannot be changed.
• Name - the name of the experiment. We are using hello here.
• Description - an optional field to describe the experiment.

After clicking Submit, the experiment is created. You will see your new experiment listed.

Experiment Details

Now that the experiment exists, we can view the details by clicking the name of the experiment, hello, in the Experiments tab. This will take us to a page the experiment’s details page.

In addition to the information we’ve already seen, this display shows us two new things.

• The address for a Merge-hosted Git repository for our experiment https://git.mod.deterlab.net/murphy/hello
• The revisions of the experiment models associated with this experiment and any live realizations based on the model revisions.

The Git repository is created just for your experiment. It’s a normal git repository so you can store experiment related data there. The Portal also uses it to store the model and all model revisions for your experiment. So let’s create a new model and push it to our new hello experiment.

Compiling and Pushing an Experiment Model

The experiment model expressed as a python file. We create the file then push it to the experiment Git repository. You can push it using standard git tools if you’d like, but Launch can do this for you.

Launch has a built-in model editor. Click the Model Editor tab to get to it.

If you have an existing model, you could Browse to it or drop it in the editor. We’re starting from scratch though, so click Start from scratch to load an empty file. Cut and paste the following into the editor.

from mergexp import *

# Create a network topology object. This network will automatically
# add IP addresses to all node interfaces and configure static routes
# between all experiment nodes. 
net = Network('hello-world', addressing==ipv4, routing==static)

# Create three nodes.
a,b,c = [net.node(name) for name in ['a', 'b', 'c']]

# Create a link connecting the three nodes.
link = net.connect([a,b,c])

# Make this file a runnable experiment based on our three node topology.
experiment(net)

Note that on the upper left of the editor frame, we now have more options. In addition to standard file operations like Clear, Upload, Download, etc, there are two model specific buttons.

• compile - this button sends the current file in the editor to the Portal for compilation. If there is an error in the model, the GUI will display the error. If the compilation is successful the GUI displays the model topology under the editor.
• push - this button pushes the current file in the editor to the experiment repository. The portal will see that a new model has been pushed, and compile the model in the background. The compiled model is then added as a Revision to the experiment.

Warning

Once a model is pushed to the experiment it is committed to the experiment repository. This is true even if the model has an error and does not compile. So it is a good idea to always compile the model and confirm it compiles correctly and the model topology is what you expect before you push the model. before ! If a bad model is pushed, it is not the end of the world. You will just have a revision of the model that it not useful as it cannot be realized.

Here’s an example compilation failure. (I changed the N in Network to an M, making the symbol invalid.)

And here’s a successful compilation. Note the success message and the model topology.

Once you confirm that the model compiles, push the model to your experiment by pushing the button.

This dialog currently has no editable entries. It is purely informative. It shows the experiment the model is being pushed to along with the branch and tag the model is under in the repository. Now push the model by clicking Push.

Above the editor you will see a confirmation message that the model was successfully pushed. (Or an error if it was not).

You can click on the experiment name to be taken to the experiment details page or the commit hash of the revision to be taken to the revision details page. Click the experiment name hello.murphy.

We see that our model revision has been added to the experiment at the bottom of the page. The revision compiled successfully, but has not been realized. This is the next step to create a working experiment.

Realizing the Experiment

Realization is the act of finding a suitable set of resources for your experiment, and allocating those resources for your experiment’s exclusive use. The specific resources are reserved just for your experiment. The realization is a reservation on all the hardware needed to run the actual experiment. To free the resources for use by others, you need to relinquish the realization. But now we will realize the single experiment revision in our experiment.

Go to the Experiments tab and click on the name of the single experiment, hello. Click on the Revision hash we want to realize. There is only one as we’ve only pushed a single revision. In the images this is b969bb8b. For you it will be something else. This takes you to the revision details page.

Click on the revision tabs to see the model in different formats. (Source code, experiment Representation (XIR), Graphviz topology, etc). Click on the topology tools to get a sense of what’s available in the topology display. When you’re ready to realize the model, click the Realize Revision button.

Give the realization a name, world. And click Submit. This will start the realization process. If the process started successfully, a “realization successful” pop up will be displayed.

Assessing Realization Status

Realization, like most everything on a merge portal, is an asynchronous process. (Realization for this model should take less than a second.) To check the status of your realization, click the Realizations tab. If the realization is complete you are greeted with a single successful realization.

If the realization failed, the Succeeded entry in the table would be a Failed button. Clicking the button will show you why the realization failed. A failed realization is usually due to the testbed not currently having the resources requested available at the time of realization. (You can click Resources on the left-side menu to see the total and available resources on the testbeds attached to the portal.)

You can see the realization details by clicking on the first entry in the row, the name, world.

The realization details are

• Name - the hostname of the node as given the model
• Experiment Address - the address on the experiment network (if given in the model)
• Intranet Address - the address on the command and control network used by Merge
• Kind - the kind of node. Physical or virtual
• Resource - the name of the machine in the facility being used
• Image - the OS the node is running
• Facility - which facility the node exists in

Materializing the Realization

Materialization is the process of taking the resources reserved by your realization and making them exist in the world. Nodes (bare metal nodes and/or virtual machines) in the experiment are turned on, imaged with the correct operating system and configured according to experiment specifications. The network to implement the experiment topology is created (along with an command and control “infrastructure” network that the portal uses for node management).

To materialize your realization, click the kabab menu and choose Materialize. This will kick off the materialization process. A pop-up will let you know the process has started. (The Relinquish command will delete the realization and free its resources for use to others.)

To see the materialization details (and status) click the Materializations tab.

Now click the single materialization shown, world.hello.murphy.

The materialization information is very similar to the realization information. Note though that this table has a kebab action menu, indicating that actions may be taken on individual nodes in the materialization. Also note though that the actions listed (Console and Reboot) are not clickable as these actions are still under development. Once complete, you will be able to reboot or open a web console on the node. TThe Status field shows the current status of your materialization. Until the experiment nodes and networks are fully configured, this will be Pending.

Once fully materialized, the status changes to Success.

You can see all materialization tasks (how the experiment nodes and networks were configured) and the materialization details by clicking the name of the materialization, which in this case is world.hello.murphy. The Status tab shows status (of course) and the Details tab shows the details.

.

Interacting with the Experiment

Create an XDC

The entry point to your experiment is an eXperiment Development Container (XDC). This container runs inside the Portal and serves as a home base for experiment development, configuration, and analysis. The container exists in a project space and can “attach” to any materialization in the same project that it is in. The attach process creates a tunnel from the XDC to the “infrapod” inside your experiment. From there, you can connect and interact with nodes in your experiment.

Click the XDCs tab on the left then the New XDC button. Fill out the dialog specifying the name of the XDC and the project under which it resides. (The projects drop down menu is populated with projects you have access to. At the moment, that is the single murphy project.) Click Submit to create the XDC. The XDCs table will reload and show the new XDC xdc.murphy.

The XDC details are as follows:

• name - the short name of the XDC. The long name, which you’ll see references to elsewhere is <xdc name>.<project name>. In our case this is xdc.murphy
• Project - which project of the XDC
• Attached - the materialization to which the XDC is attached, if any
• URL - This is a link to the Jupyter Lab instance running on the XDC. (Note that the link has an embedded token in it. Do not share this link with anyone you do not want to have access to the XDC.)
• SSH Name - the ssh hostname of the node. This is the name used to ssh to the XDC.
• Creator - who created the XDC. In this case, it’s you.
• Memory Limit - the maximum amount of memory in GB the XDC can use
• CPU Limit - maximum limit of CPU time given to the XDC
• Image - the container image

Login to the XDC

Now that the XDC is running, you can log into it. When an XDC is created user accounts are added to it for all members of the project in which the XDC exists. The merge generated users ssh keys are placed in these account. There are two ways to login to an XDC: 1) open a console in the XDC’s jupyter instance or 2) login via ssh. Logging in via ssh is covered here in the XDC documentation) so we’ll cover the first way.

Click the XDCs tab, then click the Jupyter link. This will spawn a new window with a Juptyer lab instance.

Click the Terminal button. This will open a terminal on the XDC with you logged in as root. Use sudo to login as yourself sudo su - murphy. You are now logged into the XDC. The merge command line utility, mrg is installed on all XDCs. (It is also preconfigured to communicate with the portal.) You use this to build a secure tunnel between the XDC and your experiment. This is covered in the next section.

Attaching to the Experiment

Attaching to an experiment is the process of creating a secure tunnel between an XDC and a materialization’s network. Once attached, an XDC can connect directly to any node in the materialization. To attach the XDC to a materialization, click the XDCs tab. Choose the row of XDC to attach. In our case there is only one, the xdc.murphy XDC. Click the action menu and choose Attach.

This spawns a dialog. The XDC name is the one you chose. Now choose the materialization to attach to. The Materialization drop-down menu is populated with all the materializations you have access to in that project. There is only one at the moment, world.hello.murphy so that is chosen for you automatically. Click Attach.

The attach process takes a few moments. The XDC list will refresh, but before the attach process completes. So wait a moment then refresh the XDCs page. After this you can see the Attached column shows that your XDC xdc.murphy is attached to the world.hello.murphy materialization.

Configuring and using the Experiment Materializations

Please see the documentation on Experiment Automation.

Rebooting or Reimaging Nodes of a Materialization

After an experiment has materialized, you can reboot or reimage nodes at any time. Nodes can be warm rebooted, in which the node OS attempts to gracefully shutdown and restart, or hard reset through a power cycle operation. Nodes can also be reimaged, which corresponds to a power cycle operation plus resetting the node to its base image specified in your experiment model.

Navigate to the materialization page for your materialization, click to the ‘Details’ page, and then click the hamburger button on the right to reboot or reimage an individual node.

It is also possible to reboot or reimage every node in your materialization. Click on the global ‘Materializations’ tab, find your materialization, and click on the hamburger button on the right.

Experiment Cleanup

Once you’re done using the your experiment, you should free the resources for others. Merge is smart about experiment clean up and will free and relinquish all resources when an experiment is deleted. We go through the steps here though for informational purposes and in case you ever need to partially cleanup an experiment. Knowing these steps are useful.

Detach XDCs

To detach an XDC, navigate to the XDCs tab. Find the XDC to detach (in our case there is only one). Choose Detach from the action menu.

Note that the Attached column is now empty. (If it is not, refresh the page.)

Dematerialize the Experiment

Dematerializing an experiment does the opposite of materialization. The nodes are reimaged and the experiment networks are torn down. Note that this does not free the resources for use to others. The realization is resource reservation. We’ll relinquish the resources in the next step.

Click the Materializations tab on the left. Find the row that corresponds to the materialization you want to dematerialize. In our case there is only one, world.hello.murphy. Click the kabab menu at the end of the row and choose Dematerialize.

Relinquish the Resources

Now we’ll free the resources you’ve reserved with your realization.

Click the Realizations tab. Find the realization you want to relinquish. In our case there is only one. Click the kabab menu on the far right of the row and choose Relinquish. The table will refresh and that entry is gone. The realization is relinquished.

(If you want to re-realize, just browse to the hello.murphy experiment page, choose the model revision, and click the Realize Revision button.)

Support

The Support tab in Launch has links to a few ways to get support. For immediate help the best way to contact us is via Chat. Bug reports or feature requests can be given to the Merge Testbed Gitlab repository issues system.