Installation
Tip
This guide assumes a facility software stack based on Mars. See the detailed discussion on the design and implementation of Mars in the “Development” documentation.Model in hand, it’s now time to generate installation contents for your Merge facility. Merge provides a custom self-contained installer that generates most of what you need to get your facility up and running.
Overview
The image above depicts the inputs to and outputs from the facility installer. Inputs include:
- The facility XIR model
- A user configuration file
The outputs include:
- A configuration for Ground Control (discussed below)
- A configuration for the Mars core services
- A set of Ignition Configs that automate installation of infrastructure servers, storage servers, network emulators, and any other facility assets running Fedora CoreOS (FCOS).
- A set of Zero Touch Provisioning Scripts (ZTP) that automate installation of ZTP enabled platforms, such as Cumulus Linux.
- Miscellaneous installation artifacts, including SSL certs/keys for the Mars core services and an SSH key for the facility operator that will be installed on each facility asset
Ground Control
Ground Control is a one stop shop for provisioning assets in a Mars facility. Ground Control provides DHCP and DNS for the facility management network, allowing an operator to easily reach facility assets and provision them using network based installation procedures. Ground Control implements a PXE server and provides installation assets over the management network for FCOS and ZTP based systems. With a single Ground Control config file, an operator can provision all the infrastructure-level systems on the testbed such as infrastructure servers, storage servers, network emulators and switches.
Configuration
Ground Control is configured through a single YAML file. When Ground Control starts up, the YAML file will be marshaled into the following Golang structure:
type Config struct {
Domain string
Cluster string // unused
Subnet string
SubnetDynamic string
ForwardDNS string
Nodes []*NodeConfig
Switches []*SwitchConfig
Services []*facility.DanceService // deprecated
Dns *DnsConfig
StaticFiles []StaticFile
}
We briefly describe the key options one by one. Note that some options are unused and/or deprecated and may disappear from a future release.
Name | Description |
---|---|
Domain | Search domain for the facility management network |
Subnet | IPv4 subnet for the facility management network |
SubnetDynamic | IPv4 subnet that will be used exclusively for dynamic IP allocation for infrapods. This must be a subset of Subnet |
ForwardDNS | Where Ground Control should forward DNS requests to |
Nodes | A set of NodeConfig definitions for the static server assets in the facility |
Switches | A set of SwitchConfig definitions for the static switch assets in the facility |
Dns | Additional name/IPv4 address pairs that Ground Control should provide DNS for |
StaticFiles | Files that Ground Control should serve via HTTP on the management network |
Tip
The entire Ground Control configuration can be auto-generated by the facility installer. However, you can customize pieces of the configuration via theuser-config.yml
input to the installer,
which lets you tailor the system to your needs and/or tastes.Mars Configuration
The behavior of the Mars core services can be customized through a Mars YAML configuration file. As in the Ground Control case, this file is marshaled into a Golang structure:
type Config struct {
Services ServicesConfig
Images ImagesConfig
Network NetworkConfig
Ops CredentialsWithSsh
IPMI Credentials
//Alerts []AlertConfig `yaml:",omitempty" json:",omitempty"`
}
We briefly describe these fields:
Name | Description |
---|---|
Services | Addresses, ports, keys and credentials used for communication between Mars services |
Images | Registries to use and names/tags of containers for the Mars services |
Network | Miscellaneous network configuration of Mars services |
Ops | Credentials and SSH key for the human operator (notionally, the “ops” user) |
IPMI | Credentials to access IPMI on facility assets |
Tip
As with Ground Control, the Mars configuration will be auto-generated with reasonable defaults by the facility installer, and you can customize pieces as needed through theuser-config.yml
input
to the installer.Working Example
We will now continue with the example of the Phobos testbed using the model developed in the previous section and a user configuration input file.
Create user-config.yml
Consider the following installer configuration:
ground-control:
domain: phobos.example.com
subnet: 10.0.0.0/16
subnetdynamic: 10.0.128.0/17
forwarddns: 8.8.8.8
staticfiles:
- sourceurl: https://gitlab.com/mergetb/facility/mars/-/jobs/artifacts/v1.0.3/raw/build/mars-install?job=make
destfilename: mars-install
- sourceurl: https://gitlab.com/mergetb/facility/mars/-/jobs/artifacts/v1.0.3/raw/build/canopy?job=make
destfilename: canopy
- sourceurl: https://gitlab.com/mergetb/facility/mars/-/jobs/artifacts/v1.0.3/raw/build/canopy.arm5?job=make
destfilename: canopy.arm5
mars:
images:
defaultregistry:
address: registry.gitlab.com
noverify: false
defaulttag: v1.0.3
infrapod:
foundry:
registry:
address: registry.gitlab.com
noverify: false
name: mergetb/tech/foundry/foundry
tag: v1.1.0
services:
portal:
address: grpc.mergetb.example.net
port: 443
credentials:
username: murphy
password: mergetb
infrapod:
foundry:
tls:
cert: |
-----BEGIN CERTIFICATE-----
MIIBlzCCATygAwIBAgIBATAKBggqhkjOPQQDAjAvMRAwDgYDVQQKEwdNZXJnZVRC
MRswGQYDVQQDExJwaG9ib3MuZXhhbXBsZS5jb20wHhcNMjIwMjE3MjMwODMxWhcN
MjYwMjE2MjMwODMxWjAvMRAwDgYDVQQKEwdNZXJnZVRCMRswGQYDVQQDExJwaG9i
b3MuZXhhbXBsZS5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQpH52tnEoF
z1NhQ/nlWPUwKqoggVln2Qrh3llPBziQLymPk9npDmZcJfHFttDc4DNtYoStkp2w
Zi/y4ZzjhDDbo0kwRzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwDAYDVR0TAQH/BAIwADASBgNVHREECzAJggdmb3VuZHJ5MAoGCCqGSM49BAMC
A0kAMEYCIQDpiLuASX8eZ8j736ky+R+mpfHU9UMS9czO0RVTNpzCNAIhAIkBFsYF
57hIfLH30iRH8S+zJeg17578ThZPqVtw6YPP
-----END CERTIFICATE-----
key: |
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIPW8LHz0D3+OWRFRS0Eaq8C2uFPNvoyxYG8zCseo5kQYoAoGCCqGSM49
AwEHoUQDQgAEKR+drZxKBc9TYUP55Vj1MCqqIIFZZ9kK4d5ZTwc4kC8pj5PZ6Q5m
XCXxxbbQ3OAzbWKErZKdsGYv8uGc44Qw2w==
-----END EC PRIVATE KEY-----
infraserver:
apiserver:
tls:
cert: |
-----BEGIN CERTIFICATE-----
MIIByTCCAW6gAwIBAgIBATAKBggqhkjOPQQDAjAvMRAwDgYDVQQKEwdNZXJnZVRC
MRswGQYDVQQDExJwaG9ib3MuZXhhbXBsZS5jb20wHhcNMjIwMjE3MjMwODMxWhcN
MjYwMjE2MjMwODMxWjAvMRAwDgYDVQQKEwdNZXJnZVRCMRswGQYDVQQDExJwaG9i
b3MuZXhhbXBsZS5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASALVDzzxP9
/Hex9W94D447lws/ULVPdLd9YJxEXr4jrDQot2QKv8byXaxOJEVgkrdn+FoLa/Z7
N5Dcglg83CApo3sweTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwDAYDVR0TAQH/BAIwADBEBgNVHREEPTA7gglhcGlzZXJ2ZXKCFm9wcy5waG9i
b3MuZXhhbXBsZS5jb22CFmFwaS5waG9ib3MuZXhhbXBsZS5jb20wCgYIKoZIzj0E
AwIDSQAwRgIhAKS4NtuFhDKfQTTYBnuuTe/UoHCMWoGpOU6LlXR3J2sjAiEAsvMu
ldjuzlZvDv5tMvwQWE928ykLvpDAN0wshsb6Pl8=
-----END CERTIFICATE-----
key: |
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIBXI1EwFCG4du7JAwK7A04g3qPDNQ0ZMbTiuFpvRLjfOoAoGCCqGSM49
AwEHoUQDQgAEgC1Q888T/fx3sfVveA+OO5cLP1C1T3S3fWCcRF6+I6w0KLdkCr/G
8l2sTiRFYJK3Z/haC2v2ezeQ3IJYPNwgKQ==
-----END EC PRIVATE KEY-----
ops:
username: ops
password: ops
ssh:
private: |
-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKCAQEAtXRkO0ESV0uVtdJVKNIDUH8LSpa+0hbTwXewud4Rlh3KP+WO
cB3jfnsHASApfGsp1fZIem6qwt4bA/vDY08ATU1Y5DWtYlAzwaWEc0GmCCqnHRmM
YQUbJCud9JxIfnbrkI1Z1XXxkcrJD8/LtCSqTP8BM7zah+QYrgLNVf2GBocck+2h
XXuPB819R/SgKj8MHrxlgVNkqWoO1ejCUBIGNaGvOs6R3vqgq0OCSOR1RDFo0LwM
wQ3HwJAjyXtl6tsuVerX/Q/7f/4E9tpQI7iOSSIZAdPmjvVqTDyJpdZ7kKHGGDlX
jcBUpydRlR0ijyUPZUMzoe50HJg6DER9BtUSHwIDAQABAoIBAQCJ6pFUSD08hIKh
0+BOSnGieS2H39Phst/4KJT2cMdFJhdXQY9lwBHtTG0ieeOOYZVGvw6PoBXCTzaR
bIYxsRY1qxhpPDMwim0MPlXRlJAxnNmv0H9WLp9WeqBwD2k6VgDtucIR8LdPbKYt
km7t2QBnzVqIjH480y2+9PpE5ZE826QPq0G5n3grid8X1JOW2rgNPq78DZ2XEWNk
FV1DhmVk9QTX4/Ot3I23+HPoAnalnMvDh2eooE0tdiP6u8dZ3ZjcVhrOImZVaj+U
96FPzLnCVMLxAELq9Qwz9kinXzokDV3Sesp0I1/6JhzIaFoX8RZeH9r+z+wPB2IX
tIhm+JNBAoGBAOrpIibpm8s6nSwtkYVTK1H0GuwdhiCcOXsd2TDWD3pc3qHMaCPi
hmdBs4FGfmoruUz2emipJDhiTNJLXpw4OAFnj3RpGZpcbG4Y8tmzcO2z+Jff7Ky5
GxAejXKBXNWA1TiCBFcZXkEsuTXQ8sLSG4zAHR8aOGbjeUa8U30aw9FZAoGBAMW+
sqNQ5mVIv4alZbKzJs9cfRttz2U0+S7beiaiwZnewJOK2OpZNRMMvDPoJ8L67mNu
cAsAyRyBa2CJPDWS9ChBMAm9ylKgXnqQPiQcC8wrQi/sDBXnvOsksAhqOglgscxj
2+wWfRE1iCj4vsjodVePomersSTuW1zRJqybINg3AoGBAOQULTuYz9IXe+kb6VGI
uePdXZJo7ASXilvWVD1NaUPGDccMnwRQeWyh5vaaijztbQGjwqABsZApKSt5UEff
cckzUp6VOO2d1Q1knc3vorK4baKQIMxnATa6Y12M6qTFbMBrxGkI0XH4SW9OC5ZK
x8L1+JBtNdv3+pVKXsl2BjApAoGBAJOcA0tIA4QNpvKCmoSnmMvndXg5IzygK3Yf
3XHjwnLyxXwdhEiDKMw8kGyK2suAdt0APmDoSDegICBisPAo71rxNkO/Sb0d+wiw
jcJ0sRXKINEW+fqMDwjUTRcq/IGtjWtOC4eTTe6/sBEthXUBmT4weQG8NzKjPs3t
TlpAPmdHAoGBAOQzwekw7fmzhNMfd8G6Ceqw+XzQYywnN7lL/4zzp6KOr56PWyr+
w5U5DBy2K3dJy8vspMzzj7RJKlRzfPQx1Vrw7fEB/UZk4gYSV7g49ZL7mLlO1SSI
Y/nUSseDNmimUC6vYH3Pr2MWVOJ/hZWZ3WnTWkemlt+I/q0/RieKXmSX
-----END RSA PRIVATE KEY-----
public: |
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1dGQ7QRJXS5W10lUo0gNQfwtKlr7SFtPBd7C53hGWHco/5Y5wHeN+ewcBICl8aynV9kh6bqrC3hsD+8NjTwBNTVjkNa1iUDPBpYRzQaYIKqcdGYxhBRskK530nEh+duuQjVnVdfGRyskPz8u0JKpM/wEzvNqH5BiuAs1V/YYGhxyT7aFde48HzX1H9KAqPwwevGWBU2Spag7V6MJQEgY1oa86zpHe+qCrQ4JI5HVEMWjQvAzBDcfAkCPJe2Xq2y5V6tf9D/t//gT22lAjuI5JIhkB0+aO9WpMPIml1nuQocYYOVeNwFSnJ1GVHSKPJQ9lQzOh7nQcmDoMRH0G1RIf
The configuration has 2 top-level fields: ground-control
and mars
. Underneath these fields are
configurations for Ground Control and Mars respectively, formatted according to the respective
configuration specs of these systems.
facility-install
generates a Ground Control config (ground-control.yml
) and a Mars config
(mars-config.yml
) using values set in user-config.yml
. In cases where a value is not set in
user-config.yml
, facility-install
will generate a value or use a reasonable default. For
example, facility-install
will always configure a Ground Control subnet of “10.0.0.0/16” if none
is specified, and if TLS keying information is not provided for the Mars apiserver
and foundry
services, facility-install
will randomly generate them.
Notable points from this configuration include:
- The management search domain is set to
phobos.example.com
- The management subnet is
10.0.0.0/16
- The management dynamic subnet for infrapods is
10.0.128.0/17
- Ground control will forward DNS requests on the management network to
8.8.8.8
- Ground control will serve static files for
mars-install
,canopy
, andcanopy.arm5
- Services on the testbed can download these via
http://ground-control/{mars-install,canopy,canopy.arm5}
- Services on the testbed can download these via
- The default registry for Mars services in
registry.gitlab.com
and the default tag isv1.0.3
- This means that for Mars services that don’t have entries in this config, the system will
default to using
registry.gitlab.com
/v1.0.3
- This means that for Mars services that don’t have entries in this config, the system will
default to using
- The foundry image should be pulled from
registry.gitlab.com/mergetb/tech/foundry/foundry:v1.1.0
- The API endpoint for the Merge portal is
grpc.mergetb.example.net:443
- The user that commissioned the facility on the Merge portal is
murphy
and their password on the portal ismergetb
- The user that commissioned the facility on the Merge portal is
- Specific TLS keying material is provided for the
apiserver
andfoundry
services- As noted above, this is optional. We use this for the phobos facility install so that repeated
runs of
facility-install
do not continuously generate new keying material
- As noted above, this is optional. We use this for the phobos facility install so that repeated
runs of
- A user with username
ops
and passwordops
will be installed on each facility machine, and the provided public SSH key will be installed into the authorized keys on each machine
Tip
While Ground Control and Mars are designed to support any valid input configuration, we highly recommend sticking with some of the defaults that nearly all of the Merge development and testing has been done with. These include:
ground-control.subnet
:10.0.0.0/16
ground-control.subnetdynamic
:10.0.128.0/17
ground-control.staticfiles
:- use all 3 static files listed, possibly updating
v1.0.3
if you are using a newer Mars release
- use all 3 static files listed, possibly updating
Download the Installer
curl -o ./facility-install -L https://gitlab.com/mergetb/facility/install/-/jobs/artifacts/v1.0.1/raw/build/facility-install?job=make
chmod +x facility-install
Run the installer
Assuming the phobos model is present in your current directory as phobos.xir
and the above
configuration is present as user-config.yml
:
./facility-install generate phobos.xir
This will create a directory called merge-install
and populate it with the following installation contents:
$ ls -l merge-install/
total 92
-rw-r--r--. 1 murphy murphy 227 Jun 8 22:43 apiserver-key.pem
-rw-r--r--. 1 murphy murphy 680 Jun 8 22:43 apiserver.pem
-rw-r--r--. 1 murphy murphy 3602 Jun 8 22:43 cumulus-infra-ztp.sh
-rw-r--r--. 1 murphy murphy 3602 Jun 8 22:43 cumulus-xp-ztp.sh
-rw-r--r--. 1 murphy murphy 11495 Jun 8 22:43 emu.ign
-rw-r--r--. 1 murphy murphy 227 Jun 8 22:43 foundry-key.pem
-rw-r--r--. 1 murphy murphy 611 Jun 8 22:43 foundry.pem
-rw-r--r--. 1 murphy murphy 2286 Jun 8 22:43 ground-control.yml
-rw-r--r--. 1 murphy murphy 22877 Jun 8 22:43 ifr.ign
-rw-r--r--. 1 murphy murphy 10306 Jun 8 22:43 mars-config.yml
-rw-r--r--. 1 murphy murphy 1679 Jun 8 22:43 ops_rsa
-rw-r--r--. 1 murphy murphy 381 Jun 8 22:43 ops_rsa.pub
Installing the Facility
With merge-install/
contents in hand, we can now install Merge on our testbed
Setting Up Ground Control
All testbeds must have at least one operations server running Ground Control. For a complete example of how to set up an operations server, refer to the Phobos ops server ansible setup script. This example assumes an operations server running Centos Stream 9, so you may need to make minor updates to support other OS distributions.
Lines 67-120 in particular setup Ground Control to manage the installation:
- name: setup ignition and ztp dirs
file:
path: /var/ground-control/{{item}}
state: directory
loop:
- ignition
- ztp
- name: copy ignition configs
copy:
src: merge-install/{{item}}
dest: /var/ground-control/ignition/{{item}}
loop:
- ifr.ign
- emu.ign
- name: copy ztp files
copy:
src: merge-install/{{item}}
dest: /var/ground-control/ztp/{{item}}
loop:
- cumulus-xp-ztp.sh
- cumulus-infra-ztp.sh
- name: copy ground-control config
copy:
src: merge-install/ground-control.yml
dest: /etc/ground-control.yml
- name: fetch ground control
get_url:
url: https://gitlab.com/mergetb/ops/ground-control/-/jobs/artifacts/{{gc_ref}}/raw/build/ground-control?job=make
dest: /usr/bin/ground-control
mode: 0755
vars:
gc_ref: bjk-dev
- name: create ground control service
copy:
src: ground-control.service
dest: /lib/systemd/system/ground-control.service
- name: setup ground control
shell: /usr/bin/ground-control setup -n eth0 -s {{ fcos_stream }} -c {{ fcos_version }}
vars:
fcos_stream: stable
fcos_version: 35.20220424.3.0
- name: enable and start ground control service
systemd:
name: ground-control
enabled: true
state: restarted
daemon_reload: yes
Network Boot Configuration
At this point you will need to configure your servers and switches to network boot over the management network. Once they do so, they will automatically download the Ignition or ZTP scripts and install without any need for human operator intervention.
The exact process to configure network booting varies based on your hardware models, BIOS versions, etc. We note that Ground Control supports machines using either UEFI or Legacy PXE boot protocols.
Attention
You may find it useful to utilize Ground Control to manage not just the infrastructure resources (infrastructure servers, storage servers, network emulators and switches) but also the testbed nodes. It can be useful to do so during the testbed installation phase in order to, for example, harvest MAC addresses needed for the facility model.
While we encourage such use of Ground Control, note that once the testbed is installed and is being operated by Mars, the testbed nodes/hypervisors must be configured to boot from the infranet before booting from the management network. Nodes that do not first attempt to boot via the infranet will not be able to use the sled imaging system.