Installation

How to install Merge on your facility

Model in hand, it’s now time to generate installation contents for your Merge facility. Merge provides a custom self-contained installer that generates most of what you need to get your facility up and running.

Overview

The image above depicts the inputs to and outputs from the facility installer. Inputs include:

  • The facility XIR model
  • A user configuration file

The outputs include:

  • A configuration for Ground Control (discussed below)
  • A configuration for the Mars core services
  • A set of Ignition Configs that automate installation of infrastructure servers, storage servers, network emulators, and any other facility assets running Fedora CoreOS (FCOS).
  • A set of Zero Touch Provisioning Scripts (ZTP) that automate installation of ZTP enabled platforms, such as Cumulus Linux.
  • Miscellaneous installation artifacts, including SSL certs/keys for the Mars core services and an SSH key for the facility operator that will be installed on each facility asset

Ground Control

Ground Control is a one stop shop for provisioning assets in a Mars facility. Ground Control provides DHCP and DNS for the facility management network, allowing an operator to easily reach facility assets and provision them using network based installation procedures. Ground Control implements a PXE server and provides installation assets over the management network for FCOS and ZTP based systems. With a single Ground Control config file, an operator can provision all the infrastructure-level systems on the testbed such as infrastructure servers, storage servers, network emulators and switches.

Configuration

Ground Control is configured through a single YAML file. When Ground Control starts up, the YAML file will be marshaled into the following Golang structure:

type Config struct {
    Domain        string
    Cluster       string // unused
    Subnet        string
    SubnetDynamic string
    ForwardDNS    string
    Nodes         []*NodeConfig
    Switches      []*SwitchConfig
    Services      []*facility.DanceService // deprecated
    Dns           *DnsConfig
    StaticFiles   []StaticFile
}

We briefly describe the key options one by one. Note that some options are unused and/or deprecated and may disappear from a future release.

NameDescription
DomainSearch domain for the facility management network
SubnetIPv4 subnet for the facility management network
SubnetDynamicIPv4 subnet that will be used exclusively for dynamic IP allocation for infrapods. This must be a subset of Subnet
ForwardDNSWhere Ground Control should forward DNS requests to
NodesA set of NodeConfig definitions for the static server assets in the facility
SwitchesA set of SwitchConfig definitions for the static switch assets in the facility
DnsAdditional name/IPv4 address pairs that Ground Control should provide DNS for
StaticFilesFiles that Ground Control should serve via HTTP on the management network

Mars Configuration

The behavior of the Mars core services can be customized through a Mars YAML configuration file. As in the Ground Control case, this file is marshaled into a Golang structure:

type Config struct {
    Services ServicesConfig
    Images   ImagesConfig
    Network  NetworkConfig
    Ops      CredentialsWithSsh
    IPMI     Credentials
    //Alerts   []AlertConfig `yaml:",omitempty" json:",omitempty"`
}

We briefly describe these fields:

NameDescription
ServicesAddresses, ports, keys and credentials used for communication between Mars services
ImagesRegistries to use and names/tags of containers for the Mars services
NetworkMiscellaneous network configuration of Mars services
OpsCredentials and SSH key for the human operator (notionally, the “ops” user)
IPMICredentials to access IPMI on facility assets

Working Example

We will now continue with the example of the Phobos testbed using the model developed in the previous section and a user configuration input file.

Create user-config.yml

Consider the following installer configuration:

ground-control:
  domain: phobos.example.com
  subnet: 10.0.0.0/16
  subnetdynamic: 10.0.128.0/17
  forwarddns: 8.8.8.8
  staticfiles:
    - sourceurl: https://gitlab.com/mergetb/facility/mars/-/jobs/artifacts/v1.0.3/raw/build/mars-install?job=make
      destfilename: mars-install
    - sourceurl: https://gitlab.com/mergetb/facility/mars/-/jobs/artifacts/v1.0.3/raw/build/canopy?job=make
      destfilename: canopy
    - sourceurl: https://gitlab.com/mergetb/facility/mars/-/jobs/artifacts/v1.0.3/raw/build/canopy.arm5?job=make
      destfilename: canopy.arm5
mars:
  images:
    defaultregistry:
      address: registry.gitlab.com
      noverify: false
    defaulttag: v1.0.3
    infrapod:
      foundry:
        registry:
          address: registry.gitlab.com
          noverify: false
        name: mergetb/tech/foundry/foundry
        tag: v1.1.0
  services:
    portal:
      address: grpc.mergetb.example.net
      port: 443 
      credentials:
        username: murphy
        password: mergetb
    infrapod:
        foundry:
            tls:
                cert: |
                    -----BEGIN CERTIFICATE-----
                    MIIBlzCCATygAwIBAgIBATAKBggqhkjOPQQDAjAvMRAwDgYDVQQKEwdNZXJnZVRC
                    MRswGQYDVQQDExJwaG9ib3MuZXhhbXBsZS5jb20wHhcNMjIwMjE3MjMwODMxWhcN
                    MjYwMjE2MjMwODMxWjAvMRAwDgYDVQQKEwdNZXJnZVRCMRswGQYDVQQDExJwaG9i
                    b3MuZXhhbXBsZS5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQpH52tnEoF
                    z1NhQ/nlWPUwKqoggVln2Qrh3llPBziQLymPk9npDmZcJfHFttDc4DNtYoStkp2w
                    Zi/y4ZzjhDDbo0kwRzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
                    AwEwDAYDVR0TAQH/BAIwADASBgNVHREECzAJggdmb3VuZHJ5MAoGCCqGSM49BAMC
                    A0kAMEYCIQDpiLuASX8eZ8j736ky+R+mpfHU9UMS9czO0RVTNpzCNAIhAIkBFsYF
                    57hIfLH30iRH8S+zJeg17578ThZPqVtw6YPP
                    -----END CERTIFICATE-----                    
                key: |
                    -----BEGIN EC PRIVATE KEY-----
                    MHcCAQEEIPW8LHz0D3+OWRFRS0Eaq8C2uFPNvoyxYG8zCseo5kQYoAoGCCqGSM49
                    AwEHoUQDQgAEKR+drZxKBc9TYUP55Vj1MCqqIIFZZ9kK4d5ZTwc4kC8pj5PZ6Q5m
                    XCXxxbbQ3OAzbWKErZKdsGYv8uGc44Qw2w==
                    -----END EC PRIVATE KEY-----                    
    infraserver:
        apiserver:
            tls:
                cert: |
                    -----BEGIN CERTIFICATE-----
                    MIIByTCCAW6gAwIBAgIBATAKBggqhkjOPQQDAjAvMRAwDgYDVQQKEwdNZXJnZVRC
                    MRswGQYDVQQDExJwaG9ib3MuZXhhbXBsZS5jb20wHhcNMjIwMjE3MjMwODMxWhcN
                    MjYwMjE2MjMwODMxWjAvMRAwDgYDVQQKEwdNZXJnZVRCMRswGQYDVQQDExJwaG9i
                    b3MuZXhhbXBsZS5jb20wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASALVDzzxP9
                    /Hex9W94D447lws/ULVPdLd9YJxEXr4jrDQot2QKv8byXaxOJEVgkrdn+FoLa/Z7
                    N5Dcglg83CApo3sweTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
                    AwEwDAYDVR0TAQH/BAIwADBEBgNVHREEPTA7gglhcGlzZXJ2ZXKCFm9wcy5waG9i
                    b3MuZXhhbXBsZS5jb22CFmFwaS5waG9ib3MuZXhhbXBsZS5jb20wCgYIKoZIzj0E
                    AwIDSQAwRgIhAKS4NtuFhDKfQTTYBnuuTe/UoHCMWoGpOU6LlXR3J2sjAiEAsvMu
                    ldjuzlZvDv5tMvwQWE928ykLvpDAN0wshsb6Pl8=
                    -----END CERTIFICATE-----                    
                key: |
                    -----BEGIN EC PRIVATE KEY-----
                    MHcCAQEEIBXI1EwFCG4du7JAwK7A04g3qPDNQ0ZMbTiuFpvRLjfOoAoGCCqGSM49
                    AwEHoUQDQgAEgC1Q888T/fx3sfVveA+OO5cLP1C1T3S3fWCcRF6+I6w0KLdkCr/G
                    8l2sTiRFYJK3Z/haC2v2ezeQ3IJYPNwgKQ==
                    -----END EC PRIVATE KEY-----                    
  ops:
    username: ops
    password: ops 
    ssh:
        private: |
            -----BEGIN RSA PRIVATE KEY-----
            MIIEpgIBAAKCAQEAtXRkO0ESV0uVtdJVKNIDUH8LSpa+0hbTwXewud4Rlh3KP+WO
            cB3jfnsHASApfGsp1fZIem6qwt4bA/vDY08ATU1Y5DWtYlAzwaWEc0GmCCqnHRmM
            YQUbJCud9JxIfnbrkI1Z1XXxkcrJD8/LtCSqTP8BM7zah+QYrgLNVf2GBocck+2h
            XXuPB819R/SgKj8MHrxlgVNkqWoO1ejCUBIGNaGvOs6R3vqgq0OCSOR1RDFo0LwM
            wQ3HwJAjyXtl6tsuVerX/Q/7f/4E9tpQI7iOSSIZAdPmjvVqTDyJpdZ7kKHGGDlX
            jcBUpydRlR0ijyUPZUMzoe50HJg6DER9BtUSHwIDAQABAoIBAQCJ6pFUSD08hIKh
            0+BOSnGieS2H39Phst/4KJT2cMdFJhdXQY9lwBHtTG0ieeOOYZVGvw6PoBXCTzaR
            bIYxsRY1qxhpPDMwim0MPlXRlJAxnNmv0H9WLp9WeqBwD2k6VgDtucIR8LdPbKYt
            km7t2QBnzVqIjH480y2+9PpE5ZE826QPq0G5n3grid8X1JOW2rgNPq78DZ2XEWNk
            FV1DhmVk9QTX4/Ot3I23+HPoAnalnMvDh2eooE0tdiP6u8dZ3ZjcVhrOImZVaj+U
            96FPzLnCVMLxAELq9Qwz9kinXzokDV3Sesp0I1/6JhzIaFoX8RZeH9r+z+wPB2IX
            tIhm+JNBAoGBAOrpIibpm8s6nSwtkYVTK1H0GuwdhiCcOXsd2TDWD3pc3qHMaCPi
            hmdBs4FGfmoruUz2emipJDhiTNJLXpw4OAFnj3RpGZpcbG4Y8tmzcO2z+Jff7Ky5
            GxAejXKBXNWA1TiCBFcZXkEsuTXQ8sLSG4zAHR8aOGbjeUa8U30aw9FZAoGBAMW+
            sqNQ5mVIv4alZbKzJs9cfRttz2U0+S7beiaiwZnewJOK2OpZNRMMvDPoJ8L67mNu
            cAsAyRyBa2CJPDWS9ChBMAm9ylKgXnqQPiQcC8wrQi/sDBXnvOsksAhqOglgscxj
            2+wWfRE1iCj4vsjodVePomersSTuW1zRJqybINg3AoGBAOQULTuYz9IXe+kb6VGI
            uePdXZJo7ASXilvWVD1NaUPGDccMnwRQeWyh5vaaijztbQGjwqABsZApKSt5UEff
            cckzUp6VOO2d1Q1knc3vorK4baKQIMxnATa6Y12M6qTFbMBrxGkI0XH4SW9OC5ZK
            x8L1+JBtNdv3+pVKXsl2BjApAoGBAJOcA0tIA4QNpvKCmoSnmMvndXg5IzygK3Yf
            3XHjwnLyxXwdhEiDKMw8kGyK2suAdt0APmDoSDegICBisPAo71rxNkO/Sb0d+wiw
            jcJ0sRXKINEW+fqMDwjUTRcq/IGtjWtOC4eTTe6/sBEthXUBmT4weQG8NzKjPs3t
            TlpAPmdHAoGBAOQzwekw7fmzhNMfd8G6Ceqw+XzQYywnN7lL/4zzp6KOr56PWyr+
            w5U5DBy2K3dJy8vspMzzj7RJKlRzfPQx1Vrw7fEB/UZk4gYSV7g49ZL7mLlO1SSI
            Y/nUSseDNmimUC6vYH3Pr2MWVOJ/hZWZ3WnTWkemlt+I/q0/RieKXmSX
            -----END RSA PRIVATE KEY-----            
        public: |
            ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1dGQ7QRJXS5W10lUo0gNQfwtKlr7SFtPBd7C53hGWHco/5Y5wHeN+ewcBICl8aynV9kh6bqrC3hsD+8NjTwBNTVjkNa1iUDPBpYRzQaYIKqcdGYxhBRskK530nEh+duuQjVnVdfGRyskPz8u0JKpM/wEzvNqH5BiuAs1V/YYGhxyT7aFde48HzX1H9KAqPwwevGWBU2Spag7V6MJQEgY1oa86zpHe+qCrQ4JI5HVEMWjQvAzBDcfAkCPJe2Xq2y5V6tf9D/t//gT22lAjuI5JIhkB0+aO9WpMPIml1nuQocYYOVeNwFSnJ1GVHSKPJQ9lQzOh7nQcmDoMRH0G1RIf

The configuration has 2 top-level fields: ground-control and mars. Underneath these fields are configurations for Ground Control and Mars respectively, formatted according to the respective configuration specs of these systems.

facility-install generates a Ground Control config (ground-control.yml) and a Mars config (mars-config.yml) using values set in user-config.yml. In cases where a value is not set in user-config.yml, facility-install will generate a value or use a reasonable default. For example, facility-install will always configure a Ground Control subnet of “10.0.0.0/16” if none is specified, and if TLS keying information is not provided for the Mars apiserver and foundry services, facility-install will randomly generate them.

Notable points from this configuration include:

  • The management search domain is set to phobos.example.com
  • The management subnet is 10.0.0.0/16
  • The management dynamic subnet for infrapods is 10.0.128.0/17
  • Ground control will forward DNS requests on the management network to 8.8.8.8
  • Ground control will serve static files for mars-install, canopy, and canopy.arm5
    • Services on the testbed can download these via http://ground-control/{mars-install,canopy,canopy.arm5}
  • The default registry for Mars services in registry.gitlab.com and the default tag is v1.0.3
    • This means that for Mars services that don’t have entries in this config, the system will default to using registry.gitlab.com/v1.0.3
  • The foundry image should be pulled from registry.gitlab.com/mergetb/tech/foundry/foundry:v1.1.0
  • The API endpoint for the Merge portal is grpc.mergetb.example.net:443
    • The user that commissioned the facility on the Merge portal is murphy and their password on the portal is mergetb
  • Specific TLS keying material is provided for the apiserver and foundry services
    • As noted above, this is optional. We use this for the phobos facility install so that repeated runs of facility-install do not continuously generate new keying material
  • A user with username ops and password ops will be installed on each facility machine, and the provided public SSH key will be installed into the authorized keys on each machine

Download the Installer

curl -o ./facility-install -L https://gitlab.com/mergetb/facility/install/-/jobs/artifacts/v1.0.1/raw/build/facility-install?job=make
chmod +x facility-install

Run the installer

Assuming the phobos model is present in your current directory as phobos.xir and the above configuration is present as user-config.yml:

./facility-install generate phobos.xir

This will create a directory called merge-install and populate it with the following installation contents:

$ ls -l merge-install/
total 92
-rw-r--r--. 1 murphy murphy   227 Jun  8 22:43 apiserver-key.pem
-rw-r--r--. 1 murphy murphy   680 Jun  8 22:43 apiserver.pem
-rw-r--r--. 1 murphy murphy  3602 Jun  8 22:43 cumulus-infra-ztp.sh
-rw-r--r--. 1 murphy murphy  3602 Jun  8 22:43 cumulus-xp-ztp.sh
-rw-r--r--. 1 murphy murphy 11495 Jun  8 22:43 emu.ign
-rw-r--r--. 1 murphy murphy   227 Jun  8 22:43 foundry-key.pem
-rw-r--r--. 1 murphy murphy   611 Jun  8 22:43 foundry.pem
-rw-r--r--. 1 murphy murphy  2286 Jun  8 22:43 ground-control.yml
-rw-r--r--. 1 murphy murphy 22877 Jun  8 22:43 ifr.ign
-rw-r--r--. 1 murphy murphy 10306 Jun  8 22:43 mars-config.yml
-rw-r--r--. 1 murphy murphy  1679 Jun  8 22:43 ops_rsa
-rw-r--r--. 1 murphy murphy   381 Jun  8 22:43 ops_rsa.pub

Installing the Facility

With merge-install/ contents in hand, we can now install Merge on our testbed

Setting Up Ground Control

All testbeds must have at least one operations server running Ground Control. For a complete example of how to set up an operations server, refer to the Phobos ops server ansible setup script. This example assumes an operations server running Centos Stream 9, so you may need to make minor updates to support other OS distributions.

Lines 67-120 in particular setup Ground Control to manage the installation:

    - name: setup ignition and ztp dirs
      file:
        path: /var/ground-control/{{item}}
        state: directory
      loop:
        - ignition
        - ztp

    - name: copy ignition configs
      copy:
        src: merge-install/{{item}}
        dest: /var/ground-control/ignition/{{item}}
      loop:
        - ifr.ign
        - emu.ign

    - name: copy ztp files
      copy:
        src: merge-install/{{item}}
        dest: /var/ground-control/ztp/{{item}}
      loop:
        - cumulus-xp-ztp.sh
        - cumulus-infra-ztp.sh

    - name: copy ground-control config
      copy:
        src: merge-install/ground-control.yml
        dest: /etc/ground-control.yml

    - name: fetch ground control
      get_url:
        url: https://gitlab.com/mergetb/ops/ground-control/-/jobs/artifacts/{{gc_ref}}/raw/build/ground-control?job=make
        dest: /usr/bin/ground-control
        mode: 0755
      vars:
        gc_ref: bjk-dev

    - name: create ground control service
      copy:
        src: ground-control.service
        dest: /lib/systemd/system/ground-control.service

    - name: setup ground control
      shell: /usr/bin/ground-control setup -n eth0 -s {{ fcos_stream }} -c {{ fcos_version }}
      vars:
        fcos_stream: stable
        fcos_version: 35.20220424.3.0

    - name: enable and start ground control service
      systemd:
        name: ground-control
        enabled: true
        state: restarted
        daemon_reload: yes

Network Boot Configuration

At this point you will need to configure your servers and switches to network boot over the management network. Once they do so, they will automatically download the Ignition or ZTP scripts and install without any need for human operator intervention.

The exact process to configure network booting varies based on your hardware models, BIOS versions, etc. We note that Ground Control supports machines using either UEFI or Legacy PXE boot protocols.


Last modified June 23, 2022: misc updates (5d70d91)