Install an engine

keal-install turns a fresh Ubuntu 24.04 server into a Keal engine that is also its own node, so it runs VMs from the start. keal-install --uninstall takes it off again and leaves the server the way it was. That makes it easy to try Keal out on a spare machine.

Before you start

• Ubuntu 24.04, preferably freshly installed. The installer carries on on another release, but says to expect trouble.
• Root, through sudo.
• Virtualisation switched on in the BIOS, so there is a /dev/kvm. Without it Keal installs, but VMs will not start.
• A default route. The card it goes through is the engine's main card: the one you reach Keal on, and the one VMs get out through.
• A way to the Ubuntu package mirrors, for the packages it installs.
• Optionally, a second network card for the nodes, with nothing else on it.

Installing

From a published release, on the server:

curl -fsSL https://registry.keal.sh/downloads/install-engine.sh | sudo bash
curl -fsSL https://registry.keal.sh/downloads/install-engine.sh | sudo bash -s -- --admin you   # with options

This fetches the latest release, checks its signature and checksum, and runs the keal-install inside it. KEAL_VERSION=v0.7.0 picks another release.

Or from your own machine, over ssh, with the command-line client:

keal engine install --check root@server     # check the server first; changes nothing
keal engine install root@server

This leaves keal signed in to the new engine.

The installer installs the tree it is run from, so a tarball works just as well, for example on a server with no way to reach the registry. Use a release downloaded by hand, or one made from a checkout:

git archive --prefix=keal/ HEAD | gzip > keal.tar.gz      # on a checkout
scp keal.tar.gz root@server:
tar xzf keal.tar.gz && sudo keal/support/keal-install      # on the server

Publishing a release

support/make-release builds a release from the checked-out commit, which must be clean and is versioned by its tag. A release is the engine tarball, the CLI for every platform, and their checksums, signed with the release key (~/.keal-release.key). The key's public half is support/keal-release.pub, and the install scripts carry a copy of it.

git tag v0.7.0
support/make-release --publish origo@registry.keal.sh:/var/www/keal-downloads

It copies the release into its own directory and changes latest last, so a download never gets a half-copied release. A commit with no tag of its own is published under a name like v0.7.0-3-gabc1234, but is never made the latest.

At the end it prints where Keal is and the password of its first administrator:

== Done
   Keal is at https://engine.example.com/  (or https://192.0.2.10/)
   Sign in as admin with the password  Xq3...
   It is shown this once - change it under Your settings.

Options

sudo keal/support/keal-install --help prints the same.

Running it again

Each step looks at what is there before changing anything. If the installer stops, it says what went wrong, and running it again after the fix carries on from there. Nothing that is already right is changed: the configuration, the database, the passwords and the certificate are all kept.

Everything it prints also goes to /var/log/keal-install.log.

What it does

The steps are numbered in its output as well.

1. Checks the machine: the release, KVM, the main card and the node card.
2. The Keal Registry: asks for an account, and with one a name and whether to link the engine (see A name and a certificate).
3. Packages: installs whatever is missing: the web server, MariaDB, Perl and Python, QEMU and libvirt, the network tools, rdiff-backup and Prometheus. It records which ones it installed (see Uninstalling).
4. The keal user: Keal runs as keal, with its home in /var/lib/keal. It has an ssh key, which it uses to reach its nodes, this engine included.
5. Keal: copies the release into /opt/keal and sets up its Python environment. Everything Keal is lives there, movepiston and the scripts that run as root (/opt/keal/sbin) included. The whole tree belongs to root and is read-only to the keal user, apart from its Python environment (.venv). That way the account Keal runs as cannot change anything that runs as root.
6. Directories: /etc/keal for the configuration, /var/log/keal for the logs, and /mnt/keal for storage: the VMs' disks, the images and the backups.
7. Configuration: writes /etc/keal/config.cfg with fresh passwords, and the secret Keal signs sign-ins with.
8. Database: creates the register in MariaDB, its database user, and the first administrator.
9. Web server: Apache serves Keal over HTTPS, with a self-signed certificate in /etc/keal/tls - or, with a registry account, the registry's certificate for the engine's name. The default site is switched off. With --nginx it is nginx, set up the same way; with --no-apache there is none, for one of your own. Either way it needs ports 80 and 443, and the installer stops at once if another program has them.
10. Virtualisation: starts libvirt.
11. This engine as its own node: sets up the node network and enters the engine in the register as a node.
12. Keal's services: keal-api (Keal itself) and movepiston (the engine reporting to itself as a node). With a registry account, and if you said yes, the engine is linked to the registry here.

A name and a certificate

With an account on the Keal Registry, the installer links the engine to the registry and gives it a name in DNS with a real certificate. It asks for the account before it installs anything, so a name that cannot be had is said at once:

== 2. The Keal Registry
   Keal Registry account (Enter to skip): you@example.com
   Password for you@example.com:
   signed in to https://registry.keal.sh as you@example.com
   Link this engine to the registry, owned by you@example.com? [Y/n]
   Zones you may use: uncloud.co (shared), example.com (owner)
   Name for this engine [kealbuild.uncloud.co]:
   The registry sees this engine as 193.200.44.101 (this machine's own: 10.3.1.22).
   Address for kealbuild.uncloud.co [193.200.44.101]:

• The name: any name in your own zones, or a name in a shared domain such as uncloud.co that nobody else has. The installer checks it with the registry first and asks again if it is taken or reserved.
• The address: what the record points at. The default is the address the registry sees the engine call from, which is its public one behind NAT. A private address works just as well: the certificate is issued with DNS validation, so the engine does not have to be reachable from the internet.
• The certificate: the registry gets it from Let's Encrypt and renews it. The installer waits up to three minutes for it. A daily job (/etc/cron.d/keal-tls) fetches it again once the registry has renewed it, and installs one that is still on its way after the install. It fetches it with an API key that reaches this zone alone, made for it and kept in /etc/keal/registry-dns.json.
• The link: on unless you say no, or give --no-link. It is made with the sign-in from the start, so neither the password nor a two-factor code is needed again. The password is never stored.
• No name: --no-dns-name links the engine without a name in DNS. It then keeps a self-signed certificate.

Without a terminal, say it with options instead, and give the password (and two-factor code) in KEAL_REGISTRY_PASSWORD (and KEAL_REGISTRY_CODE):

sudo KEAL_REGISTRY_PASSWORD=... keal/support/keal-install --registry-user you@example.com \
    --dns-name keal1.uncloud.co --link

With no terminal and no --registry-user, nothing is done with the registry. With --registry-user but no password in KEAL_REGISTRY_PASSWORD, the install stops rather than going on without it. With both, the engine is linked and named unless --no-link or --no-dns-name says otherwise.

--no-registry asks nothing, and the engine keeps a self-signed certificate.

Either way the installer asks the registry which address it sees the engine call from, and gives that address as the one to reach Keal by. It also puts it in the self-signed certificate. Behind NAT that is the public address rather than the machine's own.

The node network

The engine is 10.0.0.1 to its nodes and serves them DHCP, TFTP and NFS on that network. Which card that is matters, because Keal's DHCP must not answer on a network that has a DHCP server of its own.

• With one card, the address goes on a dummy interface, keal-nodes, that is used by this engine's own node alone.
• With two or more, the installer picks the first physical, wired card that is not the main one and has no address of its own. A card that already has an address is on some other network, so the installer says it is skipping it. Name it with --node-nic if you want it used anyway.

The choice is kept in /etc/keal/node-network and set up at every boot by keal-nodes-net.service, whatever manages the rest of the network.

Your own web server

By default the installer puts Apache in front of Keal and points it at a certificate. --nginx puts nginx there instead, with the configuration below. To use a web server you set up yourself, with your own certificate, install with --no-apache. Then there is no Apache and no self-signed certificate, and Keal listens on http://127.0.0.1:8000 for your web server to pass requests on to (--port for another port). The choice is kept (WEB_SERVER in /etc/keal/config.cfg: apache, nginx or none), so reinstalling or updating does not change it.

Whatever is in front has to pass on:

• X-Forwarded-For, always. Keal trusts a request from the engine itself that has no such header as coming from the engine's own services, such as Prometheus. A proxy that leaves it out makes every request it passes on look like one of those.
• X-Forwarded-Proto: https and the Host asked for, so the links Keal makes point back the way they came.
• WebSockets (the VM consoles and terminals), and event streams unbuffered (the UI's live updates).
• Uploads of any size (images).

For nginx, as --nginx sets it up (in /etc/nginx/sites-available/keal):

map $http_upgrade $connection_upgrade { default upgrade; '' close; }

server {
    listen 443 ssl;
    server_name keal.example.com;
    ssl_certificate     /etc/keal/tls/keal.crt;   # or your own
    ssl_certificate_key /etc/keal/tls/keal.key;

    client_max_body_size 0;
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Upgrade           $http_upgrade;
        proxy_set_header Connection        $connection_upgrade;
        proxy_buffering off;
        proxy_read_timeout 1h;
    }
}

A name and certificate from the registry work without Apache too. The certificate is installed in /etc/keal/tls and fetched again daily, as with Apache. Keal's own Apache or nginx is reloaded when it is renewed. For a web server of your own, set TLS_RELOAD in /etc/keal/config.cfg to what makes it take a renewed one, for example systemctl reload nginx.

keal engine install --no-apache cannot sign in to the new engine afterwards, as there is nothing in front of it yet. It says so, and keal login --url https://... does it once your web server is up.

Reaching Keal straight

For a quick test, --listen 0.0.0.0 has Keal listen on every address, so it can be reached at http://<the engine's address>:8000/ with nothing in front of it. The installer says the address at the end. Apache, or your own web server, still works as well.

Updating

An engine knows which Keal it runs, and asks the published releases which is the latest. When a newer one is out, administrators get a notification, and About Keal in the user menu shows the installed version and the latest, with an Update button. From the command line it is:

keal engine update --check     # which Keal, and whether there is a newer one
keal engine update             # update to the latest, and wait until it is back

• What it does: the engine fetches the release, checks its signature and checksum, and runs the release's installer over itself with --no-registry. The configuration, the data, the engine's name and its certificate all stay as they are.
• The restart: the update restarts keal-api and movepiston. VMs keep running, and the UI and keal engine update wait until the engine is back.
• Where it runs: as a systemd unit of its own, keal-engine-update, so the restart does not stop it. How it went is in /var/log/keal/update.json, and everything it did in /var/log/keal-install.log.
• Internet access: the engine needs to reach the published releases. One that cannot hears of no updates. To install a release by hand, see Installing. DOWNLOADS_URL in /etc/keal/config.cfg points the engine at another place releases are published.
• A git checkout: an engine that runs from one is not updated from a release, which would be copied over the working tree. Update it with git.

Uninstalling

sudo /opt/keal/support/keal-install --uninstall

It lists what it is about to remove: how many VMs and networks, how much storage, and which packages, if they are to go. Then it asks you to type uninstall, and anything else stops it with nothing changed.

Options

What it removes

In this order, so nothing brings back what has just gone:

1. The Keal Registry: the engine's name, and its link. The API key made for the certificate stays listed under API keys at the registry; delete it there.
2. Keal's services, keal-api and movepiston, stopped first.
3. Keal's VMs: stopped and removed from libvirt.
4. Keal's networks: their bridges, VLANs and DHCP servers.
5. Firewall rules: the engine's own and those of every VM network.
6. The node network: 10.0.0.1 and keal-nodes-net.service.
7. The system's own files, back as they were: - Keal's NFS exports; - Prometheus's configuration; - the Apache sites that were on before; - Keal's sudo rules.
8. The register: the database and its user, dumped first if you asked to keep the data.
9. Keal's files and user: /opt/keal, /etc/keal, /var/log/keal, /mnt/keal (unless kept) and the keal user with its home.
10. Packages, with --remove-packages.

VMs, networks and firewall rules that are not Keal's are left alone. So are the packages, unless --remove-packages is given. MariaDB, Apache, libvirt and Prometheus therefore stay installed, running without Keal.

The uninstall runs from a copy of itself in /tmp, because it removes the directory it was started from. Its output goes to the same log as the install, /var/log/keal-install.log.

What the installer remembers for it

The installer keeps a few notes in /var/lib/keal-install, so the uninstall puts back exactly what was there:

The uninstall removes this directory last. On an engine that was not set up with this installer there are no such notes, so the uninstall removes Keal but cannot put back what it changed. It also cannot remove packages there, because it does not know which ones Keal brought.

Installing again

After an uninstall, the installer starts from scratch: new passwords and a new administrator. --keep-data keeps the disks and the dump of the register, but a new install does not load the dump. To bring an engine back as it was, restore an engine backup instead (see Engine settings).