How to secure MAAS (snap/2.9/CLI)

billwear · 9 September 2020 22:50

As a MAAS administrator, you have the critical responsibility of hardening your installation to help repudiate attacks and malicious actors. While there are too many variables to make meaningful suggestions for your deployed machines, there are a number of steps you can take to improve the overall security of your MASS setup. This article provides a few suggestions.

Six questions you may have:

How do I setup a firewall for MAAS?
How do I configure a TLS-terminating load balancer (and what’s the impact on my MAAS setup?)
How do I use logs to identify security issues?
How do I implement PostgreSQL security?
What else can I do to harden MAAS?
Whom do I contact for MAAS security consulting?

Use a firewall

Each rack controller must be able to initiate TCP connections on the following ports:

Port(s)	Description
`5240`	HTTP communication with each region controller. Note that port `80` is typically used in high-availability environments.
`5241` - `5247`	Reserved for internal MAAS services.
`5248`	Reserved for rack HTTP communication.
`5250` - `5270`	Reserved for region workers (RPC).

Consider setting your firewall on your rack and region controllers to disallow communication on all ports except those used by MAAS. For example, assuming you have installed ufw, you could execute:

sudo ufw enable
sudo ufw default deny incoming

You could then follow that with commands similar to these:

sudo ufw allow 5240
sudo ufw allow 5248
sudo ufw allow 5241:5247/tcp
sudo ufw allow 5241:5247/udp
sudo ufw allow 5250:5270/tcp
sudo ufw allow 5250:5270/udp

Recognise that your particular configuration and version may vary, so consult the appropriate firewall manual pages for your specific MAAS host system.

Configure a TLS-terminating load balancer

One of the best steps you can take to improve both security and availability of your MAAS installation is to install TLS-terminating load balancer. For MAAS, we recommend using HAProxy . This section explains how to set one up.

What is a TLS-terminated load balancer?

In the context of MAAS, a load balancer distributes the incoming Web UI and API requests across multiple region controllers. This reduces both load on MAAS and wait times for user requests. Typically, this is known as a high-availability (HA) configuration, although there are two other HA configurations that can be enabled for MAAS: one for BMC access (for powering on machines), and one for DHCP, which enables primary and secondary DHCP instances that manage the same VLAN.

A TLS-terminated load balancer is a load balancer that carries encryption and decryption as far down the pipe as possible, in this case, all the way to the load balancer itself. Note that, even though the “SSL” keyword may be used to enable operation, the term SSL is considered obsolete. Hence we choose to use the term “TLS” instead, referring to Transport Layer Security.

TLS is meant to provide privacy and data integrity between two or more applications. Privacy is provided by symmetric cryptography , based on a shared secret and uniquely-generated keys negotiated at the start of a session (during the TLS handshake ). Identity of each app can be authenticated, though this feature can be optional. Authenticity of messages is ensured by using message authentication codes to detect tampering.

PEM file

As a first step, you’ll need an SSL certificate (mysite.com.crt) with a key pair (mysite.com.key), combined into a PEM file:

cat mysite.com.crt mysite.com.key > mysite.com.pem
sudo cp mysite.com.pem /etc/ssl/private/

Depending upon your chosen certificate authority, you may also need to copy your CA root certificate and one or more intermediate CA certificates into the same PEM file.

Install and configure HA proxy

To install HAProxy, execute the following commands:

sudo apt-get update
sudo apt-get install haproxy

Next, edit /etc/haproxy/haproxy.cfg as follows. In the global section of the file, add this line:

maxconn <number of concurrent connections>

Be aware that there’s a balance between accepting many connections and overloading the API by trying to serve too many requests. Also, you should consider adding a line like this one to the same section:

tune.ssl.default-dh-param 2048

This parameter configures the maximum size of temporary DHE keys that are generated.

Next, in the defaults section, add the following lines under mode http:

option forwardfor
option http-server-close

The option forwardfor tells HAProxy to add X-Forward-For headers to each request. The http-server-close option reduces latency between HAProxy and your users by closing connections but maintaining a keep-alive.

Finally, you’ll set the frontend and backend parameters that define the connections between HAProxy and MAAS. For the frontend, you can set parameters this way:

frontend maas
    bind *:443 ssl crt /etc/ssl/private/mysite.com.pem
    reqadd X-Forwarded-Proto:\ https
    retries 3
    option redispatch
    default_backend maas

This stanza defines a frontend name maas; tells HAProxy to handle incoming traffic to port 443, providing SSL encryption, enabled by the certificates and keys you earlier concatenated into your PEM file; allows three retries and redispatch; and forwards these requests to maas as the backend server, which is defined something like this:

backend maas
    timeout server 90s
    balance source
    hash-type consistent
    server localhost localhost:5240 check
    server maas-api-1 <ip-address-of-a-region-controller>:5240 check
    server maas-api-2 <ip-address-of-another-region-controller>:5240 check

This stanza defines a backend server group named maas; sets consistent hashing, 90 second timeout, and balanced sources; sets the server address and port (localhost:5240); and engages two region controllers, named maas-api-1 and maas-api-2. Note that the “1” and “2” designations are completely arbitrary, as there is no sense of “primary” or “secondary” associated with this configuration.

Finally, restart the (already-running) load balancer so that these changes can take effect and the HAProxy will begin to forward requests:

sudo systemctl restart haproxy

Note that you can also enable HAProxy logging if desired. This logging is an optional feature of the HAProxy tool and is thus left to your discretion.

If desired, you can bypass the use of SSL in your HAProxy. Alternatively, you can set up TLS encryption on your MAAS web UI without implementing HAProxy.

Use logs to identify security issues

There are four categories of log files that you can use to help identify security issues:

firewall logs
Web server logs
MAAS log files
system log files

This section will offer some advice, as well as links to more detailed information on these categories.

Firewall logs

The Ubuntu firewall, UFW , is a front-end for iptables , so the UFW log output is very similar to what you’ll encounter in iptables itself. If you want to secure your MAAS installation, it’s very important to periodically review your UFW logs, found in /var/log/ufw*.

Learning to recognise issues in the UFW/iptables log is an art form, so we’re not going to give an extended tutorial here. Still, there are some key indicators that might help you spot security issues.

You might look for something probing a port that’s not supporting an application service. Attackers use port scanners to look for openings. You might see entries like these:

blocked incoming tcp connection request from 96.39.208.43:8240 to 128.17.92.85:6002
blocked incoming tcp connection request from 96.39.208.43:8240 to 128.17.92.85:6003
blocked incoming tcp connection request from 96.39.208.43:8240 to 128.17.92.85:6004

You can also compare attempts on unusual port numbers against well-known hacker tools . For instance, repeated attempts against port 12361 might mean that someone is attempting to attack with the Whack-a-mole exploit.

Also suspicious are repeated, unsuccessful access attempts, against the same port or service, from the same domain, IP address, or subnet. These attempts may be spread out in time (grep is your friend, here). For example, a group of login attempts that look like this may indicate that an attacker is trying to disguise port scans by switching IP addresses within a block of addresses available to them:

blocked incoming tcp connection request from 96.39.208.43:49343 to 64.242.119.18:31337
blocked incoming tcp connection request from 96.39.208.62:49343 to 64.242.119.18:31337
blocked incoming tcp connection request from 96.39.209.243:49343 to 64.242.119.18:31337
blocked incoming tcp connection request from 96.39.208.135:49343 to 64.242.119.18:31337
.
.
.
blocked incoming tcp connection request from 96.39.208.208:49343 to 64.242.119.18:31337

Watch out for suspicious messages or connections originating inside your network, which may indicate that you have a Trojan residing inside your UFW:

blocked outgoing tcp packet from 192.168.23.100:5240 to 96.38.231.18:443 as FIN:ACK received, but there is no active connection.

This message will usually be repeated a number of times, since Trojans are fairly persistent.

Look for source-routed packets, that is, packets with a source address internal to your network, but which originate from outside your network, indicating that someone is trying to spoof one of your internal addresses.

Review the IP addresses that are being rejected and dropped. Try to identify them with a ping -a <IP address>. Spoofed addresses won’t have an owner (and you can block them). Real addresses have a whois entry, so it’s possible you can contact the ISP to report and resolve this issue.

There are many other firewall log analysis techniques, and a number of good open-source and commercial log analysis programs. If you decide to analyse directly, though, you’re basically looking for blocked connection issues, connections to (potentially) open ports you’re not using, and suspicious-looking outbound connections.

Web server logs

Detecting malicious activity directed toward your Web server is best done with a log analysis tool. If you want to review the raw logs directly, you can look for them in two places:

/var/log/httpd/, /var/log/apache, or /var/log/apache2 (in the case of Apache), or
the path given in /etc/nginx/nginx.conf or given in your site configuration file, which itself is found at the path /etc/nginx/sites-available (in the case of nginx – look for the access_log directive).

Web server log analysis is also an art form, so we don’t plan to offer a comprehensive tutorial here, but here are few examples of things to look for in your logs:

multiple requests in less than one second, or some other appropriate time-frame.
multiple secure/login page accesses in a one-minute window, especially when they fail.
attempts to access non-existent pages using different paths or query parameters (e.g., 135.25.48.19:5250/maas/index.html).
look out for SQL injection attacks, for example:

84.55.41.57- - [14/Apr/2016:08:22:13 0100]
“GET /wordpress/wp-content/plugins/custom_plugin/check_user.php?userid=1
AND (SELECT 6810 FROM(SELECT COUNT(*),CONCAT(0x7171787671,(SELECT
(ELT(6810=6810,1))),0x71707a7871,FLOOR(RAND(0)*2))x FROM
INFORMATION_SCHEMA.CHARACTER_SETS GROUP BY x)a) HTTP/1.1” 200 166 “-” “Mozilla/5.0
(Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR
3.5.30729)”
attempts to run a Web shell, for instance:

192.168.1.102 29/Oct/2018:14:52:16 GET /b374k.php HTTP/1.1 200 2125 Mozilla/5.0

As mentioned above, there are a large number of Web server exploits, and this document does not propose to enumerate them all. If you want to ensure secure operation, though, it’s useful to familiarise yourself with these kinds of Web server attacks.

MAAS log files

Presently, your primary use of MAAS log files to improve security is to periodically check log files for login failures. You can check for this activity in the regiond.log file, found at /var/snap/maas/common/log/regiond.log. For reference, a valid login request looks like this entry:

Presently, your primary use of MAAS log files to improve security is to periodically check log files for login failures. You can check for this activity in the regiond.log file, found at /var/log/maas/regiond.log. For reference, a valid login request looks like this entry:

2020-03-31 21:17:56 regiond: [info] 10.132.172.1 GET /MAAS/accounts/login/ HTTP/1.1
--> 200 OK (referrer: http://10.132.172.231:5240/MAAS/r/; agent: Mozilla/5.0 (X11;
Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149
Safari/537.36)````

If a login fails due to bad input (username/password), the regiond log will contain an entry something like this one:

2020-03-31 21:18:08 regiond: [info] 10.132.172.1 POST /MAAS/accounts/login/ HTTP/1.1
--> 400 BAD_REQUEST (referrer: http://10.132.172.231:5240/MAAS/r/; agent:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/80.0.3987.149 Safari/537.36)````

An entry like this one would also be suspect, since it involves omitting username/password entries at the login prompt:

2020-03-31 21:18:45 regiond: [info] 10.132.172.1 POST /MAAS/accounts/login/ HTTP/1.1
--> 204 NO_CONTENT (referrer: http://10.132.172.231:5240/MAAS/r/; agent: Mozilla/5.0
(X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149
Safari/537.36)````

The key differentiator to distinguish problems from routine failures is the frequency. If you notice a lot of these last two entries in a given period of time, you may want to investigate more thoroughly.

System log files

You can also use the standard system logs to detect malicious activity, though this subject is largely beyond the scope of this document. As a simple example, consider using journalctl to detect and source an SSH brute force attack:

[root@maasserver ~]# journalctl -u sshd | grep "Failed password"
Jun 06 13:45:19 router sshd[2487]: Failed password for root from 234.19.184.6 port 42258 ssh2
Jun 06 13:45:24 router sshd[2487]: Failed password for root from 234.19.184.6 port 42258 ssh2
Jun 06 13:45:35 router sshd[2487]: Failed password for root from 234.19.184.6 port 38834 ssh2
Jun 06 13:45:48 router sshd[2487]: Failed password for root from 234.19.184.6 port 35444 ssh2

From here, you can either use whois to locate the attacker and work with the ISP to block them, or simply use your UFW firewall to block them directly.

As mentioned, this subject is far too complex for a detailed tutorial in this section. For more information, try the Ubuntu journalctl manpage or another, similar source .

Implement PostgreSQL security

PostgreSQL contains secrets, and should be encrypted for maximum protection. You should consider full disk encryption . Also recommended is TLS encryption between MAAS and PostgreSQL .

Other things you can do to harden MAAS

In addition to the items mentioned above, you should be aware of a few other points about hardening MAAS.

Good passwords

You should pick good passwords and store them securely (e.g. in a KeePassX password database). Perform user administration only via the web UI. Only share the maas and root user passwords with administrators.

File permissions

MAAS configuration files should be set to have permission 640: readable by logins belonging to the maas group and writeable only by the root user. Currently, the regiond.conf file contains the login credentials for the PostgreSQL database used by MAAS to keep track of all machines, networks, and configuration.

chmod 640 /var/snap/maas/current/rackd.conf
chmod 640 /var/snap/maas/current/regiond.conf

After:

-rw-r----- 1 root maas   90 Sep 27 14:13 rackd.conf
-rw-r----- 1 root maas  157 Sep 27 14:14 regiond.conf

About snap security

Since snaps are fully confined or “sandboxed,” they bring a lot of inherent security to the contained application. More detailed information can be found in [this snap blog ](https://snapcraft.io/blog/where-eagles-vs Snapnap-security-overview).

chmod 640 /etc/maas/rackd.conf
chmod 640 /etc/maas/regiond.conf

After:

-rw-r----- 1 root maas   90 Sep 27 14:13 rackd.conf
-rw-r----- 1 root maas  157 Sep 27 14:14 regiond.conf

Shared secrets

When you add a new rack or region controller, MAAS asks for a shared secret it will use to communicate with the rest of MAAS. This secret is also exposed in the web UI when you click the ‘Add rack controller’ button on the Controllers page. MAAS automatically generates this secret when your first region controller installed, and stores the secret in a plain text file. This file is automatically protected with the correct permissions, so there is no need for any action on your part.

Whom to contact about MAAS security consulting

If you need help implementing MAAS security, please contact us. We will be happy to assist you in arranging security consulting appropriate to your needs.