MAAS nginx module not properly forwarding 'X-Forwarded-Proto'

[miclosedi]

Hello.

I’ve recently integrated MAAS with TLS. I wasn’t able to use the already-available CLI command for enabling this (maas config-tls), because I also needed some additional control for enforcing rate limiting and providing security headers. So I decided to use nginx. Configuration below:

limit_req_zone $binary_remote_addr zone=maas_login_ratelimit:10m rate=10r/m;

server {
      listen 5443 ssl;
      server_name _;
      server_tokens off;

      ssl_certificate     /var/snap/maas/common/certs/maas.crt;
      ssl_certificate_key /var/snap/maas/common/certs/maas.key;

      ssl_protocols TLSv1.2 TLSv1.3;
      ssl_ciphers HIGH:!aNULL:!MD5;

      add_header Strict-Transport-Security "max-age=15552000; includeSubdomains" always;
      add_header X-Content-Type-Options    "nosniff" always;
      add_header Referrer-Policy           "strict-origin-when-cross-origin" always;
      add_header Content-Security-Policy   "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self'
  data:;" always;

      location /MAAS/accounts/login/ {
              limit_req zone=maas_login_ratelimit burst=3 nodelay;

              proxy_pass http://localhost:5240/MAAS/accounts/login/;
              proxy_set_header X-Forwarded-Host $http_host;
              proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
              include /etc/nginx/proxy_params;
      }

      location / {
          proxy_set_header X-Forwarded-Host $http_host;
          proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
          proxy_pass http://localhost:5240;
          include /etc/nginx/proxy_params;
      }

      location /MAAS/ws {
          proxy_pass http://localhost:5240/MAAS/ws;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
          proxy_set_header Connection "Upgrade";
      }
  }

I have an external service that makes use of MAAS API to deploy and release machines. The service is accessing port 5443, which forwards requests towards port 5240 (MAAS service), along with all the headers. However, the MAAS service nginx is not consuming the X-Forwarded-Proto header, and forces the protocol to HTTP, causing authentication to fail with code 401:

requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://10.5.4.220:5443/MAAS/api/2.0/machines/cx63b6/

After investigations, I’ve found the MAAS regiond.nginx.conf file at:

/var/snap/maas/current/http/regiond.nginx.conf

The fix was to change the X-Forwarded-Proto directive from $scheme to $http_x_forwarded_proto.

Before:

proxy_http_version 1.1;
proxy_buffering off;
proxy_read_timeout 900s; # to match the Twisted one
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;

After:

proxy_http_version 1.1;
proxy_buffering off;
proxy_read_timeout 900s; # to match the Twisted one
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header X-Forwarded-Host $http_host;

This works.
I just wanted to make sure whether this is a bug, am I doing something wrong or it’s intended to work this way. Using sudo snap restart maas resets the regiond.nginx.conf file, but this can be overcome with a simple bash script that changes the directive after restart. It’s kind of a workaround.

Thank you!
Edi

[troyanov]

Hello Edi,

I guess when I introduced this configuration (4 years ago) I was not considering such use-case and simply copy-pasted existing config used by rackd.

I don’t remember if there is any conditional logic dependant on X-Forwarded-Protobut I guess the main intent for using $scheme originally was to prevent malicious user from manually sending a header like X-Forwarded-Proto: https even if they are on a plain http connection.

Anyway, your concern is valid and it is worth submitting a bug report on Launchpad

I just wanted to make sure whether this is a bug, am I doing something wrong or it’s intended to work this way. Using sudo snap restart maas resets the regiond.nginx.conf file, but this can be overcome with a simple bash script that changes the directive after restart.

This is not a bug, but an expected behaviour. Nginx configuration file is templated by the regiond during startup, so you have to modify the template to make it persistent.

Thanks a lot for the provided workaround!

[miclosedi]

Hello.

Thanks a lot for your response.

I’ll submit a bug report today for the X-Forwarded-Proto directive.