Since upgrading MAAS to 2.5.0, i’ve noticed that often machines will take forever to download both the initial kernel/initrd images when PXE booting as well as the root.tmp.img later in boot. This takes so long that the MAAS often times out the operation and marks the machine as failed.
The only fixes i’ve been able to determine for this are to either reboot the machine or… run tcpdump. As soon as I fire up a tcpdump session the image starts transferring at full speed! The fact that running tcpdump fixes the issue makes it rather tricky to work out what’s going on - i’ve not yet been able to mirror the port to another machine and capture MAAS’s traffic that way.
My MAAS machine itself is an ESXi VM running Ubuntu 18.04, it is a single machine running both the region and rack controller. Nothing of relevance seems to pop up in MAAS’s logs, journalctl or dmesg.
Right, so having been looking at this for weeks and getting nowhere I did a bit more investigation after posting and may have found a fix, but I don’t quite understand why it works yet.
Editing MAAS’s nginx config at /var/lib/maas/http/nginx.conf and disabling sendfile on; then restarting the maas-http service appears to have completely resolved this. Downloads of the kernel images, initramfs and squashfs all go at full speed (1Gbit/sec). I can also download from multiple hosts in parallel.
Before disabling this, I took a Wireshark capture on my desktop when doing the download, and a tcpdump from another non-MAAS machine when doing a download. Both showed an initial high throughput, which quickly slowed down to about 1Mbit/sec with huge numbers of Out-of-Order TCP segments, TCP retransmissions and other TCP errors.
The question now is why does disabling sendfile improve things? I did also try setting sendfile_max_chunk in nginx’s configuration to 512k with no change - so far only completely disabling sendfile fixes things. From what I understand of sendfile, it should improve performance as it means I/O can remain in the Kernel rather than having to transition to userspace.
Further host details if there’s potentially something related here;
Linux maas 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
nginx version: nginx/1.14.0 (Ubuntu)
ESXi VM - 2 vCPU, 4GB RAM, 1x NIC using VMXNET3, portgroup with VLAN.
I’m really puzzled as to why tcpdump/disabling sendfile fixes this, so i’m eager to hopefully find out what the root cause is! For now, i’ll keep sendfile disabled in MAAS’s nginx config.
