Invalid Checksums Pulling Images from maas.io

Features
1

Here is an example of what I am seeing in the logs when I try to download 20.04 for amd64 arch. The same issue occurs with 22.04 as well.

Running maas 3.1 from PPA. I’ve used this exact setup numerous times and had no issues until the end of last week, roughly 9/15 or so. It is occuring on all of my currently running maas systems.

2022-09-20 13:18:54 stderr: [error] Exception in thread Thread-2:
2022-09-20 13:18:54 stderr: [error] Traceback (most recent call last):
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
2022-09-20 13:18:54 stderr: [error] self.run()
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 870, in run
2022-09-20 13:18:54 stderr: [error] self._target(*self._args, **self._kwargs)
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 795, in write_content_thread
2022-09-20 13:18:54 stderr: [error] if write_chunk():
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 756, in call_within_transaction
2022-09-20 13:18:54 stderr: [error] return func_outside_txn(*args, **kwargs)
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 559, in retrier
2022-09-20 13:18:54 stderr: [error] return func(*args, **kwargs)
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3.8/contextlib.py”, line 75, in inner
2022-09-20 13:18:54 stderr: [error] return func(*args, **kwds)
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 781, in write_chunk
2022-09-20 13:18:54 stderr: [error] buf = reader.read(self.read_size)
2022-09-20 13:18:54 stderr: [error] File “/usr/lib/python3/dist-packages/simplestreams/contentsource.py”, line 284, in read
2022-09-20 13:18:54 stderr: [error] raise checksum_util.invalid_checksum_for_reader(self)
2022-09-20 13:18:54 stderr: [error] simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://images.maas.io/ephemeral-v3/stable/focal/amd64/20220914/hwe-20.04/lowlatency/boot-kernel. Found
297721c359f4cd1b37e78f004e0862cff41646fe06cc1c8136cf302a89d8b670. Expected 17e36603a96a1033de98e0ae2334e6efe4a18b02bf7e0a411189092e4b305a90. read 906 bytes expected 11470944 bytes. (size 906 expected 11470944)
2022-09-20 13:19:02 maasserver.regiondservices.active_discovery: [info] Active network discovery: Active scanning is not enabled on any subnet. Skipping periodic scan.
2022-09-20 13:19:14 stderr: [error] Exception in thread Thread-5:
2022-09-20 13:19:14 stderr: [error] Traceback (most recent call last):
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
2022-09-20 13:19:14 stderr: [error] self.run()
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 870, in run
2022-09-20 13:19:14 stderr: [error] self._target(*self._args, **self._kwargs)
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 795, in write_content_thread
2022-09-20 13:19:14 stderr: [error] if write_chunk():
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 756, in call_within_transaction
2022-09-20 13:19:14 stderr: [error] return func_outside_txn(*args, **kwargs)
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 559, in retrier
2022-09-20 13:19:14 stderr: [error] return func(*args, **kwargs)
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3.8/contextlib.py”, line 75, in inner
2022-09-20 13:19:14 stderr: [error] return func(*args, **kwds)
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 781, in write_chunk
2022-09-20 13:19:14 stderr: [error] buf = reader.read(self.read_size)
2022-09-20 13:19:14 stderr: [error] File “/usr/lib/python3/dist-packages/simplestreams/contentsource.py”, line 284, in read
2022-09-20 13:19:14 stderr: [error] raise checksum_util.invalid_checksum_for_reader(self)
2022-09-20 13:19:14 stderr: [error] simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://images.maas.io/ephemeral-v3/stable/focal/amd64/20220914/hwe-20.04/generic/boot-kernel. Found 29
7721c359f4cd1b37e78f004e0862cff41646fe06cc1c8136cf302a89d8b670. Expected 8cecc7f0f135561881f6fd17a68a1c26a8db0019b4e63aceb8d1b1efd77a1b0f. read 906 bytes expected 11431744 bytes. (size 906 expected 11431744)
2022-09-20 13:19:21 stderr: [error] Exception in thread Thread-7:
2022-09-20 13:19:21 stderr: [error] Traceback (most recent call last):
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
2022-09-20 13:19:21 stderr: [error] self.run()
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 870, in run
2022-09-20 13:19:21 stderr: [error] self._target(*self._args, **self._kwargs)
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 795, in write_content_thread
2022-09-20 13:19:21 stderr: [error] if write_chunk():
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 756, in call_within_transaction
2022-09-20 13:19:21 stderr: [error] return func_outside_txn(*args, **kwargs)
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 559, in retrier
2022-09-20 13:19:21 stderr: [error] return func(*args, **kwargs)
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3.8/contextlib.py”, line 75, in inner
2022-09-20 13:19:21 stderr: [error] return func(*args, **kwds)
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 781, in write_chunk
2022-09-20 13:19:21 stderr: [error] buf = reader.read(self.read_size)
2022-09-20 13:19:21 stderr: [error] File “/usr/lib/python3/dist-packages/simplestreams/contentsource.py”, line 284, in read
2022-09-20 13:19:21 stderr: [error] raise checksum_util.invalid_checksum_for_reader(self)
2022-09-20 13:19:21 stderr: [error] simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://images.maas.io/ephemeral-v3/stable/focal/amd64/20220914/ga-20.04/lowlatency/boot-kernel. Found
297721c359f4cd1b37e78f004e0862cff41646fe06cc1c8136cf302a89d8b670. Expected 44267ad7edd52a0c6a5e5187d822c0df5f2cc5a7edb81522b5ac7186a902d89a. read 906 bytes expected 13717760 bytes. (size 906 expected 13717760)
2022-09-20 13:19:27 regiond: [info] 127.0.0.1 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2022-09-20 13:19:32 stderr: [error] Exception in thread Thread-10:
2022-09-20 13:19:32 stderr: [error] Traceback (most recent call last):
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
2022-09-20 13:19:32 stderr: [error] self.run()
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3.8/threading.py”, line 870, in run
2022-09-20 13:19:32 stderr: [error] self._target(*self._args, **self._kwargs)
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 795, in write_content_thread
2022-09-20 13:19:32 stderr: [error] if write_chunk():
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 756, in call_within_transaction
2022-09-20 13:19:32 stderr: [error] return func_outside_txn(*args, **kwargs)
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/utils/orm.py”, line 559, in retrier
2022-09-20 13:19:32 stderr: [error] return func(*args, **kwargs)
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3.8/contextlib.py”, line 75, in inner
2022-09-20 13:19:32 stderr: [error] return func(*args, **kwds)
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3/dist-packages/maasserver/bootresources.py”, line 781, in write_chunk
2022-09-20 13:19:32 stderr: [error] buf = reader.read(self.read_size)
2022-09-20 13:19:32 stderr: [error] File “/usr/lib/python3/dist-packages/simplestreams/contentsource.py”, line 284, in read
2022-09-20 13:19:32 stderr: [error] raise checksum_util.invalid_checksum_for_reader(self)
2022-09-20 13:19:32 stderr: [error] simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://images.maas.io/ephemeral-v3/stable/focal/amd64/20220914/ga-20.04/generic/boot-kernel. Found 297
721c359f4cd1b37e78f004e0862cff41646fe06cc1c8136cf302a89d8b670. Expected 059ae2f5af20cfc19c825105c61981e44dfda648f845a2612abac1e4a44c258f. read 906 bytes expected 13660416 bytes. (size 906 expected 13660416)

Any guidance would be greatly appreciated. Will continue to update here as I find more info!

2

Did you ever get any help with this? I am having the ame problem, and it is a huge blocker for me.

3

A work around could be to try a different kernel. The Lowlatency kernel for focal works fine.

4

Sorry for this late response. I found this to be related to our firewall blocking these downloads. In our specific case it was being blocked based on a file category that our firewall uses to inspect downloaded files and block based on certain extensions. Once the traffic was allowed via our firewall my issues disappeared. I removed the images and re-downloaded them many times, never saw the issue again.