I added a new rack controller to my maas environment, but the images will not sync to the rack controller.
The events tab on the rack controller shows the following message
Rack import error - Unable to import boot images: (‘Connection broken: IncompleteRead(0 bytes read)’, IncompleteRead(0 bytes read))
The sync is successful on other rack controllers.
Both region and rack controller are on version 2.6.1
Any guidance to help tracking down the cause of this error is appreciated
/var/log/maas/rackd.log
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:centos:amd64:generic:centos66: to_add=['20190701_01'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:centos:amd64:generic:centos70: to_add=['20190701_01'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:custom:amd64:generic:centos7_weekly: to_add=['20180920'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:custom:amd64:generic:clv_test1: to_add=['20181025'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:custom:amd64:generic:cory_fedora: to_add=['20180921'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:custom:amd64:generic:esxi: to_add=['20180920'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:grub-efi-signed:amd64:generic:uefi: to_add=['20190404.0'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:grub-efi:arm64:generic:uefi: to_add=['20190404.0'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:grub-ieee1275:ppc64el:generic:open-firmware: to_add=['20190506.0'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:pxelinux:i386:generic:pxe: to_add=['20180807.0'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-16.04-lowlatency:xenial: to_add=['20191024'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-16.04:xenial: to_add=['20191024'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-18.04-lowlatency:bionic: to_add=['20191029'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:ga-18.04:bionic: to_add=['20191029'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-edge:xenial: to_add=['20191024'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency-edge:xenial: to_add=['20191024'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency:xenial: to_add=['20191024'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04:xenial: to_add=['20191024'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-18.04-edge:bionic: to_add=['20191029'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-18.04-lowlatency-edge:bionic: to_add=['20191029'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-18.04-lowlatency:bionic: to_add=['20191029'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-18.04:bionic: to_add=['20191029'] to_remove=[]
2019-11-04 15:11:12 sstreams: [info] maas:v2:download/maas:boot:windows:amd64:generic:win2016: to_add=['20180606'] to_remove=[]
2019-11-04 15:11:13 sstreams: [info] maas:v2:download/maas:boot:centos:amd64:generic:centos66: to_add=['20190701_01'] to_remove=[]
2019-11-04 15:11:25 sstreams: [info] maas:v2:download/maas:boot:centos:amd64:generic:centos70: to_add=['20190701_01'] to_remove=[]
2019-11-04 15:11:47 sstreams: [info] maas:v2:download/maas:boot:custom:amd64:generic:centos7_weekly: to_add=['20180920'] to_remove=[]
2019-11-04 15:11:47 provisioningserver.rackdservices.image_download_service: [critical] Downloading images failed.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback
self._startRunCallbacks(fail)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
self._runCallbacks()
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult
_inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/python3/dist-packages/provisioningserver/rackdservices/image_download_service.py", line 124, in maybe_start_download
yield self._start_download()
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/python3/dist-packages/provisioningserver/rackdservices/image_download_service.py", line 115, in _start_download
get_proxy_url("http"), get_proxy_url("https"))
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/python3/dist-packages/provisioningserver/rpc/boot_images.py", line 158, in _import_boot_images
yield deferToThread(_run_import, sources, maas_url, **proxies)
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 234, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/rpc/boot_images.py", line 126, in _run_import
imported = boot_resources.import_images(sources)
File "/usr/lib/python3/dist-packages/provisioningserver/import_images/boot_resources.py", line 197, in import_images
sources, storage, product_mapping)
File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 337, in download_all_boot_resources
keyring_file=source.get('keyring')),
File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 283, in download_boot_resources
writer.sync(reader, rpath)
File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
return self.sync_index(reader, path, data, content)
File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
self.sync(reader, path=epath)
File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
return self.sync_products(reader, path, data, content)
File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 341, in sync_products
self.insert_item(item, src, target, pgree, ipath_cs)
File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 223, in insert_item
self.store, filename, tag, checksums, size, contentsource)
File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 61, in insert_file
store.insert(tag, content_source, checksums, mutable=False, size=size)
File "/usr/lib/python3/dist-packages/simplestreams/objectstores/__init__.py", line 142, in insert
buf = reader.read(self.read_size)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line 275, in read
buf = self.cs.read(size)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line 136, in read
return self.fd.read(size)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line 401, in read
return self._read(size)
File "/usr/lib/python3/dist-packages/simplestreams/contentsource.py", line 431, in read_compressed
ret += next(self.r_iter)
File "/usr/lib/python3/dist-packages/requests/models.py", line 748, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
/var/log/maas/regiond.log
2019-11-04 09:11:12 regiond: [info] 10.58.69.88 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2019-11-04 09:11:12 regiond: [info] 10.58.69.88 GET /MAAS/images-stream/streams/v1/index.json HTTP/1.1 --> 200 OK (referrer: -; agent: python-simplestreams/0.1)
2019-11-04 09:11:12 regiond: [info] 10.58.69.88 GET /MAAS/images-stream/streams/v1/maas:v2:download.json HTTP/1.1 --> 200 OK (referrer: -; agent: python-simplestreams/0.1)
2019-11-04 09:11:12 regiond: [info] 10.58.69.88 GET /MAAS/images-stream/streams/v1/index.json HTTP/1.1 --> 200 OK (referrer: -; agent: python-simplestreams/0.1)
2019-11-04 09:11:13 regiond: [info] 10.58.69.88 GET /MAAS/images-stream/streams/v1/maas:v2:download.json HTTP/1.1 --> 200 OK (referrer: -; agent: python-simplestreams/0.1)
2019-11-04 09:11:25 regiond: [info] 10.65.156.28 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2019-11-04 09:11:25 regiond: [info] 10.58.69.88 GET /MAAS/images-stream/centos/amd64/generic/centos66/20190701_01/root-tgz HTTP/1.1 --> 200 OK (referrer: -; agent: python-simplestreams/0.1)
2019-11-04 09:11:42 regiond: [info] 10.58.69.88 GET /MAAS/rpc/ HTTP/1.1 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2019-11-04 09:11:47 regiond: [info] 10.58.69.88 GET /MAAS/images-stream/centos/amd64/generic/centos70/20190701_01/root-tgz HTTP/1.1 --> 200 OK (referrer: -; agent: python-simplestreams/0.1)
2019-11-04 09:11:47 -: [critical] WSGI application error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 87, in callWithContext
self.contexts.pop()
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 894, in callInContext
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 522, in run
self.started = True
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 498, in run
for elem in appIterator:
File "/usr/lib/python3/dist-packages/django/utils/text.py", line 329, in compress_sequence
for item in sequence:
File "/usr/lib/python3/dist-packages/maasserver/bootresources.py", line 182, in __next__
self._set_up()
File "/usr/lib/python3/dist-packages/maasserver/bootresources.py", line 176, in _set_up
'rb', connection=self._connection)
File "/usr/lib/python3/dist-packages/maasserver/fields.py", line 460, in open
self.oid, mode, 0, new_file)
psycopg2.OperationalError: ERROR: large object 123170 does not exist