I have a box which has been running for a few years now, with Trusty and Maas 1.9.5, which is used as the cloud for a Juju model. The Juju model was upgraded from 1.25 to 2.x (currently 2.7), and is working fine - however we wish to upgrade the host running Maas from Trusty and have hit a roadblock.
I’ve built a test environment, with a copy of the production db on Trusty - and when I tried upgrading it to Xenial the Maas upgrade fell over during the migration steps - I raised https://bugs.launchpad.net/maas/+bug/1880495 to track that.
I also tried installing a fresh Bionic and Focal box, with the restored Postgres db, and installing maas - those failed also, which was no surprise, but in different ways:
Focal, Maas 2.7 in snap, Postgres on the same host but not in the snap:
MAAS URL [default=http://10.231.8.145:5240/MAAS]:
Database host: 127.0.0.1
Database name: maasdb
Database user: maas
Database password: redacted
<snip>
Running migrations:
Applying contenttypes.0001_initial...Traceback (most recent call last):
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
psycopg2.ProgrammingError: relation "django_content_type" already exists
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/snap/maas/5229/bin/maas-region", line 11, in <module>
load_entry_point('maas==2.7.0', 'console_scripts', 'maas-region')()
File "/snap/maas/5229/lib/python3.6/site-packages/maasserver/region_script.py", line 76, in run
run_django(is_snap, is_devenv)
File "/snap/maas/5229/lib/python3.6/site-packages/maasserver/region_script.py", line 65, in run_django
management.execute_from_command_line()
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/snap/maas/5229/lib/python3.6/site-packages/maasserver/management/commands/dbupgrade.py", line 117, in handle
call_command("migrate", interactive=False)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 131, in call_command
return command.execute(*args, **defaults)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 204, in handle
fake_initial=fake_initial,
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/migrations/operations/models.py", line 97, in database_forwards
schema_editor.create_model(model)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 303, in create_model
self.execute(sql, params or None)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/backends/base/schema.py", line 120, in execute
cursor.execute(sql, params)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/snap/maas/5229/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 62, in execute
return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "django_content_type" already exists
On Bionic:
Setting up maas-region-controller (2.4.2-7034-g2f5deb8b8-0ubuntu1) ...
dbconfig-common: writing config to /etc/dbconfig-common/maas-region-controller.conf
Creating config file /etc/dbconfig-common/maas-region-controller.conf with new version
creating postgres user maas: already exists.
resetting password: success.
creating database maasdb: already exists.
dbconfig-common: flushing administrative password
Operations to perform:
Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running migrations:
Applying contenttypes.0001_initial... FAKED
Applying auth.0001_initial... FAKED
Applying auth.0002_auto_20151119_1629... OK
Applying auth.0003_django_1_11_update... OK
Applying auth.0004_user_email_allow_null... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying piston3.0001_initial... FAKED
Applying maasserver.0001_initial... FAKED
Applying metadataserver.0001_initial... FAKED
Applying maasserver.0002_remove_candidate_name_model... OK
Applying maasserver.0003_add_node_type_to_node... OK
Applying maasserver.0004_migrate_installable_to_node_type...Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
psycopg2.DataError: malformed array literal: ""
LINE 1: UPDATE "maasserver_node" SET "routers" = '', "node_type" = 1...
^
DETAIL: Array value must start with "{" or dimension information.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/lib/python3/dist-packages/maasserver/__main__.py", line 9, in <module>
raise SystemExit(execute_from_command_line())
File "/usr/lib/python3/dist-packages/maasserver/__init__.py", line 56, in execute_from_command_line
management.execute_from_command_line()
File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/usr/lib/python3/dist-packages/maasserver/management/commands/dbupgrade.py", line 426, in handle
fake_initial=self._south_was_performed(database))
File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 131, in call_command
return command.execute(*args, **defaults)
File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 204, in handle
fake_initial=fake_initial,
File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/lib/python3/dist-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
self.code(from_state.apps, schema_editor)
File "/usr/lib/python3/dist-packages/maasserver/migrations/builtin/maasserver/0004_migrate_installable_to_node_type.py", line 18, in con
vert_installable_to_node_type
node.save(update_fields=['node_type'])
File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 214, in save
return super(CleanSave, self).save(*args, **kwargs)
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 808, in save
force_update=force_update, update_fields=update_fields)
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 838, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line 278, in _save_table
update_fields=update_fields)
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 905, in _save_table
forced_update)
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 955, in _do_update
return filtered._update(values) > 0
File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 664, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1204, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
raise original_exception
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 889, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.DataError: malformed array literal: ""
LINE 1: UPDATE "maasserver_node" SET "routers" = '', "node_type" = 1...
^
DETAIL: Array value must start with "{" or dimension information.
dpkg: error processing package maas-region-controller (--configure):
installed maas-region-controller package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of maas:
maas depends on maas-region-controller (= 2.4.2-7034-g2f5deb8b8-0ubuntu1); however:
Package maas-region-controller is not configured yet.
dpkg: error processing package maas (--configure):
dependency problems - leaving unconfigured
Processing triggers for rsyslog (8.32.0-1ubuntu4) ...
No apport report written because the error message indicates its a followup error from a previous failure.
Errors were encountered while pr
ocessing:
maas-region-controller
maas
E: Sub-process /usr/bin/dpkg returned an error code (1)
So, after all that, my question: What’s the preferred, safe, way to upgrade from 1.9.5 on Trusty to the current version? Is there a way to troubleshoot these migrations to help get them working for this db - maybe a checklist of things to look at in the db that’s causing these failures?