Migration from 3.0 to 3.1 failing (column iface.node_id does not exist)

Hello.

I’m encountering a problem when upgrading to maas 3.1 from 3.0. A DB migration task is failing and I’m really not sure what to do.
Thanks for any assistance!

snap refresh --channel=3.1 maas
2023-04-12T22:33:58+03:00 INFO Waiting for "snap.maas.supervisor.service" to stop.
error: cannot perform the following tasks:
- Run post-refresh hook of "maas" snap if present (run hook "post-refresh":
-----
Operations to perform:
  Apply all migrations: auth, contenttypes, maasserver, metadataserver, piston3, sessions, sites
Running migrations:
  No migrations to apply.
Traceback (most recent call last):
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.UndefinedColumn: column iface.node_id does not exist
LINE 39: JOIN maasserver_node node ON node.id = iface.node_id
                                                ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/snap/maas/26273/bin/maas-region", line 8, in <module>
    sys.exit(run())
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/region_script.py", line 77, in run
    run_django(is_snap, is_devenv)
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/region_script.py", line 66, in run_django
    management.execute_from_command_line()
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py", line 130, in handle
    self._perform_view_installation(database)
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/management/commands/dbupgrade.py", line 97, in _perform_view_installation
    dbviews.register_all_views()
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/utils/orm.py", line 756, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/utils/orm.py", line 559, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/dbviews.py", line 387, in register_all_views
    _register_view(view_name, view_sql)
  File "/snap/maas/26273/lib/python3.8/site-packages/maasserver/dbviews.py", line 38, in _register_view
    cursor.execute(view_sql)
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/snap/maas/26273/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: column iface.node_id does not exist
LINE 39: JOIN maasserver_node node ON node.id = iface.node_id
                                                ^

Traceback (most recent call last):
  File "/snap/maas/26273/bin/maas", line 8, in <module>
    sys.exit(main())
  File "/snap/maas/26273/lib/python3.8/site-packages/maascli/__init__.py", line 39, in main
    options.execute(options)
  File "/snap/maas/26273/lib/python3.8/site-packages/maascli/snap.py", line 448, in __call__
    raise exc
  File "/snap/maas/26273/lib/python3.8/site-packages/maascli/snap.py", line 445, in __call__
    self.handle(options)
  File "/snap/maas/26273/lib/python3.8/site-packages/maascli/snap.py", line 939, in handle
    sys.exit(migrate_db())
  File "/snap/maas/26273/lib/python3.8/site-packages/maascli/snap.py", line 353, in migrate_db
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/snap/maas/26273/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1.
-----)
1 Like

It feels like database migration went sideways and slammed against the bulkhead while upgrading MAAS from 3.0 to 3.1. The migration script is trying to access a column node_id on the iface table that doesn’t exist, but I can’t tell if it really doesn’t exist, or something got butchered during the migration process.

Hopefully, you have a complete backup of your MAAS database. If so, did you try reverting to the 3.0 version of MAAS (snap revert maas)? If you revert, does MAAS work right?

And if that works right, could you try the migration again (assuming you’re not already trapped in this loop) to see if something just sneezed during the process?