Downgrading/upgrading MAAS fails with alembic.script.revision.ResolutionError: No such revision or branch '0007'

[merlijn-sebrechts]

I was running MAAS 3.6 using the snap, and upgraded to a manually built snap based on 3.8.

Now i’m trying to downgrade to 3.7.

• Downgrading 3.8 → 3.7 didn’t work
• Downgrading to 3.6 works without any issues.
• Upgrading 3.6 → 3.7 fails with the same issue.

The issue is this:

- Run post-refresh hook of "maas" snap if present (run hook "post-refresh": 
-----
File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/script/revision.py", line 566, in get_revisions
    return tuple(
           ^^^^^^
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/script/revision.py", line 567, in <genexpr>
    self._revision_for_ident(rev_id, branch_label)
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/script/revision.py", line 638, in _revision_for_ident
    raise ResolutionError(
alembic.script.revision.ResolutionError: No such revision or branch '0007'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/snap/maas/40925/bin/maas-region", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/snap/maas/40925/lib/python3.12/site-packages/maasserver/region_script.py", line 81, in run
    run_django(is_snap, is_devenv)
  File "/snap/maas/40925/lib/python3.12/site-packages/maasserver/region_script.py", line 70, in run_django
    management.execute_from_command_line()
  File "/snap/maas/40925/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/snap/maas/40925/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/snap/maas/40925/usr/lib/python3/dist-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/snap/maas/40925/usr/lib/python3/dist-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/snap/maas/40925/lib/python3.12/site-packages/maasserver/management/commands/dbupgrade.py", line 279, in handle
    command.upgrade(alembic_cfg, "head")
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/snap/maas/40925/lib/python3.12/site-packages/maasservicelayer/db/alembic/env.py", line 116, in <module>
    run_migrations_online()
  File "/snap/maas/40925/lib/python3.12/site-packages/maasservicelayer/db/alembic/env.py", line 110, in run_migrations_online
    asyncio.run(run_async_migrations())
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/snap/maas/40925/lib/python3.12/site-packages/maasservicelayer/db/alembic/env.py", line 75, in run_async_migrations
    await connection.run_sync(do_run_migrations)
  File "/snap/maas/40925/usr/lib/python3/dist-packages/sqlalchemy/ext/asyncio/engine.py", line 886, in run_sync
    return await greenlet_spawn(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/snap/maas/40925/usr/lib/python3/dist-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
    result = context.switch(value)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/snap/maas/40925/lib/python3.12/site-packages/maasservicelayer/db/alembic/env.py", line 58, in do_run_migrations
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/runtime/migration.py", line 615, in run_migrations
    for step in self._migrations_fn(heads, self):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/command.py", line 392, in upgrade
    return script._upgrade_revs(revision, rev)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/script/base.py", line 447, in _upgrade_revs
    with self._catch_revision_errors(
  File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/snap/maas/40925/usr/lib/python3/dist-packages/alembic/script/base.py", line 283, in _catch_revision_errors
    raise util.CommandError(resolution) from re
alembic.util.exc.CommandError: Can't locate revision identified by '0007'
Traceback (most recent call last):
  File "/snap/maas/40925/bin/maas", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/snap/maas/40925/lib/python3.12/site-packages/maascli/__init__.py", line 38, in main
    options.execute(options)
  File "/snap/maas/40925/lib/python3.12/site-packages/maascli/snap.py", line 433, in __call__
    raise exc
  File "/snap/maas/40925/lib/python3.12/site-packages/maascli/snap.py", line 430, in __call__
    self.handle(options)
  File "/snap/maas/40925/lib/python3.12/site-packages/maascli/snap.py", line 983, in handle
    sys.exit(migrate_db())
             ^^^^^^^^^^^^
  File "/snap/maas/40925/lib/python3.12/site-packages/maascli/snap.py", line 338, in migrate_db
    subprocess.check_call(
  File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/snap/maas/40925/bin/maas-region', 'dbupgrade']' returned non-zero exit status 1.
-----)

Did I break my setup by upgrading to 3.8? Or is this salvageable?

[r00ta]

I assume by 3.8 you mean latest/edge.

latest/edge is built on master and it’s the development branch where migrations might break. So yeah, you messed up your env

[r00ta]

Also, downgrade is not supported in any release (except point releases, where we do our best not to introduce db migrations)

[merlijn-sebrechts]

Yes, I upgraded to a snap I build myself from the main branch. That’s what I meant with 3.8.

I understand downgrading is not officially supported, but I’m a bit confused that upgrading from 3.6 to 3.7 now also fails. Is this because the downgrade broke something, or is there more going on?

[r00ta]

hard to say. What’s the error?

[merlijn-sebrechts]

alembic.script.revision.ResolutionError: No such revision or branch '0007'

My limited understanding says that this might be a broader issue.

[r00ta]

@merlijn-sebrechts

0007

3.7 does not have that migration at all. Your env might be more broken than what you think that migration is only in master. If you are rebuilding MAAS from the source make sure you actually make snap-tree-clean && make snap-clean && make snap-tree when you move from one branch to another

[merlijn-sebrechts]

The weird thing is, I’m getting this error simply when running

sudo snap refresh maas --channel=3.7/stable

I’m now on the regular 3.6/stable channel, and trying to move to the regular 3.7/stable channel. It’s not clear to me how snap could still have issues from a manually installed snap two revisions ago.

[r00ta]

if you are on 3.6, can you sudo -u postgres psql -d maasdb -c "\dt" --pset pager=0?

[merlijn-sebrechts]

@r00ta

sudo -u postgres psql -d maasdb -c “\dt” --pset pager=0

Note the name of our db is xnl, not maasdb.

sudo -u postgres psql -d xnl -c "\dt" --pset pager=0
                              List of relations
 Schema |                       Name                       | Type  |  Owner   
--------+--------------------------------------------------+-------+----------
 public | alembic_version                                  | table | xnladmin
 public | auth_group                                       | table | xnladmin
 public | auth_group_permissions                           | table | xnladmin
 public | auth_permission                                  | table | xnladmin
 public | auth_user                                        | table | xnladmin
 public | auth_user_groups                                 | table | xnladmin
 public | auth_user_user_permissions                       | table | xnladmin
 public | django_content_type                              | table | xnladmin
 public | django_migrations                                | table | xnladmin
 public | django_session                                   | table | xnladmin
 public | django_site                                      | table | xnladmin
 public | maasserver_agent                                 | table | xnladmin
 public | maasserver_blockdevice                           | table | xnladmin
 public | maasserver_bmc                                   | table | xnladmin
 public | maasserver_bmcroutablerackcontrollerrelationship | table | xnladmin
 public | maasserver_bootresource                          | table | xnladmin
 public | maasserver_bootresourcefile                      | table | xnladmin
 public | maasserver_bootresourcefilesync                  | table | xnladmin
 public | maasserver_bootresourceset                       | table | xnladmin
 public | maasserver_bootsource                            | table | xnladmin
 public | maasserver_bootsourcecache                       | table | xnladmin
 public | maasserver_bootsourceselection                   | table | xnladmin
 public | maasserver_bootstraptoken                        | table | xnladmin
 public | maasserver_cacheset                              | table | xnladmin
 public | maasserver_config                                | table | xnladmin
 public | maasserver_controllerinfo                        | table | xnladmin
 public | maasserver_defaultresource                       | table | xnladmin
 public | maasserver_dhcpsnippet                           | table | xnladmin
 public | maasserver_dnsdata                               | table | xnladmin
 public | maasserver_dnspublication                        | table | xnladmin
 public | maasserver_dnsresource                           | table | xnladmin
 public | maasserver_dnsresource_ip_addresses              | table | xnladmin
 public | maasserver_domain                                | table | xnladmin
 public | maasserver_event                                 | table | xnladmin
 public | maasserver_eventtype                             | table | xnladmin
 public | maasserver_fabric                                | table | xnladmin
 public | maasserver_filestorage                           | table | xnladmin
 public | maasserver_filesystem                            | table | xnladmin
 public | maasserver_filesystemgroup                       | table | xnladmin
 public | maasserver_forwarddnsserver                      | table | xnladmin
 public | maasserver_forwarddnsserver_domains              | table | xnladmin
 public | maasserver_globaldefault                         | table | xnladmin
 public | maasserver_interface                             | table | xnladmin
 public | maasserver_interface_ip_addresses                | table | xnladmin
 public | maasserver_interfacerelationship                 | table | xnladmin
 public | maasserver_iprange                               | table | xnladmin
 public | maasserver_largefile                             | table | xnladmin
 public | maasserver_licensekey                            | table | xnladmin
 public | maasserver_mdns                                  | table | xnladmin
 public | maasserver_neighbour                             | table | xnladmin
 public | maasserver_node                                  | table | xnladmin
 public | maasserver_node_tags                             | table | xnladmin
 public | maasserver_nodeconfig                            | table | xnladmin
 public | maasserver_nodedevice                            | table | xnladmin
 public | maasserver_nodedevicevpd                         | table | xnladmin
 public | maasserver_nodegrouptorackcontroller             | table | xnladmin
 public | maasserver_nodekey                               | table | xnladmin
 public | maasserver_nodemetadata                          | table | xnladmin
 public | maasserver_nodeuserdata                          | table | xnladmin
 public | maasserver_notification                          | table | xnladmin
 public | maasserver_notificationdismissal                 | table | xnladmin
 public | maasserver_numanode                              | table | xnladmin
 public | maasserver_numanodehugepages                     | table | xnladmin
 public | maasserver_oidc_provider                         | table | xnladmin
 public | maasserver_ownerdata                             | table | xnladmin
 public | maasserver_packagerepository                     | table | xnladmin
 public | maasserver_partition                             | table | xnladmin
 public | maasserver_partitiontable                        | table | xnladmin
 public | maasserver_physicalblockdevice                   | table | xnladmin
 public | maasserver_podhints                              | table | xnladmin
 public | maasserver_podhints_nodes                        | table | xnladmin
 public | maasserver_podstoragepool                        | table | xnladmin
 public | maasserver_rack                                  | table | xnladmin
 public | maasserver_rbaclastsync                          | table | xnladmin
 public | maasserver_rbacsync                              | table | xnladmin
 public | maasserver_rdns                                  | table | xnladmin
 public | maasserver_regioncontrollerprocess               | table | xnladmin
 public | maasserver_regioncontrollerprocessendpoint       | table | xnladmin
 public | maasserver_regionrackrpcconnection               | table | xnladmin
 public | maasserver_reservedip                            | table | xnladmin
 public | maasserver_resourcepool                          | table | xnladmin
 public | maasserver_rootkey                               | table | xnladmin
 public | maasserver_script                                | table | xnladmin
 public | maasserver_scriptresult                          | table | xnladmin
 public | maasserver_scriptset                             | table | xnladmin
 public | maasserver_secret                                | table | xnladmin
 public | maasserver_service                               | table | xnladmin
 public | maasserver_space                                 | table | xnladmin
 public | maasserver_sshkey                                | table | xnladmin
 public | maasserver_sslkey                                | table | xnladmin
 public | maasserver_staticipaddress                       | table | xnladmin
 public | maasserver_staticroute                           | table | xnladmin
 public | maasserver_subnet                                | table | xnladmin
 public | maasserver_tag                                   | table | xnladmin
 public | maasserver_template                              | table | xnladmin
 public | maasserver_userprofile                           | table | xnladmin
 public | maasserver_vaultsecret                           | table | xnladmin
 public | maasserver_versionedtextfile                     | table | xnladmin
 public | maasserver_virtualblockdevice                    | table | xnladmin
 public | maasserver_virtualmachine                        | table | xnladmin
 public | maasserver_virtualmachinedisk                    | table | xnladmin
 public | maasserver_virtualmachineinterface               | table | xnladmin
 public | maasserver_vlan                                  | table | xnladmin
 public | maasserver_vmcluster                             | table | xnladmin
 public | maasserver_zone                                  | table | xnladmin
 public | piston3_consumer                                 | table | xnladmin
 public | piston3_nonce                                    | table | xnladmin
 public | piston3_token                                    | table | xnladmin
(108 rows)

[r00ta]

@merlijn-sebrechts

alembic_version

I believe you tried to downgrade anyways your DB. If you had a clean 3.6 backup it would be impossible for you to have the table

alembic_version

[merlijn-sebrechts]

Correct, this is still the same 3.6 → 3.8 (self-compiled) → 3.6 server.

Would the best course of action be to go back to 3.8 and wait until MAAS catches up? I still have a backup from before the upgrade to 3.8, but then I’d lose a bunch of machines that have already been added.

[r00ta]

We had to revert some migrations in master and we still have to fix one of them, so no way 3.8 will self-heal your DB. The only way to recover your DB is to look into the migrations that were applied when you migrated and try to fix the tables manually reverting some changes and restoring the columns that were dropped with their values from the old backup.

If I were you, I would restore the 3.6 backup and redo the work

[merlijn-sebrechts]

Ok, thanks for your help!

Just curious, is there a less destructive way for me to test my patches in our testbed?

Would it be less dangerous to simply apply my patches to the latest stable release and switch back and forth between upstream stable and my patched snap?

[r00ta]

If you are developing patches for your “production” environment always patch the stable release you are using.

Instead if you want to contribute upstream or try out the master branch you should do it in a controlled/testing environment, never in production for the reasons I explained above