How to back up MAAS

[billwear]

Errors or typos? Topics missing? Hard to read? Let us know.

MAAS uses standard command-line utilities to keep backups simple and familiar.

Clean reset backup

This procedure does a clean reset backup using pg_dumpall. MAAS and PostgreSQL are fully overwritten on restore. You can also do a limited back and restore if other services are sharing your PostgreSQL database.

Generally, using pg_dumpall for a database dump doesn’t require stopping PostgreSQL. It works with live databases, ensuring consistent backups and minimal service disruption. The PostgreSQL transactional model guarantees dump accuracy. You should be able to keep PostgreSQL running – skipping the optional steps below – but you should stop MAAS first to avoid conflicts during the dump. If you’re concerned at all, just use the optional steps to stop and restart PostgreSQL.

Please note that the order of steps is important.

Clean Snap backup

1. Use this command, if needed, to confirm the local PostgreSQL service name:

   sudo systemctl list-units --type=service | grep postgres

2. Create an epoch-stamped directory on your external media to store the backup:

   cd <external-backup-media>/maas-backups/
   mkdir $(date +%s)

3. Backup the database to your external backup media:

   sudo -u postgres pg_dumpall -c > "<external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_dump.sql"

4. Stop MAAS:

   sudo snap stop maas

5. (Optional) Confirm that no other PostgreSQL sessions are active:

   sudo -u postgres psql -c  "SELECT * FROM pg_stat_activity"

6. (Optional) Stop PostgreSQL:

   sudo systemctl stop postgresql.service

7. Take a snapshot of MAAS and note the snapshot ID:

   sudo snap save maas

8. Verify the snapshot:

   sudo snap check-snapshot <snapshot-id>

9. Export the snapshot to your external backup media:

   sudo snap export-snapshot <snapshot-id> <external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_snapshot_<snapshot-id>

10. (Optional) Restart PostgreSQL:

   sudo systemctl start postgresql.service

11. (Optional) Verify that PostgreSQL is running:

   sudo systemctl status postgresql

12. Restart MAAS:

   sudo snap restart maas

Clean Snap restore

1. Stop MAAS to clear any cache:

   sudo snap stop maas 

2. Remove the current MAAS instance:

   sudo snap remove maas

3. Do a clean restore of the database:

   sudo -u postgres psql -f <external-backup-media>/maas-backups/<backup-timestamp>/<dump_epoch>_dump.sql postgres 

4. (Optional) Confirm that no other PostgreSQL sessions are active:

   sudo -u postgres psql -c  "SELECT * FROM pg_stat_activity"

5. (Optional) Stop PostgreSQL:

   sudo systemctl stop postgresql.service

6. Import the MAAS snapshot from external media:

   sudo snap import-snapshot <external-backup-media>/maas-backups/<backup-timestamp>/<snapshot-epoch>_maas_snapshot_<snapshot-id>

7. Restore the MAAS snapshot:

   sudo snap restore <snapshot-id>

8. (Optional) Restart PostgreSQL:

   sudo systemctl start postgresql.service

9. (Optional) Verify that PostgreSQL is running:

   sudo systemctl status postgresql

10. Restart MAAS:

   sudo snap restart maas

Clean package backup

1. Use this command, if needed, to confirm the local PostgreSQL service name:

   sudo systemctl list-units --type=service | grep postgres

2. Create an epoch-stamped directory on your external media to store the backup:

   cd <external-backup-media>/maas-backups/
   mkdir $(date +%s)

3. Backup the database to your external backup media:

   sudo -u postgres pg_dumpall -c > "<external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_dump.sql"

4. Stop MAAS services:

   sudo systemctl stop maas-dhcpd.service
   sudo systemctl stop maas-rackd.service
   sudo systemctl stop maas-regiond.service

5. (Optional) Confirm that no other PostgreSQL sessions are active:

   sudo -u postgres psql -c  "SELECT * FROM pg_stat_activity"

6. (Optional) Stop PostgreSQL:

   sudo systemctl stop postgresql.service

7. Write a verified archive of the key MAAS files to your external backup:

   sudo tar cvpzWf <external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_backup.tgz --exclude=/var/lib/maas/boot-resources /etc/maas /var/lib/maas

8. (Optional) Restart PostgreSQL:

   sudo systemctl start postgresql.service

9. (Optional) Verify that PostgreSQL is running:

   sudo systemctl status postgresql

10. Restart MAAS:

   sudo snap restart maas

Clean package restore

1. (Optional, but recommended if possible) Begin with a fresh Ubuntu install.

2. Make sure the required PostgreSQL version is installed.

3. Use this command, if needed, to confirm the local PostgreSQL service name:

   sudo systemctl list-units --type=service | grep postgres

4. Do a clean restore of the database:

   sudo -u postgres psql -f <external-backup-media>/maas-backups/<backup-timestamp>/<dump_epoch>_dump.sql postgres 

5. Install MAAS from packages.

6. Once fully operational, stop these MAAS services:

   sudo systemctl stop maas-dhcpd.service
   sudo systemctl stop maas-rackd.service
   sudo systemctl stop maas-regiond.service

7. (Optional) Confirm that no other PostgreSQL sessions are active:

   sudo -u postgres psql -c  "SELECT * FROM pg_stat_activity"

8. (Optional) Stop PostgreSQL:

   sudo systemctl stop postgresql.service

9. Untar the backup to a temporary directory:

   mkdir /tmp/maas_backup
   sudo tar xvzpf <external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_backup.tgz -C /tmp/maas_backup

10. Create epoch-stamped backup directories:

   backup_dir="/var/backups/maas_backup/$(date +%s)"
   sudo mkdir -p "$backup_dir"

11. Move current MAAS directories to backup:

   sudo mv /etc/maas "$backup_dir/etc_maas"
   sudo mv /var/lib/maas "$backup_dir/var_lib_maas"

12. Restore MAAS configuration from backup:

   sudo cp -prf /tmp/maas_backup/etc/maas /etc/
   sudo cp -prf /tmp/maas_backup/var/lib/maas /var/lib/

13. Do a clean restore of the database:

   sudo -u postgres psql -f <external-backup-media>/maas-backups/<backup-timestamp>/<dump_epoch>_dump.sql postgres 

14. (Optional) Restart PostgreSQL:

   sudo systemctl start postgresql.service

15. (Optional) Verify that PostgreSQL is running:

   sudo systemctl status postgresql

16. Restart MAAS services:

   sudo systemctl stop maas-dhcpd.service
   sudo systemctl stop maas-rackd.service
   sudo systemctl stop maas-regiond.service

Limited backup

For MAAS and PostgreSQL setups which are shared with other services, follow these backup and restore guidelines.

Limited Snap backup

1. Use this command, if needed, to confirm the local PostgreSQL service name:

   sudo systemctl list-units --type=service | grep postgres

2. Create an epoch-stamped directory on your external media to store the backup:

   cd <external-backup-media>/maas-backups/
   mkdir $(date +%s)

3. Identify the MAAS database that you want to back up:

   sudo -u postgres psql
   \l
   # identifying the MAAS database is up to you
   \q 

4. Backup the database to your external backup media:

   sudo -u postgres pg_dump [maas_database_name] > "<external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_only_backup.sql"

5. Stop MAAS:

   sudo snap stop maas

6. Take a snapshot of MAAS and note the snapshot ID:

   sudo snap save maas

8. Verify the snapshot:

   sudo snap check-snapshot <snapshot-id>

9. Export the snapshot to your external backup media:

   sudo snap export-snapshot <snapshot-id> <external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_snapshot_<snapshot-id>

10. Restart MAAS:

   sudo snap restart maas

Limited Snap restore

1. Stop MAAS to clear any cache:

   sudo snap stop maas 

2. Remove the current MAAS instance:

   sudo snap remove maas

3. Do a limited restore of the MAAS database only:

   sudo -u postgres psql [maas_database_name] < <external-backup-media>/maas-backups/<backup-timestamp>/<dump_epoch>_maas_only_backup.sql

4. Import the MAAS snapshot from external media:

   sudo snap import-snapshot <external-backup-media>/maas-backups/<backup-timestamp>/<snapshot-epoch>_maas_snapshot_<snapshot-id>

5. Restore the MAAS snapshot:

   sudo snap restore <snapshot-id>

6. Restart MAAS:

   sudo snap restart maas

Limited package backup

1. Use this command, if needed, to confirm the local PostgreSQL service name:

   sudo systemctl list-units --type=service | grep postgres

2. Create an epoch-stamped directory on your external media to store the backup:

   cd <external-backup-media>/maas-backups/
   mkdir $(date +%s)

3. Identify the MAAS database that you want to back up:

   sudo -u postgres psql
   \l
   # identifying the MAAS database is up to you
   \q 

4. Backup the database to your external backup media:

   sudo -u postgres pg_dump [maas_database_name] > "<external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_only_backup.sql"

5. Stop MAAS services:

   sudo systemctl stop maas-dhcpd.service
   sudo systemctl stop maas-rackd.service
   sudo systemctl stop maas-regiond.service

6. Write a verified archive of the key MAAS files to your external backup:

   sudo tar cvpzWf <external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_backup.tgz --exclude=/var/lib/maas/boot-resources /etc/maas /var/lib/maas

7. Restart MAAS:

   sudo snap restart maas

Limited package restore

1. Use this command, if needed, to confirm the local PostgreSQL service name:

   sudo systemctl list-units --type=service | grep postgres

2. Do a limited restore of the MAAS database only:

   sudo -u postgres psql [maas_database_name] < <external-backup-media>/maas-backups/<backup-timestamp>/<dump_epoch>_maas_only_backup.sql

3. Untar the MAAS backup to a temporary directory:

   mkdir /tmp/maas_backup
   sudo tar xvzpf <external-backup-media>/maas-backups/<backup-timestamp>/$(date +%s)_maas_backup.tgz -C /tmp/maas_backup

4. Create epoch-stamped backup directories:

   backup_dir="/var/backups/maas_backup/$(date +%s)"
   sudo mkdir -p "$backup_dir"

5. Move current MAAS directories to backup:

   sudo mv /etc/maas "$backup_dir/etc_maas"
   sudo mv /var/lib/maas "$backup_dir/var_lib_maas"

6. Remove the current MAAS installation

   sudo apt-get remove --purge maas
   sudo apt-get autoremove

6. Install MAAS from packages.

7. Once fully operational, stop these MAAS services:

   sudo systemctl stop maas-dhcpd.service
   sudo systemctl stop maas-rackd.service
   sudo systemctl stop maas-regiond.service

8. Restore MAAS configuration from backup:

   sudo cp -prf /tmp/maas_backup/etc/maas /etc/
   sudo cp -prf /tmp/maas_backup/var/lib/maas /var/lib/

9. Restart MAAS services:

   sudo systemctl stop maas-dhcpd.service
   sudo systemctl stop maas-rackd.service
   sudo systemctl stop maas-regiond.service

[marosg]

This document assumes MAAS is installed on Bionic and it is missing instructions how to backup on Focal.
Those maas-*.service which should be stopped are not present in MAAS 3.1 on Focal.

[billwear]

thanks, @marosg. making a note to test backup on focal and see what else might have changed.

[vultaire]

Found a bug.

I had a borked snap upgrade of MAAS - it was an upgrade from MAAS 2.9 to 3.0 on Bionic, and it got stuck on a migration which I presume isn’t totally compatible with Postgres 10. The migrations which did run look harmless enough, but as a side effect of the upgrade errors, views were dropped.

I found the reference to “sudo maas-region dbupgrade” on this page, however for the snap what I really needed was “sudo snap run --shell maas -c ‘maas-region dbupgrade’” (which I found from others internally). After running that, the views were restored.

TL;DR: The docs should note the appropriate “maas-region dbupgrade” command to use when running MAAS from snaps.

[jfleach]

The following lines in the documentation are numbered the same and are out of order. If you run the second command to restore the database, the server isn’t up because it was stopped in step #3.

#8. Restore the PostgreSQL dump with the following command:
sudo -u postgres psql -f dump.sql postgres

#8. Restart the PostgreSQL service:
sudo systemctl start postgresql.service

These should be swapped and re-numbered to #8 and #9.

[billwear]

@jfleach, edited; thank you for pointing out this error.

[camille.rodriguez]

The backup procedure of the deb package install mentions a snap command to start maas again. It should be systemctl commands instead.

[billwear]

@camille.rodriguez :: gaah! thanks for the close review! will fix.

[dilyn-corner]

I believe we are missing a step in “Do a clean restore of a snap install”;

@billwear

Restart MAAS:

MAAS isn’t reinstalled during this process, so this command should fail (restoring a snapshot doesn’t reinstall the snap, it merely restore $SNAP_COMMON and things). Add a step to reinstall MAAS (with perhaps a link-out to the install and upgrade pages?)

This likewise appears in “Do a minimal downtime restore of a MAAS snap install”.