Debugging Commissioning and Testing Scripts

Development
1

Overview of how scripts are executed

MAAS runs commissioning and testing scripts in an ephemeral Ubuntu environment. When Ubuntu boots MAAS returns cloud-init user data which includes a tool which downloads and executes all selected commissioning and testing scripts.

The cloud-init script extracts the tool, maas-run-remote-scripts, to /tmp/user_data.sh*/bin and executes it. The tool then asks MAAS for a tar containing all scripts to be run and an index of the scripts to be run and assoicated metadata. This tar is downloaded from MAAS_URL/MAAS/metadata/2012-03-01/maas-scripts/ and extracted into /tmp/user_data.sh*.

maas-run-remote-scripts executes scripts in order of hardware type(CPU, Memory, Storage, Node). In each hardware type single threaded tests are run first, followed by instance(storage tests run on multiple drives at once), and finally any(run in parallel with any other test of the same hardware type).

Before each script is run all specified package are installed. If any package fails to install the script is marked failed and installation output is sent to MAAS for logging. Because packages are installed before executing the script a script may be written in any language as long as the interrupter. If the script contains a storage parameter MAAS will map the specified storage device to what is available on the system using the model and serial number if available. The script is then executed with the following environment variables added.

  • OUTPUT_COMBINED_PATH - Location of the combined output log path
  • OUTPUT_STDOUT_PATH - Location of STDOUT log path
  • OUTPUT_STDERR_PATH - Location of STDERR log path
  • RESULT_PATH - Location of where script results should be stored
  • DOWNLOAD_PATH - Location of where any url packages have been downloaded and extracted to
  • RUNTIME - How long the script has to run
  • HAS_STARTED - Whether or not the script has previously run(e.g a reboot has occurred)

While the script is running maas-run-remote-scripts pings the region every 2 minutes to indicate scripts are still running. maas-run-remote-scripts will also keep track of how long the current script has been running. If the script has run 5 minutes past its given runtime it will be killed.

Once the script finishes or times out the return code, all script output, and if written any YAML results are uploaded to MAAS.

maas-run-remote-scripts debug features

When run in a terminal all script output is printed.

Disable pinging and sending results to MAAS

# /tmp/user_data.sh.*/bin/maas-run-remote-scripts --no-send /tmp/user_data.sh.*

Execute currently downloaded scripts, do not redownload

# /tmp/user_data.sh.*/bin/maas-run-remote-scripts --no-download /tmp/user_data.sh.*

Manually executing scripts after download

  • All commissioning scripts are stored in /tmp/user_data.sh.*/scripts/commissioning
  • All testing scripts are stored in /tmp/user_data.sh.1dRUUY/scripts/testing

Logs

  • /tmp/user_data.sh*/out/
  • /var/log/cloud-init-output.log
  • /var/log/cloud-init.log
3 Likes