Semi-automated system updates, reboots, and poweroff using the NAS box and database

[hamishmb]

Just a thread to document how this system works, so we can use it after deployment and eventually put this in whichever document it's meant to go in. If any part of this explanation is unclear, let me know and I will update it.

Summary of feature

In the last week, I implemented the ability to trigger a system reboot, update, or shutdown for any pi (or the NAS box) through the database. I also enabled a way to do this using the commandline for convenience until we have the Staff GUI. This allows easy updating of the river system software, as well as clean reboots and shutdowns with proper river system software shutdown procedures.

Shutting down any one pi/the NAS box

Using the database

To use the database, edit the *Control table that matches the site_id for that device to read, for example:

Code: Select all

Record ID	Device ID	Device Status	Request	Locked By
1	SUMP	Locked	Shutdown	NAS
<... other records ...>

Look for the site ID for that device (eg could be "NAS" or "VALVE6" instead of "SUMP"). Case sensitivity for the Request column is not important.

Using the commandline

SSH in to that device and run:

Code: Select all

touch /tmp/.shutdown

After a few minutes, the river system software will stop and the device will power down safely.

Rebooting any one pi/the NAS box

Using the database

To use the database, edit the *Control table that matches the site_id for that device to read, for example:

Code: Select all

Record ID	Device ID	Device Status	Request	Locked By
1	SUMP	Locked	Reboot NAS
<... other records ...>

Look for the site ID for that device (eg could be "NAS" or "VALVE6" instead of "SUMP"). Case sensitivity for the Request column is not important.

Using the commandline

SSH in to that device and run:

Code: Select all

touch /tmp/.reboot

After a few minutes, the river system software will stop and the device will power down safely.

Shutting down the whole system

Using the database

To use the database, edit the NASControl table to read:

Code: Select all

Record ID	Device ID	Device Status	Request	Locked By
1	NAS	Locked	Shutdownall	NAS
<... other records ...>

Case sensitivity for the Request column is not important.

Using the commandline

SSH in to the NAS box (required your public key to be installed) and run:

Code: Select all

touch /tmp/.shutdownall

After a few minutes, the river system software will stop on the NAS box. Then, the NAS box will use the database to prompt all the other pis to power down. After they have powered down, the NAS box will power down. Wait at least 2-3 minutes before removing power from the NAS box.

Rebooting the whole system

Using the database

To use the database, edit the NASControl table to read:

Code: Select all

Record ID	Device ID	Device Status	Request	Locked By
1	NAS	Locked	Rebootall	NAS
<... other records ...>

Case sensitivity for the Request column is not important.

Using the commandline

SSH in to the NAS box (required your public key to be installed) and run:

Code: Select all

touch /tmp/.rebootall

After a few minutes, the river system software will stop on the NAS box. Then, the NAS box will use the database to prompt all the other pis to reboot. After they have started to reboot, the NAS box will reboot. The whole system will take up to around 5 minutes to come back online.

Updating the whole system

NOTE: This method does not allow you to update only a single pi - we don't want to do that during normal operation. Only an engineer should be updating individual pis, and they will have the know-how to do that without this convenience procudure.

Preparation

Create a gzipped tarball (.tar.gz file extension) containing the new software. This should unpack to create a "rivercontrolsystem" directory with the framework inside. Place this file at "/mnt/HD/HD_a2/rivercontrolsystem.tar.gz" on the NAS box (use scp or SSH in and use a USB stick to do this).

Using the database

To use the database, edit the NASControl table to read:

Code: Select all

Record ID	Device ID	Device Status	Request	Locked By
1	NAS	Locked	Update	NAS
<... other records ...>

Case sensitivity for the Request column is not important.

Using the commandline

SSH in to the NAS box (required your public key to be installed) and run:

Code: Select all

touch /tmp/.update

After a few minutes, the river system software will stop on the NAS box. Then, the NAS box will start hosting the update file on its webserver and use the database to prompt all the other pis to download and install the update. The pis will connect the the webserver, download the update, move the old software to "rivercontrolsystem.old" as a backup, and then extract the new software. Once the pis signal that they have downloaded the update and are rebooting, the NAS box will update itself the same way and then reboot. The whole system will take up to around 5 minutes to come back online.