General Software Improvements (3rd June 2020 onwards)

[hamishmb]

More improvements:

• Start system software on boot on NAS box.
• Round CPU usage to 2 decimal places (sometimes still shows more but this is due to limitations of binary precision).
• Watch for /tmp/.shutdown, /tmp/.reboot and /tmp/.update files to trigger shutdown, reboot, and software update.
  • This is a convenience feature to make it easy for us to apply updates with SSH.
• Guard database queries on NAS box with try, except clauses too (sometimes can't connect to database first try, prevents a crash).
• Wait 60 seconds for NAS box to finish boot-up before fully starting up the river system software.
• Make pi wait up to 3 minutes for a system tick on startup.
  • This is because the NAS box takes ~2 minutes to boot, and then 1 more minute to settle, but the pis take only around 1 minute to boot.
  • Seems like a long time, but usually a tick is obtained in around 10 seconds if things are already up and running.
• Add support for applying software updates.
• Watch database records for prompts to shutdown, reboot, or apply updates too.
  • This is provision to allow the Staff GUI initiate an update later, in case we ever want that.
• Wait for all pis to download update (from NAS box) before rebooting the NAS box.
• Several complex tests performed to make sure this is all working (more to come)

NB: The "software updates" I reference here are river control software updates, not OS updates.

[TerryJC]

Make pi wait up to 3 minutes for a system tick on startup.

• This is because the NAS box takes ~2 minutes to boot, and then 1 more minute to settle, but the pis take only around 1 minute to boot.

In the case of SumpPi, what will the circulation pump be doing during this period? It may not be a problem, but if there's lots of water in the sump, the river could overtop it.

Having said that, we generally need someone on hand during SumpPi boot up anyway.

[hamishmb]

Good point Terry. Once we have the startup routines, I suppose each pi would be doing those during this time, but for the moment, it would mean the pumps are off until either a tick is obtained or 3 minutes pass.

I guess a temporary solution to that is have someone present during reboots, which as you say is what we do anyway at the moment.

[hamishmb]

More changes:

• Fully tested the semi-automated update procedure:
  • (with the NAS box and 3 virtual pis).
  • Requires database and NAS box to be installed (can't use it for the next few updates sadly).
• Add the ability to reboot or shutdown individual pis cleanly and easily using the database or by touching a file in /tmp.
• Add the ability to reboot or shutdown all pis (and the NAS box) cleanly and easily by using the database or by touching one file in /tmp on the NAS box.

I'm not quite sure where to put the documentation for this yet, so I'll just make us a new thread here in the Software section for now.

UPDATE: Documentation here: viewtopic.php?f=14&t=256

[hamishmb]

Changes to master branch:

• Add new probe limits for all hall effect probes.
• Add probe voltage debugging:
  • Enabled for 10 minutes from startup by specifying the -d flag, otherwise remains off as before.

[hamishmb]

Changes to engineer gui:

• Sort readings by date descending as well as tick and probe
  • This way, even if a reading is from the same probe and tick, the newer one will display closer to the top of the table.
• Use the visitoriface credentials, now that they are set up, instead of the temporary admin credentials.
• Set version to 1.0 (no further changes expected).

[hamishmb]

Note for the future (not sure where to put this, suggestions welcome):

We probably don't want to update the River System Pis beyond Debian Buster (10) without careful consideration. I have found after upgrading my desktop that it cannot connect to the NAS box's database because the mysqlclient library is too new and no longer supports the version of MySQL (5.1.x) that the database runs. I have previously attempted to upgrade the NAS box to use a newer database package, but was unsuccessful.

So to summarise, all upgrades within Debian Buster (10) should be okay, but upgrading past Buster is likely to cause us problems, unless:

• Debian Bullseye (11) uses an old enough version of libmysqlclient at release time/has one available.
• We can upgrade the database software on the NAS box (ideal solution).
• We can downgrade the mysqlclient library on the Pis (might be a security risk).
• We can use some kind of translation layer or compatibility mode (not aware of anything like this existing))

EDIT: Link: https://bugs.mysql.com/bug.php?id=90994

[hamishmb]

More changes:

• Use logiccoretools (translation layer) in more places instead of config.DBCONNECTION.
• Don't attempt to use sockets monitors on the NAS box (not needed, wasting CPU time).
• No longer stop the main loop if thee are no monitors running.
• Use new generic control logic (just update the record in SystemStatus table) if no other logic is available for each site in config.py.
• Avoid event duplication in EventLog table.
• Store readings to send to database later if the connection dies.
  • Please note that the tick may be wrong (or 0) if this happens.
  • The engineer GUI sorts by Tick, so these readings probably won't be visible (in top 50 results).
  • ^ However, the software will still use them when they arrive, as it's queries sort by record ID, not tick.

Pretty much just little bug fixes and documentation now - finishing touches

[TerryJC]

We probably don't want to update the River System Pis beyond Debian Buster (10) without careful consideration. I have found after upgrading my desktop that it cannot connect to the NAS box's database because the mysqlclient library is too new and no longer supports the version of MySQL (5.1.x) that the database runs. I have previously attempted to upgrade the NAS box to use a newer database package, but was unsuccessful.

This may not be too much of an issue going forward. Clearly 'if it ain't broke, don't fix it'. which has been the attitude that I've taken for all projects at WMT, including this one until we got Internet access through the VPN.

There isn't much in the River System Network that would interest the average hacker, apart from perhaps the vandal who thinks that it's fun to damage other peoples property. Obviously we don't want anything like that to happen, so we should keep an eye on the VPN software and ensure that the VPN Server is fully up to date on a fairly frequent basis. (Even if it does, it's not hard to restore the system, even if we have to go round with a bagful of SD Cards. We should also be able to keep the Webserver OS up to date, but the authors of nodogsplash seem to have a penchant for breaking changes, so we might wish to think twice before updating that.

[hamishmb]

I guess we'll have to cross that bridge when we come to it. I might well be able to find an alternative solution, or it might just not be a problem with Raspbian 11 Bullseye anyway. By the time it comes out (who knows when) we might not want to install further updates anyway if all is pretty much said and done.