GitLab CI (Continuous Integration) is now fixed. Patrick, if you want this in your branch, you can probably cherry-pick the commit.
The NAS box scripts have been improved to log output from the river control system in syslog.
Syslog is accessible only via the web interface on the NAS box. I'm not sure where the file is but I haven't had a thorough search yet.
Patrick's new Stage Pi software was deployed and seems to be working fine.
Unresolved issues:
Race condition / other anomaly with site-wide updater (avoid using for now).
Python/river control system software seems to periodically and randomly crash on the NAS box.
NAS box startup sequence is now excessively slow as it also checks the indexes when checking table integrity. A minor annoyance only, as the river system is operational during that time, but improvements can probably be made.
CI pipeline status added to README.md - easier to see current status that way.
Install updates on the WMT Pis to pre-emptively mitigate a network reliability issue.
Added more information on notifications to help avoid communication issues.
Unresolved issues:
Race condition / other anomaly with site-wide updater (avoid using for now).
Python/river control system software seems to periodically and randomly crash on the NAS box.
NAS box startup sequence is now excessively slow as it also checks the indexes when checking table integrity. A minor annoyance only, as the river system is operational during that time, but improvements can probably be made.
hamishmb wrote: ↑05/08/2020, 15:31
Note for the future (not sure where to put this, suggestions welcome):
We probably don't want to update the River System Pis beyond Debian Buster (10) without careful consideration. I have found after upgrading my desktop that it cannot connect to the NAS box's database because the mysqlclient library is too new and no longer supports the version of MySQL (5.1.x) that the database runs. I have previously attempted to upgrade the NAS box to use a newer database package, but was unsuccessful.
So to summarise, all upgrades within Debian Buster (10) should be okay, but upgrading past Buster is likely to cause us problems, unless:
Debian Bullseye (11) uses an old enough version of libmysqlclient at release time/has one available.
We can upgrade the database software on the NAS box (ideal solution).
We can downgrade the mysqlclient library on the Pis (might be a security risk).
We can use some kind of translation layer or compatibility mode (not aware of anything like this existing))
Raspbian 11 Bullseye is now out, and I have not yet confirmed whether this problem will exist. However, in order to keep the software versions on the pis more in sync with what the NAS box runs, I propose we keep the river system pis on Raspbian 10 Buster, to make maintenance easier. This also means we will not have to rebuild the SD cards for every river system pi.
For some reason I wasn't subscribed to this Topic.
I agree with you that we shouldn't mess with what is after all, a production system. It is important that all the Pis in the River System are kept up to date with security fixes etc, but a jump to a new Version would be difficult to manage without a major rework of the whole system.
During the Winter Maintenance period I intend to rework the Minster Bells and Music hardware because we have to remove it anyway to upgrade the network interface. That system doesn't connect to the NAS Box, so it probably wouldn't be a problem if I upgraded the OS of those two Pis, but there is probably little advantage in doing that until the Raspberry Pi Foundation stop doing routine security updates to Buster.
Similarly, I'm probably not going to do anything to the Tiny TVs because they aren't coming out. In the Railway Room, one of the systems is coming out, but none of those Pis are connected to the network, so as long as they still work I see no need to upgrade them.
Hmm, the security updates thing is a concern - I don't think older releases of Raspbian ever got an EOL date, I think they might have just immediately stopped getting updates. The VPN and Webserver will need to be updated though as they are internet-facing and protect the rest of the site-wide network.
Note: If we create a separate partition for the software and logs, we will probably need it to be at least a couple of hundred megabytes: the logs files from the last 11 months are taking up over 3 gigs uncompressed on each pi!
I'm going to change the software to compress old logs, which should help alleviate this problem, but still food for thought.
The database backup USB stick on the NAS box is also full. I'm emptying it now, but I have some suggestions which can help us avoid this problem reoccurring later. I will shortly make a new thread for planning my latest set of changes.