General Software Improvements (3rd June 2020 onwards)

[hamishmb]

NB: NAS box scp is now working: viewtopic.php?p=4178#p4178

[PatrickW]

Don't update status and log event if the status is the same as last time.

Did you implement this in the DatabaseConnection class, or somewhere else? (I could look at the code, it's quicker to ask. )

Where I'm coming from is: Do I need to make sure the Stage Pi logic doesn't update status and log event if the status is the same as last time, or will it happen automatically?

[hamishmb]

This is in the DatabaseConnection class - you don't need to change anything

EDIT: Same goes for the device controlling thing - all automatically done and complexity hidden in DatabaseConnection.

[hamishmb]

This week's changes:

• Another related, but simple, Sump Pi crash was resolved. This only occurred when the database went offline and is unrelated to the other trouble this week. I have tested that there are no more related issues.

Most of my WMT time this week was spent trying to diagnose a potential memory issue on the NAS, and a potentially-related software crash with an unexplained cause. I am still investigating the cause of this issue and trying to understand exactly what happened.

For now, I'm going to monitor memory usage periodically with "free" and see if the readings shown in the engineer GUI still have new ticks (indicates that the river system software is still running on the NAS box). If anyone else could periodically log in to the VPN and check these things too that would be useful.

[hamishmb]

I have made the following change:

• Error handling when attempting to connect to the database has been improved (turned out not to be the cause of any issues, but good to have nevertheless)

I also identified the cause of the VALVE12 issue:

It seems Raspbian comes bundled with an old version of the database connection module (1.3.10 from Feb 2017), and using the latest one from pylint (2.0.1) fixes a potential hang when connecting/reconnecting. This was meant to have been fixed earlier (https://github.com/PyMySQL/PyMySQL/issues/338) but I guess further work was since done. I have tested this fix on VALVE12, but would also like to apply it to the other pis pre-emptively. This means the package from the repositories for this needs to be removed, but all dependencies kept in place. The installation spec that notes these dependencies and the order of installation will also require an update to note that we need to install with pylint but install the dependencies (I will provide a list) first.

It should be easy enough to revert this change if it causes issues with other pis.

[hamishmb]

This week's improvements:

• config.py: Comment out VALVES1-3 and uncomment VALVE12 to match current on-site configuration
• Fix an issue where system tick wasn't set if the database was empty.
• Further improvements to sump pi error handling.

I also applied the above update to mysqlclient. All seems fine. The NAS box is stuck on v1.4.4 unless I recompile it, but I'll do that only if I see a need to - f I go down that avenue I could spend all my time just recompiling newer stuff for the NAS box

Below are the commands I used to update mysqlclient. We'll need these when we power up the Hanham valves again:

Code: Select all

sudo apt purge -y python3-mysqldb
sudo apt install -y libmariadb3 mariadb-common mysql-common
sudo pip3 install mysqlclient==2.0.1

[hamishmb]

Next set of changes:

• Merged Patrick's code into master.
• Tested resulting code in VMs and ran unit tests - all good.
• Moved old control logic and new integration and setup functions into new file Logic/controllogic.py.
• Fixed the corresponding unit tests to use the new file.
• Integrated stagepilogic tests into the test suites.
• Show results for all unit tests at the end.
  • I did this because we have all missed test failures due to running them all at once and not scrolling back up to check each suite passed.
  • It takes around 5 minutes to run the whole suite.
• Update README to reflect changes to unit tests.

Please note, I very much want to test this code in VMs and maybe on Terry's test rig a little bit prior to deploying - moving control logic around makes me nervous that I somehow broke something, despite the unit tests passing.

It's also worth noting that test coverage has decreased since July (last major change to the figure), and that we may need to fix this before deploying the next set of changes. Details are at: viewtopic.php?p=4464#p4464

Thank you for the control logic Patrick, and I apologise for it continuing to take me so long to get it properly merged, tested, and integrated. I'm still very short of time at the moment I'm afraid, but today's couple of hours let me get a lot done and a lot of future work organised.

NOT fixed/done:

• Broken sumppi logic tests (still using pre-database design).
• Remove the extraneous arguments to control logic functions now that we're using the database.

[hamishmb]

Next set of changes:

• Put stage pi logic docstrings onto one line so they display properly when executing the unit tests (for the paragraphs, I just put the first sentence on one line).
• Fix the sump pi unit tests (test failure and also coverage).
• Remove an old sockets-based unit test that isn't relevant any more.
• Sort out documentation generation for all modules under the logic package.
• Add Patrick Wigmore to copyright information and update version and release date prior to deployment.
• Run all unit tests again.
• Test code in complete virtual WMT network with all currently powered-on pis and the NAS box.

All ready to go for deployment stage 1 on Thursday

[hamishmb]

Set of changes from last week:

• Partially set up Gitlab CI/CD as previously discussed.
  • Currently disabled as unit tests are not yet working with the system.
• Improve logging and error handling of site-wide updater.
• Add a packaging script to make it easier and more reliable to use the site-wide updater.
• Drop repeated requests for new system ticks in sockets code (lower network and system load).
• Make sure old rivercontrolsystem.tar.gz file is removed after updating.

Changed on site last week:

• Security of VPN and Webservers improved.
• All river system pis updated to the latest version of Raspberry Pi OS.
• All river system pis updated to latest river control system (after the above changes were done).
• Stage Valve (VALVE12) now online.

Unresolved issues:

• Race condition / other anomaly with site-wide updater (avoid using for now).
• Python/river control system software seems to periodically and randomly crash on the NAS box.

[hamishmb]

Set of changes this week:

• Progress made in diagnosing periodic random python crashes on the NAS box (details not there yet though).
• Add various indexes to the database in the NAS box to improve performance of the engineer GUI, and of the river control system itself.

Queued to change next week:

• Deploy Patrick's updated Stage Pi code with refactoring and general fixes and improvements.
• Begin diagnosis of site-wide updater issues.

Unresolved issues:

• Race condition / other anomaly with site-wide updater (avoid using for now).
• Python/river control system software seems to periodically and randomly crash on the NAS box.
• NAS box startup sequence is now excessively slow as it also checks the indexes when checking table integrity. A minor annoyance only, as the river system is operational during that time, but improvements can probably be made.