General Software Improvements (2nd October 2019 onwards)

[hamishmb]

I've started a new thread for this, seeing as the general software changes tend to get a bit fragmented across different topics.

I made the following general improvements today:

• Increased gate valve closed tolerance to 2% to reduce hunting.
• Use 24-hour time in logging messages for better clarity.
• Log files:
  • Rotate log files each night at midnight (note the names are slightly different to the readings files, but it's still very clear).
  • (the names are different because there's a built-in way to do this).
  • Automatically recreate log files if they are moved or deleted.

[hamishmb]

I made the following improvements this morning:

• Configuration for pis that are not yet installed has been commented out.
  • (this is the cause of the network weirdness after our Canford school meeting last week)
• There is more information in the logging messages from socket tools - this should help us diagnose our issues.
• The logger levels have been changed in socket tools - more messages are now visible when not running in debug mode.
• MySQLdb is no longer imported on startup - it is not yet present on the pis at WMT.
• System load monitoring has been implemented.
  • This is done every 30 seconds.
  • CPU load and memory usage are monitored.
  • This is done in "normal" mode as well as debug mode.
  • This depends on the "psutil" Python module.

Hopefully this will help us determine the source of our problems.

Terry, can you verify that the psutil Python module is available on the SD cards installed at WMT? I didn't have to install it manually on my system, so I'm assuming it's probably okay, but worth checking.

[TerryJC]

Terry, can you verify that the psutil Python module is available on the SD cards installed at WMT? I didn't have to install it manually on my system, so I'm assuming it's probably okay, but worth checking.

I created this script called psutil.py:

Code: Select all

#! /usr/bin/python3
import psutil
print("Done")

and ran it using

Code: Select all

python3 psutil.py

On this machine (my desktop), it works. On the Pi that I use for my local SumpPi, it gives an import error.

How do you want to proceed? I assume that I could install it here by connecting the Pi to the internet, but that wouldn't help the 4+1 Pis deployed (Sump, Wendy, Stage, Wendy Gate + Stage Gate).

Is there a way to install it via SSH with the module installed on a laptop?

Otherwise, we'' have to make 4 (5) new SD Cards with psutil installed and I haven't really got enough spare SD Cards to do that in one go.

[hamishmb]

We could download the .deb package for Raspbian 9 and 10, copy it across with SSH, and install it. It's probably called python3-psutil, but I don't have a pi in front of me at the moment, so hard to be sure. Edit: dependencies might also get annoying, if there are lots of them.

We could stagger it, and then install updates on the existing SD cards and mysqlclient on them too. Do you have to make them from scratch, or can you run apt update and apt upgrade?

Alternatively, if we could get an internet connection to the pis temporarily then we could do updates and install software in-situ.

[TerryJC]

We could download the .deb package for Raspbian 9 and 10, copy it across with SSH, and install it. It's probably called python3-psutil, but I don't have a pi in front of me at the moment, so hard to be sure. Edit: dependencies might also get annoying, if there are lots of them.

That might well work, but as you say, could be a bit fraught.

We could stagger it, and then install updates on the existing SD cards and mysqlclient on them too. Do you have to make them from scratch, or can you run apt update and apt upgrade?

I only have one SD Card here that is configured with Raspbian 10 and has the same software as the deployed system devices; that's SumpPi. I also have the old Wendy Butts Pi, that was removed when I installed the upgraded hardware. That will have Raspbian 9 on it.

I can make new SD Cards in a number of ways:

1. From scratch, as we discussed on the Network issues thread, with the absolute, most up-to date version of Raspbian 10.
2. I can use the latest backup of the Gate Valve SD Card (2019-09-11_V4_Gate_Valve_Backup.zip), which as you can see was made around a month or so ago. I would then modify each card to be a Gate Valve or SAC as needed.
3. I could also use the SumPi SD Card that I have here as the starting point, but I'm not 100% sure exactly when I made that (probably late August).

Whichever method I choose, the workload would be much the same, so I would probably prefer to use option 1. I would then install psutil and mysqlclient at the same time as I install all the other Python modules and utilities.

Alternatively, if we could get an internet connection to the pis temporarily then we could do updates and install software in-situ.

We probably could do that, but we'd need to use a tethered phone or something.

[hamishmb]

Quite possibly best to do them all from scratch then. That way they'll all be running the exact same version of Raspbian.

[TerryJC]

OK.

I'm downloading Raspbian Lite now. (It's dated 2019-09-26, which is about 1 month newer than the one I used for the current Raspbian 10 cards.)

[TerryJC]

OK. I'm currently updating/upgrading the installation.

Which software framework version should I install? The latest or an earlier version?

[hamishmb]

I'd go with the latest version (ID 8863a103 for future reference). Note that it won't work without psutil.

[TerryJC]

OK. I'll start with a new SumpPi, the Wendy Pi, then Wendy Gate Valve, then Stage Pi and finally, when I've retrieved the old SD Cards, Stage Gate Valve.