Investigation of Level Sensor Anomalies

[hamishmb]

For the G6 monitors, I checked the current version of main.py for the setup code, and it is missing. These are still hardcoded rather than gleaned from config.py, so if you add the lines:

Code: Select all

monitors.append(SocketsMonitor(sockets["SOCK6"], "G6", "FS0"))
monitors.append(SocketsMonitor(sockets["SOCK6"], "G6", "M0"))

To main.py where you find he similar ones (probably around line 215), that should fix that problem. I'm not having the other problem, and your config.py looks okay to me. I wonder if this is something I fixed in the last few updates, but can't say because I don't know what the problem is :/. Would it be good for me to come around to your place at some point, and then I can see it in situ?

[TerryJC]

Hamish,

Thanks for that. I'll try this and if it fixes the problem great. If not, then you are welcome to come round.

[TerryJC]

Hamish,

That was the problem. (I now recall that we had to add a line for the Gate Valve when we integrated that back in April, but I didn't have to do it for the updated Wendy Butts Pi because the entries were already there from the earlier type probe. My memory (or lack of) strikes again.)

I'm not sure if we discussed this, but would it be possible to get the code in main.py to add the monitor for all probes / devices in config.py, so we don't have to remember all these little details? I can put them into the Installation Spec of course, but getting the code to do it automagically would save a fair bit of re-configuring.

We still have two more Butts Interfaces and nine Gate Valves to install (and my memory, since you are unlikely to be around for all of them).

[TerryJC]

BTW. No need to come here. I'll return to monitoring the system over a long period.

[hamishmb]

Yes, we can do that, it's on my list, I just haven't gotten to it yet It might be worth adding to to the Spec for now, and then removing it later?

Glad it's working now.

[TerryJC]

I was just about to add it to the spec.

[TerryJC]

Hamish,

Once you had sorted out my connection problems, I recommenced the original tests. Both Pis seem to be working OK, apart from an apparent loss of control over SSH that came up at around 1730 BST yesterday (that's when I first noticed it).

When I restarted the testing I connected to both Pis via SSH and started top on them. The CPU activity was around 20% and has remained at that ever since (total uptime currently 1 day, 21:11, so less than when I first noticed the very high CPU activity a few days ago). I maintained these two connections whenever my desktop was on.

As mentioned, I wanted to shut down my desktop yesterday afternoon at around 5:30. so I rechecked the CPUs and they were looking good. However, when I typed 'Q' to close down top; nothing! Both Pis exhibited the same problem, but the task display was apparently still updating. I opened another shell and attempted to log into SumpPi again, but there was no prompt for the password. I tried the same with SButtsPi and got a 'no response' error. In the end, I had to simply close the console windows, so that I could shut down the machine. I had no time to investigate further.

This morning, I started my PC and logged into the two Pis as normal and they both seem fine. I'm now left wondering if the problem was due to my PC (in some way), the router (it's in a benign environment) or the sheer volume of traffic that had built up over the day.

A few minutes ago I downloaded the logs and readings from both Pi's (see attached), but I can see nothing that would indicate a lockup at that time. The results file for G6:M0 shows the correct values being stored as I set the magnet at the start, (I did this a few times) then lots of dots. Both G6:M0 files (Sump and Butts) appear to show the same things.

rivercontrolsystem.log still makes no mention of connections are records nothing after the initial startup, so there is still something amiss there.

This lockup may be indicative of what is happening at WMT and it may be another red herring. Either way, we still have no explanation of the high CPU percentage I saw at the start or the problems at WMT, so I'll carry on testing.

[hamishmb]

That's really bizarre, never had that happen to me!

I'm not having this at my end - I'm running a test as well - and all seems fine so far (up for 3 days). I'll let it run for longer and see what happens. At least we're getting the correct readings anyway

[TerryJC]

Sitrep:

Both systems have now been running for just over four days with no further problems. Currently, the CPU usage for both Pis remains at ~20%.

I am going to be very busy this weekend because I have two family birthdays to attend (one of which is in the Midlands), so I won't be monitoring the Pis quite so closely. However, I will leave them running and checkout their status when I return on Monday.

[TerryJC]

Uptime is now just over 5 days, with no new anomalies.

Attached are the results that I downloaded from both Pis a few minutes ago. There is still something not quite right about rivercontrolsystem.log which still makes no mention of connections being made and records nothing after the initial startup. However, the size of G6:M0-2019-06-23 09:37:43.csv in the results directory of Stage Butts Pi is identical in size to G6:M0-2019-06-23 09:36:55.csv in the results directory of Sump Pi and appears to contain exactly the same data, so the connection is clearly working.

As mentioned elsewhere, I am off to the Midlands tomorrow morning (the M27 is closed ), so won't be monitoring this system until Monday afternoon/evening. By then it will have run untouched for well over a week.