July 2020 onwards - River System Status

[Penri]

The river control system was restarted on Sunday 28th. It took two or three hard resets of the Wendy Street and Sump Pi's and much shaking of the Sump and Wendy Street probes to get the system up and running satisfactorily but it appeared to then function as required. As system operation in this way will soon be superseded by a new control architecture I only note this here for interest.

The general river level when left was reasonable b ut time manual manipulation may be required to re-establish an equilibrium.

Before restart all the butts and sump were full and the river / bog garden at about 1/2 to 3/4 capacity.

[TerryJC]

Now the VPN has been installed, I've been able to download Results and Logs from the Pis for the first time since late Winter. All of the old files have been deleted on the Pis, so the next batch will contain files from this morning onward.

See attached zipped up file of zipped up files.

This is a pretty big download. Future results should be much smaller.

[Penri]

On arrival river not flowing.
Wendy valve RPi showed no activity, Sump and Wendy Butts RPis showed normal activity.
Soft reset Valve RPi, came up showing normal activity.
BUT despite probe waggling system did not restart.
Have just hard reset and now waiting for outcome.

[Penri]

... that worked, now up and running.
Currently manually routing water from Hanham and stage butts to fill river.

[TerryJC]

This seems very odd. Why would the river stop flowing under the circumstances we've had? (ie plenty of rain.) It shouldn't be necessary to waggle probes after a Gate Valve reboot.

[Penri]

No pumps running, sump full to overflowing.

[hamishmb]

What's the status with this? I'm freeish this morning so if needs be (and it's relatively quiet), I can come in to investigate.

[TerryJC]

This morning's attempt to carry out the Planned work to roll out the additional devices (viewtopic.php?p=3782#p3782) was only partially successful for a number of reasons. However, we learned a few things and I finally (I hope) solved the problem with the need to recycle the Big Switch each time a Pi is rebooted, but we are still having issues with the Sump SAC not discovering the true level of the Sump at startup and industriously emptying it.

First the problem with the Big Switch. What I found was that some of the Pis, namely SumpPi and Wendy Gate Valve still had a couple or routes setup which I added (on the advice of the author of the VPN Server code) while trying to get it to work. I thought that I had taken them all out, but I obviously missed a couple. As of this afternoon, I am able to log directly into all of the running Pis on the site, (which now includes Stage Butts Pi).

The second problem is a bit more complicated. We always had an issue when rebooting Pis that were part of an SAC because if the water level is between sensors, then the returned result would be 0 mm. If this is the Sump, then the Sump SAC turns off the re-circulation pump and fully opens the Wendy Gate Valve with the result that the Sump rapidly fills and overflows and the Sump SAC never catches up, so we lose all of the water. We have planned to resolve that part of the problem as one of the activities to be completed going forward. In the past the solution has been to 'waggle' the probe to ensure that a sensor is passed and thus the SAC has something to work with.

Just recently however, we've seen something slightly different and at this time we aren't sure what is going on. Even when the probe is waggled, it seems to take a very long time (and sometimes never) for the SAC to catch up. With all the other issues going on, it's been a bit difficult to see what's been happening, but I've downloaded the logs for the period when it happened this morning and you can perhaps see what might be the problem.

The Sump Pi file without a time is the stuff that covers the time since I downloaded on Friday.

At the time of writing the Sump SAC still seems to be having problems with it mostly thinking that it is empty but then thinking that it's full.

[Penri]

Just looking at the Sump results. On the 24th everything seemed to be fairly orderly but with the probe returning, with only 2 exceptions, reading in 100mm increments. Yet on the 28th (today) we're seeing 25s, 50s and 75s with a few 100s increments and I know the water was at 500 or above from when I went in (river stopped, sump overflowing) to when I left at 12:45ish.
By comparison probe G4 seems a lot better behaved returning a good mix of x25, x50, x75 and x00 on both the 24th and 28th.
Is the sump probe H/W misbehaving, based on the 24th readings I would guess at the x25, x50 and x75 resistor strings as being suspect, but today's reading don't have that same behaviour pattern, the x25, x50 and x75 resistor strings are returning values. Would water in the probe cause this and if there was water wouldn't we see 00 reported as well and how does that explain the 24th readings?

One other observation / question, does something happen as 00:00 each day that could cause the probes to stop reporting new readings both probes go to the end of the 24th and then from the 25th onward report the the last reading taken on the 24th until we reset the system this morning.

[TerryJC]

Penri,

I don't know the answer to these questions at the moment. I can think of some scenarios:

1. There is a problem with the probe. This might be borne out by the fact that tailing the log shows that the Pi thinks the level is 25, 75, etc as you mentioned, but every few minutes it thinks there is 500 or 600 mm. The changes happen in a matter of seconds and I don't believe that the actual level can change that fast.
2. Coincidentally, the VPN Server came fully on line around the 24th, (although the first connections happened on the 18th, albeit with multiple issues). The VPN Server seems to be working reliably now, so that may point to it being a problem, or the opposite; I simply don't know. Earlier today the V4 (Wendy) Gate Valve was flagging connection errors, but I think that was related to the erroneous routes that had been set up. There are no socket errors at the moment.
3. If the above behaviour was caused by being started with the probe float between levels, then surely all the pumping out and letting in of water that seems to be occuring at the moment would naturally take the level past a sensor and the system would then stabilise?
4. On my Test Rig I have seen some anomalous outputs when the magnet was right at the top of the pole, but I wouldn't have expected it to look like this.

My only suggestions are that we shut down the VPN on Thursday, when you are in and also that I bring the spare probe for you to try.