March 2022 onwards - River System Status

[hamishmb]

Terry posted:

When Penri arrived at the WMT this morning, the river had stopped running. I checked the logs and sure enough, the Sump ran out of water at approx 2355 and the pump was turned off.

The problem was that in the intervening period the river had emptied itself into the sump and it was full (and had presumably been overflowing for some time). Penri turned on the Manual control and the river settled down again. The bad news is that SumpPi failed to register the changes in level (the readings had been 275 mm since last night) and the main circulation pump remained turned off. A reboot made it work again.

So we have a problem. For some reason SumpPi doesn't recover from an extremely low level situation and appears to either:

Stops measuring and uses the last known reading.
No longer obtain the correct reading for some reason.

This has to be a software issue, since a reboot solves it. Hamish can you shed any light on what's happening?

PS. I suspect that this issue has been there for a long time, but didn't emerge because we rarely completely ran out of water.

I'm interested to see what the problem might be, but will need to come in probably to do so.

[hamishmb]

I suspect this might be a situation which has arisen due to the network being down, but I shall see when I can get a hold of some logs.

[TerryJC]

I suspect this might be a situation which has arisen due to the network being down, but I shall see when I can get a hold of some logs.

This all occurred when the network was up.

[hamishmb]

Ah, okay then.

I'll have a look soon and get back to you.

[hamishmb]

A quick look at the readings show fluctuations around 250-300mm from midnight until 02:22 and then it gets stuck at 275 until the reboot at 10:14, and then starts reading 475 and fluctuates up and down (up till 14:07, when I'm writing this message).

Corroborating that with the log file, it seems to have emptied the (EDIT: Wendy butts) trying to refill itself, and then never seen another level until being rebooted. I can't see any signs of a crash or anything, it's very bizarre.

This is almost definitely a software problem of some kind, but have we checked that probe? I'm fairly sure we have recovered from empty several times, but perhaps I misremember.

At any rate, I do intend to make improvements to error handling and reporting, and that should help us fix the remaining issues.

[hamishmb]

I should add, I've been very out of the loop since March last year - could we possibly do a meeting sometime to catch me up on the things that have happened since then?

I apologise if the probe checks have all been done already, I simply can't remember. The quality of the software has never been quite how I wanted it to be, but hopefully I can change that now.

[Penri]

A meeting will be great but please make it after WMT re-opens (02Apr22), I've got my hands full currently with getting things ready for the off.

I can't attest to the probes being bomb proof (and I'm talking about the stuff in the plastic tubes here) but any issues seen in situ should be reasonably hard (not intermittent) and/or probably have some pattern to them. So failures at the X25 levels: 025, 125, 225, 325, ..., or a totally missing level would probably be down to the probe (the stuff in the plastic tube).

[hamishmb]

Okay, seems fair. Let's wait until then and then organise something when you have more time.

I agree, I don't think this is a probes/interfacing issue - a reboot definitely wouldn't fix that, as far as I understand how the hardware works. Hopefully it won't happen again until I figure out what the cause is. I need to spend some time reacquainting myself with the software.

[TerryJC]

Picking up on my previous posting at viewtopic.php?p=5701#p5701:

I've soaked the board in cleaning solution and when my hand-held vacuum cleaner has recharged, I should be able to complete the cleaning and carry out the repair.

Probably tomorrow now.

In fact it wasn't just the main power link on the Veroboard that had gone brittle. As with the Lady Hanham Gate Valves, anywhere I had soldered header wire to Veropins ended up falling off when I tried to disassemble the valve. This means that I've had to totally disassemble all the modules to allow me to get access, and re-assembly won't be quite so straightforward because I'll have to remake all the links that use DuPont headers at one end. I got tied up with helping my son repair his bike this morning and I won't have any spare time this afternoon.

Hopefully I'll be able to do this tomorrow.

I predict that we are going to have a lot of problems with Gate Valves going forward, especially the ones that have been deployed a long time. The Stage Valve wasn't working the other day (but neither was the Stage SAC, so that wasn't a priority). I have little doubt that the fault with the Stage Valve stems from the same problem.

[hamishmb]

That's a shame - they've been pretty rock-solid until this point.

Just so I know, which systems are parts of the network are online and working right now? Have we got the webserver working again yet?

I guess we maybe don't want to deploy the hanham valves I have until we've waterproofed them better then?