All,
During a telephone conversation with Penri this afternoon about an unrelated topic; we discussed these errors. What I discovered is that this problem is not anything to do with the Sump Pi; I observed the same errors in the Lady Hanham, Wendy Butts and Stage Butts Pi logss. Just for completeness the Wendy Butts Gate Valve log is also exhibiting these errors. I may be wrong but it looks like the error is originating at the NAS Box which is refusing connections.
Hamish and /or Patrick, Can you shed any light on this?
July 2020 onwards - River System Status
Re: July 2020 onwards - River System Status
Yes, the error originates at the NAS box. It is the same error we've been seeing sporadically for months with little explanation, but hopefully more data was collected this time so as long as it isn't rebooted before I look, I should be able to see what happened.
Hamish
Re: July 2020 onwards - River System Status
I can be in on this Saturday afternoon (01May21) from around 14:00 if that helps.
Re: July 2020 onwards - River System Status
Hello
Went into WMT this PM, river stopped, sump full, cirulation pump off, all measuring floats floating.
Performed hard reset on Sump and Wendy control boxes, circulation pump restarted.
I'm currently feeding water into the Hanman butts via a hose, will go back in shortly to remove hose and let water into Wendy butts / Sump.
Went into WMT this PM, river stopped, sump full, cirulation pump off, all measuring floats floating.
Performed hard reset on Sump and Wendy control boxes, circulation pump restarted.
I'm currently feeding water into the Hanman butts via a hose, will go back in shortly to remove hose and let water into Wendy butts / Sump.
Re: July 2020 onwards - River System Status
As I left WMT the river was flowing into the sump and the control system operating normally.
Re: July 2020 onwards - River System Status
Went into WMT yesterday afternoon (02May21), river was still operating normally although the sump level was pretty low so I topped is up via the Hanham butts, manually, using a hose.
Re: July 2020 onwards - River System Status
Hopefully it will continue working fine from now on. I'm hoping to look into this soon but I won't be able to test any hypotheses for a while because my VM storage fell over and I need to reinstall everything, so that will take a while.
Hamish
Re: July 2020 onwards - River System Status
Sorry, a bit late responding to this. (I made the mistake of not logging in when I last got a notification for this thread, which kills subsequent notifications until next log-in.)
Just to elaborate on the error that originates in the NAS box, which Hamish has already addressed: This is the one where the control software on the NAS crashes, for as yet mysterious and unknown NAS-ey reasons that only crop up on the actual NAS. That part of the overall system is currently responsible for setting and restoring the system tick and monitoring the temperature of the NAS box and its drives.
If it crashes, the rest of the system can continue to operate without it, but the system tick will stop incrementing in the logged readings. (I think that's the limit of the consequences. Hamish probably understands the behaviour relating to the system tick better than I do.) It also means that, if a Pi gets restarted while the error condition is present, that Pi will be delayed by a one-off 3 minute timeout trying to get the tick, before it gives up and carries on without it.
Just to elaborate on the error that originates in the NAS box, which Hamish has already addressed: This is the one where the control software on the NAS crashes, for as yet mysterious and unknown NAS-ey reasons that only crop up on the actual NAS. That part of the overall system is currently responsible for setting and restoring the system tick and monitoring the temperature of the NAS box and its drives.
If it crashes, the rest of the system can continue to operate without it, but the system tick will stop incrementing in the logged readings. (I think that's the limit of the consequences. Hamish probably understands the behaviour relating to the system tick better than I do.) It also means that, if a Pi gets restarted while the error condition is present, that Pi will be delayed by a one-off 3 minute timeout trying to get the tick, before it gives up and carries on without it.
Re: July 2020 onwards - River System Status
Yeah, that's right Patrick. I'm thinking maybe I'll implement a basic watchdog next just to restart it when it crashes - still have had no luck finding out why it crashes.
In other news, the spare HDD was installed into the NAS box last Thursday and has finished re-syncing itself and all seems fine. I am running tests on the one that failed to determine if it can be fixed and be used as a spare, but hopefully the new drive should last a long time.
In other news, the spare HDD was installed into the NAS box last Thursday and has finished re-syncing itself and all seems fine. I am running tests on the one that failed to determine if it can be fixed and be used as a spare, but hopefully the new drive should last a long time.
Hamish
Re: July 2020 onwards - River System Status
Noticing the torrential downpour outside, I have just checked the water levels in the river system.
SUMP peaked at 815mm at 03:15 (4:15am BST), fell to 775, and has risen again to 850mm over the last 20 minutes at the time of writing (05:29 UTC/06:29 BST). This correlates well with the online rain radar for the past 2 hours, which showed two bands of heavy rain passing over, each ending at approximately the time when the water level peaked.
The SUMP control logic is requesting both pumps to be enabled, so water should be getting extracted into Wendy butts.
G4 Wendy has slowly risen from 275 at midnight (1am) through 300, 325 and first reached 350 at 05:05 (06:05). A smaller rise than I might have expected, but I don't have a good handle on how fast it should fill in these conditions.
G6 Stage has risen steadily from 50 to 125 over the same period at essentially exactly the same rate as the rise in Wendy, which makes sense to me, given that V12 is open and there is not much height difference between the butts (i.e. it's two measurements of the same water level, offset by 225mm.)
The Lady Hanham butts have been up and down. The G3:M0 reading fell sharply from 300 to 100 at 04:58 (05:58 BST), then rose haltingly up to 325 by 05:03, with three drops back to 100mm and then a sudden rise up to to 325 that took less than 1 minute, before holding steady. Interesting. (G3:M1 and G3:M2 are unchanging at 25mm and 150mm respectively.)
SUMP peaked at 815mm at 03:15 (4:15am BST), fell to 775, and has risen again to 850mm over the last 20 minutes at the time of writing (05:29 UTC/06:29 BST). This correlates well with the online rain radar for the past 2 hours, which showed two bands of heavy rain passing over, each ending at approximately the time when the water level peaked.
The SUMP control logic is requesting both pumps to be enabled, so water should be getting extracted into Wendy butts.
G4 Wendy has slowly risen from 275 at midnight (1am) through 300, 325 and first reached 350 at 05:05 (06:05). A smaller rise than I might have expected, but I don't have a good handle on how fast it should fill in these conditions.
G6 Stage has risen steadily from 50 to 125 over the same period at essentially exactly the same rate as the rise in Wendy, which makes sense to me, given that V12 is open and there is not much height difference between the butts (i.e. it's two measurements of the same water level, offset by 225mm.)
The Lady Hanham butts have been up and down. The G3:M0 reading fell sharply from 300 to 100 at 04:58 (05:58 BST), then rose haltingly up to 325 by 05:03, with three drops back to 100mm and then a sudden rise up to to 325 that took less than 1 minute, before holding steady. Interesting. (G3:M1 and G3:M2 are unchanging at 25mm and 150mm respectively.)