Results Database Server

[hamishmb]

Quick query: did we decide how the NAS box should determine what reading/tick interval to set?

[TerryJC]

We did have a conversation about this, but I'm not sure a consensus was reached. It's my view that the System Tick should be at least as often as the shortest reading interval so that any two readings from a particular sensor would come in during different tick values.

[hamishmb]

Oh yes, I remember. I think we decided (at least on a preliminary basis) to set the system tick to 15 seconds because of that.

Doing the reading interval would be more complex, but an initial idea I have is to get all the pis to respond with their desired interval and pick the lowest for the next interval. This will mean lower intervals than many sites need at any particular time, but I don't think we want different intervals at different sites, that could become messy very quickly.

[TerryJC]

The problem with having a site wide interval is that the only one that would work is the fastest.

I think that if we stick to the current approach of having reading intervals to suit the local conditions it should be possible to interpolate values that haven't changed because the reading interval is longer than the tick. Just repeat the last value.

[hamishmb]

Yeah, I suppose we could do that too.

Not sure what you mean about the only interval that would work being the fastest one though - the site-wide reading interval could just be the lowest interval any of the pis wanted.

I think either approach would work. Different intervals at different sites would be more efficient but would also need more complex logic.

Example:

Sump Pi is pumping to G4 and needs to monitor the levels there closely. The NAS box would need to send both G4 and the SUMP a shorter interval to make sure there are no overflows. At the same time, G3 might also be trying to fill G4, but be asking for G4 to have a slightly lower reading interval, so the NAS box would have to be very smart about this, and all the requests would have to come in at the right time.

This is all perfectly possible, but would it be making things unnecessarily complicated/prone to fail in complex ways?

[TerryJC]

Not sure what you mean about the only interval that would work being the fastest one though - the site-wide reading interval could just be the lowest interval any of the pis wanted.

Let me give you an example of my own before I address your example below:

There are two approaches to your suggestion. The first approach is that the NAS has to be made aware of the reading interval values in use by all of the Pis (by the Pis writing the value to an extra record in the database) and then increase all Pis to the fastest. For example; the Sump is running out of water so the reading interval would be reduced from 60s to 30s or even 15s. The NAS Box would see this and tell all the other Pis to change their reading intervals. This would vastly increase the volume of data to be handled because Pis that have seen no change in levels (say) would be sending the same numbers far more frequently than they do now.

Conversely, the NAS Box would check the timing of all of the Readings coming in and calculate the reading interval. When the Sump's reading became more frequent it would increase the site wide reading interval accordingly, with the same increase in data throughput.

I think either approach would work. Different intervals at different sites would be more efficient but would also need more complex logic.

Example:

Sump Pi is pumping to G4 and needs to monitor the levels there closely. The NAS box would need to send both G4 and the SUMP a shorter interval to make sure there are no overflows. At the same time, G3 might also be trying to fill G4, but be asking for G4 to have a slightly lower reading interval, so the NAS box would have to be very smart about this, and all the requests would have to come in at the right time.

This is all perfectly possible, but would it be making things unnecessarily complicated/prone to fail in complex ways?

I don't think it would work quite like that. At the moment, there is no linkage between the reading intervals of the Sump and the G4 Butts. Clearly there is a cause and effect, because as the Sump starts pumping, the G4 Butts would fill and, depending on the amount of water available, the G4 Pi might have to increase its reading interval. However, in this scenario, the Butts hold far more water than the Sump, so it isn't inevitable.

Left to myself, I would go for scenario that I've proposed because of the reduction in data throughput and disc usage. Yes, there is a penalty in that the readings won't line up, but in your scenario the NAS has to be made aware of the reading interval of each and every Pi in order to:
decide what to tell all the other Pis to do. With my approach the NAS simply stores the values and the timings are sorted out in the graphing software.

[PatrickW]

I think this boils down to a difference in our individual understandings of which code gets to decide the reading interval.

To me, it is the job of the three specified control logic instances to choose reading intervals. That is:

• the Sump Pi control logic
• the Stage Pi control logic, and
• the Lady Hanham Pi control logic

I think maybe Hamish has been thinking roughly the same thing?

Since the three control logic instances depend on readings from other Pis, I think they all need to be able to specify the reading interval for those other Pis. This would either be by "voting" for a global interval or by "voting" for individual intervals for each other Pi they are interested in.

Your (Terry's) second scenario in the above post strongly implies to me that each butt group Pi should choose its own reading interval independently and heuristically, based purely on its own water level(s). The three specified control logic instances would then get what they're given and have no say in the reading interval on other Pis, relying on each Pi using a good heuristic to set its reading interval.

It is easy enough to think of a heuristic that makes sure the reading interval is always fast when it needs to be fast: just speed it up as the water level approaches overflow or empty.

It's harder to think of a heuristic that makes sure the reading interval is not fast when no other Pis actually need the readings at that moment. Just because a butt group is nearly full or nearly empty doesn't mean the control logic needs a short interval. Covering these edge cases would turn into an exercise in duplicating the existing control logic, but in lots of different places, creating a maintenance headache. A cleaner method would be to get the framework to run all of the control logic on all of the Pis, but pass in a parameter that says whether it's the "main" instance, so that only the "main" instance actually issues commands to change valve and pump states. Then, each instance could return the reading interval it wants for that specific Pi.

I like the global reading interval solution, at least for an initial implementation.

For a subsequent refinement, I like the idea of each control logic function casting a vote for the reading interval for each site it is interested in.

[hamishmb]

I was thinking of using the sockets code for the pis to vote on the reading interval - that would be almost immediate and would represent very little load on the systems and network, and latency would also be low. The sockets code also seems to be very reliable now - what we've had at WMT for months has been fine network-wise, and since then I've made further improvements.

Note that currently the G4 interval is actually directly tied to what Sump Pi is doing - it sets the reading interval much faster for all sites when it is pumping water to the G4 butts. This is because when we didn't do this, we found the default interval was high enough that the butts would overflow.

The software currently deployed at WMT uses Sump Pi to dictate all the intervals. I think I'm with Patrick on this one - at least in the first instance, I think we should have a global reading interval. I'm going to have to work hard to ensure things do stay in sync, because I think anything else is a recipe for massive over-engineering and for things to go wrong.

In the future I'd be amenable to making it more intelligent, but I think making too many big changes at once would be a bad plan.

[hamishmb]

I should note that I think all of these are ideas with great promise, but the limiting factors are my time and motivation, and my angst to deliver something that works well!

[TerryJC]

Note that currently the G4 interval is actually directly tied to what Sump Pi is doing - it sets the reading interval much faster for all sites when it is pumping water to the G4 butts. This is because when we didn't do this, we found the default interval was high enough that the butts would overflow.

The software currently deployed at WMT uses Sump Pi to dictate all the intervals. I think I'm with Patrick on this one - at least in the first instance, I think we should have a global reading interval. I'm going to have to work hard to ensure things do stay in sync, because I think anything else is a recipe for massive over-engineering and for things to go wrong.

In the future I'd be amenable to making it more intelligent, but I think making too many big changes at once would be a bad plan.

All,

I think we're in danger of overthinking this. Let us assume for the moment that we didn't use the global reading interval; why would things get out of sync? The NAS Box would store everything at the same rate as the system tick, which would be the same speed as the fastest possible reading rate (ie 15 seconds). The whole reason that I proposed the system tick in the first place was to make it easy to keep things in sync; trying to do that with time stamps would be a nightmare. My original thought were to make it easier to build graphs, but the system tick is equally useful when trying to keep the system together.

So having said that; how would this work? The following indicates the additional Tasks to be performed by each device in the system:

• NAS Box - None. The data will be stored in the database and utilised by the Pis as necessary. The difference between the reading rate and the system tick at any given time is only half the max and min reading rates (eg (60 - 15)/2 = 22.5 seconds). In the scheme of things that is negligble from a reporting point of view.
• Sump Pi - None. It simply carries on doing what it's been doing all along.
• Wendy Butts Pi - None. It simply carries on doing what it's been doing all along.
• Stage Butts Pi - Minimal. Assuming your new code is basically a modified replica of the Sump Pi algorithm Patrick, the only changes should be to utilise the database instead of the sockets to obtain readings.
• Lady Hanham Butts Pi - A suitable algorithm will be needed based on the one that Sump Pi and Stage Butts Pi is (or will be) running.

Something else that would have to be done by the NAS Box or the Webserver is that the graphing tool would need to interpolate values which haven't been reported during any system tick period. A time stamp can be obtained from the system tick which is being generated anyway.

On the other hand, if the global reading interval was implemented, then each device would need to do the following additional tasks:

• NAS Box - An additional table or column to indicate the current reading intervals of each Pi (or some other method of discovering the fastest by polling or something).
• NAS Box - An additional table or column to indicate the new reading interval to each Pi. This would require some calculation or sorting of the incoming values to realise.
• Pis - Continuously monitor and change their current reading interval in response to the instruction from the NAS Box. This could result in some unintended side-effects. For example, if the global reading interval was reduced from 30 seconds to 60 seconds just as the Stage Butts reached the empty threshold, then all Pis would be switched to the higher rate and there is a possibility of undesirable fluctuations with additional computational load on all Pis and additional data throughput as mentioned earlier.

Have I missed the point?