Summary of meeting 2020-05-23 about NAS box and control algorithms

[hamishmb]

Here follows a quick (and probably incomplete) summary of what we discussed.

We talked about:

• How using a site-wide global reading interval would be a simple solution, but would be inefficient.
  • At first I thought this wasn't a big problem.
  • Now I see that we'd have lots of duplicate readings on the NAS box and this would make the data harder to crunch.
  • We'd also have to purge the database of old readings more frequently - not something we want to do often.
• How using a local reading interval for each pi would work better, but be more complex.
• ^ We outlined a number of ways of dealing with this:
  • Storing the reading intervals in the database alongside the readings.
    • This would work, but be high latency and cause more CPU load on the NAS box - not ideal.
    • Could also cause large time drift when polling for intervals if the NAS box responds slowly/is busy.
  • Improving the sockets code to add a "message bus" type feature.
    • This would allow the NAS box (or any other device) to forward messages between any of the pis connected to it.
    • This would be quicker, cause less system and network load, and use less storage space on the NAS box.
    • I (Hamish) also think I could implement this without too much difficulty - it seems like a good solution.
    • It essentially allows peer to peer comms (as far as the pis are concerned) without the complexity of many sockets directly connecting the pis.
    • But note that this does make the NAS box a single point of failure(!)

We decided:

• We do not want a global reading interval for the reasons outlined above.
• To give the message bus idea with the sockets a go first, to see how workable the idea is for sending reading intervals between Pis.
• Failing that, modify the database spec and update the code to use an extra column in the readings tables to store the interval there instead.

I'm sure I've forgotten something, but hopefully that covers most of it.

[PatrickW]

That sounds about right to me.

Here is a link to the RabbitMQ website, to explain what I was referring to when I started talking about message buses. We don't even need half the features RabbitMQ has, so I think it is entirely reasonable to build something based on our existing sockets code instead of having to go down the rabbit hole (ha, pun not intended) of figuring out how to integrate RabbitMQ. Posting for background information only at this point, not as a proposal.

[hamishmb]

Cool, I shall have a read.

One thing I have forgotten is what we were going to use the interval information at the neighbouring sites for, but I guess that goes into control algorithms and is somewhat outside the scope of what I'm doing.

[TerryJC]

Patrick,

MTQQ is the protocol that i immediately thought of when you mentioned RabbitMQ. It is covered at https://en.wikipedia.org/wiki/MQTT.

I do remember discussing this very briefly early on in the design cycle but I think that it was simply a conversation that we had one morning at WMT. I can't recall exactly how it went, but we decided that we didn't need it and it may have been poorly supported in the Raspberry Pi at the time. I see that MQTT is covered in RabbitMQ.

Hamish,

Thanks for doing this. I can't see any problems or ommisions.

[hamishmb]

Makes sense, and you're welcome Terry.

Has just occurred to me: if the NAS box is going to store the current intervals of the pis in variables, do we actually need the message bus? I might have lost the plot here - I can't remember if we wanted it for something else as well.

[TerryJC]

I think the conversation about RabbitMQ was a suggestion rather than a decision on how we should do it. When you said that would add "a "message bus" type feature" I assumed that was how you would achieve the tweaking of the variables, not a decision to use some kind of formal message bus.

If you think that you can do it without implementing a formal bus protocol other than the sockets code, lets keep it simple!

[PatrickW]

(In reply to post #3 from Hamish, regarding "what we were going to use the information at the neighbouring sites for". Wrote this last night and didn't submit it.)

We touched upon whether it is actually necessary to vary the reading interval at all, with Hamish pointing out that there was a problem with the Wendy Butts overflowing if the reading interval wasn't sped up by the algorithm on Sump Pi.

I think Terry also came to the conclusion that pretty much all the algorithms at least have an interest in the water level in G4. Whether that means they actually need to change the interval at G4 seems to be a question that can't be answered without testing on the physical water system, so I think we just carried on the discussion based on the assumption that it would be necessary, based on Hamish's past experience.

We didn't really discuss the content of the messages to be passed via the sockets message bus, and where they come from and go to in the code. I would expect the control algorithms to set out any intervals they "want". On the receiving end, something in the main loop would check for incoming messages and then set that Pi's reading interval to be some function (i.e. the minimum) of the intervals requested.

I think each message would need to consist of:

• a site ID to send the message to,
• something to say that it's a reading interval request (in case we ever want to send other kinds of messages), and
• the requested interval itself.

Presumably there will be an object that handles sending and receiving messages, which gets initialised on each Pi, taking the address of the 'message broker' from the config file?

Each time the control algorithm identifies a need to specify a reading interval, it would call whatever function is used to send a message, specifying a site ID and a reading interval to request.

Then, in the main loop, after the control algorithm executes, main.py identify the minimum interval among:

• any incoming messages that were sent to the local site ID from the other Pis
• a default interval and
• any requests that the collector gathered for the local site ID.

The local interval can then be set to that minimum.

Does that seem in line with your expectations?

[PatrickW]

As for the "message bus" implementation on the NAS box, I suppose the code would be similar to what I just posted, except:

• Create multiple message send/receive type objects, one for each Pi in the config (or have a socket open for connection requests from any Pi, and create a send/receive object for each request that comes in)
• Instead of picking up messages only for itself, it picks up all the messages that arrive
• Where the Pis would use the received messages to determine their own reading interval, it should read the site ID of each message and forward it down the message sending channel to that site's Pi.
• I suppose ideally it should loop a bit faster than the Pis (free-running?) so that it has a chance to pick up messages and forward them within the space of one loop iteration on the Pis? Otherwise it could take two Pi loop iterations to deliver the messages. Not sure how much it matters really.

Or, perhaps it works out neater to have a special 'message broker' kind of object that handles the send/receive on the NAS, and which contains the logic for "routing" them to the right destination, so it's a case of repeatedly calling some "routeMessages" method on the object, and that method does all the work.

It might also be sensible to run all the NAS-side message broking in its own thread, rather than in the main loop. That way, if we were ever to decide that, actually, one of the Pis should be the message broker it could take on the dual roles of broker and client? The message broking thread would have no need to communicate with the rest of the program, except via the network, so I guess there are no or few concerns about concurrent access to shared state? (He says, looking at the concept of threads through narrowed, untrusting eyes.)

For all their apparent detail, these are obviously just broad, hand-wavy ideas of mine, I figure it takes actually trying to implement the thing to pin down the real details.

[PatrickW]

Hamish, when you said "if the NAS box is going to store the current intervals of the pis in variables, do we actually need the message bus?", were you thinking that the Pis should poll the NAS box to see the current state of the variables, rather than having to wait for it send them messages about it? I guess that could work. More like a set of pigeonholes than a postal service.

Ohhh! I just realised my immediate previous post about the broker implementation on the NAS is wrong!

When I described the client side of things that would run on every Pi, I was actually describing something that would need to work as a pigeonhole system.

When I started trying to think how the broker side would work, I got confused and was thinking it would have to push messages out to the clients, in contradiction to the fact I'd just said the clients would have to reach out and fetch their own messages. Allowing the NAS to push the messages to the Pis would need each Pi to run a separate thread to receive messages, which seems pointlessly complex when there can just be pigeonholes, so I guess I'm on board with a pigeonhole system if that's what you meant.

Whether a set of digital pigeonholes qualifies as a message bus is probably neither here nor there, if it gets the job done.

Sorry, typical Patrick filling the thread with verbose waffle that I end up backtracking on!

[hamishmb]

That all makes sense I think, but my brain is a bit frazzled today so I'll have to reply properly later I'm certainly guilty of waffling too - don't worry about it

I should note that each socket does actually have a separate thread, so messages can indeed be pushed to devices.