Summary of meeting 2020-05-23 about NAS box and control algorithms

[hamishmb]

Okay, brain is now working.

I think each message would need to consist of:

• a site ID to send the message to,
• something to say that it's a reading interval request (in case we ever want to send other kinds of messages), and
• the requested interval itself.

Presumably there will be an object that handles sending and receiving messages, which gets initialised on each Pi, taking the address of the 'message broker' from the config file?

Yes, that's the Sockets object (Tools/sockettools.py), along with the SocketHandlerThread object that gets created for each end of each socket.

Then, in the main loop, after the control algorithm executes, main.py identify the minimum interval among:

• any incoming messages that were sent to the local site ID from the other Pis
• a default interval and
• any requests that the collector gathered for the local site ID.

The local interval can then be set to that minimum.

Does that seem in line with your expectations?

This seems like a reasonable way to do things.

Or, perhaps it works out neater to have a special 'message broker' kind of object that handles the send/receive on the NAS, and which contains the logic for "routing" them to the right destination, so it's a case of repeatedly calling some "routeMessages" method on the object, and that method does all the work.

It might also be sensible to run all the NAS-side message broking in its own thread, rather than in the main loop. That way, if we were ever to decide that, actually, one of the Pis should be the message broker it could take on the dual roles of broker and client? The message broking thread would have no need to communicate with the rest of the program, except via the network, so I guess there are no or few concerns about concurrent access to shared state? (He says, looking at the concept of threads through narrowed, untrusting eyes.)

Yes, I was thinking of using the SocketHandlerThread class to manage this, and hide the complexity from the main loop. Then, as you say, any pi/other device running the sockets code could then be a message broker/forwarder. This would also stop the main loops from being able to "snoop" on messages being delivered to other connected devices, but that aren't intended for the current device.

The thread programming is a bit tricky and has caused us problems in the past, but Python's Global Interpreter Lock (https://wiki.python.org/moin/GlobalInterpreterLock) helps us a little bit in this case.

Reply to your last message Patrick:

Yes I suppose we don't need a message-bus-type-thing for that, but by the sounds of it, it's something that may well be useful to have later, so I might go ahead and implement it anyway - it really shouldn't be too hard.

[PatrickW]

Yes, that's the Sockets object (Tools/sockettools.py), along with the SocketHandlerThread object that gets created for each end of each socket.

Just had another look at the Sockets class. I never realised, or forgot, quite how much of this is already written, and that the Sockets object deals in terms of messages. The name leads me astray, because usually I'd think of sockets as referring to the actual, raw sockets underneath, whereas messages would be an abstraction on top of sockets, but here Sockets is the abstraction on top of sockets. Sorry if I've come across as trying to reinvent the wheel.

any requests that the collector gathered for the local site ID.

↗ These words that I wrote don't make sense. I should have written "any requests that the local control algorithm wanted for the local site". "Collector" is a holdover from a slightly different suggestion I was writing, which turned out not to make sense. Clearly I didn't edit it completely out of my post.

Yes I suppose we don't need a message-bus-type-thing for that, but by the sounds of it, it's something that may well be useful to have later, so I might go ahead and implement it anyway - it really shouldn't be too hard.

It seems to me that message-bus-type-things could have all sorts of features, but you must be talking about something specific. If I have to guess, I think you might mean you will implement something 'client side' to gather the messages, so that they will arrive in the in tray on each Pi's desk, ready and waiting for the main loop to read them, rather than making the main loop walk all the way to the post room and back?

I think the individual Pis should have the responsibility of connecting to the forwarder/broker and subscribing to have messages delivered to them, rather than the forwarder/broker just blindly connecting and pushing messages out to everything that's configured to receive them. I think that will make it easier to deal with any Pis going offline and having to reconnect. (In either case, the config comes from config.py.)

In order to discard stale messages, the recipient will need to be able to know in what order the messages from a given origin were sent. I think your code already deals with this.

There are some details to work out for the sending and receiving ends, about the semantics of the messages; "single order" versus "standing order with timeout", but that doesn't affect how message delivery itself is implemented.

[hamishmb]

Just had another look at the Sockets class. I never realised, or forgot, quite how much of this is already written, and that the Sockets object deals in terms of messages. The name leads me astray, because usually I'd think of sockets as referring to the actual, raw sockets underneath, whereas messages would be an abstraction on top of sockets, but here Sockets is the abstraction on top of sockets. Sorry if I've come across as trying to reinvent the wheel.

You weren't and don't worry. Because I wrote a lot of this code back in 2017/2018 I have to go back every now and then because I forget what's been done too

It seems to me that message-bus-type-things could have all sorts of features, but you must be talking about something specific. If I have to guess, I think you might mean you will implement something 'client side' to gather the messages, so that they will arrive in the in tray on each Pi's desk, ready and waiting for the main loop to read them, rather than making the main loop walk all the way to the post room and back?

I think the individual Pis should have the responsibility of connecting to the forwarder/broker and subscribing to have messages delivered to them, rather than the forwarder/broker just blindly connecting and pushing messages out to everything that's configured to receive them. I think that will make it easier to deal with any Pis going offline and having to reconnect. (In either case, the config comes from config.py.)

This describes what I was thinking of doing, you've just put it in a clearer way than I could manage.

In order to discard stale messages, the recipient will need to be able to know in what order the messages from a given origin were sent. I think your code already deals with this.

There are some details to work out for the sending and receiving ends, about the semantics of the messages; "single order" versus "standing order with timeout", but that doesn't affect how message delivery itself is implemented.

Yes, the order of the messages is preserved if the connection is lost and resumed later. You've lost me a bit with the "single order" vs "standing order with timeout" bit, but otherwise I think we're in agreement.

[PatrickW]

You've lost me a bit with the "single order" vs "standing order with timeout" bit

I was badly adapting the banking terminology: single payments and standing orders.

• Single order or single request: When a Pi is using the messages it has received to make a decision about its reading interval, on every loop iteration, there has to be a fresh message ready and waiting in the "inbox" to renew each reading interval request.
  • requires tight synchronisation between the loops on every Pi
  • loops must run at the same rate; decouple loop interval/tick from reading interval
  • sensitive to the exact timing of message processing within the loop body on each Pi -- may need to add arbitrary delays to make it work -- slight drift in synchronisation could cause intermittent glitches (although each loop interval is long, most of the processing occurs in a quick flurry, so the timescales involved could be very short)
  • renewal messages must never take longer than 1 loop interval to arrive (defined by the loop on the receiving end)
• Standing order with timeout: Requests for reading intervals stand until they are either cancelled by a subsequent contrary message or they time out. Messages would still be sent every loop iteration, since the timeout would be short.
  • loop synchronisation not required at all
  • recipients implement timeout timers
  • renewal messages must never take longer than the timeout to arrive
  • a timeout of twice the longest loop interval in the system is the maximum required to deal with any amount of loop desynchronisation
  • longer timeouts only necessary to cover extreme delays in message delivery
  • if a Pi crashes, the system will continue to heed that Pi's reading interval requests until the timeout expires (minor disadvantage)

[hamishmb]

Ah okay, I think I understand now.

Sounds like we probably want something closer to a standing order. What I've been thinking of is a bit of a hybrid approach between the two I think.

I do quite like the publisher-subscriber model, though it probably is a bit complex for our needs. Perhaps I'll stick that on the list of things to do later on? I can see us using it, but maybe not worth delaying things to implement it right now?