Webserver Development

[TerryJC]

Currently, as reported in viewtopic.php?p=4618#p4618 I have two programs; one for each Pi in the system which perform the basic functions specified in the requirements; eg play music and ring bells. I have an outstanding issue with the sockets code but I have temporarily put that on the back burner while I look at the use of Flask to produce a Webserver that can be used to control and monitor those functions.

I have now spent several days working through several web-based tutorials and a pretty useful text book called 'Developing Web Applications with Python'. My takeaway from this study to date is that:

• Flask is quite powerful because it comes with numerous modules and extensions that make the webserver code relatively simple.
• Apart from the online Tutorials and textbooks, the Flask Documentation (https://flask.palletsprojects.com/en/1.1.x/) appears to be well-written and comprehensive.
• I think that the basic requirement of producing a web form (or some forms) will be relatively straightforward, (although it will take me some time to develop such a thing owing to my non-programming background).
• All of the Tutorials and textbooks that I have found major on accessing information in a database and the Flask module flask-sqlalchemy seems to provide some great features, especially when combined with other modules to aid formatting etc.
• Other tutorials that I've found give some insights into social networking development and blogging, but they all seem to rely on an underlying database technology as you would expect.
• I've found nothing that fetches data using sockets code in a web form although there are a couple of Flask modules:
  • flask-sockets which provides a websocket library
    and
  • flask-socketsio which gives Flask applications access to low latency bi-directional communications between the clients and the server. The client-side application can use any of the SocketIO official clients libraries in Javascript, C++, Java and Swift, or any compatible client to establish a permanent connection to the server

I'm assuming that something can be produced which uses sockets code to retrieve and write data instead of SQL queries.

One question though:

Hamish, I understand that your sockets code is in an enhanced Python module to make using it easier but it uses the standard Python sockets module underneath. Do you think it will be compatible with the flask-socketsio module which seems to be what we want?

[TerryJC]

Today I stumbled across an excellent Flask Tutorial on YouTube (https://www.youtube.com/watch?v=zdgYw-3tzfI). It's a video of a lecture given at Harvard by someone who I consider to be an extremely good teacher.

It's about 1.5 hours in duration, but it has cleared up a number of points that previously I had been struggling to understand.

It may be useful to others who wish to use Flask elsewhere.

[hamishmb]

Terry,

I suppose it should be compatible with Flask's socketsio module, as long as socketsio doesn't expect any special protocol to be used (which it may well do).

However, this might be a somewhat cumbersome way to talk to the bells software from Flask because:

• Flask is running on Bells Pi (?) so a simpler form of inter-process communication such as pipes can probably be used.
  • If this is not the case then the rest of my thoughts here are totally useless
• My Sockets class expects the other end to use a specific (though simple) protocol, so you'll need to make sure the Flask end uses it correctly and/or extend the protocol as needed.
• Using any kind of sockets-based communication between programs on the same computer is somewhat inefficient unless it's very optimised for this (which my code isn't).

Would it potentially be possible to start the bells software from Flask and use stdin and stdout to talk to it? That might be a lot faster and simpler.

Regardless, a sockets-based method will definitely work if this is the best/only way to do it, it might just be a little inconvenient/overcomplicated if the bells software and the flask webserver are on the same pi.

[TerryJC]

I suppose it should be compatible with Flask's socketsio module, as long as socketsio doesn't expect any special protocol to be used (which it may well do).

I made a mistake in my post; the Flask module is flask-socketsio, not flask-sockets. I've done a bit more research and it seems as if it should be possible. However, I presume that we could load the sockettools module into the server code and use the Python delimiters within the Flask routes to access the sockets running on the other Pi (see below).

However, this might be a somewhat cumbersome way to talk to the bells software from Flask because:

• Flask is running on Bells Pi (?) so a simpler form of inter-process communication such as pipes can probably be used.
  • If this is not the case then the rest of my thoughts here are totally useless

Actually the Webserver code will be running on the Music Pi because that is the Pi 3 so it has more cores, etc.

I was thinking about using inter-process communication between the Music Pi software and the Webserver. However the Bells Pi needs to be monitored and controlled by the Webserver on the Music Pi so some form of network based comms is needed here. This was behind my earlier query about using a database (as per the way we do it in the River System).

If we use inter-process communication between the Webserver and the MP3 Player on the same Pi, then it may be that we could run a database on the Music Pi, which the Bells Pi has access to. That may be 'good enough' since the traffic will be low and there are no commands etc that need to have an instant response in the same way as the Music Player.

[*]My Sockets class expects the other end to use a specific (though simple) protocol, so you'll need to make sure the Flask end uses it correctly and/or extend the protocol as needed.[*]Using any kind of sockets-based communication between programs on the same computer is somewhat inefficient unless it's very optimised for this (which my code isn't).[/list]

Would it potentially be possible to start the bells software from Flask and use stdin and stdout to talk to it? That might be a lot faster and simpler.

It would but it would have to be the Music software started from the Webserver on the Music Pi.

Regardless, a sockets-based method will definitely work if this is the best/only way to do it, it might just be a little inconvenient/overcomplicated if the bells software and the flask webserver are on the same pi.

Agreed.

[hamishmb]

I made a mistake in my post; the Flask module is flask-socketsio, not flask-sockets. I've done a bit more research and it seems as if it should be possible. However, I presume that we could load the sockettools module into the server code and use the Python delimiters within the Flask routes to access the sockets running on the other Pi (see below).

I think this is probably more ideal.

Actually the Webserver code will be running on the Music Pi because that is the Pi 3 so it has more cores, etc.

My bad. Replace "Bells Pi" with "Music Pi" for everything I said then

I was thinking about using inter-process communication between the Music Pi software and the Webserver. However the Bells Pi needs to be monitored and controlled by the Webserver on the Music Pi so some form of network based comms is needed here. This was behind my earlier query about using a database (as per the way we do it in the River System).

If we use inter-process communication between the Webserver and the MP3 Player on the same Pi, then it may be that we could run a database on the Music Pi, which the Bells Pi has access to. That may be 'good enough' since the traffic will be low and there are no commands etc that need to have an instant response in the same way as the Music Player.

I feel it might be better to have the Webserver use IPC to talk to the software on Music Pi, and then that can forward any required messages to Bells Pi using sockettools.py. Does that seem reasonable? The idea there is that Bells Pi would be controlled the same way regardless of whether the command is coming from Music Pi or the web interface.

It would but it would have to be the Music software started from the Webserver on the Music Pi.

Of course

[TerryJC]

After yesterday evening's discussion about this topic at the online LUG Meeting, I think it's worth re-iterating a summary of what was said. Building on the above discussion, it was agreed that:

• The Music Pi, as the main server, will be configured as follows:
  • It will run the main music MP3 Player.
  • It will be configured as a sockets server using the socketstools module from the River System.
  • It will host the control and monitoring Webserver for both the Music Pi and the Bells Pi.
  • Communication between the Webserver and the music software will use inter-process communication.
  • Communication between the music software and the Bells software (on Bells Pi), will use sockets code.
  • The music software will maintain a list containing the status of all the functions in both Pis.
  • The status list will be passed to the Webserver when required.
  • On receipt of control messages for the MP3 Player from the Webserver it will invoke the appropriate functions.
  • On receipt of control messages for the Bells Player from the Webserver it will pass them to the Bells Pi using sockets code.
• The Bells Pi will be configured as follows:
  • It will run the bells MP3 Player.
  • It will be a sockets client and will provide status information to the Music software when required.
  • On receipt of control messages for the Bells Player from the Music Pi it will invoke the appropriate functions.

If there are any comments on this, please let me know.

I will now study the methods available to achieve inter-process communication between the Webserver, (built using Flask) and the music player software written in Python.

[hamishmb]

Sounds right to me.

[TerryJC]

I will now study the methods available to achieve inter-process communication between the Webserver, (built using Flask) and the music player software written in Python.

Well. I've now spent several days looking at various ways to implement IPC between the Webserver and the Music Player software and have reached several conclusions:

1. They all add quite a bit of complexity to the code base. My studies haven't been particularly useful because all the Tutorials and discussions that I've found assume a degree of understanding of Computer Science that I don't possess. (This isn't particularly surprising because this sort of thing is fairly advanced. For example, most contributors to the Raspberry Pi Forums on this topic, tend to talk in terms of what can be done, not how to do it.
   The how is left to the reader...)
2. I asked the question on the Raspberry Pi Forums and got very few responses. The one detailed answer I got suggested using Remote Procedure Calls with Apache Thrift. This was initially quite attractive to me because if I can simply call a function in the Music software from the Webserver, I would get the answer directly. However, despite the detailed response, I'm finding the going heavy.
3. One response on the RPi Forums suggested that if we were already using sockets code between the two Pis, then we should continue to use it between the two programs on the Music Pi. His point was that despite to networking code being in the loop, the response should be pretty quick.

Hamish,

What do you think? I've always been a pretty KISS sort of engineer and reusing the sockets code everywhere would be pretty KISS. It also occurred to me that any delays introduced by the sockets code between the Webserver and the Music software, would be minimal compared to the delays between the User's device and the Webserver.

Either way, I'll need a solution to the sockets problem that I raised in viewtopic.php?p=4618#p4618.

[hamishmb]

Just fixed that problem for you

Okay then, if IPC is complicated/difficult to do I agree it probably isn't worth doing. Using the sockets code is probably fine. Hopefully you can also use sockettools.py (it may have to be installed under /usr/lib/python3.x/site-packages/ for Flask to pick it up, depending on how imports from Flask work).

[TerryJC]

Just fixed that problem for you

Yes. Thanks.

Okay then, if IPC is complicated/difficult to do I agree it probably isn't worth doing. Using the sockets code is probably fine. Hopefully you can also use sockettools.py (it may have to be installed under /usr/lib/python3.x/site-packages/ for Flask to pick it up, depending on how imports from Flask work).

I didn't actually say it was complicated, just that it was complicated for me.

Once I've got sockettools working in the main programs, I'll implement it in the Webserver. One advantage will be that I should be able to get the data direct from Bells Pi without using Music Pi as a staging post.