System Software Design

[hamishmb]

Following the style Terry used here, here is our work-in-progress system software design documentation.

It's enormous, so make sure you have time before you start reading

Programming language and development tools

Language: Python 3 (3.4+ only).

Tools I've used when developing this software are:

• pylint (pylint 3)
  • Pylint is used to help identify faults and room for improvement in Python code.
  • Make sure you use Pylint 3 for Python 3.
• A text editor, Xed (similar to Gedit).
  • This is a good text editor that uses syntax highlighting for easy reading.
  • It is by no means required; you could instead use JEdit, or Notepad++, for example.
• Git, with github.com.
  • The github repository for this project is at https://github.com/wmtprojectsteam/rivercontrolsystem.
  • To learn how to use git, have a look at https://githowto.com/

Design Philosophy

I have taken a very object-oriented design philosophy for this project, because of the advantages it can bring us later as the project grows and we add functionality.

As such, I have created a framework which we can build our software on.

• This framework is designed to make use of complicated software features, like sockets, simple.
• This framework is flexible and extendible, such that more features can be added to it without disrupting existing functionality.
• The framework is capable of automatic error handling and self-monitoring, making use of advanced concepts such as threads to dynamically manage and fix problems it encounters, with no repercussions for higher layers of the software suite.

Components and files

There are a lot of components in this framework, and more will be created as it grows and matures into what will become our production software.

Starting from the start, there are various files in this project. The three main files are (wait for it) main.py, universal_standalone_monitor.py, and universal_standalone_monitor_config.py.

main.py

This is the main part of the control software, and it currently manages balancing water between the butts and the sump using a resistance probe, and a solid state relay to control the butts pump. This software runs on sumppi. It communicates with buttspi over the network to gather float switch readings.

universal_standalone_monitor.py

This file is run to provide the single monitors that communicate with the master pi (in this case sumppi). On buttpi, this software is run in "Float Switch" mode, and so it configures itself to take readings from a float switch, connected using the settings found in universal_standalone_monitor_config.py. It then sends these readings over the network to the master pi (sumppi).

universal_standalone_monitor_config.py

This file holds the configuration for the universal standalone monitor. This is all pre-agreed stuff like pin numbers, reading intervals, and some framework-specific stuff which will be discussed below.

Tests
There are a number of basic system tests as well. Note that these are used to check the system setup, rather thn unit tests, which will be provided later on.

The other files found in the "Tools" folder are as follows:

__init__.py

Used to initialise the python package, not really a part of the framework.

coretools.py

This file holds:

• "greet_and_get_filename", A function to greet the user and get a file name to save readings to. This is used by the universal monitor and main.py
• "do_control_logic", The function that is given the readings ad control of the devices. It does all the decision-making for the main control software (main.py).

standalone_shared_functions.py

This file is a left-over from when we had multiple standalone monitors for each type of probe. Everything in here will probably end up in universal_standalone_monitor.py.

It holds:

• "usage", A usage function for the universal monitor.
• "handle_cmdline_options". This function handles all of the commandline options for the universal monitor.

sensorobjects.py

This file holds all of the classes for different device types. It makes use of class derivation to reduce code duplication. As such, all the classes have a common base, and then build on it for functionality specific to the device (e.g. a capacitive probe, or an SSR-controlled motor)

monitortools.py

This file holds the universal monitor. This is essentially a separate thread that takes readings when required and makes them available in a queue for the rest of the program to read when it desires. A separate thread is used because otherwise readings would delay execution of the rest of the code, and the rest of the code would be made much more complicated.

sockettools.py

A collection of tools, including a handler thread, to make it easier to use sockets. This was originally a port of some C++ code I was writing for my latest project, Stroodlr. This works in a very similar way to the monitor, except it is two-way communication taking place here. The handler thread handles disconnections and reconnections once started, and the socket can be used easily by pushing messages to an "outgoing" queue, and checking for messages on the "incoming" queue, using methods provided by the interface class. All of the underlying complexity and error handling is hidden from the user.

Interface classes and methods required to use the software.

Constructors, methods, and functions are discussed here with the following format:

<return-type> name(<type> arg1, <type> arg2, <type1>/<type2> arg3, <type> arg4=<default-value> [, <type> optional-arg6])

sensorobjects.py

All the constructors in this module take one argument: str name.

For example:

my_probe = FloatSwitch("butts_switch")

Classes:

• BaseDeviceClass
  • Is the base class for all of the devices.
  • Everything else here inherits from this.
  • Interface methods:
    • none set_pins(int/list pins, bool _input=True).
      • Sets the pins the device will use (from low to high if resistance probe).
    • str get_name().
      • Returns the name of the device this object is representing.
    • tuple get_pins().
      • Returns the pins of the device (from low to high if a resistance probe).
• Motor
  • Used to represent and control a motor using a solid-state relay.
  • Interface Methods:
    • none set_pwm_available(bool pwm_available, int pwm_pin).
      • Enables/Disables PWM support for this motor, and sets PWM pin. If disabling, set pin to -1.
    • bool pwm_supported().
      • Returns True if PWm supports, else False.
    • bool enable().
      • Turns the motor on. Return True if successful, else False.
    • bool disable().
      • Turns the motor off. Returns True if successful, else False.
• FloatSwitch
  • Used to represent and get readings from a float switch.
  • Interface Methods:
    • none set_active_state(bool state).
      • Sets the active state for the pin(s). True for active high, False for active low.
    • tuple get_reading().
      • Returns the state of the switch. True = triggered, else False.
      • First part of tuple is bool, 2nd str.
• Capacitive Probe
  • Used to represent and get readings from a capacitive probe.
  • Interface methods:
    • tuple get_reading().
      • Returns the frequency from the probe and status text.
      • Will eventually return level in mm, but not calibrated yet.
      • Fault detection not yet implemented so status text is always "OK".
      • First part of tuple is int, 2nd part is str.
• ResistanceProbe
  • Used to represent and get readings from a resistance probe.
  • Interface methods:
    • none set_active_state(bool state).
      • Sets the active state for the pins. True = active high, else false.
    • bool get_active_state().
      • Returns the active state for the pins. As above.
    • tuple get_reading().
      • Gets the level of water in the probe.
      • First part of tuple is an int, 2nd is a str.
      • Includes fault detection.
• HallEffectDevice.
  • Used to represent and get readings from a hall effect device (frequency as RPM only).
  • Interface classes:
    • tuple get_reading().
      • Returns the rate at which a water-wheel is rotating in RPM.
      • First part of tuple is int, 2nd is str.

sockettools.py

Constructor for Sockets:
Sockets(str the_type)

the_type *_MUST_* be "Plug" or "Socket".

A "Plug" is used with a client, a "Socket" is used with a server. With a plug, you must declare the server address using the relevant method (see below).

For example:

ServerSocket = Socket("Socket")

OR

ClientSocket = Socket("Plug")

Classes:

• Sockets
  • High level representation of a socket.
  • Used to hide complexity of underlying socket from programmer, and provide convenience classes.
  • Interface methods:
    • none set_portnumber(int port_number).
      • Sets the port number for the socket.
    • none set_server_address(str server_address).
      • Sets the server address for the socket.
      • Only required if creating a client socket, a "Plug" using my terminology.
    • none set_console_output(bool state).
      • Used to silence/reveal cmdline messages from the sockets handler.
      • If not used, we default to showing them.
    • none reset()
      • used to bring the socket back to pre-connection config.
      • Useful when closing the socket.
      • Used used internally when handling disconnection/reconnection.
    • bool is_ready()
      • Returns True if socket is ready to transmit, else false.
    • bool just_reconnected().
      • Returns True if socket just reconnected to server, else false.
    • none wait_for_handler_to_exit().
      • Makes the current thread wait until the handler has exited.
      • NOTE: Doesn't ask the handler to exit by itself.
    • bool handler_has_exited()
      • Returns true if handler has exited.
    • none start_handler()
      • Starts the handler thread and the returns.
    • none request_handler_exit.
      • Asks the handler to exit, and then returns immediately.
    • none write(str data)
      • Queues some data to be sent over the socket.
    • none send_to_peer(str data)
      • Sends data to peer, and waits for confirmation of delivery.
      • Currently not used or supported.
    • bool has_data
      • Returns True if there's data to read from the socket's incoming queue.
    • str read().
      • Returns the next element from the incoming queue.
    • none pop().
      • Removes the oldest message from the incoming queue.
• SocketHandlerThread
  • Used to handle socket r/w and connection/disconnection.
  • Used internally only.
  • Do _NOT_ call this directly.
  • Only use it through <Sockets-obj>.start_handler().

monitortools.py

Constructor for Monitor:

Monitor(str type, <probe-obj> probe, int num_readings, int reading_interval)

For example:

monitor = Monitor("Float Switch", probe_instance, 50, 300)

to monitor a float switch every 5 mins, 50 times.

OR

monitor = Monitor("Resistance Probe", another_instance_of_a_probe, 0, 600)

to monitor a resistance probe every 10 mins for ever.

Classes:

• BaseMonitorClass
  • Base class for the monitors.
  • All monitors derive from this.
  • Interface methods:
    • bool is_running().
      • Returns True if monitor is running, else False.
    • bool has_data.
      • Returns True if the monitor has new data on the queue to be read.
    • str get_reading()
      • Returns a reading from the queue, and deletes it from the queue.
    • none set_reading_interval(int interval)
      • Sets the reading interval for this monitor.
      • Takes immediate effect.
    • none request_exit([wait=False]).
      • Asks the monitor thread to exit.
      • Waits before returning if specified.
• Monitor
  • A universal monitor for all devices.
  • No interface classes.

coretools.py

Functions:

• file-obj greet_and_get_filename(str module_name[, str file_name])
  • Used to greet the user and get a filename if not specified.
  • Returns a file handle if it could open the file, otherwise exits the program.
• none do_control_logic()
  • Not documented because subject to rapid change.
  • Handles decision making and setting reading intervals.
  • Currently tuned for the was-August-now-September deployment.

Whew!

Hopefully this is what you were thinking of Terry XD

Hamish

[TerryJC]

I haven't studied this in detail yet, but it is definitely along the right lines.

We can chat about this tomorrow morning if you're in and then I'll comment on it in detail.

I think it would make sense to think about producing proper documents from this and the Requirements. Again, we'll talk about it tomorrow morning.

[TerryJC]

Hamish,

Now that the system has been deployed again, I've returned to the documentation. I have taken your design description from above and reformatted it into a Design Specification in LibreOffice. Once I've finished it, I'll post it for general comment.

In order to do this, I have worked my way through much of your code and have developed a brief description of the two algorithms that you are using. Before I use them I'd like your comments on my interpretation of the code and your thoughts on my comments that I've embedded {thus}. First the description in text form:

Master Pi Processing:
The processing sequence is as follows:

• Check arguments and output an error message if they are not correct.
• Setup and check the connection for the clients to connect to on port number 30000. {I thought you were going to change this so that we would still get the results written to the file even if the connection was lost}
• Create the Results file and open it to write.
• Create the Python objects for the devices sump_probe and butts_pump using the sensor objects class.
• Setup the devices (GPIO pins, direction and state).
• Setup the Reading Interval for the hall-effect sensor. {I would suggest that this is done in one place; eg a config file like you are using for the Butts. We also discussed taking the readings much more frequently than every 5 minutes but only writing the result if it is a new level.}
• Start taking readings from the hall-effect sensor and the float switch state reported by the buttspi. Wait until the system has synchronised.
• Enter a monitor loop. At each iteration:
  • Check for a new reading and write the value to the file and stdout.
  • If the sump water level is between 400 mm and 600 mm, do nothing.
  • If the sump water level is over 600 mm, start the Butts Pump.
  • Continue to pump water to the Butts until the sump level falls below 600 mm or the buttspi reports that the Float Switch has activated. Stop the Butts Pump.

Butts Pi Processing
The processing sequence is as follows:

• Check arguments. {I couldn't actually see what happens if the argument is wrong, although I didn't drill down into you code to far. Is there any reason why this action is done differently to sumppi?}
• Setup and check the connection for the clients to connect to on port number 30000.
• Create the Results file and open it to write. {Same comment as for the Sump}
• Create the Python objects for the Float Switch device and set up the Reading Interval from the _config file. {It would make some sense if this config file was actually located at the Master Pi and the Reading Interval sent out from one place}
• Setup the device (GPIO pin, direction and state).
• Start taking readings from the float switch.
• Enter a monitor loop. At each iteration:
  • Check for a new reading and write the value to the file and stdout.
  • Broadcast the float switch state to sumppi.

Once you are happy with this, I was intending to produce a flow diagram of this (with less detail) and put those in the document too.

All comments gratefully received

[hamishmb]

Hi,

Thanks Terry, I wasn't sure how I was going to make that document

You've pretty much got it, and here are my comments:

Master Pi:
2 - We still set the connection up and wait, but all this is done by another thread now - SocketsHandlerThread, see Tools/sockets.py if you like.

That way the connection is ready to go, but the program continues without it if it disconnects. Reconnection is handled this way too.

6. - Good idea, will add to my list.

Note that the reading interval is changed in the monitor loop too if needed.

Butts Pi:
1 - The program will stop with an Assertion error. Different because we take different arguments (fewer), and a relic from when we had monitors for each sensor type, should change it really.

4 - It is - if the interval changes at sump pi, it is sent to buttspi, but we need an interval to start with in case there's no connection - we don't want to wait for one as you said above

Hopefully that clears it up, but I just tapped it in my phone so hopefully it's not too terse

Hamish

[TerryJC]

Hamish,

Thanks for this. I'll pick this up tomorrow and you can review the full document.

[TerryJC]

Hamish,

Here is my first go at representing the Sump Pi processing in the form of a flowchart. I know and appreciate that the modern way to do this is to use UML or a state diagram, but those types of diagram are less likely to be understood by those who are not software engineers.

BTW. I used Dia to produce the original of this, but the Flowchart symbols covered by that tool don't cover the full range defined by ISO, so I've had to make do a bit.

Anyway, let me know what you think:

SumpPi_Flowchart.png (57.03 KiB) Viewed 1344 times

Once you're OK with this, I'll do one for Butts Pi.

Note that I've not attempted to show the Exception handling or any other 'housework' activities. The idea of this is only in part to document the software for future developers. My main goal was so that anyone on the team could understand what was going on without needing to read OO Python!

[Penri]

Hello

Thanks for this picture of the of the inner workings of the SumpPi S/W, very informative for a Luddite like me.

Am I right in saying that the S/W, as is, effectively tries to maintain the water level at 600mm +/- 100mm, given enough water in the butts, etc.

I'm drawing that conclusion on the assumption that a reading of 600mm is treated at "= 600mm" and that "> 600mm" is 700mm or above and "< 600mm" is 500mm and below.


Hwyl

Penri

[TerryJC]

Penri,

I'll let Hamish give the definitive answer to that, but my interpretation of the code is that the software does nothing if the level is between 400 mm and 600 mm. Below 400 mm, it sends a series of Warnings, each getting more urgent. Above 600 mm, it turns on the Butts pump until the level in the sump falls to 600 mm or the butts become full.

[hamishmb]

Hi,

Terry, that looks really good

Yep Penri, you're right. Terry, should I be doing these diagrams and suchlike, or are you happy to do them, because I appreciate that this must have taken quite a lot of your time?

Hamish

[TerryJC]

Terry, that looks really good

OK. I'll do the buttspi algorithm and add both to the Design Spec. Once that's done, I'll post it this afternoon for comments.

Terry, should I be doing these diagrams and suchlike, or are you happy to do them, because I appreciate that this must have taken quite a lot of your time?

In an engineering company there isn't usually anyone to do the documentation other than the developer, which is right and proper because it is their design.

However, this is not an engineering company and I don't mind doing them. Obviously, I need enough information from you to allow me to sort out the rest. One useful thing that came out of this is that it gave me an opportunity to review your software (at least at a high level). I've made a couple of observations, which I'll tell you about tomorrow. There's nothing horrendous; just some pointers on how to present your code.