Investigation of Level Sensor Anomalies

Acts as an interface between the other forums here. Used to coordinate overall direction of the project.
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Investigation of Level Sensor Anomalies

Post by TerryJC »

To be honest, I'm not sure what is going on at the moment.

When I started the previous test, SumpPi was started after SButtsPi. We always knew that SumpPi needed to be up so that it could connect to the peers when they came up. That's why we put a 30 s delay into the script in rc.local on the remote Pis, just before sending the rdate command, to ensure that SumpPi was well and truly up before trying to do anything.

I was aware of this, but thought that it would have no effect on the performance of SButtsPi, since it would simply run standalone.

I now want to confirm that by starting the two devices together and ensuring that they saw each other. Unfortunately, I seem to have mixed results with this. SButtsPi clearly can see SumpPi, because the clock was set by rdate. However, SumpPi doesn't seem to be able to get a connection from SButtsPi after main.py has started. Here is what it thinks:

Code: Select all

tail rivercontrolsystem.log 
22/06/2019 09:27:03 AM - Tools.sockettools - INFO: Sockets._connect_socket(): Attempting to connect to the requested socket...
22/06/2019 09:27:03 AM - Tools.sockettools - INFO: Sockets._create_socket(): Creating the socket...
22/06/2019 09:27:03 AM - Tools.sockettools - INFO: Sockets._create_socket(): Done!
22/06/2019 09:27:03 AM - Tools.sockettools - INFO: Sockets._connect_socket(): Attempting to connect to the requested socket...
22/06/2019 09:27:03 AM - Tools.sockettools - INFO: Sockets._create_socket(): Creating the socket...
22/06/2019 09:27:03 AM - Tools.sockettools - INFO: Sockets._create_socket(): Done!
22/06/2019 09:27:03 AM - Tools.sockettools - INFO: Sockets._connect_socket(): Attempting to connect to the requested socket...
22/06/2019 09:27:03 AM - River System Control Software - INFO: Starting to take readings...
22/06/2019 09:27:03 AM - River System Control Software - INFO: Waiting for client(s) to connect...
22/06/2019 09:28:16 AM - Tools.sockettools - INFO: Sockets._connect_socket(): Done!
There is no message to say it has connected to peers.

You've seen the version of config.py that I'm using on SButtsPi. This is what I get from SumPi when I cat it:

Code: Select all

   #Settings for the G6 site (client pi behind the stage).
    "G6":
        {
            "ID": "G6",

            #Local probes.
            "Probes":
                {

                    "G6:M0":
                    {
                        "Type":             "Hall Effect Probe2",
                        "ID":               "G6:M0",
                        "Name":             "Stage Butts Probe",
                        "Class":            Tools.deviceobjects.HallEffectProbe2,
                        "HighLimits":       (0.07, 0.17, 0.35, 0.56, 0.73, 0.92, 1.22, 1.54, 2.1, 2.45),
                        "LowLimits":        (0.05, 0.15, 0.33, 0.53, 0.7, 0.88, 1.18, 1.5, 2, 2.4),
                        "Depths100s":       (0, 100, 200, 300, 400, 500, 600, 700, 800, 900),
                        "Depths25s":        (25, 125, 225, 325, 425, 525, 625, 725, 825, 925),
                        "Depths50s":        (50, 150, 250, 350, 450, 550, 650, 750, 850, 950),
                        "Depths75s":        (75, 175, 275, 375, 475, 575, 675, 775, 875, 975),
                        "Default Interval": 10
                    },

                    "G6:FS0":
                    {
                        "Type": "Float Switch",
                        "ID":   "G6:FS0",
                        "Name": "Stage Butts Switch",
                        "Class": Tools.deviceobjects.FloatSwitch,
                        "Pins":  (8),
                        "Default Interval": 30
                    }
                },

            "ServerAddress": "192.168.0.2",
            "ServerPort": 30006,
            "ServerName": "SumpPi"
        },
I can see nothing wrong with that (or any difference to the SButtsPi copy, so why isn't it connecting?

I'm still monitoring both Pis and CPU usage is much the same after an uptime of 1 hour 17 minutes.
Terry
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Investigation of Level Sensor Anomalies

Post by TerryJC »

TerryJC wrote: 22/06/2019, 11:45There is no message to say it has connected to peers.
I've just spotted that the screen messages on the monitor connected to SButtsPi says:

Code: Select all

Connected to peer (SumpPi).
but as related above, SumpPi is not reporting the connection.

I've also noticed that the readings directory on SumPi contains no results for G6.

Any ideas?

CPU usage is still low.
Terry
hamishmb
Posts: 1891
Joined: 16/05/2017, 16:41

Re: Investigation of Level Sensor Anomalies

Post by hamishmb »

What about at the top of config.py on sumppi, is the sockets configuration there? This is weird because it's working in a Debian test wmt-sryle network I created with virtual machines, and it works at WMT. I have to start my shift now, but I will reply again after 7:00 or so. If it says connected on one pi, it'd be weird for it not to be connected at the other end.

In main.py, does it set up Sockets monitor classes for the probes on the G6 site?

Hamish
Hamish
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Investigation of Level Sensor Anomalies

Post by TerryJC »

Yes. The sockets configuration there:

Code: Select all

SITE_SETTINGS = {

    #Settings for the SUMP site (master pi).
    "SUMP":
        {
            "ID": "SUMP",

            #Sockets to host.
            "Sockets":
                {

                    #For connection to Wendy Street Butts Pi.
                    "Wendy Street Buttspi Socket":
                        {
                            "ID":           "SOCK4",
                            "Name":         "Wendy Street Buttspi Socket",
                            "PortNumber":   30004
                        },

                    #For connection to Wendy Street Stage Pi.
                    "Wendy Street Stagepi Socket":
                        {
                            "ID":           "SOCK6",
                            "Name":         "Wendy Street Stagepi Socket",
                            "PortNumber":   30006
                        },

                    #For connection to Wendy Street Butts Pi Gate Valve.
                    "Wendy Street Butts Gate Valve Socket":
                        {
                            "ID":           "SOCK14",
                            "Name":         "Wendy Street Butts Gate Valve V4 Socket",
                            "PortNumber":   30014
                        }
                },
I'm not sure how to check that it sets up Sockets monitor classes for the probes on the G6 site.

I'm going to stop SButtsPi in an hour or two and then restart it with "G4" as the ID, to see if that works.

CPU usage is still low.
Terry
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Investigation of Level Sensor Anomalies

Post by TerryJC »

TerryJC wrote: 22/06/2019, 14:03I'm going to stop SButtsPi in an hour or two and then restart it with "G4" as the ID, to see if that works.
I get almost the same behaviour when I identify the remote as G4. There is still no message in rivercontrolsystem.log to say that the peer(s) have connected. By the same token universalmonitor.log doesn't say anything about connecting to SumpPi; it just says;

Code: Select all

INFO: Sockets._connect_plug(): Done!.
However, the screen does say that it's connected to SumpPi, exactly the same as when the device is identified as G6.

Where it is different though is that SumpPi now generates Results files prefixed with G4.

I intend to change it back to see if i can spot the difference.
Terry
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Investigation of Level Sensor Anomalies

Post by TerryJC »

Ok. I changed the ID back to "G6", cleared the files from the readings and logs directories of both Pis and rebooted.

Still no peers connections in the log files, but here's the thing. The Readings directory contains sensor files for G4 and not G6. So it looks like the system is misconfigured somehow, with the following anomalies:
  1. SumpPi is not identifying that it has connected to the peer in its log file (could this be because it is not seeing valid readings from the sensor?).
  2. However, SButtsPi believes it has connected.
  3. SButtsPi is recording valid levels for the Hall Effect Sensor in its readings director.
  4. SumpPi thinks that it is connecting to G4, even though it has apparently not managed to achieve it. The Results files only have the column headings and not the readings.
I can't do much more today, so I'll leave it overnight.

SumpPi CPU currently between 16 and 22%
SButtsPi CPU currently between 15 and 20%
Terry
hamishmb
Posts: 1891
Joined: 16/05/2017, 16:41

Re: Investigation of Level Sensor Anomalies

Post by hamishmb »

Yeah, I think the SocketsMonitors aren't being set up then. The messages from the Sockets class don't say which peer they have connected to - I should change that. This does sound like a setup issue at your end, could you post the full logs for me?
Hamish
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Investigation of Level Sensor Anomalies

Post by TerryJC »

See attached. I took these about 5 minutes ago.
Attachments
SSumpPi_Testing.zip
(4.62 KiB) Downloaded 99 times
Terry
hamishmb
Posts: 1891
Joined: 16/05/2017, 16:41

Re: Investigation of Level Sensor Anomalies

Post by hamishmb »

That's weird, there should definitely be more stuff than that in the Sump Pi log, it looks like most of the startup sequence didn't work.
Hamish
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Investigation of Level Sensor Anomalies

Post by TerryJC »

Hamish,

Just to be absolutely certain, I copied the contents of my git repository on to both SD Cards and restarted the system. The result was exactly the same.

I did a find on my git repository to see which files had been changed since the system was last changed and known to work (15th May when Wendy Butts Pi was integrated). Here is the result:

Code: Select all

terry@OptiPlex:~/Development/Wimborne_Model_Town/River_System/Code/git_repository/rivercontrolsystem$ find -newermt 2019-05-15
.
./gate_valve.py
./config.py
./tests
./tests/gate_valve.py
./tests/test_socketsclass_client.py
./tests/hall_effect_probe2.py
./tests/test_solidstate_relay.py
./tests/test_socketsclass_server.py
./tests/level_shifter_test.py
./.git
./.git/FETCH_HEAD
./.git/objects
./.git/objects/03
./.git/objects/03/c9d4d7e3706c85078fe861efa9d383d8bc2212
./.git/objects/50
./.git/objects/50/d844b15404582d56fa7e8a132e7b65b632bbb5
./.git/objects/c4
./.git/objects/c4/a498d2d3f9c8bea342d83a5f0fbcbc9edd4541
./.git/objects/b3
./.git/objects/b3/6bd30da9da8c9db307190e3391a039989abf84
./.git/objects/ab/d0627b0feae9571f97ccac209fbcb1b21d383f
./.git/objects/2c
./.git/objects/2c/b03bf883218b96a411c603e90cbebbc9e00283
./.git/objects/3f
./.git/objects/3f/9408222cdb080ac2ca02a20c658e18f4423c7e
./.git/objects/94
./.git/objects/94/f8fc11a5bb06d56cfeaeac93b0a9ac4544cb66
./.git/objects/99
./.git/objects/99/1061827cb57e537d442848fbfc4cf825d9213c
./.git/objects/f6
./.git/objects/f6/41cffe895598e577c5bcdee55a466c7e042b3c
./.git/objects/f4
./.git/objects/f4/b8ca0428dd0ffbab470b864bffeb0e72c2efef
./.git/ORIG_HEAD
./.git/index
./.git/COMMIT_EDITMSG
./.git/logs/HEAD
./.git/logs/refs/remotes/origin/master
./.git/logs/refs/heads/master
./.git/refs/remotes/origin
./.git/refs/remotes/origin/master
./.git/refs/heads
./.git/refs/heads/master
AFAICT, only config.py is relevant to what I am trying to do. I can't see anything wrong with it, but I've attached it here because you might spot the problem. It should be identical to the version that I pushed two weeks ago when I added the configuration for Stage Butts.

Other than there being something amiss in there I can't see what might be going wrong. AFAICT, the hardware is working OK because I can SSH into both Pi's from my desktop machine. Also, SButtsPi's clock is updated by rdate, so there is communication going on outside of the Software Framework.

Any other ideas? Clearly I can't deploy Stage Butts Pi until this is resolved (along with the lock-ups in level measurement that we seem to be getting on site)>
Attachments
config.py
(12.35 KiB) Downloaded 98 times
Terry
Post Reply