Maintenance

This Forum is to cover work carried out to re-engineer the existing Minster Bells and MP3 Player System, which has a number of issues caused by the original design.
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Maintenance

Post by TerryJC »

Today's log file us quite revealing:

Code: Select all

pi@minster-music:~/logs $ cat Minster-Music.log
09/08/2021 00:51:00 - Minster Software - INFO: 192.168.0.30 Up
09/08/2021 00:51:00 - Minster Software - INFO: 192.168.0.253 Up
09/08/2021 00:51:00 - Minster Software - INFO: 192.168.0.254 Up
09/08/2021 00:51:00 - Minster Software - INFO: WiFi Up
09/08/2021 01:51:00 - Minster Software - INFO: 192.168.0.30 Up
09/08/2021 01:51:00 - Minster Software - INFO: 192.168.0.253 Up
09/08/2021 01:51:00 - Minster Software - INFO: 192.168.0.254 Up
09/08/2021 01:51:00 - Minster Software - INFO: WiFi Up
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.30 Down
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.253 Down
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.254 Down
09/08/2021 02:51:00 - Minster Software - INFO: WiFi Down - Restarting WiFi
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.30 Down
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.253 Down
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.254 Down
09/08/2021 02:51:00 - Minster Software - INFO: WiFi Down - Restarting WiFi
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.30 Down
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.253 Down
09/08/2021 02:51:00 - Minster Software - INFO: 192.168.0.254 Down
09/08/2021 02:51:00 - Minster Software - INFO: Unable to Restart WiFi
09/08/2021 03:51:00 - Minster Software - INFO: 192.168.0.30 Up
09/08/2021 03:51:00 - Minster Software - INFO: 192.168.0.253 Up
09/08/2021 03:51:00 - Minster Software - INFO: 192.168.0.254 Up
09/08/2021 03:51:00 - Minster Software - INFO: WiFi Up
09/08/2021 04:51:00 - Minster Software - INFO: 192.168.0.30 Up
09/08/2021 04:51:00 - Minster Software - INFO: 192.168.0.253 Up
09/08/2021 04:51:00 - Minster Software - INFO: 192.168.0.254 Up
09/08/2021 04:51:00 - Minster Software - INFO: WiFi Up
09/08/2021 05:51:00 - Minster Software - INFO: 192.168.0.30 Up
09/08/2021 05:51:00 - Minster Software - INFO: 192.168.0.253 Up
09/08/2021 05:51:00 - Minster Software - INFO: 192.168.0.254 Up
09/08/2021 05:51:00 - Minster Software - INFO: WiFi Up
09/08/2021 06:51:00 - Minster Software - INFO: 192.168.0.30 Up
09/08/2021 06:51:00 - Minster Software - INFO: 192.168.0.253 Up
09/08/2021 06:51:00 - Minster Software - INFO: 192.168.0.254 Up
09/08/2021 06:51:00 - Minster Software - INFO: WiFi Up
The first entry of the day at nine minutes to one shows that the WiFi was up. At nine minutes to two it was still up, but at nine minutes to three it was down. The code tried to restart the WiFi but failed until the next attempt at nine minutes to four, when the WiFi was up again. My theory for this is that the heavy rain overnight (I heard it at around 2 am) caused the WiFi to lose the signal. The code to stop /restart the interface all failed to enable a reconnect because it was still raining. However, the stop /restart code resets the stats that the driver / adaptor use to take the WiFi down, so when the rain stopped the re connection occurred (sometime between 0251 and 0351) and communications were restored.

All this leads me to a further theory:
  1. When the signal is lost it will reconnect by design; just as a laptop does when you bring it back into range of a known AP.
  2. The adaptor/driver collects the number of dropouts and stores the count as 'Invalid misc: ". For whatever reason (poor sensitivity, screening from trees. etc) this adaptor disconnects when the Invalid misc: count become excessive. It then stops reconnecting when (if) conditions improve.
I shall monitor this for a few more days and then remove the code that triggers the daily 9 am reboot.
Terry
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Maintenance

Post by TerryJC »

No rain last night and no WiFi disconnects! The forecast is for little rain over the coming days so I have commented out the 9 am reboot code. This means that the WiFi will remain as is until (presumably) it builds a backlog of dropouts and disconnects. The stop /restart code should then trigger a reconnect.

I'll repost here once that has happened (or not as the case may be).
Terry
Penri
Posts: 1284
Joined: 18/05/2017, 21:28

Re: Maintenance

Post by Penri »

I noticed this morning that the gardeners have trimmed back the branches / leaves around the gazebo antenna but are yet to tackle the hedge behind the bus shelter.

The gazebo antenna signal path to the Minster is clearer but there are couple of trees away from the perimeter that may cause shading that's going to be difficult to do anything about.
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Maintenance

Post by TerryJC »

Well in some ways I hope that the signal is degraded enough to cause a drop-out. Otherwise I may have to wait for the next heavy rain to find out if my new code is a real solution.

I just checked the signal using iwconfig and the strength and link quality don't seem any better. The current Invalid misc: count is 72, but of course the system was rebooted at approximately 0930 this morning after I added the code change.
Terry
Penri
Posts: 1284
Joined: 18/05/2017, 21:28

Re: Maintenance

Post by Penri »

Terry

Last night I was at WMT to support an event and tried to log into the Minster to turn the music / bells on but could not.

I didn't contact you as it wasn't an issue but is that that you'd expect, does the system "go to sleep" after close play?

I tried to connect several time from around 17:20'ish to around 18:15 perhaps.

Penri
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Maintenance

Post by TerryJC »

Penri,

Re our telcon just now. I had detected a problem yesterday evening, but didn't think that anything could be done after hours....

Since you were able to connect to the WiFi Antenna, but not log on to the Minster, I suspect that there may be a routing problem at the Big Switch.
Terry
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Maintenance

Post by TerryJC »

OK. It seems that I can't ping hosts through the VPN, but I can ping them if I log into another host inside the network. So that was another red herring.

When I examined the logs I found that the WiFi had gone down overnight and my reconnect code hadn't worked. I have now modified the code to trigger a reboot when the WiFi disconnects, but only if the event is detected outside Opening Hours. If the problem occurs inside Opening Hours, the reboot will occur at the next WiFi test funcyion failure after the site closes; eg 1751 or 2351 if Extended Hours was triggered.

I waited until after the site closed and then rebooted the Music Pi to load my modified code.

I'll keep you posted.
Terry
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Maintenance

Post by TerryJC »

Penri wrote: 10/08/2021, 12:41I noticed this morning that the gardeners have trimmed back the branches / leaves around the gazebo antenna but are yet to tackle the hedge behind the bus shelter.

The gazebo antenna signal path to the Minster is clearer but there are couple of trees away from the perimeter that may cause shading that's going to be difficult to do anything about.
As mentioned back then, the cutting back of the trees has improved of the signal a fair bit. The level is now 5-6 dB stronger than it was; the signal quality has improved to around 50% but the 'Invalid misc: ' is still counting up steadily (It's at around 1800 ATM).

The WiFi has now stayed up for well over 4 days (an International Record I think) which is good news, except that I've been unable to test the behaviour of my new code if the signal goes down during Opening Hours. Still we can't have everything.

I suspect if we get heavy rain the WiFi will disconnect, so I just have to hope that it occurs between 1000 and 1700!
Terry
TerryJC
Posts: 2616
Joined: 16/05/2017, 17:17

Re: Maintenance

Post by TerryJC »

The WiFi disconnected last night sometime between 2051 and 2151; that was over 6 days uptime. The code detected this, rebooted the Pi and access was restored. I'm not sure what happened between those two times weather wise, but Lynn tells me that she noticed that it was quite windy at around that time. There was also rain some time prior to that; perhaps the remaining branches were thrashing around and being wet, they attenuated the signal enough to trigger the disconnect.

I have not yet been able to test the part of the code that covers the situation where the WiFi goes down during Opening Hours, but a code inspection hasn't revealed anything wrong. I have committed the latest version and pushed it to GitLab and if any problems do emerge I'll fix them as they arise of course.

I think that is the best that I can do at present until I can strip the system down during the Winter Maintenance period and put in a better WiFi Adaptor or convert the system to Ethernet.
Terry
Penri
Posts: 1284
Joined: 18/05/2017, 21:28

Re: Maintenance

Post by Penri »

The music and bells have worked faultlessly throughout, as was expected, but its been great to confirm have it confirmed in practise.

When you have the opportunity I'd appreciate it if you could turn the volume up on the music, from comments made to me and my own observations a number, perhaps the majority, of the tracks are quite quiet. I don't have any quantitative measurements to say if the sound level is lower or perhaps the background noise level higher, it's sometimes difficult to hear unless you stick your head through the Minsters door.
I believe I remember and volume adjustment is via a SIM card change.

Penri
Post Reply