Network issues investigation

[TerryJC]

Did you think about my suggestion back in August; to add extra logging of CPU Usage etc (viewtopic.php?f=13&t=187&p=2697&hilit=CPU+Usage#p2697). It might give us a few more clues as to the events leading up to the freeze. You could implement it so it only happens in debug mode.

[hamishmb]

I completely forgot about that actually. Yeah, we could do that. I'll add that when I have a little time and headspace.

[TerryJC]

Good.

[TerryJC]

When we re-installed the old Wendy Butts probe (see viewtopic.php?f=15&t=193) after Penri had repaired it, we had more connectivity problems:

1. When Penri arrived Wendy Butts Pi wasn't responding to his iPhone App. Everything else seemed OK.
2. After the work was done we tried a number of reboots and also powered the remotes down (we left Sump Pi up). Wendy Butts then started responding, but tailing the logs revealed that the Wendy Butts Gate Valve wasn't connecting to the Sump (see attached).
3. After more reboots, we still couldn't see any connections, so we power cycled everything. Wendy Butts Pi and Stage Butts Pi were behaving normally, but the Gate Valve still wasn't connecting.

Since I could tail the Gate Valve log and 'ip a' showed the normal response, I couldn't (and still can't) see how this was the hardware. I checked config.py for the Sump and could see nothing wrong. The Gate Valve has worked since we last changed the code, so I can't think of anything that could account for this.

[hamishmb]

Ahh that's a pain :/

The gate valve has been working previosuly recently, so that's kinda perplexing. The gate valve and Wendy butta are both on Raspbian 10...

However, Raspbian 10 is also what I've been testing with for a long time. Maybe I should do a longer test over a week or something like that? I don't really like leaving stuff on because of power usage, but it might help us determine the cause of the problem.

Maybe there's a networking bug or USB issue in older versions of Raspbian 10? We do need to add the mysqlclient python module anyway, so perhaps we should do the latest updates and see if that helps.

[TerryJC]

Hamish,

I still have a few SD Cards. so I could create replacements for the Wendy Gate Valve and Wendy Pi and swap them on Tuesday. However, Penri and I are planning to change the connectors on those devices on Tuesday morning, so maybe we should put off one or the other exercise so that when (if) the problem is solved, we have some idea what solved it.

Alternatively, we could build the SD Cards using Raspbian 9, so that all the devices are running the same version of Raspbian.

[hamishmb]

Yes, I think doing one thing at a time is probably a good plan.

I'd rather not to Raspbian 9, because I haven't tested on that for a while. It might also be a good idea to update Raspbian with the latest updates on the other pis so they're running a more similar software stack overall.

[TerryJC]

So do we want to change the connectors first or build SD Cards with the latest version of Raspbian?

[hamishmb]

I'd probably do the connectors first, but it's your choice. I'm also making some software changes today that will give us more debugging information.

[TerryJC]

OK. Connectors it is.