Summer / Autumn 2017 Prototype Deployment

[TerryJC]

All,

I popped into the WMT earlier this afternoon and took a copy of all the files written since we deployed, see attached.

The first thing that hits you is that the log files are enormous! (They are far bigger than the log files on any of my Linux boxes ) I think that we ought to consider only writing to the log when something changes (that would mean only a few lines in the Float_Readings.txt file instead of over 4000 ) Secondly, there is a lot of repetition within the log files. For example:

Code: Select all

04/10/2017 11:41:46 AM - River System Control Software 0.9.1 - DEBUG: Sockets.read_pending_messages(): Waiting for data...
04/10/2017 11:41:47 AM - River System Control Software 0.9.1 - DEBUG: Sockets.send_pending_messages(): Sending any pending messages...
04/10/2017 11:41:47 AM - River System Control Software 0.9.1 - DEBUG: Sockets.send_pending_messages(): Done.
04/10/2017 11:41:47 AM - River System Control Software 0.9.1 - DEBUG: Sockets.read_pending_messages():

gets repeated many times and so does:

Code: Select all

Attempting to read from socket...
04/10/2017 11:43:31 AM - River System Control Software 0.9.1 - DEBUG: Sockets.read_pending_messages(): Waiting for data...
04/10/2017 11:43:32 AM - River System Control Software 0.9.1 - DEBUG: Sockets.send_pending_messages(): Sending any pending messages...
04/10/2017 11:43:32 AM - River System Control Software 0.9.1 - DEBUG: Sockets.send_pending_messages(): Done.
04/10/2017 11:43:32 AM - River System Control Software 0.9.1 - DEBUG: Sockets.read_pending_messages():

Could we cut down on this a bit?

Finally, the total size on disc of all the files is around half a gig, but compressed they come down to 10.8 MB. We probably need to think about compressing each file every week or so and starting a new one (as is done in /var/log).

Other than that the system does seem to be working; there are readings between 400 and 800 mm in the Levels file, so that's good.

One other thing. Hamish, I think that when you set up the software in .bashrc to autostart, you forgot this:

Code: Select all

if [ $(tty) == /dev/tty1 ]; then
	cd rivercontrolsystem-master
	./master.py -f <filename>
fi

and:

Code: Select all

if [ $(tty) == /dev/tty1 ]; then
	rdate -v 192.168.0.2
	cd rivercontrolsystem-master
	./universal_standalone_monitor.py -f <filename>
fi

because the software restarts when I SSH into the two Pis. We'll need to add that to the two .bashrc files on Tuesday.

[hamishmb]

Hi,

Wow! Okay, yep we can cut this down, I just need to disable debug logging mode, that should fix it

I forgot about that, will change it soon then, and log rotation sounds like a good idea.

Hamish

[TerryJC]

Hamish,

If you don't think we need to keep the logs, then we probably don't need to compress the files. We can talk about this tomorrow morning.

[hamishmb]

Noted in the meetings forum.

[hamishmb]

Hi everyone,

Attached are the latest readings and logs from our deployment. It mostly stays around 500mm, so feel free to skip to the end where you can see that Penri and I turned the valve, and we had around a 50% chance that the probe picked the change up

However, the software is working fine, and we know what's wrong with the probe We know our network connection is okay as well - there are no recorded dropouts, and all messages appeared when expected.

Much better with debugging off, but there are still changes to make as we discussed

Hamish

[Penri]

Hello

I've just spent some time analysing the logs and writing some observations but for some reason the forum timed out and I lost what was written, however, the gist was:

Can the format of the logs be made basically the same so it's easier to compare one with the other, I like the way you formatted the rivercontrolsystem one.
The messages in rivercontrolsystem are a little misleading, if you follow the sequence from the pump turning on, there is no turn pump off message, yet it must be happening because the water level measurement shows the level decending through 500 to 400 and then going back to >600 again.

I agree with your comments on the general operation of the S/W and on the probes reliability too.


Hwyl

Penri

[hamishmb]

Hi,

1) Yep I can do that, it's on the list

2) Doing it now

Okay, so I guess we need the new probe before we open the valve when we aren't there, but we can have it running to store rainwater at the moment.
Hamish

[hamishmb]

Hi,

Okay Penri, I have updated the program to fix the problem you mentioned about the pump.

Terry, I have changed the way the program reports new readings as you suggested with the '.'s. However, the logger doesn't support logging without a newline to it will look different in the log file. This should be fine for now, but later one we should try to engineer a solution or come up with another idea.

Hamish

[Penri]

Hello

Thank you Hamish.

A quick sit rep on the river system today, butts almost full, sump within acceptable range. Butts level Float switch not floating at the required level, even with the addition of the spare stand-offs, I am back at WMT late afternoon so will add a couple more standoff and see if that does the trick.

New magnetic probe is built but failed during testing (my fault), I'm awaiting some new hall effect switches to correct the issue plus working on an improved float design to accommodate the ring magnet.

Hwyl

Penri

[TerryJC]

Terry, I have changed the way the program reports new readings as you suggested with the '.'s. However, the logger doesn't support logging without a newline to it will look different in the log file. This should be fine for now, but later one we should try to engineer a solution or come up with another idea.

That will be fine. The idea was to minimise the size of the file, but more importantly to make it easier to read.

This should do the trick nicely. If there are no changes for three hours (say), then the file will have about thirty or so lines containing only dots and then the new reading and so on. The number of dots will convey how stable the level is.