NAS failsafe and disaster recovery options

[Penri]

Hello

I've read along with this thread with interest if little understanding of the S/W techniques and intricacies involved.

As things stand we had no budget for additional spend above what we identified and shared with Stuart some time ago. The current closure of WMT, with resulting lack of revenue, further exacerbates the issue, in fact the board of trustees have placed a moratorium on all spend apart from the most essential.

That being the case I would ask you to implement the cheapest and most straightforward solution for now, let me know what the cost are before committing to anything.

I would be interested in exploring how we could go forward with a more elegant solution in the future, especially if we could wrap it up as a new extension project or as part of one and perhaps get a grant for it; now, however, is not the time for doing that.

Hope that helps, stay safe and healthy.

Penri

[PatrickW]

I don't have a great deal of insight to contribute here regarding different database backup strategies.

However, it did occur to me that another mechanism by which backups could be taken would be by downloading a file through the staff GUI. It's probably not a very good idea, but it came to mind nevertheless. Downsides: it's manual; it's a burden on someone to remember to do it; it can't be implemented until a staff GUI is implemented. Upsides: the backup (probably) ends up in a different building from the NAS; uses existing equipment (assuming sufficient space to store backup files).

It also occurred to me that fault diagnosis after a disaster will be important. I don't think we can assume in advance that there will be a pristine set of hardware waiting for us to restore our backup onto it. Someone will probably have to determine which parts are damaged, which are fully functional and which can (potentially) be brought back to life with some magic incantations. I think part of the disaster recovery plan needs to be to make sure that diagnosis is as easy as possible, or to render it unnecessary.

If the NAS hardware were to fail, how easy would it be for an unskilled volunteer to replace it? A hard drive replacement might not be too hard, since that might be a standard component*. But suppose the NAS' power supply fails or its main board gets zapped or succumbs to corrosion: can a replacement for that be sourced? If not, are the NAS installation instructions general enough that they can apply to whatever hardware is readily available, or are they very closely tied to this specific NAS hardware? (I realise that the final instructions don't yet exist. These are rhetorical questions.)

Some disasters could result in loss of not only the NAS but also the building that houses it. In that scenario, we would not be able to instate a replacement NAS immediately in the same location. Hopefully that can be regarded as a relatively extreme situation, so we might accept more downtime for that level of devastation.

*Though, older NAS hardware can have some compatibility limitations for hard drives. I have a NAS from circa 2006 that cannot take a drive larger than 4TB, nor one without 512B sectors (4kB is common now).

[TerryJC]

However, it did occur to me that another mechanism by which backups could be taken would be by downloading a file through the staff GUI. It's probably not a very good idea, but it came to mind nevertheless. Downsides: it's manual; it's a burden on someone to remember to do it; it can't be implemented until a staff GUI is implemented. Upsides: the backup (probably) ends up in a different building from the NAS; uses existing equipment (assuming sufficient space to store backup files).

The problem with mechanical backups is that they rely on someone being ready, willing and able to do them.

A case in point. Until recently, I've been in the habit of copying all the results and log files using FTP. Currently of course, that is impractical.

There is no need to store the backups in another building, I simply zipped them up and uploaded them to this forum.

It also occurred to me that fault diagnosis after a disaster will be important. I don't think we can assume in advance that there will be a pristine set of hardware waiting for us to restore our backup onto it. Someone will probably have to determine which parts are damaged, which are fully functional and which can (potentially) be brought back to life with some magic incantations. I think part of the disaster recovery plan needs to be to make sure that diagnosis is as easy as possible, or to render it unnecessary.

If the NAS hardware were to fail, how easy would it be for an unskilled volunteer to replace it? A hard drive replacement might not be too hard, since that might be a standard component*. But suppose the NAS' power supply fails or its main board gets zapped or succumbs to corrosion: can a replacement for that be sourced? If not, are the NAS installation instructions general enough that they can apply to whatever hardware is readily available, or are they very closely tied to this specific NAS hardware? (I realise that the final instructions don't yet exist. These are rhetorical questions.)

This is something that has worried me from the start and the reason why I spend so much time on writing User Manuals. All of the projects that I've been involved in to date have had a User Manual or at the least a Composite Decoment that details design, installation and other relevant details. I've also tried to provide spares for items that are critical or not easy to source (especially since some components may become obsolete).

Some disasters could result in loss of not only the NAS but also the building that houses it. In that scenario, we would not be able to instate a replacement NAS immediately in the same location. Hopefully that can be regarded as a relatively extreme situation, so we might accept more downtime for that level of devastation.

Personally, if the disaster resulted in the loss of the building then i suspect that replacing the NAS Box would be the least of out worries.

[hamishmb]

As far as cost goes, re-partitioning the SD cards costs us nothing. As for USB backup, that can be done with some old spare USB drive - the files are tiny, especially when compressed. I swear I could make the website backups simple, and probably use almost no bandwidth, so unless billed per MB or something that would be okay. However, the security issues are more of a concern.

Replacing the NAS box would not be a simple process, even if it was like-for-like. Installing the software and configuring it requires quite a lot of steps. If a replacement box also used ARMv5 or higher, the software I compiled would probably work, though.

[hamishmb]

Note that we have a spare hard drive already - an older but well-working 1 TB drive I used to use.

[TerryJC]

Replacing the NAS box would not be a simple process, even if it was like-for-like. Installing the software and configuring it requires quite a lot of steps. If a replacement box also used ARMv5 or higher, the software I compiled would probably work, though.

My comment was perhaps a bit obscure. When I said that the loss of the NAS Box would be the least of our worries, what I meant was the loss of the building would result in the loss of the whole Model Railway Layout as well as the building itself.

Yes it won't be easy to replicate the NAS Box if it was lost, but if the cause was the building burning down, we would have plenty of time.

Of course if the NAS Box failed due to a simple hardware failure, then that's a different story. However, the main reason for the amount of work you had to do was the complete lack of support by D-Link for the software on the box; everything was years out of date. That's why I changed it in the first place and got myself a Netgear ReadyNAS for use here at home. If we had to replace the NAS Box, then experience would tell us to avoid D-Link devices like the plague.

[hamishmb]

Yes, D-Link are pretty awful. As long as the replacement was ARM-based the software would work. Some of it might also be available in some form from the manufacturer. It all depends on what we end up doing with it really. Auto-starting scripts on boot would need some figuring out.

However, whatever replacement we would hypothetically get would probably be far easier to set up than this NAS box . The version of gzip it has is from 2003! I doubt it'll fail any time soon because it looks mint inside to be honest - almost no dust before I cleaned it and no corrosion or anything. I reckon it'll be fine for a while

[TerryJC]

As long as the replacement was ARM-based the software would work. Some of it might also be available in some form from the manufacturer. It all depends on what we end up doing with it really. Auto-starting scripts on boot would need some figuring out.

The replacement for the D-Link at home was a Netgear ReadyNAS RN212, which has a 1.4 GHz ARM Cortex A15 Quad Core processor. I paid ~£330 for it (without discs) in January 2018 and it still gets fairly regular updates.

I just ssh'd into it and got this:

Code: Select all

TerryJColes@ReadyNAS:~$ cat /proc/version
Linux version 4.4.190.alpine.1 (root@blocks) (gcc version 8.3.0 (Debian 8.3.0-2) ) #1 SMP Mon Oct 28 02:01:23 UTC 2019

I think that's a bit better than the D-Link was. This box is still available for around £240 brand new or it might be possible to pick one up second-hand, although eBay has none at present.

[PatrickW]

It looks like (some) NASes get better software support than I realised. I was imagining that they would all be like the D-Link. In ten years' time, they might be even better!

If we're talking £240, then there are also options involving single board computers, some of which will run a standard PC Linux distribution for minimal software hassle. (e.g. PC Engines APU2, available from LinITX.)

I don't think there's a shortage of hardware options. The hard part is identifying suitable hardware and then installing the software onto it. A certain amount of know-how and/or documentation is required. There's a trade-off between documenting specific hardware in enough detail to hand-hold a novice through the process, and documenting the process in a general way that can be applied to multiple hardware options. I'd go for a combination of both: specific for the hardware we've got and general for the hardware someone else might choose in the future.

[TerryJC]

It looks like (some) NASes get better software support than I realised. I was imagining that they would all be like the D-Link. In ten years' time, they might be even better!

Most NASs have better support than the D-Link. When I bought the one that Hamish now has, I only ever got one update even though they continued selling the model for quite some time.

If we're talking £240, then there are also options involving single board computers, some of which will run a standard PC Linux distribution for minimal software hassle. (e.g. PC Engines APU2, available from LinITX.)

I don't think there's a shortage of hardware options. The hard part is identifying suitable hardware and then installing the software onto it. A certain amount of know-how and/or documentation is required. There's a trade-off between documenting specific hardware in enough detail to hand-hold a novice through the process, and documenting the process in a general way that can be applied to multiple hardware options. I'd go for a combination of both: specific for the hardware we've got and general for the hardware someone else might choose in the future.

There is even a Raspberry Pi option; https://www.ionos.co.uk/digitalguide/se ... ry-pi-nas/. If you scroll down a bit, then it even shows how to use the suggested tool (OpenMediaVault) to configure the media as RAID.

The problem with building it yourself is that the physical construction becomes a bigger problem than the hardware, eg case, slots, cooling etc, but it's all possible.