The real "silent killer" I see here is the reliance on mirroring. One-failure protection, with a 2x expansion factor. As it happens, I work on large storage systems, where 2x is our maximum expansion factor and for that we get resistance to as many as nine simultaneous failures. Across power and network failure domains, with multiple kinds of background scrubbing to detect loss of that redundancy. Oh, and 60PB is something we might add to an existing cluster for a day to absorb transient I/O load. There's also a bunch of monitoring and automation stuff that should be considered "table stakes" for storage at these scales. Seems like an opportunity to use what I and others have learned for a good cause, to make this valuable resource more efficient and more durable all at once.
Did you get to the part where they explain that they are strapped for resources and in need of donations? They have plenty of expertise, money is what they need.
If they're only getting one-failure protection for a 2x expansion factor, then they clearly do need more expertise and more money won't solve their durability problem. With more money they could expand to 100x their size and have the exact same vulnerability to data loss. That's a problem worth solving.
Is your argument that they could have had 10x less vulnerability to data loss while operating at the same cost and backing up the same amount of total data? (If so, then this really is a valuable skill)
Or are you saying that they should have backed up 10x less data in order to have the 10x more redundant copies? That's a value decision on what to prioritize (more data saved vs 'good enough' resiliency).
Donations would be the best way to solve this problem so that they don't have to make this trade off in the first place.
The decision here was: For the given budget we have, where do we make the reliability vs data backed-up trade off? Increasing one necessarily decreases the other.
The options are:
- Increase reliability but decrease how much data capacity they have
- Increase data capacity at the cost of lower reliability
- Remove the "given the budget we have" constraint via fundraising. With more funding they can afford to increase both
You can see the decision they made for the fixed budget option. Now we can help them remove their constraint with extra donations (Every $5 helps. I've already donated)
Simply not true. Encoding schemes represent a whole different set of tradeoff possibilities which you don't seem to have considered, and simple replication is worse than other options in terms of both reliability and efficiency.
It's different techniques (erasure coding). As it turns out, applying those techniques would reduce need for additional physical resources as well, since it allows data to be stored more efficiently with the same physical resources. It does require more CPU and memory relative to bytes stored or spindles to store them on, but it's easy to make that tradeoff and still come out ahead.
No doubt Facebook considers theirs a competitive advantage, but I'm kind of surprised there isn't a reasonably robust OSS erasure-coding based object store available in 2019. Maybe LizardFS?
Ceph has an object-store interface (which might still account for the majority of its usage) and some people I used to work with on Gluster are now at Minio, so those are the two I'd look at if I needed something in that space.
