Data Hoarder

11 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago

MODERATORS

communick@selfhosted.forum

Is a 20 disk raidz3 really a bad idea, for media archiving use cases? (alien.top)

submitted 1 year ago by goldcakes@alien.top to c/datahoarder@selfhosted.forum

7 comments fedilink hide all child comments

I am archiving a vast amount of media files that are rarely accessed. I'm writing large sequential files, at peaks of about 100MB/s.

I want to maximise storage space primarily; I have 20x 18TB HDDs.

I've been told that large (e.g. 20 disk) vdevs are bad because resilvers will take a very long time, which creates higher risk of pool failure. How bad of an idea is this?

you are viewing a single comment's thread
view the rest of the comments

[–] EchoGecko795@alien.top 1 points 1 year ago

I have a few 24x drive RAIDz3 pools, and as long as you can live with the longer scrub and resilver time they make a good archive or backup pool, but I would not really want it as an always on active pool. If you want to know what the estimated failure rate here is a calculator.

Not sure if its broken or if my mobile firefox browser just doesn't like it, but I seem to be getting an error of 0% failure rates, there are other calculators if you google them though.

https://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/

I've been told that large (e.g. 20 disk) vdevs are bad because resilvers will take a very long time, which creates higher risk of pool failure. How bad of an idea is this?

I normally only have to replace 1 drive at a time, but with RAIDz3 you have to lose 4 drives at the same time for data loss to happen. If you are using a mixed batches of drive (not all from the same run) this happening is very low, and usually happening due to some other event (overheating, fire, cow attacking the disk shelf) In the 5 years I have had these pools, the worst was losing 1 drive, and having errors pop up on another drive, which were still corrected because RAIDz3 has 3 drives of protection.