this post was submitted on 04 Dec 2023
1 points (100.0% liked)

Data Hoarder

11 readers
1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 1 year ago
MODERATORS
 

In the past, I have used a USB to transfer a few video files from one computer to another. This time I have over 1 terabyte of movies and shows I'd like to transfer. What is the best method?

I know it will take a very long time

top 28 comments
sorted by: hot top controversial new old
[–] Bob_Spud@alien.top 3 points 11 months ago

Quickest -

  1. Take the lid off the source PC put the HDD in it
  2. Copy the stuff.
  3. Pull it out
  4. Put it the other PC,
  5. Your done

Formatting the HDD my be optional.

[–] cmnybo@discuss.tchncs.de 3 points 11 months ago

Connect both computers over ethernet and rsync the data across. It will still take several hours over gigabit ethernet though.

[–] NyaaTell@alien.top 2 points 11 months ago (1 children)
  1. Take the drive from PC 1 and attach to PC 2 => copy / paste

  2. External drive

[–] tididew@alien.top 1 points 11 months ago

What file system do you get the most compatibility out of?

[–] beholdthepineapple@alien.top 2 points 11 months ago

Parallel port would be faster than serial, but if you’re using serial I’d go with Z modem for the compression and error correction.

[–] BmanUltima@alien.top 1 points 11 months ago

You could go over a wired network.

[–] emalvick@alien.top 1 points 11 months ago

1 Tb shouldn't be too bad. You could try an external HD, but that becomes a double operation (1 copy to the drive, 1 copy off) or use your home network.

Create a share of the folder(s) you want to copy and then access from the new PC to copy and paste to. Speed will depend on your network speed AND drive speeds.

[–] Terakahn@alien.top 1 points 11 months ago

I would write it to an nvme drive and swap that drive to the new pc

[–] Celcius_87@alien.top 1 points 11 months ago

External usb hdd

[–] WikiBox@alien.top 1 points 11 months ago

USB 3.1 Gen2 is hard to beat. 10Gbps. But for HDDs the HDDs themselves will be the bottleneck. For sustained file transfers, perhaps no more than up to 3Gbps, even if installed in the same PC. Good SATA SSDs may manage closer to 6Gbps. NVMe SSDs are likely to saturate USB 3.1 Gen2. Then PCI-e directly between installed drives is the fastest. Normal cabled network is only 1Gbps.

File transfers to/from external multibay USB DAS can be done in parallel, involving multiple HDDs, and come close to saturating 10Gbps USB3.1 Gen2 speeds.

Then there is thunderbolt.

Start a rsync transfer when you go to bed. In the morning it will most likely be done.

[–] adragontattoo@alien.top 1 points 11 months ago

robocopy or rsync if you can't place the drive in the target or an external enclosure.

[–] TADataHoarder@alien.top 1 points 11 months ago

What is the best method?

Sometimes the best method isn't a transfer at all. Don't forget transporting is an option.
Do you need to transfer the files? or could you just use an external you share between different PCs?
If an external works, but seems inconvenient, you can fix that by having the drive connected to a USB switch. When connected to a switch you can press a button or use a remote to change which PC the drive is connected to giving you instant access to the drive contents without having to sync/copy anything over.

Links below for examples of USB switches
https://www.amazon.com/Cable-Matters-Sharing-Computers-Peripherals/dp/B083JKDNRJ/
https://www.amazon.com/Sabrent-Computers-Peripherals-Indicators-USB-SW30/dp/B074TYDJK2/

[–] 4thelulzgamer@alien.top 1 points 11 months ago

If you want the bare minimum fast transfer and both PC are in local network, use Dukto-R6. It's super basic file copying, you just need to drag and drop.

For more advanced options,the others have already pointed it out.

[–] Sopel97@alien.top 1 points 11 months ago

This will depend on the relative placement of these PCs

[–] smstnitc@alien.top 1 points 11 months ago (1 children)

I have a PC in a spot of my house I can't get a cable to. But that's the PC I do my Blu Ray ripping and encoding on, so I keep a 5tb external drive for moving the files to my NAS afterwards. Sometimes it would take days over WiFi instead of minutes to hours over USB 3 twice. (and piss off my wife for saturating the wifi, heh)

[–] Beaver_On_Fire@alien.top 1 points 11 months ago (1 children)

So consider a powerline network adapter, or move the PC to a place you can run a cable.

[–] smstnitc@alien.top 1 points 11 months ago

My experience with powerline is pretty terrible. And there is no better place for it sadly.

[–] p3dal@alien.top 1 points 11 months ago

I've been sending 10TB from one PC to another on my network over the last 2 days, it's nearly done. It's a wired network, and while sometimes I use tools like robocopy for more granular control over the the copy settings and logging of any copy errors, I generally find the standard windows copy tool to be faster.

[–] RedEyed__@alien.top 1 points 11 months ago

rsync over sshfs over Ethernet.
I transferred 10 Tb over night, but it depends on how many files do you have.

[–] Sea_Caterpillar5296@alien.top 1 points 11 months ago

Shouldn't actually take too long... I'd just take a couple sata cables. I transferred 6tb in about 12 or 18 hours, idr.

[–] Frankensteins_Chick@alien.top 1 points 11 months ago

I just transferred 12tb from one hdd to another via usb 3.0. Went at roughly ~180mbps and took 1.5 days

[–] ZeroSulu@alien.top 1 points 11 months ago

Over LAN with flavored solution of your choice.
Other than that, I use one of my spare drives and my USB docking station if I need to move it outside of my own house. I never really had a need for dedicated external drives but a docking station is handy for many things, this being one of them.

[–] diamondsw@alien.top 1 points 11 months ago

Fastest "normal" way would be write it to a USB 3 drive on machine 1, read from drive later on machine 2. An SSD would be best. Even with having to do the write and read in two phases, I'd still expect it to be faster than a 1G network.

(Less normal ways would be things like Thundrerbolt SSDs, 10G or 40G ethernet for direct transfer, etc. But if you have that kind of tech already in place, then you're not asking. ;)

[–] glhughes@alien.top 1 points 11 months ago

10 GbE. 17 minutes.

Of course you need disks that can read/write at that speed (1.25 GB/s). Should not be a problem for NVMe SSDs.

[–] Carnildo@alien.top 1 points 11 months ago

rsync over a wired network. That way, I can stop and resume at any time.

[–] pavoganso@alien.top 1 points 11 months ago
[–] Beaver_On_Fire@alien.top 1 points 11 months ago

Just use gigabit networking. You are looking ~4h/TB give or take depending on a few factors.

[–] dlarge6510@alien.top 1 points 11 months ago

Over a network.

Not WiFi though!

You could hook a single network cable between them. Give each machine an IP address, share the folder that you are copying to or copying from (your choice) and drag the files over. Or you could use RoboCopy to potentially do it faster.

Or just get a 2tb usb external drive and use that, it could be slower or perhaps faster than the network option depending on many factors but also more expensive as you have to buy the drive!