Move, and split up, 5TB of dataUseful Command-line Commands on WindowsExclude only some files with the same...
Plagiarism of code by other PhD student
Was it really inappropriate to write a pull request for the company I interviewed with?
Why do members of Congress in committee hearings ask witnesses the same question multiple times?
Every subset equal to original set?
Why did the Cray-1 have 8 parity bits per word?
What type of investment is best suited for a 1-year investment on a down payment?
Canadian citizen, on US no-fly list. What can I do in order to be allowed on flights which go through US airspace?
Is divide-by-zero a security vulnerability?
Why is working on the same position for more than 15 years not a red flag?
How can I handle a player who pre-plans arguments about my rulings on RAW?
Make me a metasequence
For a 1-action spell, do I need to take a turn to ready the spell before I can cast it, or can I cast it immediately?
How can I be pwned if I'm not registered on the compromised site?
What is the difference between a forward slip and a side slip?
Second-story floor leveling question for new home
Can a space-faring robot still function over a billion years?
In which way proportional valves are controlled solely by current?
Why did John Williams use a march to symbolise Indiana Jones?
Movie: Scientists travel to the future to avoid nuclear war, last surviving one is used as fuel by future humans
Is there a frame of reference in which I was born before I was conceived?
How to evaluate the limit where something is raised to a power of x?
Is it possible to make a clamp function shorter than a ternary in JS?
Why can't we make a perpetual motion machine by using a magnet to pull up a piece of metal, then letting it fall back down?
The need of reserving one's ability in job interviews
Move, and split up, 5TB of data
Useful Command-line Commands on WindowsExclude only some files with the same name when copying using robocopySplit Brain DNS and DNS forwardingHow do I use robocopy to list all files over a certain size recursively?Mirror directory, only purging files in destination directory older than 7 daysRobocopy Server 2003 does not support the /mt optionRobocopy and NTFS PermissionsHow can I take a one time full backup of a Windows Server without the need for a restore program?ERROR 1307 (0x0000051B) - While copying with Robocopy - None of the known sollutions workrobocopy log all but only display header/summary
Here is the scenario:
I have 5 TB (yes, that's a T) of files on a Windows server that I need to migrate to a new server in as short and efficient time as possible. (Think: Robocopy, Rsync, etc as I plan to use differentials to do this over time). The files are in ~41,000 directories under a single parent directory (d:filesfolder1, d:filesfolder2, etc).
Since these are migrating to a new server, I want to split this up so that they are not all in the same "files" directory, but are instead split as logically as possible across multiple drive (trying to stay at about 2TB drive sizes for backup and replication purposes).
Robocopy doesn't have a regex option. Rsync would require a linux server, which isn't impossible, but adds overhead: this is a Windows to Windows move. I've found a way to loop through the 41,000 directories using powershell and initiate Robocopy for each directory individually, thus allowing me to specify destination...but this seems inefficient.
One other option I've considered is migrating everything at once, and then scripting out a copy to the other drives as needed. This would mean copying 2/3 of the files twice.
Have I missed anything obvious?
filesystems windows robocopy
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
|
show 3 more comments
Here is the scenario:
I have 5 TB (yes, that's a T) of files on a Windows server that I need to migrate to a new server in as short and efficient time as possible. (Think: Robocopy, Rsync, etc as I plan to use differentials to do this over time). The files are in ~41,000 directories under a single parent directory (d:filesfolder1, d:filesfolder2, etc).
Since these are migrating to a new server, I want to split this up so that they are not all in the same "files" directory, but are instead split as logically as possible across multiple drive (trying to stay at about 2TB drive sizes for backup and replication purposes).
Robocopy doesn't have a regex option. Rsync would require a linux server, which isn't impossible, but adds overhead: this is a Windows to Windows move. I've found a way to loop through the 41,000 directories using powershell and initiate Robocopy for each directory individually, thus allowing me to specify destination...but this seems inefficient.
One other option I've considered is migrating everything at once, and then scripting out a copy to the other drives as needed. This would mean copying 2/3 of the files twice.
Have I missed anything obvious?
filesystems windows robocopy
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
2
1. 5TB isn't as much as you make it sound like, this isn't the 90s - heck you can buy a 6TB SATA drive now. 2. Why split this up over time? It'll take <12 hours on GbE 3. DeltaCopy isrsyncfor Windows. 4. RoboCopy can exclude directories - are you sure it wont work? 5. Splitting the files to multiple disks isn't necessary for most backup systems, and complicates RAID if you're concerned about availability in single failure scenarios.
– Chris S
Aug 20 '14 at 20:00
2
Have I missed anything obvious?Yes, yes you have. Attach a >5 TB drive to the server (you can get 6 and 8 TB USB drives these days), copy the files to the new drive (or do a block-level clone, if you prefer), move the USB drive to the other server, and copy the files back. A lot faster than doing a network transfer, and as ChrisS noted, you're only making things more complicated and difficult on yourself by trying to limit your drive size to 2TB - there's just no reason to do that these days (and if there is, you need to replace the old PoS that has a 2TB limit).
– HopelessN00b
Aug 20 '14 at 20:12
I agree with Chris, 5TB isn't so much data. However, 41000 directories is a little horrible because I assume you've got lots of little files. That will make the copy process a tad slow.
– Matt
Aug 20 '14 at 20:28
I'd go with the hard drive copy idea. Anything over the network is going to be slow.
– Matt
Aug 20 '14 at 20:54
Don't forget that Robocopy has the/MT(multi-threaded) switch. This can dramatically speed it up if you have lots of small files.
– Zoredache
Aug 20 '14 at 21:15
|
show 3 more comments
Here is the scenario:
I have 5 TB (yes, that's a T) of files on a Windows server that I need to migrate to a new server in as short and efficient time as possible. (Think: Robocopy, Rsync, etc as I plan to use differentials to do this over time). The files are in ~41,000 directories under a single parent directory (d:filesfolder1, d:filesfolder2, etc).
Since these are migrating to a new server, I want to split this up so that they are not all in the same "files" directory, but are instead split as logically as possible across multiple drive (trying to stay at about 2TB drive sizes for backup and replication purposes).
Robocopy doesn't have a regex option. Rsync would require a linux server, which isn't impossible, but adds overhead: this is a Windows to Windows move. I've found a way to loop through the 41,000 directories using powershell and initiate Robocopy for each directory individually, thus allowing me to specify destination...but this seems inefficient.
One other option I've considered is migrating everything at once, and then scripting out a copy to the other drives as needed. This would mean copying 2/3 of the files twice.
Have I missed anything obvious?
filesystems windows robocopy
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
Here is the scenario:
I have 5 TB (yes, that's a T) of files on a Windows server that I need to migrate to a new server in as short and efficient time as possible. (Think: Robocopy, Rsync, etc as I plan to use differentials to do this over time). The files are in ~41,000 directories under a single parent directory (d:filesfolder1, d:filesfolder2, etc).
Since these are migrating to a new server, I want to split this up so that they are not all in the same "files" directory, but are instead split as logically as possible across multiple drive (trying to stay at about 2TB drive sizes for backup and replication purposes).
Robocopy doesn't have a regex option. Rsync would require a linux server, which isn't impossible, but adds overhead: this is a Windows to Windows move. I've found a way to loop through the 41,000 directories using powershell and initiate Robocopy for each directory individually, thus allowing me to specify destination...but this seems inefficient.
One other option I've considered is migrating everything at once, and then scripting out a copy to the other drives as needed. This would mean copying 2/3 of the files twice.
Have I missed anything obvious?
filesystems windows robocopy
filesystems windows robocopy
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
edited Aug 20 '14 at 19:37
Deer Hunter
90841624
90841624
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
asked Aug 20 '14 at 19:12
Chad JohnsonChad Johnson
163
163
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 1 min ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
2
1. 5TB isn't as much as you make it sound like, this isn't the 90s - heck you can buy a 6TB SATA drive now. 2. Why split this up over time? It'll take <12 hours on GbE 3. DeltaCopy isrsyncfor Windows. 4. RoboCopy can exclude directories - are you sure it wont work? 5. Splitting the files to multiple disks isn't necessary for most backup systems, and complicates RAID if you're concerned about availability in single failure scenarios.
– Chris S
Aug 20 '14 at 20:00
2
Have I missed anything obvious?Yes, yes you have. Attach a >5 TB drive to the server (you can get 6 and 8 TB USB drives these days), copy the files to the new drive (or do a block-level clone, if you prefer), move the USB drive to the other server, and copy the files back. A lot faster than doing a network transfer, and as ChrisS noted, you're only making things more complicated and difficult on yourself by trying to limit your drive size to 2TB - there's just no reason to do that these days (and if there is, you need to replace the old PoS that has a 2TB limit).
– HopelessN00b
Aug 20 '14 at 20:12
I agree with Chris, 5TB isn't so much data. However, 41000 directories is a little horrible because I assume you've got lots of little files. That will make the copy process a tad slow.
– Matt
Aug 20 '14 at 20:28
I'd go with the hard drive copy idea. Anything over the network is going to be slow.
– Matt
Aug 20 '14 at 20:54
Don't forget that Robocopy has the/MT(multi-threaded) switch. This can dramatically speed it up if you have lots of small files.
– Zoredache
Aug 20 '14 at 21:15
|
show 3 more comments
2
1. 5TB isn't as much as you make it sound like, this isn't the 90s - heck you can buy a 6TB SATA drive now. 2. Why split this up over time? It'll take <12 hours on GbE 3. DeltaCopy isrsyncfor Windows. 4. RoboCopy can exclude directories - are you sure it wont work? 5. Splitting the files to multiple disks isn't necessary for most backup systems, and complicates RAID if you're concerned about availability in single failure scenarios.
– Chris S
Aug 20 '14 at 20:00
2
Have I missed anything obvious?Yes, yes you have. Attach a >5 TB drive to the server (you can get 6 and 8 TB USB drives these days), copy the files to the new drive (or do a block-level clone, if you prefer), move the USB drive to the other server, and copy the files back. A lot faster than doing a network transfer, and as ChrisS noted, you're only making things more complicated and difficult on yourself by trying to limit your drive size to 2TB - there's just no reason to do that these days (and if there is, you need to replace the old PoS that has a 2TB limit).
– HopelessN00b
Aug 20 '14 at 20:12
I agree with Chris, 5TB isn't so much data. However, 41000 directories is a little horrible because I assume you've got lots of little files. That will make the copy process a tad slow.
– Matt
Aug 20 '14 at 20:28
I'd go with the hard drive copy idea. Anything over the network is going to be slow.
– Matt
Aug 20 '14 at 20:54
Don't forget that Robocopy has the/MT(multi-threaded) switch. This can dramatically speed it up if you have lots of small files.
– Zoredache
Aug 20 '14 at 21:15
2
2
1. 5TB isn't as much as you make it sound like, this isn't the 90s - heck you can buy a 6TB SATA drive now. 2. Why split this up over time? It'll take <12 hours on GbE 3. DeltaCopy is
rsync for Windows. 4. RoboCopy can exclude directories - are you sure it wont work? 5. Splitting the files to multiple disks isn't necessary for most backup systems, and complicates RAID if you're concerned about availability in single failure scenarios.– Chris S
Aug 20 '14 at 20:00
1. 5TB isn't as much as you make it sound like, this isn't the 90s - heck you can buy a 6TB SATA drive now. 2. Why split this up over time? It'll take <12 hours on GbE 3. DeltaCopy is
rsync for Windows. 4. RoboCopy can exclude directories - are you sure it wont work? 5. Splitting the files to multiple disks isn't necessary for most backup systems, and complicates RAID if you're concerned about availability in single failure scenarios.– Chris S
Aug 20 '14 at 20:00
2
2
Have I missed anything obvious? Yes, yes you have. Attach a >5 TB drive to the server (you can get 6 and 8 TB USB drives these days), copy the files to the new drive (or do a block-level clone, if you prefer), move the USB drive to the other server, and copy the files back. A lot faster than doing a network transfer, and as ChrisS noted, you're only making things more complicated and difficult on yourself by trying to limit your drive size to 2TB - there's just no reason to do that these days (and if there is, you need to replace the old PoS that has a 2TB limit).– HopelessN00b
Aug 20 '14 at 20:12
Have I missed anything obvious? Yes, yes you have. Attach a >5 TB drive to the server (you can get 6 and 8 TB USB drives these days), copy the files to the new drive (or do a block-level clone, if you prefer), move the USB drive to the other server, and copy the files back. A lot faster than doing a network transfer, and as ChrisS noted, you're only making things more complicated and difficult on yourself by trying to limit your drive size to 2TB - there's just no reason to do that these days (and if there is, you need to replace the old PoS that has a 2TB limit).– HopelessN00b
Aug 20 '14 at 20:12
I agree with Chris, 5TB isn't so much data. However, 41000 directories is a little horrible because I assume you've got lots of little files. That will make the copy process a tad slow.
– Matt
Aug 20 '14 at 20:28
I agree with Chris, 5TB isn't so much data. However, 41000 directories is a little horrible because I assume you've got lots of little files. That will make the copy process a tad slow.
– Matt
Aug 20 '14 at 20:28
I'd go with the hard drive copy idea. Anything over the network is going to be slow.
– Matt
Aug 20 '14 at 20:54
I'd go with the hard drive copy idea. Anything over the network is going to be slow.
– Matt
Aug 20 '14 at 20:54
Don't forget that Robocopy has the
/MT (multi-threaded) switch. This can dramatically speed it up if you have lots of small files.– Zoredache
Aug 20 '14 at 21:15
Don't forget that Robocopy has the
/MT (multi-threaded) switch. This can dramatically speed it up if you have lots of small files.– Zoredache
Aug 20 '14 at 21:15
|
show 3 more comments
1 Answer
1
active
oldest
votes
I've done window 2 windows rsync using the CygWin framework a couple of years back.
Rsync + sshd is definently doable.
I also found this and it looks like it might make windows-rsyncing easier than ever:
http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
We usually clone the data to through physical disks first to make the "first sync" as fast as possible and then use rsync afterwards to only move the deltas/differences over the network.
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
3
I've found the windows rsync port with cygwin performs very poorly unfortunately. It's not that surprising. It also misses file permissions.
– Matt
Aug 20 '14 at 20:25
Well as soon as your rsyncing over ssh, i think the SSHd is the bottleneck (or used to be anyway).
– thelogix
Aug 20 '14 at 20:34
not really, it just underperformed. Worse was the fact that it missed file permissions and other metadata. Don't use it on windows until someone comes up with a native port that overcomes the limitations. Acrosync claims to be a native port.
– Matt
Aug 20 '14 at 20:53
Yes dont use if you need to replicate the windows file permissions. But a complete disparagement is a bit harsh. I've used it to syncronize avatar image-files in a webserver cluster and IIS logfiles to a central logparsing-server. Original permissions were unimportant and new file followed the umask of cygwin. Perfect tool for both jobs.
– thelogix
Aug 20 '14 at 21:04
Yeah it might be a bit harsh. My memory of it was that it was underwhelmed. I was considering re-writing it but I was just too busy with other things.
– Matt
Aug 20 '14 at 21:20
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "2"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f622860%2fmove-and-split-up-5tb-of-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I've done window 2 windows rsync using the CygWin framework a couple of years back.
Rsync + sshd is definently doable.
I also found this and it looks like it might make windows-rsyncing easier than ever:
http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
We usually clone the data to through physical disks first to make the "first sync" as fast as possible and then use rsync afterwards to only move the deltas/differences over the network.
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
3
I've found the windows rsync port with cygwin performs very poorly unfortunately. It's not that surprising. It also misses file permissions.
– Matt
Aug 20 '14 at 20:25
Well as soon as your rsyncing over ssh, i think the SSHd is the bottleneck (or used to be anyway).
– thelogix
Aug 20 '14 at 20:34
not really, it just underperformed. Worse was the fact that it missed file permissions and other metadata. Don't use it on windows until someone comes up with a native port that overcomes the limitations. Acrosync claims to be a native port.
– Matt
Aug 20 '14 at 20:53
Yes dont use if you need to replicate the windows file permissions. But a complete disparagement is a bit harsh. I've used it to syncronize avatar image-files in a webserver cluster and IIS logfiles to a central logparsing-server. Original permissions were unimportant and new file followed the umask of cygwin. Perfect tool for both jobs.
– thelogix
Aug 20 '14 at 21:04
Yeah it might be a bit harsh. My memory of it was that it was underwhelmed. I was considering re-writing it but I was just too busy with other things.
– Matt
Aug 20 '14 at 21:20
add a comment |
I've done window 2 windows rsync using the CygWin framework a couple of years back.
Rsync + sshd is definently doable.
I also found this and it looks like it might make windows-rsyncing easier than ever:
http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
We usually clone the data to through physical disks first to make the "first sync" as fast as possible and then use rsync afterwards to only move the deltas/differences over the network.
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
3
I've found the windows rsync port with cygwin performs very poorly unfortunately. It's not that surprising. It also misses file permissions.
– Matt
Aug 20 '14 at 20:25
Well as soon as your rsyncing over ssh, i think the SSHd is the bottleneck (or used to be anyway).
– thelogix
Aug 20 '14 at 20:34
not really, it just underperformed. Worse was the fact that it missed file permissions and other metadata. Don't use it on windows until someone comes up with a native port that overcomes the limitations. Acrosync claims to be a native port.
– Matt
Aug 20 '14 at 20:53
Yes dont use if you need to replicate the windows file permissions. But a complete disparagement is a bit harsh. I've used it to syncronize avatar image-files in a webserver cluster and IIS logfiles to a central logparsing-server. Original permissions were unimportant and new file followed the umask of cygwin. Perfect tool for both jobs.
– thelogix
Aug 20 '14 at 21:04
Yeah it might be a bit harsh. My memory of it was that it was underwhelmed. I was considering re-writing it but I was just too busy with other things.
– Matt
Aug 20 '14 at 21:20
add a comment |
I've done window 2 windows rsync using the CygWin framework a couple of years back.
Rsync + sshd is definently doable.
I also found this and it looks like it might make windows-rsyncing easier than ever:
http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
We usually clone the data to through physical disks first to make the "first sync" as fast as possible and then use rsync afterwards to only move the deltas/differences over the network.
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
I've done window 2 windows rsync using the CygWin framework a couple of years back.
Rsync + sshd is definently doable.
I also found this and it looks like it might make windows-rsyncing easier than ever:
http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
We usually clone the data to through physical disks first to make the "first sync" as fast as possible and then use rsync afterwards to only move the deltas/differences over the network.
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
answered Aug 20 '14 at 20:19
thelogixthelogix
37917
37917
3
I've found the windows rsync port with cygwin performs very poorly unfortunately. It's not that surprising. It also misses file permissions.
– Matt
Aug 20 '14 at 20:25
Well as soon as your rsyncing over ssh, i think the SSHd is the bottleneck (or used to be anyway).
– thelogix
Aug 20 '14 at 20:34
not really, it just underperformed. Worse was the fact that it missed file permissions and other metadata. Don't use it on windows until someone comes up with a native port that overcomes the limitations. Acrosync claims to be a native port.
– Matt
Aug 20 '14 at 20:53
Yes dont use if you need to replicate the windows file permissions. But a complete disparagement is a bit harsh. I've used it to syncronize avatar image-files in a webserver cluster and IIS logfiles to a central logparsing-server. Original permissions were unimportant and new file followed the umask of cygwin. Perfect tool for both jobs.
– thelogix
Aug 20 '14 at 21:04
Yeah it might be a bit harsh. My memory of it was that it was underwhelmed. I was considering re-writing it but I was just too busy with other things.
– Matt
Aug 20 '14 at 21:20
add a comment |
3
I've found the windows rsync port with cygwin performs very poorly unfortunately. It's not that surprising. It also misses file permissions.
– Matt
Aug 20 '14 at 20:25
Well as soon as your rsyncing over ssh, i think the SSHd is the bottleneck (or used to be anyway).
– thelogix
Aug 20 '14 at 20:34
not really, it just underperformed. Worse was the fact that it missed file permissions and other metadata. Don't use it on windows until someone comes up with a native port that overcomes the limitations. Acrosync claims to be a native port.
– Matt
Aug 20 '14 at 20:53
Yes dont use if you need to replicate the windows file permissions. But a complete disparagement is a bit harsh. I've used it to syncronize avatar image-files in a webserver cluster and IIS logfiles to a central logparsing-server. Original permissions were unimportant and new file followed the umask of cygwin. Perfect tool for both jobs.
– thelogix
Aug 20 '14 at 21:04
Yeah it might be a bit harsh. My memory of it was that it was underwhelmed. I was considering re-writing it but I was just too busy with other things.
– Matt
Aug 20 '14 at 21:20
3
3
I've found the windows rsync port with cygwin performs very poorly unfortunately. It's not that surprising. It also misses file permissions.
– Matt
Aug 20 '14 at 20:25
I've found the windows rsync port with cygwin performs very poorly unfortunately. It's not that surprising. It also misses file permissions.
– Matt
Aug 20 '14 at 20:25
Well as soon as your rsyncing over ssh, i think the SSHd is the bottleneck (or used to be anyway).
– thelogix
Aug 20 '14 at 20:34
Well as soon as your rsyncing over ssh, i think the SSHd is the bottleneck (or used to be anyway).
– thelogix
Aug 20 '14 at 20:34
not really, it just underperformed. Worse was the fact that it missed file permissions and other metadata. Don't use it on windows until someone comes up with a native port that overcomes the limitations. Acrosync claims to be a native port.
– Matt
Aug 20 '14 at 20:53
not really, it just underperformed. Worse was the fact that it missed file permissions and other metadata. Don't use it on windows until someone comes up with a native port that overcomes the limitations. Acrosync claims to be a native port.
– Matt
Aug 20 '14 at 20:53
Yes dont use if you need to replicate the windows file permissions. But a complete disparagement is a bit harsh. I've used it to syncronize avatar image-files in a webserver cluster and IIS logfiles to a central logparsing-server. Original permissions were unimportant and new file followed the umask of cygwin. Perfect tool for both jobs.
– thelogix
Aug 20 '14 at 21:04
Yes dont use if you need to replicate the windows file permissions. But a complete disparagement is a bit harsh. I've used it to syncronize avatar image-files in a webserver cluster and IIS logfiles to a central logparsing-server. Original permissions were unimportant and new file followed the umask of cygwin. Perfect tool for both jobs.
– thelogix
Aug 20 '14 at 21:04
Yeah it might be a bit harsh. My memory of it was that it was underwhelmed. I was considering re-writing it but I was just too busy with other things.
– Matt
Aug 20 '14 at 21:20
Yeah it might be a bit harsh. My memory of it was that it was underwhelmed. I was considering re-writing it but I was just too busy with other things.
– Matt
Aug 20 '14 at 21:20
add a comment |
Thanks for contributing an answer to Server Fault!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f622860%2fmove-and-split-up-5tb-of-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
1. 5TB isn't as much as you make it sound like, this isn't the 90s - heck you can buy a 6TB SATA drive now. 2. Why split this up over time? It'll take <12 hours on GbE 3. DeltaCopy is
rsyncfor Windows. 4. RoboCopy can exclude directories - are you sure it wont work? 5. Splitting the files to multiple disks isn't necessary for most backup systems, and complicates RAID if you're concerned about availability in single failure scenarios.– Chris S
Aug 20 '14 at 20:00
2
Have I missed anything obvious?Yes, yes you have. Attach a >5 TB drive to the server (you can get 6 and 8 TB USB drives these days), copy the files to the new drive (or do a block-level clone, if you prefer), move the USB drive to the other server, and copy the files back. A lot faster than doing a network transfer, and as ChrisS noted, you're only making things more complicated and difficult on yourself by trying to limit your drive size to 2TB - there's just no reason to do that these days (and if there is, you need to replace the old PoS that has a 2TB limit).– HopelessN00b
Aug 20 '14 at 20:12
I agree with Chris, 5TB isn't so much data. However, 41000 directories is a little horrible because I assume you've got lots of little files. That will make the copy process a tad slow.
– Matt
Aug 20 '14 at 20:28
I'd go with the hard drive copy idea. Anything over the network is going to be slow.
– Matt
Aug 20 '14 at 20:54
Don't forget that Robocopy has the
/MT(multi-threaded) switch. This can dramatically speed it up if you have lots of small files.– Zoredache
Aug 20 '14 at 21:15