Efficient rsyncrsync time comparison - what's the precision of the Modified times comparisonrsync and backup...
Why airport relocation isn't done gradually?
Is there a name of the flying bionic bird?
Doomsday-clock for my fantasy planet
Can I find out the caloric content of bread by dehydrating it?
How would photo IDs work for shapeshifters?
How can I fix this gap between bookcases I made?
Is a vector space a subspace of itself?
How can I add custom success page
Patience, young "Padovan"
Shall I use personal or official e-mail account when registering to external websites for work purpose?
How to answer pointed "are you quitting" questioning when I don't want them to suspect
What do the Banks children have against barley water?
Is ipsum/ipsa/ipse a third person pronoun, or can it serve other functions?
Information to fellow intern about hiring?
COUNT(*) or MAX(id) - which is faster?
Unbreakable Formation vs. Cry of the Carnarium
Why was the "bread communication" in the arena of Catching Fire left out in the movie?
New order #4: World
How to make payment on the internet without leaving a money trail?
Why did the Germans forbid the possession of pet pigeons in Rostov-on-Don in 1941?
What is the offset in a seaplane's hull?
Why doesn't a const reference extend the life of a temporary object passed via a function?
Why is the design of haulage companies so “special”?
Is there any use for defining additional entity types in a SOQL FROM clause?
Efficient rsync
rsync time comparison - what's the precision of the Modified times comparisonrsync and backup and changing timezonersync vs mtime and ctimeUsing rsync, how can I know what attribute of source file differed from the dest which caused the trasferrsync hangs unless I defrag ext4 filesystemSmarter filetransfers than rsync?Different hash value of large rsynced file on centos and ubuntu?Usage of --remove-source-files option of rsyncWhat differences are between using `-u` and not with rsync?rsync --update with symlinks
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
Now 5 years after this question was orignially asked, is there a way for rsync
to do the following?
- same time and same size ► skip file (no transfer, no checksum)
- different sizes ► transfer file (no checksum)
- different times and same size ► perform checksum ► transfer only if checksums differ
In my case, I have a network share that is accessible through multiples OS (one that uses UTC and another that uses RTC - I do not have admin/root access on either, so I cannot change it). Modifying the file in one OS changes the time so that it is a few hours ahead/behind the "real" time, depending on which OS I browse the files. If I modify a file, the timestamp will be "incorrect".
rsync timestamps size
add a comment |
Now 5 years after this question was orignially asked, is there a way for rsync
to do the following?
- same time and same size ► skip file (no transfer, no checksum)
- different sizes ► transfer file (no checksum)
- different times and same size ► perform checksum ► transfer only if checksums differ
In my case, I have a network share that is accessible through multiples OS (one that uses UTC and another that uses RTC - I do not have admin/root access on either, so I cannot change it). Modifying the file in one OS changes the time so that it is a few hours ahead/behind the "real" time, depending on which OS I browse the files. If I modify a file, the timestamp will be "incorrect".
rsync timestamps size
rync
's option-u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.) -- This is not exactly what you want, but should help when the time differs 'one way'. So if you 'dry run'rsync
with swapped source and target you should be able to catch time differences both ways.
– sudodus
Jun 18 '18 at 4:16
add a comment |
Now 5 years after this question was orignially asked, is there a way for rsync
to do the following?
- same time and same size ► skip file (no transfer, no checksum)
- different sizes ► transfer file (no checksum)
- different times and same size ► perform checksum ► transfer only if checksums differ
In my case, I have a network share that is accessible through multiples OS (one that uses UTC and another that uses RTC - I do not have admin/root access on either, so I cannot change it). Modifying the file in one OS changes the time so that it is a few hours ahead/behind the "real" time, depending on which OS I browse the files. If I modify a file, the timestamp will be "incorrect".
rsync timestamps size
Now 5 years after this question was orignially asked, is there a way for rsync
to do the following?
- same time and same size ► skip file (no transfer, no checksum)
- different sizes ► transfer file (no checksum)
- different times and same size ► perform checksum ► transfer only if checksums differ
In my case, I have a network share that is accessible through multiples OS (one that uses UTC and another that uses RTC - I do not have admin/root access on either, so I cannot change it). Modifying the file in one OS changes the time so that it is a few hours ahead/behind the "real" time, depending on which OS I browse the files. If I modify a file, the timestamp will be "incorrect".
rsync timestamps size
rsync timestamps size
asked Jun 18 '18 at 1:29
wjwrpoyobwjwrpoyob
61
61
rync
's option-u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.) -- This is not exactly what you want, but should help when the time differs 'one way'. So if you 'dry run'rsync
with swapped source and target you should be able to catch time differences both ways.
– sudodus
Jun 18 '18 at 4:16
add a comment |
rync
's option-u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.) -- This is not exactly what you want, but should help when the time differs 'one way'. So if you 'dry run'rsync
with swapped source and target you should be able to catch time differences both ways.
– sudodus
Jun 18 '18 at 4:16
rync
's option -u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.) -- This is not exactly what you want, but should help when the time differs 'one way'. So if you 'dry run' rsync
with swapped source and target you should be able to catch time differences both ways.– sudodus
Jun 18 '18 at 4:16
rync
's option -u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.) -- This is not exactly what you want, but should help when the time differs 'one way'. So if you 'dry run' rsync
with swapped source and target you should be able to catch time differences both ways.– sudodus
Jun 18 '18 at 4:16
add a comment |
2 Answers
2
active
oldest
votes
Amazingly, this does not look possible with a built-in rsync option!
This may work:
rsync -an --info=name src dest
| rsync -an --info=name --checksum --include-from - --exclude="*" src dest
Explanation:
The first rsync command uses the default rsync mode of checking time and size. It outputs any directory or filename with a different time or size. Because of the -n flag, no files are transferred.
The second rsync command takes the list of files with a changed time or size from the first rsync and then runs rsync again on just those files with the --checksum argument.
The second command has -n for dry run mode, so only the file names will be printed. Remove -n from the second command to actually transfer the files.
New contributor
add a comment |
On the one hand, the linked question is very confusing. The answer by 9mjb is correct.
On the other hand, if you are using rsync on a local mount of a network filesystem, there is no way it can checksum the remote file without downloading the whole file anyway! So you are going to be stuck with slowness here. Assuming you have less network bandwidth available than the disk speed.
Except I don't understand what problem you are trying to explain about times, either. If you use -a
, it should transfer the exact times from the source system. If you don't use -a
, it won't transfer the original times from the source filesystem, therefore the times will not match exactly for subsequent transfers anyway. It sounds like you don't want -u
behaviour, but then you can just not use -u
:-).
rsync
is efficient by default. (At the cost of "if time and size both match, the chance that the files are different is insignificant, I'm willing to take the risk not to transfer").
[rsync] is famous for its
delta-transfer algorithm, which reduces the amount of data sent over
the network by sending only the differences between the source files
and the existing files in the destination. Rsync is widely used for
backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size
or in last-modified time.
The important thing to understand is that the rsync delta-transfer algorithm, and the "quick check", are separate.
It sounds like you don't want the behaviour of --checksum
. --checksum
disables the "quick check" part. In that case, don't use --checksum
.
-c, --checksum
This changes the way rsync checks if the files have been changed
and are in need of a transfer. Without this option, rsync uses
a "quick check" that (by default) checks if each file’s size and
time of last modification match between the sender and receiver.
This option changes this to compare a 128-bit checksum for each
file that has a matching size. Generating the checksums means
that both sides will expend a lot of disk I/O reading all the
data in the files in the transfer (and this is prior to any
reading that will be done to transfer changed files), so this
can slow things down significantly.
The sending side generates its checksums while it is doing the
file-system scan that builds the list of the available files.
The receiver generates its checksums when it is scanning for
changed files, and will checksum any file that has the same size
as the corresponding sender’s file: files with either a changed
size or a changed checksum are selected for transfer.
Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a
whole-file checksum that is generated as the file is trans‐
ferred, but that automatic after-the-transfer verification has
nothing to do with this option’s before-the-transfer "Does this
file need to be updated?" check.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f450361%2fefficient-rsync%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Amazingly, this does not look possible with a built-in rsync option!
This may work:
rsync -an --info=name src dest
| rsync -an --info=name --checksum --include-from - --exclude="*" src dest
Explanation:
The first rsync command uses the default rsync mode of checking time and size. It outputs any directory or filename with a different time or size. Because of the -n flag, no files are transferred.
The second rsync command takes the list of files with a changed time or size from the first rsync and then runs rsync again on just those files with the --checksum argument.
The second command has -n for dry run mode, so only the file names will be printed. Remove -n from the second command to actually transfer the files.
New contributor
add a comment |
Amazingly, this does not look possible with a built-in rsync option!
This may work:
rsync -an --info=name src dest
| rsync -an --info=name --checksum --include-from - --exclude="*" src dest
Explanation:
The first rsync command uses the default rsync mode of checking time and size. It outputs any directory or filename with a different time or size. Because of the -n flag, no files are transferred.
The second rsync command takes the list of files with a changed time or size from the first rsync and then runs rsync again on just those files with the --checksum argument.
The second command has -n for dry run mode, so only the file names will be printed. Remove -n from the second command to actually transfer the files.
New contributor
add a comment |
Amazingly, this does not look possible with a built-in rsync option!
This may work:
rsync -an --info=name src dest
| rsync -an --info=name --checksum --include-from - --exclude="*" src dest
Explanation:
The first rsync command uses the default rsync mode of checking time and size. It outputs any directory or filename with a different time or size. Because of the -n flag, no files are transferred.
The second rsync command takes the list of files with a changed time or size from the first rsync and then runs rsync again on just those files with the --checksum argument.
The second command has -n for dry run mode, so only the file names will be printed. Remove -n from the second command to actually transfer the files.
New contributor
Amazingly, this does not look possible with a built-in rsync option!
This may work:
rsync -an --info=name src dest
| rsync -an --info=name --checksum --include-from - --exclude="*" src dest
Explanation:
The first rsync command uses the default rsync mode of checking time and size. It outputs any directory or filename with a different time or size. Because of the -n flag, no files are transferred.
The second rsync command takes the list of files with a changed time or size from the first rsync and then runs rsync again on just those files with the --checksum argument.
The second command has -n for dry run mode, so only the file names will be printed. Remove -n from the second command to actually transfer the files.
New contributor
edited yesterday
New contributor
answered Apr 3 at 21:39
presto8presto8
1012
1012
New contributor
New contributor
add a comment |
add a comment |
On the one hand, the linked question is very confusing. The answer by 9mjb is correct.
On the other hand, if you are using rsync on a local mount of a network filesystem, there is no way it can checksum the remote file without downloading the whole file anyway! So you are going to be stuck with slowness here. Assuming you have less network bandwidth available than the disk speed.
Except I don't understand what problem you are trying to explain about times, either. If you use -a
, it should transfer the exact times from the source system. If you don't use -a
, it won't transfer the original times from the source filesystem, therefore the times will not match exactly for subsequent transfers anyway. It sounds like you don't want -u
behaviour, but then you can just not use -u
:-).
rsync
is efficient by default. (At the cost of "if time and size both match, the chance that the files are different is insignificant, I'm willing to take the risk not to transfer").
[rsync] is famous for its
delta-transfer algorithm, which reduces the amount of data sent over
the network by sending only the differences between the source files
and the existing files in the destination. Rsync is widely used for
backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size
or in last-modified time.
The important thing to understand is that the rsync delta-transfer algorithm, and the "quick check", are separate.
It sounds like you don't want the behaviour of --checksum
. --checksum
disables the "quick check" part. In that case, don't use --checksum
.
-c, --checksum
This changes the way rsync checks if the files have been changed
and are in need of a transfer. Without this option, rsync uses
a "quick check" that (by default) checks if each file’s size and
time of last modification match between the sender and receiver.
This option changes this to compare a 128-bit checksum for each
file that has a matching size. Generating the checksums means
that both sides will expend a lot of disk I/O reading all the
data in the files in the transfer (and this is prior to any
reading that will be done to transfer changed files), so this
can slow things down significantly.
The sending side generates its checksums while it is doing the
file-system scan that builds the list of the available files.
The receiver generates its checksums when it is scanning for
changed files, and will checksum any file that has the same size
as the corresponding sender’s file: files with either a changed
size or a changed checksum are selected for transfer.
Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a
whole-file checksum that is generated as the file is trans‐
ferred, but that automatic after-the-transfer verification has
nothing to do with this option’s before-the-transfer "Does this
file need to be updated?" check.
add a comment |
On the one hand, the linked question is very confusing. The answer by 9mjb is correct.
On the other hand, if you are using rsync on a local mount of a network filesystem, there is no way it can checksum the remote file without downloading the whole file anyway! So you are going to be stuck with slowness here. Assuming you have less network bandwidth available than the disk speed.
Except I don't understand what problem you are trying to explain about times, either. If you use -a
, it should transfer the exact times from the source system. If you don't use -a
, it won't transfer the original times from the source filesystem, therefore the times will not match exactly for subsequent transfers anyway. It sounds like you don't want -u
behaviour, but then you can just not use -u
:-).
rsync
is efficient by default. (At the cost of "if time and size both match, the chance that the files are different is insignificant, I'm willing to take the risk not to transfer").
[rsync] is famous for its
delta-transfer algorithm, which reduces the amount of data sent over
the network by sending only the differences between the source files
and the existing files in the destination. Rsync is widely used for
backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size
or in last-modified time.
The important thing to understand is that the rsync delta-transfer algorithm, and the "quick check", are separate.
It sounds like you don't want the behaviour of --checksum
. --checksum
disables the "quick check" part. In that case, don't use --checksum
.
-c, --checksum
This changes the way rsync checks if the files have been changed
and are in need of a transfer. Without this option, rsync uses
a "quick check" that (by default) checks if each file’s size and
time of last modification match between the sender and receiver.
This option changes this to compare a 128-bit checksum for each
file that has a matching size. Generating the checksums means
that both sides will expend a lot of disk I/O reading all the
data in the files in the transfer (and this is prior to any
reading that will be done to transfer changed files), so this
can slow things down significantly.
The sending side generates its checksums while it is doing the
file-system scan that builds the list of the available files.
The receiver generates its checksums when it is scanning for
changed files, and will checksum any file that has the same size
as the corresponding sender’s file: files with either a changed
size or a changed checksum are selected for transfer.
Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a
whole-file checksum that is generated as the file is trans‐
ferred, but that automatic after-the-transfer verification has
nothing to do with this option’s before-the-transfer "Does this
file need to be updated?" check.
add a comment |
On the one hand, the linked question is very confusing. The answer by 9mjb is correct.
On the other hand, if you are using rsync on a local mount of a network filesystem, there is no way it can checksum the remote file without downloading the whole file anyway! So you are going to be stuck with slowness here. Assuming you have less network bandwidth available than the disk speed.
Except I don't understand what problem you are trying to explain about times, either. If you use -a
, it should transfer the exact times from the source system. If you don't use -a
, it won't transfer the original times from the source filesystem, therefore the times will not match exactly for subsequent transfers anyway. It sounds like you don't want -u
behaviour, but then you can just not use -u
:-).
rsync
is efficient by default. (At the cost of "if time and size both match, the chance that the files are different is insignificant, I'm willing to take the risk not to transfer").
[rsync] is famous for its
delta-transfer algorithm, which reduces the amount of data sent over
the network by sending only the differences between the source files
and the existing files in the destination. Rsync is widely used for
backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size
or in last-modified time.
The important thing to understand is that the rsync delta-transfer algorithm, and the "quick check", are separate.
It sounds like you don't want the behaviour of --checksum
. --checksum
disables the "quick check" part. In that case, don't use --checksum
.
-c, --checksum
This changes the way rsync checks if the files have been changed
and are in need of a transfer. Without this option, rsync uses
a "quick check" that (by default) checks if each file’s size and
time of last modification match between the sender and receiver.
This option changes this to compare a 128-bit checksum for each
file that has a matching size. Generating the checksums means
that both sides will expend a lot of disk I/O reading all the
data in the files in the transfer (and this is prior to any
reading that will be done to transfer changed files), so this
can slow things down significantly.
The sending side generates its checksums while it is doing the
file-system scan that builds the list of the available files.
The receiver generates its checksums when it is scanning for
changed files, and will checksum any file that has the same size
as the corresponding sender’s file: files with either a changed
size or a changed checksum are selected for transfer.
Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a
whole-file checksum that is generated as the file is trans‐
ferred, but that automatic after-the-transfer verification has
nothing to do with this option’s before-the-transfer "Does this
file need to be updated?" check.
On the one hand, the linked question is very confusing. The answer by 9mjb is correct.
On the other hand, if you are using rsync on a local mount of a network filesystem, there is no way it can checksum the remote file without downloading the whole file anyway! So you are going to be stuck with slowness here. Assuming you have less network bandwidth available than the disk speed.
Except I don't understand what problem you are trying to explain about times, either. If you use -a
, it should transfer the exact times from the source system. If you don't use -a
, it won't transfer the original times from the source filesystem, therefore the times will not match exactly for subsequent transfers anyway. It sounds like you don't want -u
behaviour, but then you can just not use -u
:-).
rsync
is efficient by default. (At the cost of "if time and size both match, the chance that the files are different is insignificant, I'm willing to take the risk not to transfer").
[rsync] is famous for its
delta-transfer algorithm, which reduces the amount of data sent over
the network by sending only the differences between the source files
and the existing files in the destination. Rsync is widely used for
backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check"
algorithm (by default) that looks for files that have changed in size
or in last-modified time.
The important thing to understand is that the rsync delta-transfer algorithm, and the "quick check", are separate.
It sounds like you don't want the behaviour of --checksum
. --checksum
disables the "quick check" part. In that case, don't use --checksum
.
-c, --checksum
This changes the way rsync checks if the files have been changed
and are in need of a transfer. Without this option, rsync uses
a "quick check" that (by default) checks if each file’s size and
time of last modification match between the sender and receiver.
This option changes this to compare a 128-bit checksum for each
file that has a matching size. Generating the checksums means
that both sides will expend a lot of disk I/O reading all the
data in the files in the transfer (and this is prior to any
reading that will be done to transfer changed files), so this
can slow things down significantly.
The sending side generates its checksums while it is doing the
file-system scan that builds the list of the available files.
The receiver generates its checksums when it is scanning for
changed files, and will checksum any file that has the same size
as the corresponding sender’s file: files with either a changed
size or a changed checksum are selected for transfer.
Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a
whole-file checksum that is generated as the file is trans‐
ferred, but that automatic after-the-transfer verification has
nothing to do with this option’s before-the-transfer "Does this
file need to be updated?" check.
edited yesterday
answered yesterday
sourcejedisourcejedi
25.8k445113
25.8k445113
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f450361%2fefficient-rsync%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
rync
's option-u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.) -- This is not exactly what you want, but should help when the time differs 'one way'. So if you 'dry run'rsync
with swapped source and target you should be able to catch time differences both ways.– sudodus
Jun 18 '18 at 4:16