Check if file exists in S3 BucketChecking for the existence of files against a listCheck whether files in a...
How much code would a codegolf golf if a codegolf could golf code?
C++20 constexpr std::copy optimizations for run-time
Why in most German places is the church the tallest building?
Sleeping solo in a double sleeping bag
Why did this happen to Thanos's ships at the end of "Avengers: Endgame"?
Are modern clipless shoes and pedals that much better than toe clips and straps?
Are illustrations in novels frowned upon?
Why don't electrons take the shorter path in coils?
Why did MS-DOS applications built using Turbo Pascal fail to start with a division by zero error on faster systems?
Can't stopover at Sapporo when going from Asahikawa to Chitose airport?
Start from ones
Accent on í misaligned in bibliography / citation
Are there account age or level requirements for obtaining special research?
Avoiding racist tropes in fantasy
How should I face my manager if I make a mistake because a senior coworker explained something incorrectly to me?
Why do all fields in a QFT transform like *irreducible* representations of some group?
Is “I am getting married with my sister” ambiguous?
Can realistic planetary invasion have any meaningful strategy?
Using `With[...]` with a list specification as a variable
In an emergency, how do I find and share my position?
Are there any music source codes for sound chips?
Notepad++ - How to find multiple values on the same line in any permutation
Does travel insurance for short flight delays exist?
Fried gnocchi with spinach, bacon, cream sauce in a single pan
Check if file exists in S3 Bucket
Checking for the existence of files against a listCheck whether files in a file list exist in a certain directoryLinux Check Image Exists at Apache URLBranch File Server Caching- LinuxCheck if a file exists on a remote machine with spaces in the filename and/or pathUntil loop will not closexargs with aws cli (amazon s3 copy)Check if file exist, isn't empty and equal to anotherUpdate file with multiple values automaticallybash how to check size of the swap
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
This directory /data/files/ has thousands files like:
1test
2test
3test
[...]
60000test
60001test
I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be offline and because of that the file is skipped.
How can I check if the file that exists in /data/files/ is also in the S3 Bucket? and if not copy the missing file to S3?
I would prefer to do this using BASH. Also if I need to change the AWS CLI for another one, can be.
bash files aws amazon-s3 amazon
add a comment |
This directory /data/files/ has thousands files like:
1test
2test
3test
[...]
60000test
60001test
I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be offline and because of that the file is skipped.
How can I check if the file that exists in /data/files/ is also in the S3 Bucket? and if not copy the missing file to S3?
I would prefer to do this using BASH. Also if I need to change the AWS CLI for another one, can be.
bash files aws amazon-s3 amazon
There are a bunch of command-line tools that talk to S3 such ass3cmdands4cmdand FUSE filesystems such as s3fs and s3ql. There are also things likerclonewhich probably solve your entire problem for you. What are you currently using to talk to S3?
– derobert
Jan 23 '17 at 22:49
@derobert i'm using theaws cli- If you have an example to help please feel free to answer the question.
– Patrick B.
Jan 23 '17 at 23:12
I'd thinkrclone copy /data/files whatever:would do everything for you... But anyway, you should edit your question to clarify which software you're using to talk to AWS. And if you're open to switching.
– derobert
Jan 23 '17 at 23:13
add a comment |
This directory /data/files/ has thousands files like:
1test
2test
3test
[...]
60000test
60001test
I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be offline and because of that the file is skipped.
How can I check if the file that exists in /data/files/ is also in the S3 Bucket? and if not copy the missing file to S3?
I would prefer to do this using BASH. Also if I need to change the AWS CLI for another one, can be.
bash files aws amazon-s3 amazon
This directory /data/files/ has thousands files like:
1test
2test
3test
[...]
60000test
60001test
I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be offline and because of that the file is skipped.
How can I check if the file that exists in /data/files/ is also in the S3 Bucket? and if not copy the missing file to S3?
I would prefer to do this using BASH. Also if I need to change the AWS CLI for another one, can be.
bash files aws amazon-s3 amazon
bash files aws amazon-s3 amazon
edited Jan 23 '17 at 23:32
Patrick B.
asked Jan 23 '17 at 22:16
Patrick B.Patrick B.
291 gold badge1 silver badge4 bronze badges
291 gold badge1 silver badge4 bronze badges
There are a bunch of command-line tools that talk to S3 such ass3cmdands4cmdand FUSE filesystems such as s3fs and s3ql. There are also things likerclonewhich probably solve your entire problem for you. What are you currently using to talk to S3?
– derobert
Jan 23 '17 at 22:49
@derobert i'm using theaws cli- If you have an example to help please feel free to answer the question.
– Patrick B.
Jan 23 '17 at 23:12
I'd thinkrclone copy /data/files whatever:would do everything for you... But anyway, you should edit your question to clarify which software you're using to talk to AWS. And if you're open to switching.
– derobert
Jan 23 '17 at 23:13
add a comment |
There are a bunch of command-line tools that talk to S3 such ass3cmdands4cmdand FUSE filesystems such as s3fs and s3ql. There are also things likerclonewhich probably solve your entire problem for you. What are you currently using to talk to S3?
– derobert
Jan 23 '17 at 22:49
@derobert i'm using theaws cli- If you have an example to help please feel free to answer the question.
– Patrick B.
Jan 23 '17 at 23:12
I'd thinkrclone copy /data/files whatever:would do everything for you... But anyway, you should edit your question to clarify which software you're using to talk to AWS. And if you're open to switching.
– derobert
Jan 23 '17 at 23:13
There are a bunch of command-line tools that talk to S3 such as
s3cmd and s4cmd and FUSE filesystems such as s3fs and s3ql. There are also things like rclone which probably solve your entire problem for you. What are you currently using to talk to S3?– derobert
Jan 23 '17 at 22:49
There are a bunch of command-line tools that talk to S3 such as
s3cmd and s4cmd and FUSE filesystems such as s3fs and s3ql. There are also things like rclone which probably solve your entire problem for you. What are you currently using to talk to S3?– derobert
Jan 23 '17 at 22:49
@derobert i'm using the
aws cli - If you have an example to help please feel free to answer the question.– Patrick B.
Jan 23 '17 at 23:12
@derobert i'm using the
aws cli - If you have an example to help please feel free to answer the question.– Patrick B.
Jan 23 '17 at 23:12
I'd think
rclone copy /data/files whatever: would do everything for you... But anyway, you should edit your question to clarify which software you're using to talk to AWS. And if you're open to switching.– derobert
Jan 23 '17 at 23:13
I'd think
rclone copy /data/files whatever: would do everything for you... But anyway, you should edit your question to clarify which software you're using to talk to AWS. And if you're open to switching.– derobert
Jan 23 '17 at 23:13
add a comment |
4 Answers
4
active
oldest
votes
If you do aws s3 ls on the actual filename. If the filename exists, the exit code will be 0 and the filename will be displayed, otherwise, the exit code will not be 0:
aws s3 ls s3://bucket/filname
if [[ $? -ne 0 ]]; then
echo "File does not exist"
fi
1
The problem with this is thats3 lswill list the file and give a return code of 0 (success) even if you provide a partial path. For example,aws s3 ls s3://bucket/filenwill list the files3://bucket/filename.
– Donnie Cameron
Mar 2 at 2:02
add a comment |
first answer is close but in cases where you use -e in shebang, the script will fail which you would most like not want. It is better to use wordcount. So you can use the below command:
wordcount=`aws s3 ls s3://${S3_BUCKET_NAME}/${folder}/|grep $${file}|wc -c`
echo wordcount=${wordcount}
if [[ "${wordcount}" -eq 0 ]]; then
do something
else
do something
fi
1
$${file}will expand to the PID and{file}, probably not what you meant.
– Benjamin W.
Aug 23 '18 at 20:55
add a comment |
I was able to do it using rclone[1] as @derobert has suggested.
The command is very simple:
rclone check sourcepath remote:s3bucketname
Example:
Let's imagine you want to check if the S3 bucket (bucket name: tmp_data_test_bucket) has all the files that this directory has: /tmp/data/
Command:
rclone check /tmp/data/ remote:tmp_data_test_bucket
[1] http://rclone.org/
rclone is not a native call, I don't consider this the best solution.
– Mike Q
2 days ago
add a comment |
I created two functions as an example because I figured I might want to know the size of the file and I may want to know if the file exists.
This function gets the size of the file and "returns" it as an echo:
s3_file_size() {
if command -v aws &> /dev/null; then
echo "$(aws s3 ls "${1}" --summarize | grep "Total.*Size" | grep -o -E '[0-9]+')"
return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
This function uses the other one to determine if it receives a file size of 0 which will imply the file is essentially not there. (yes it will treat a file of size 0 as not there)
s3_does_file_exist() {
if command -v aws &> /dev/null; then
[[ $(s3_file_size "${1}") -lt 1 ]] && return 1 || return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f339650%2fcheck-if-file-exists-in-s3-bucket%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
4 Answers
4
active
oldest
votes
4 Answers
4
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you do aws s3 ls on the actual filename. If the filename exists, the exit code will be 0 and the filename will be displayed, otherwise, the exit code will not be 0:
aws s3 ls s3://bucket/filname
if [[ $? -ne 0 ]]; then
echo "File does not exist"
fi
1
The problem with this is thats3 lswill list the file and give a return code of 0 (success) even if you provide a partial path. For example,aws s3 ls s3://bucket/filenwill list the files3://bucket/filename.
– Donnie Cameron
Mar 2 at 2:02
add a comment |
If you do aws s3 ls on the actual filename. If the filename exists, the exit code will be 0 and the filename will be displayed, otherwise, the exit code will not be 0:
aws s3 ls s3://bucket/filname
if [[ $? -ne 0 ]]; then
echo "File does not exist"
fi
1
The problem with this is thats3 lswill list the file and give a return code of 0 (success) even if you provide a partial path. For example,aws s3 ls s3://bucket/filenwill list the files3://bucket/filename.
– Donnie Cameron
Mar 2 at 2:02
add a comment |
If you do aws s3 ls on the actual filename. If the filename exists, the exit code will be 0 and the filename will be displayed, otherwise, the exit code will not be 0:
aws s3 ls s3://bucket/filname
if [[ $? -ne 0 ]]; then
echo "File does not exist"
fi
If you do aws s3 ls on the actual filename. If the filename exists, the exit code will be 0 and the filename will be displayed, otherwise, the exit code will not be 0:
aws s3 ls s3://bucket/filname
if [[ $? -ne 0 ]]; then
echo "File does not exist"
fi
edited Feb 20 at 13:07
Mr Chow
32 bronze badges
32 bronze badges
answered Jun 13 '17 at 17:55
onetwopunchonetwopunch
1811 silver badge3 bronze badges
1811 silver badge3 bronze badges
1
The problem with this is thats3 lswill list the file and give a return code of 0 (success) even if you provide a partial path. For example,aws s3 ls s3://bucket/filenwill list the files3://bucket/filename.
– Donnie Cameron
Mar 2 at 2:02
add a comment |
1
The problem with this is thats3 lswill list the file and give a return code of 0 (success) even if you provide a partial path. For example,aws s3 ls s3://bucket/filenwill list the files3://bucket/filename.
– Donnie Cameron
Mar 2 at 2:02
1
1
The problem with this is that
s3 ls will list the file and give a return code of 0 (success) even if you provide a partial path. For example, aws s3 ls s3://bucket/filen will list the file s3://bucket/filename.– Donnie Cameron
Mar 2 at 2:02
The problem with this is that
s3 ls will list the file and give a return code of 0 (success) even if you provide a partial path. For example, aws s3 ls s3://bucket/filen will list the file s3://bucket/filename.– Donnie Cameron
Mar 2 at 2:02
add a comment |
first answer is close but in cases where you use -e in shebang, the script will fail which you would most like not want. It is better to use wordcount. So you can use the below command:
wordcount=`aws s3 ls s3://${S3_BUCKET_NAME}/${folder}/|grep $${file}|wc -c`
echo wordcount=${wordcount}
if [[ "${wordcount}" -eq 0 ]]; then
do something
else
do something
fi
1
$${file}will expand to the PID and{file}, probably not what you meant.
– Benjamin W.
Aug 23 '18 at 20:55
add a comment |
first answer is close but in cases where you use -e in shebang, the script will fail which you would most like not want. It is better to use wordcount. So you can use the below command:
wordcount=`aws s3 ls s3://${S3_BUCKET_NAME}/${folder}/|grep $${file}|wc -c`
echo wordcount=${wordcount}
if [[ "${wordcount}" -eq 0 ]]; then
do something
else
do something
fi
1
$${file}will expand to the PID and{file}, probably not what you meant.
– Benjamin W.
Aug 23 '18 at 20:55
add a comment |
first answer is close but in cases where you use -e in shebang, the script will fail which you would most like not want. It is better to use wordcount. So you can use the below command:
wordcount=`aws s3 ls s3://${S3_BUCKET_NAME}/${folder}/|grep $${file}|wc -c`
echo wordcount=${wordcount}
if [[ "${wordcount}" -eq 0 ]]; then
do something
else
do something
fi
first answer is close but in cases where you use -e in shebang, the script will fail which you would most like not want. It is better to use wordcount. So you can use the below command:
wordcount=`aws s3 ls s3://${S3_BUCKET_NAME}/${folder}/|grep $${file}|wc -c`
echo wordcount=${wordcount}
if [[ "${wordcount}" -eq 0 ]]; then
do something
else
do something
fi
edited Sep 7 '18 at 22:57
Rui F Ribeiro
41.3k16 gold badges94 silver badges158 bronze badges
41.3k16 gold badges94 silver badges158 bronze badges
answered Aug 23 '18 at 20:33
Anuj AgrawalAnuj Agrawal
211 bronze badge
211 bronze badge
1
$${file}will expand to the PID and{file}, probably not what you meant.
– Benjamin W.
Aug 23 '18 at 20:55
add a comment |
1
$${file}will expand to the PID and{file}, probably not what you meant.
– Benjamin W.
Aug 23 '18 at 20:55
1
1
$${file} will expand to the PID and {file}, probably not what you meant.– Benjamin W.
Aug 23 '18 at 20:55
$${file} will expand to the PID and {file}, probably not what you meant.– Benjamin W.
Aug 23 '18 at 20:55
add a comment |
I was able to do it using rclone[1] as @derobert has suggested.
The command is very simple:
rclone check sourcepath remote:s3bucketname
Example:
Let's imagine you want to check if the S3 bucket (bucket name: tmp_data_test_bucket) has all the files that this directory has: /tmp/data/
Command:
rclone check /tmp/data/ remote:tmp_data_test_bucket
[1] http://rclone.org/
rclone is not a native call, I don't consider this the best solution.
– Mike Q
2 days ago
add a comment |
I was able to do it using rclone[1] as @derobert has suggested.
The command is very simple:
rclone check sourcepath remote:s3bucketname
Example:
Let's imagine you want to check if the S3 bucket (bucket name: tmp_data_test_bucket) has all the files that this directory has: /tmp/data/
Command:
rclone check /tmp/data/ remote:tmp_data_test_bucket
[1] http://rclone.org/
rclone is not a native call, I don't consider this the best solution.
– Mike Q
2 days ago
add a comment |
I was able to do it using rclone[1] as @derobert has suggested.
The command is very simple:
rclone check sourcepath remote:s3bucketname
Example:
Let's imagine you want to check if the S3 bucket (bucket name: tmp_data_test_bucket) has all the files that this directory has: /tmp/data/
Command:
rclone check /tmp/data/ remote:tmp_data_test_bucket
[1] http://rclone.org/
I was able to do it using rclone[1] as @derobert has suggested.
The command is very simple:
rclone check sourcepath remote:s3bucketname
Example:
Let's imagine you want to check if the S3 bucket (bucket name: tmp_data_test_bucket) has all the files that this directory has: /tmp/data/
Command:
rclone check /tmp/data/ remote:tmp_data_test_bucket
[1] http://rclone.org/
edited Jan 29 '17 at 21:20
Jeff Schaller♦
49.1k11 gold badges72 silver badges163 bronze badges
49.1k11 gold badges72 silver badges163 bronze badges
answered Jan 24 '17 at 19:01
Patrick B.Patrick B.
291 gold badge1 silver badge4 bronze badges
291 gold badge1 silver badge4 bronze badges
rclone is not a native call, I don't consider this the best solution.
– Mike Q
2 days ago
add a comment |
rclone is not a native call, I don't consider this the best solution.
– Mike Q
2 days ago
rclone is not a native call, I don't consider this the best solution.
– Mike Q
2 days ago
rclone is not a native call, I don't consider this the best solution.
– Mike Q
2 days ago
add a comment |
I created two functions as an example because I figured I might want to know the size of the file and I may want to know if the file exists.
This function gets the size of the file and "returns" it as an echo:
s3_file_size() {
if command -v aws &> /dev/null; then
echo "$(aws s3 ls "${1}" --summarize | grep "Total.*Size" | grep -o -E '[0-9]+')"
return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
This function uses the other one to determine if it receives a file size of 0 which will imply the file is essentially not there. (yes it will treat a file of size 0 as not there)
s3_does_file_exist() {
if command -v aws &> /dev/null; then
[[ $(s3_file_size "${1}") -lt 1 ]] && return 1 || return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
add a comment |
I created two functions as an example because I figured I might want to know the size of the file and I may want to know if the file exists.
This function gets the size of the file and "returns" it as an echo:
s3_file_size() {
if command -v aws &> /dev/null; then
echo "$(aws s3 ls "${1}" --summarize | grep "Total.*Size" | grep -o -E '[0-9]+')"
return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
This function uses the other one to determine if it receives a file size of 0 which will imply the file is essentially not there. (yes it will treat a file of size 0 as not there)
s3_does_file_exist() {
if command -v aws &> /dev/null; then
[[ $(s3_file_size "${1}") -lt 1 ]] && return 1 || return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
add a comment |
I created two functions as an example because I figured I might want to know the size of the file and I may want to know if the file exists.
This function gets the size of the file and "returns" it as an echo:
s3_file_size() {
if command -v aws &> /dev/null; then
echo "$(aws s3 ls "${1}" --summarize | grep "Total.*Size" | grep -o -E '[0-9]+')"
return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
This function uses the other one to determine if it receives a file size of 0 which will imply the file is essentially not there. (yes it will treat a file of size 0 as not there)
s3_does_file_exist() {
if command -v aws &> /dev/null; then
[[ $(s3_file_size "${1}") -lt 1 ]] && return 1 || return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
I created two functions as an example because I figured I might want to know the size of the file and I may want to know if the file exists.
This function gets the size of the file and "returns" it as an echo:
s3_file_size() {
if command -v aws &> /dev/null; then
echo "$(aws s3 ls "${1}" --summarize | grep "Total.*Size" | grep -o -E '[0-9]+')"
return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
This function uses the other one to determine if it receives a file size of 0 which will imply the file is essentially not there. (yes it will treat a file of size 0 as not there)
s3_does_file_exist() {
if command -v aws &> /dev/null; then
[[ $(s3_file_size "${1}") -lt 1 ]] && return 1 || return 0
else
echo "Warn-${FUNCNAME[0]}, AWS command missing."
return 1
fi
}
answered 2 days ago
Mike QMike Q
1114 bronze badges
1114 bronze badges
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f339650%2fcheck-if-file-exists-in-s3-bucket%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There are a bunch of command-line tools that talk to S3 such as
s3cmdands4cmdand FUSE filesystems such as s3fs and s3ql. There are also things likerclonewhich probably solve your entire problem for you. What are you currently using to talk to S3?– derobert
Jan 23 '17 at 22:49
@derobert i'm using the
aws cli- If you have an example to help please feel free to answer the question.– Patrick B.
Jan 23 '17 at 23:12
I'd think
rclone copy /data/files whatever:would do everything for you... But anyway, you should edit your question to clarify which software you're using to talk to AWS. And if you're open to switching.– derobert
Jan 23 '17 at 23:13