Mdthbs

German equivalent to "going down the rabbit hole"

I was reported to HR as being a satan worshiper

Why are JWST optics not enclosed like HST?

Are sweatpants frowned upon on flights?

What is a "hashed transaction" in SQL Server Replication terminology?

Did the Apollo Guidance Computer really use 60% of the world's ICs in 1963?

Journal published a paper, ignoring my objections as a referee

How do I keep my animals from eating my people food?

Ask one verbal question to figure out who is blind and who is mute among three persons

“all of who” or “all of whom”?

Can UV radiation be safe for the skin?

In what language did Túrin converse with Mím?

What's the origin of the concept of alternate dimensions/realities?

Could a complex system of reaction wheels be used to propel a spacecraft?

Unexpected behavior after assignment of function object to function wrapper

Is it good practice to speed up and slow down where not written in a song?

Eshet Chayil in the Tunisian service

Necessity of tenure for lifetime academic research

What is the motivation behind designing a control stick that does not move?

How to differentiate between two people with the same name in a story?

LWC: Is it safe to rely on window.location.href to get the page url?

Sum and average calculator

Break down the phrase "shitsurei shinakereba naranaindesu"

Magnetic thread storage?

Launching Multiple Queries With Bash Script

parallel processing using xargsSave password in bash script for multiple queriescURL download with multiple rangescurl download multiple files with brace syntaxHow do I decode a list of base64-encoded file names?Mysql queries from bash filecurl with variables on bash scriptHow to access http server from bash script with existing tcp connection?Wireshark not showing HTTP queries

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

1

I have this script (based on this post) to check the http-status of domains list (000, 418, 404 etc) and i would like to improve:

#!/bin/bash

# change path to your url list



yourlist=$(pwd)/url-list.txt



advance=$(cat advance.txt 2>/dev/null || echo 1)

while read LINE; do

  curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"

  echo " $LINE"

  advance=$(($advance+1))

  echo $advance > advance.txt

done < <(tail -n +$advance $yourlist) >> out

infile:

amazon.com

google.com

outfile:

301 google.com

301 amazon.com

If the script is interrupted for any reason, it starts from the last processed line.

Problem: It is very slow since it verifies line by line.

Tests: I have already tried other alternatives, such as fping (very limited given the size of the list), pyfunceble (freezes PC), wget (doesn't do what i expect), GNU parallel, etc etc. None has convinced me.

Without test: A user recommended this answer based on 'xarg -P' (xargs -P 4 -L 1 -n 1), but I don't know how to implement it because I don't have knowledge about xarg. I also found another similar answer in bash, but it hasn't worked so far.

Question: How can I launch multiple queries (parallel processing) with my script so that I could process many lines at the same time (if it would be possible to set the number of lines to be processed manually, avoiding freeze or blocking the script or PC)?

bash curl http

share|improve this question

edited 1 hour ago

ajcg

asked 4 hours ago

ajcgajcg

29244 silver badges1717 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Probably helpful: correct xargs parallel usage and parallel processing using xargs and similar questions.
  
  – Kusalananda♦
  4 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @Kusalananda very interesting. checking. thk
  
  – ajcg
  3 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @Kusalananda It would be something like enclosing my script inside xarg: xargs -P 4 -L 1 sh -c '<my script curl> $0 || exit 255' < url-list.txt > etc. But I have no idea. I know little about xarg
  
  – ajcg
  3 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens. linux.die.net/man/1/xargs
  
  – ajcg
  2 hours ago
  

  
  
  
  

add a comment | 

1

I have this script (based on this post) to check the http-status of domains list (000, 418, 404 etc) and i would like to improve:

#!/bin/bash

# change path to your url list



yourlist=$(pwd)/url-list.txt



advance=$(cat advance.txt 2>/dev/null || echo 1)

while read LINE; do

  curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"

  echo " $LINE"

  advance=$(($advance+1))

  echo $advance > advance.txt

done < <(tail -n +$advance $yourlist) >> out

infile:

amazon.com

google.com

outfile:

301 google.com

301 amazon.com

If the script is interrupted for any reason, it starts from the last processed line.

Problem: It is very slow since it verifies line by line.

Tests: I have already tried other alternatives, such as fping (very limited given the size of the list), pyfunceble (freezes PC), wget (doesn't do what i expect), GNU parallel, etc etc. None has convinced me.

Without test: A user recommended this answer based on 'xarg -P' (xargs -P 4 -L 1 -n 1), but I don't know how to implement it because I don't have knowledge about xarg. I also found another similar answer in bash, but it hasn't worked so far.

Question: How can I launch multiple queries (parallel processing) with my script so that I could process many lines at the same time (if it would be possible to set the number of lines to be processed manually, avoiding freeze or blocking the script or PC)?

bash curl http

share|improve this question

edited 1 hour ago

ajcg

asked 4 hours ago

ajcgajcg

29244 silver badges1717 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Probably helpful: correct xargs parallel usage and parallel processing using xargs and similar questions.
  
  – Kusalananda♦
  4 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @Kusalananda very interesting. checking. thk
  
  – ajcg
  3 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @Kusalananda It would be something like enclosing my script inside xarg: xargs -P 4 -L 1 sh -c '<my script curl> $0 || exit 255' < url-list.txt > etc. But I have no idea. I know little about xarg
  
  – ajcg
  3 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens. linux.die.net/man/1/xargs
  
  – ajcg
  2 hours ago
  

  
  
  
  

add a comment | 

I have this script (based on this post) to check the http-status of domains list (000, 418, 404 etc) and i would like to improve:

#!/bin/bash

# change path to your url list



yourlist=$(pwd)/url-list.txt



advance=$(cat advance.txt 2>/dev/null || echo 1)

while read LINE; do

  curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"

  echo " $LINE"

  advance=$(($advance+1))

  echo $advance > advance.txt

done < <(tail -n +$advance $yourlist) >> out

infile:

amazon.com

google.com

outfile:

301 google.com

301 amazon.com

If the script is interrupted for any reason, it starts from the last processed line.

Problem: It is very slow since it verifies line by line.

Tests: I have already tried other alternatives, such as fping (very limited given the size of the list), pyfunceble (freezes PC), wget (doesn't do what i expect), GNU parallel, etc etc. None has convinced me.

Without test: A user recommended this answer based on 'xarg -P' (xargs -P 4 -L 1 -n 1), but I don't know how to implement it because I don't have knowledge about xarg. I also found another similar answer in bash, but it hasn't worked so far.

Question: How can I launch multiple queries (parallel processing) with my script so that I could process many lines at the same time (if it would be possible to set the number of lines to be processed manually, avoiding freeze or blocking the script or PC)?

bash curl http

share|improve this question

edited 1 hour ago

ajcg

asked 4 hours ago

ajcgajcg

29244 silver badges1717 bronze badges

#!/bin/bash

# change path to your url list



yourlist=$(pwd)/url-list.txt



advance=$(cat advance.txt 2>/dev/null || echo 1)

while read LINE; do

  curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"

  echo " $LINE"

  advance=$(($advance+1))

  echo $advance > advance.txt

done < <(tail -n +$advance $yourlist) >> out

amazon.com

google.com

301 google.com

301 amazon.com

share|improve this question

edited 1 hour ago

ajcg

asked 4 hours ago

ajcgajcg

29244 silver badges1717 bronze badges

share|improve this question

edited 1 hour ago

ajcg

asked 4 hours ago

ajcgajcg

29244 silver badges1717 bronze badges

asked 4 hours ago

ajcgajcg

29244 silver badges1717 bronze badges

asked 4 hours ago

ajcgajcg

29244 silver badges1717 bronze badges

1 Answer
1

active

oldest

votes

0

you could split up your url list in e.g. 10 parts and use a main script ala

./subscript1.sh &

./subscript2.sh &

...

./subscript10.sh &

to run it in parallel. Caution: They should use distinct log files.

share|improve this answer

answered 2 hours ago

mifritschermifritscher

17188 bronze badges

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f538340%2flaunching-multiple-queries-with-bash-script%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

0

you could split up your url list in e.g. 10 parts and use a main script ala

./subscript1.sh &

./subscript2.sh &

...

./subscript10.sh &

to run it in parallel. Caution: They should use distinct log files.

share|improve this answer

answered 2 hours ago

mifritschermifritscher

17188 bronze badges

add a comment | 

0

you could split up your url list in e.g. 10 parts and use a main script ala

./subscript1.sh &

./subscript2.sh &

...

./subscript10.sh &

to run it in parallel. Caution: They should use distinct log files.

share|improve this answer

answered 2 hours ago

mifritschermifritscher

17188 bronze badges

add a comment | 

you could split up your url list in e.g. 10 parts and use a main script ala

./subscript1.sh &

./subscript2.sh &

...

./subscript10.sh &

to run it in parallel. Caution: They should use distinct log files.

share|improve this answer

answered 2 hours ago

mifritschermifritscher

17188 bronze badges

./subscript1.sh &

./subscript2.sh &

...

./subscript10.sh &

share|improve this answer

answered 2 hours ago

mifritschermifritscher

17188 bronze badges

answered 2 hours ago

mifritschermifritscher

17188 bronze badges

answered 2 hours ago

mifritschermifritscher

17188 bronze badges