Execute a command once per line of piped input? The 2019 Stack Overflow Developer Survey...

Is an up-to-date browser secure on an out-of-date OS?

Why are there uneven bright areas in this photo of black hole?

Can a rogue use sneak attack with weapons that have the thrown property even if they are not thrown?

Can there be female White Walkers?

Can a flute soloist sit?

Button changing its text & action. Good or terrible?

For what reasons would an animal species NOT cross a *horizontal* land bridge?

Getting crown tickets for Statue of Liberty

Is it correct to say the Neural Networks are an alternative way of performing Maximum Likelihood Estimation? if not, why?

Pokemon Turn Based battle (Python)

Worn-tile Scrabble

Is it safe to harvest rainwater that fell on solar panels?

Is it ethical to upload a automatically generated paper to a non peer-reviewed site as part of a larger research?

Is it okay to consider publishing in my first year of PhD?

Is Cinnamon a desktop environment or a window manager? (Or both?)

Why didn't the Event Horizon Telescope team mention Sagittarius A*?

How much of the clove should I use when using big garlic heads?

Keeping a retro style to sci-fi spaceships?

Is bread bad for ducks?

How to notate time signature switching consistently every measure

How do you keep chess fun when your opponent constantly beats you?

Why can't devices on different VLANs, but on the same subnet, communicate?

Relationship between Gromov-Witten and Taubes' Gromov invariant

How to type a long/em dash `—`



Execute a command once per line of piped input?



The 2019 Stack Overflow Developer Survey Results Are InGeneral specification for command line interfaceHow a piped shell programs balance their output/input rates?Is there a line-by-line bash pipe?Queue a task in a running shellHow can I add default command line parameters to an executable in a way that will also work on GUI apps?execute command from find called by findWhy 'kill -l' gives different output in fish and bashLooking for fish theme with powerline, full path and return and carriage return after the prompt lineStart bash then autostart fish with terminatorGrep Ignoring Pipe?





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}







141















I want to run a java command once for every match of ls | grep pattern -. In this case, I think I could do find pattern -exec java MyProg '{}' ; but I'm curious about the general case - is there an easy way to say "run a command once for every line of standard input"? (In fish or bash.)










share|improve this question





























    141















    I want to run a java command once for every match of ls | grep pattern -. In this case, I think I could do find pattern -exec java MyProg '{}' ; but I'm curious about the general case - is there an easy way to say "run a command once for every line of standard input"? (In fish or bash.)










    share|improve this question

























      141












      141








      141


      36






      I want to run a java command once for every match of ls | grep pattern -. In this case, I think I could do find pattern -exec java MyProg '{}' ; but I'm curious about the general case - is there an easy way to say "run a command once for every line of standard input"? (In fish or bash.)










      share|improve this question














      I want to run a java command once for every match of ls | grep pattern -. In this case, I think I could do find pattern -exec java MyProg '{}' ; but I'm curious about the general case - is there an easy way to say "run a command once for every line of standard input"? (In fish or bash.)







      bash shell fish






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Feb 17 '11 at 4:26









      XodarapXodarap

      1,16321215




      1,16321215






















          10 Answers
          10






          active

          oldest

          votes


















          85














          That's what xargs does.



          ... | xargs command





          share|improve this answer





















          • 24





            Not quite. printf "foo barnbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?

            – Jander
            Feb 17 '11 at 5:17








          • 3





            @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.

            – Keith
            Feb 17 '11 at 5:25






          • 4





            ... | tr 'n' '' | xargs -0

            – vrdhn
            Feb 17 '11 at 11:12






          • 7





            like, "the specific circumstances which give the right answer to the question". :)

            – mattdm
            Feb 17 '11 at 18:17






          • 7





            If you want to see the proper way to do this with xargs, see my answer below.

            – Michael Goldshteyn
            Jun 30 '15 at 16:08



















          146














          The accepted answer has the right idea, but the key is to pass xargs the -n1 switch, which means "Execute the command once per line of output:"



          cat file... | xargs -n1 command


          Or, for a single input file you can avoid the pipe from cat entirely and just go with:



          <file xargs -n1 command





          share|improve this answer





















          • 1





            Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.

            – Ronan Jouchet
            Oct 24 '15 at 19:58








          • 4





            How dyou access the line inside command ?

            – B T
            Apr 14 '16 at 1:21











          • This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.

            – masterxilo
            Mar 15 '18 at 9:43






          • 1





            printf "foo barnbaz bat" | xargs -n1 echo whee splits by words and not by lines

            – Gismo Ranas
            Oct 24 '18 at 8:32



















          104














          In Bash or any other Bourne-style shell (ash, ksh, zsh, …):



          while read -r line; do command "$line"; done


          read -r reads a single line from standard input (read without -r interprets backslashes, you don't want that). Thus you can do either of the following:



          $ command | while read -r line; do command "$line"; done  

          $ while read -r line; do command "$line"; done <file





          share|improve this answer





















          • 5





            When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work

            – user16468
            Mar 10 '12 at 20:39













          • This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done

            – Den
            Sep 10 '18 at 9:27





















          19














          I agree with Keith, xargs is the most general tool for the job.



          I usually use a 3 step approach.




          • do the basic stuff until you have something you would like to work with

          • prepare the line with awk so it gets the correct syntax

          • then let xargs execute it, maybe with the help of bash.


          There are smaller and faster ways, but this ways almost always works.



          A simple example:



          ls | 
          grep xls |
          awk '{print "MyJavaProg --arg1 42 --arg2 "$1""}' |
          xargs -0 bash -c


          the 2 first lines selects some files to work with,
          then awk prepares a nice string with a command to execute and some arguments and $1 is the first column input from the pipe.
          And finally I make sure that xargs sends this string to bash that just execute it.



          It is a little bit overkill, but this recipe has helped me in a lot of places since it is very flexible.






          share|improve this answer





















          • 5





            Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args "%s"",$1)

            – glenn jackman
            Feb 17 '11 at 14:30













          • @glenn: Missed the null char, will update the answer

            – Johan
            Feb 18 '11 at 12:43











          • @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print...

            – Eric Renouf
            May 31 '16 at 15:19



















          15














          GNU Parallel is made for that kind of tasks. The simplest usage is:



          cat stuff | grep pattern | parallel java MyProg


          Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ






          share|improve this answer



















          • 1





            No real need for the cat here since grep can directly read the file

            – Eric Renouf
            May 31 '16 at 15:20






          • 3





            @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html

            – Ole Tange
            May 31 '16 at 20:29






          • 1





            Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you

            – Eric Renouf
            Jun 1 '16 at 0:32



















          8














          Also, while read loop in fish shell (I assume you want fish shell, considering you used fish tag).



          command | while read line
          command $line
          end


          Few points to note.





          • read doesn't take -r argument, and it doesn't interpret your backslashes, in order to make most common use case easy.

          • You don't need to quote $line, as unlike bash, fish doesn't separate variables by spaces.


          • command by itself is a syntax error (to catch such use of placeholder arguments). Replace it with the real command.






          share|improve this answer
























          • Doesn't while need to be paired with do & done instead of end?

            – aff
            Nov 19 '18 at 7:55











          • @aff This is specifically about fish shell, which has different syntax.

            – Konrad Borowski
            Nov 19 '18 at 14:29











          • Ah, so that's what the fish means.

            – aff
            Nov 20 '18 at 6:33



















          5














          If you need to control where exactly the input argument is inserted into your command line or if you need to repeat it several times then you need to use xargs -I{}.



          EXAMPLE #1

          Create an empty folder structure in another_folder that mirrors the subfolders in the current directory:





              ls -1d ./*/ | xargs -I{} mkdir another_folder/{}


          EXAMPLE #2

          Apply an operation on a file list coming from stdin, in this case make a copy of each .html file by appending a .bak extension:



              find . -iname "*.html" | xargs -I{} cp {} {}.bak


          From the xargs man page for MacOS/BSD:



           -I replstr
          Execute utility for each input line, replacing one or more occurrences of
          replstr in up to replacements (or 5 if no -R flag is specified) arguments
          to utility with the entire line of input. The resulting arguments, after
          replacement is done, will not be allowed to grow beyond 255 bytes; this is
          implemented by concatenating as much of the argument containing replstr as
          possible, to the constructed arguments to utility, up to 255 bytes. The
          255 byte limit does not apply to arguments to utility which do not contain
          replstr, and furthermore, no replacement will be done on utility itself.
          Implies -x.


          Linux xargs man page:



             -I replace-str
          Replace occurrences of replace-str in the initial-
          arguments with names read from standard input. Al‐
          so, unquoted blanks do not terminate input items;
          instead the separator is the newline character.
          Implies -x and -L 1.





          share|improve this answer

































            1














            When dealing with potentially unsanitized inputs, I like to see the entire job 'spelled out' line by line for visual inspection before I run it (especially when it's something destructive like cleaning people's mailbox's).



            So what I do is generate a list of parameters (ie. usernames), feed it to a file in one-record-per-line fashion, like this:



            johndoe  
            jamessmith
            janebrown


            Then I open the list in vim, and mangle it with search and replace expressions until I get a list of full commands that need to get executed, like this:



            /bin/rm -fr /home/johndoe  
            /bin/rm -fr /home/jamessmith


            This way if your regex is incomplete, you will see in what command will have potential problems (ie. /bin/rm -fr johnnyo connor). This way you can undo your regex, and try it again with a more reliable version of it. Name mangling is notorious for this, because it's hard to take care of all the edge cases like Van Gogh, O'Connors, St. Clair, Smith-Wesson.



            Having set hlsearch is useful for doing this in vim, as it will highlight all the matches, so you can easily spot if it doesn't match, or matches in an unintended way.



            Once your regex is perfect and it catches all the cases you can test for/think of, then I usually convert it to a sed expression so it can be fully automated for another run.



            For cases where the number of lines of input prevents you from doing a visual inspection, I highly recommend echoing the command to the screen (or better yet, a log) before it executes, so if it errors out, you know exactly which command caused it to fail. Then you can go back to your original regex and adjust once more.






            share|improve this answer

































              0














              If a program ignores the pipe but accepts files as arguments, then you can just point it to the special file /dev/stdin.



              I am not familiar with java, but here is an example of how you would do it for bash:



              $ echo $'pwd n cd / n pwd' |bash /dev/stdin
              /home/rolf
              /


              The $ is necessary for bash to translate n into newlines. I'm not sure why.






              share|improve this answer

































                0














                I prefer this - allowing multi-line commands and clear code



                find -type f -name filenam-pattern* | while read -r F
                do
                echo $F
                cat $F | grep 'some text'
                done


                ref https://stackoverflow.com/a/3891678/248616






                share|improve this answer


























                  Your Answer








                  StackExchange.ready(function() {
                  var channelOptions = {
                  tags: "".split(" "),
                  id: "106"
                  };
                  initTagRenderer("".split(" "), "".split(" "), channelOptions);

                  StackExchange.using("externalEditor", function() {
                  // Have to fire editor after snippets, if snippets enabled
                  if (StackExchange.settings.snippets.snippetsEnabled) {
                  StackExchange.using("snippets", function() {
                  createEditor();
                  });
                  }
                  else {
                  createEditor();
                  }
                  });

                  function createEditor() {
                  StackExchange.prepareEditor({
                  heartbeatType: 'answer',
                  autoActivateHeartbeat: false,
                  convertImagesToLinks: false,
                  noModals: true,
                  showLowRepImageUploadWarning: true,
                  reputationToPostImages: null,
                  bindNavPrevention: true,
                  postfix: "",
                  imageUploader: {
                  brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
                  contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
                  allowUrls: true
                  },
                  onDemand: true,
                  discardSelector: ".discard-answer"
                  ,immediatelyShowMarkdownHelp:true
                  });


                  }
                  });














                  draft saved

                  draft discarded


















                  StackExchange.ready(
                  function () {
                  StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f7558%2fexecute-a-command-once-per-line-of-piped-input%23new-answer', 'question_page');
                  }
                  );

                  Post as a guest















                  Required, but never shown

























                  10 Answers
                  10






                  active

                  oldest

                  votes








                  10 Answers
                  10






                  active

                  oldest

                  votes









                  active

                  oldest

                  votes






                  active

                  oldest

                  votes









                  85














                  That's what xargs does.



                  ... | xargs command





                  share|improve this answer





















                  • 24





                    Not quite. printf "foo barnbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?

                    – Jander
                    Feb 17 '11 at 5:17








                  • 3





                    @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.

                    – Keith
                    Feb 17 '11 at 5:25






                  • 4





                    ... | tr 'n' '' | xargs -0

                    – vrdhn
                    Feb 17 '11 at 11:12






                  • 7





                    like, "the specific circumstances which give the right answer to the question". :)

                    – mattdm
                    Feb 17 '11 at 18:17






                  • 7





                    If you want to see the proper way to do this with xargs, see my answer below.

                    – Michael Goldshteyn
                    Jun 30 '15 at 16:08
















                  85














                  That's what xargs does.



                  ... | xargs command





                  share|improve this answer





















                  • 24





                    Not quite. printf "foo barnbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?

                    – Jander
                    Feb 17 '11 at 5:17








                  • 3





                    @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.

                    – Keith
                    Feb 17 '11 at 5:25






                  • 4





                    ... | tr 'n' '' | xargs -0

                    – vrdhn
                    Feb 17 '11 at 11:12






                  • 7





                    like, "the specific circumstances which give the right answer to the question". :)

                    – mattdm
                    Feb 17 '11 at 18:17






                  • 7





                    If you want to see the proper way to do this with xargs, see my answer below.

                    – Michael Goldshteyn
                    Jun 30 '15 at 16:08














                  85












                  85








                  85







                  That's what xargs does.



                  ... | xargs command





                  share|improve this answer















                  That's what xargs does.



                  ... | xargs command






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Aug 27 '18 at 22:20









                  Anthony Geoghegan

                  7,98154055




                  7,98154055










                  answered Feb 17 '11 at 4:58









                  KeithKeith

                  6,4081925




                  6,4081925








                  • 24





                    Not quite. printf "foo barnbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?

                    – Jander
                    Feb 17 '11 at 5:17








                  • 3





                    @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.

                    – Keith
                    Feb 17 '11 at 5:25






                  • 4





                    ... | tr 'n' '' | xargs -0

                    – vrdhn
                    Feb 17 '11 at 11:12






                  • 7





                    like, "the specific circumstances which give the right answer to the question". :)

                    – mattdm
                    Feb 17 '11 at 18:17






                  • 7





                    If you want to see the proper way to do this with xargs, see my answer below.

                    – Michael Goldshteyn
                    Jun 30 '15 at 16:08














                  • 24





                    Not quite. printf "foo barnbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?

                    – Jander
                    Feb 17 '11 at 5:17








                  • 3





                    @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.

                    – Keith
                    Feb 17 '11 at 5:25






                  • 4





                    ... | tr 'n' '' | xargs -0

                    – vrdhn
                    Feb 17 '11 at 11:12






                  • 7





                    like, "the specific circumstances which give the right answer to the question". :)

                    – mattdm
                    Feb 17 '11 at 18:17






                  • 7





                    If you want to see the proper way to do this with xargs, see my answer below.

                    – Michael Goldshteyn
                    Jun 30 '15 at 16:08








                  24




                  24





                  Not quite. printf "foo barnbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?

                  – Jander
                  Feb 17 '11 at 5:17







                  Not quite. printf "foo barnbaz bat" | xargs echo whee will yield whee foo bar baz bat. Maybe add the -L or -n options?

                  – Jander
                  Feb 17 '11 at 5:17






                  3




                  3





                  @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.

                  – Keith
                  Feb 17 '11 at 5:25





                  @Jander The question was rather general, so I gave the general tool. True, you will have to adjust its behavior with options depending on the specific circumstances.

                  – Keith
                  Feb 17 '11 at 5:25




                  4




                  4





                  ... | tr 'n' '' | xargs -0

                  – vrdhn
                  Feb 17 '11 at 11:12





                  ... | tr 'n' '' | xargs -0

                  – vrdhn
                  Feb 17 '11 at 11:12




                  7




                  7





                  like, "the specific circumstances which give the right answer to the question". :)

                  – mattdm
                  Feb 17 '11 at 18:17





                  like, "the specific circumstances which give the right answer to the question". :)

                  – mattdm
                  Feb 17 '11 at 18:17




                  7




                  7





                  If you want to see the proper way to do this with xargs, see my answer below.

                  – Michael Goldshteyn
                  Jun 30 '15 at 16:08





                  If you want to see the proper way to do this with xargs, see my answer below.

                  – Michael Goldshteyn
                  Jun 30 '15 at 16:08













                  146














                  The accepted answer has the right idea, but the key is to pass xargs the -n1 switch, which means "Execute the command once per line of output:"



                  cat file... | xargs -n1 command


                  Or, for a single input file you can avoid the pipe from cat entirely and just go with:



                  <file xargs -n1 command





                  share|improve this answer





















                  • 1





                    Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.

                    – Ronan Jouchet
                    Oct 24 '15 at 19:58








                  • 4





                    How dyou access the line inside command ?

                    – B T
                    Apr 14 '16 at 1:21











                  • This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.

                    – masterxilo
                    Mar 15 '18 at 9:43






                  • 1





                    printf "foo barnbaz bat" | xargs -n1 echo whee splits by words and not by lines

                    – Gismo Ranas
                    Oct 24 '18 at 8:32
















                  146














                  The accepted answer has the right idea, but the key is to pass xargs the -n1 switch, which means "Execute the command once per line of output:"



                  cat file... | xargs -n1 command


                  Or, for a single input file you can avoid the pipe from cat entirely and just go with:



                  <file xargs -n1 command





                  share|improve this answer





















                  • 1





                    Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.

                    – Ronan Jouchet
                    Oct 24 '15 at 19:58








                  • 4





                    How dyou access the line inside command ?

                    – B T
                    Apr 14 '16 at 1:21











                  • This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.

                    – masterxilo
                    Mar 15 '18 at 9:43






                  • 1





                    printf "foo barnbaz bat" | xargs -n1 echo whee splits by words and not by lines

                    – Gismo Ranas
                    Oct 24 '18 at 8:32














                  146












                  146








                  146







                  The accepted answer has the right idea, but the key is to pass xargs the -n1 switch, which means "Execute the command once per line of output:"



                  cat file... | xargs -n1 command


                  Or, for a single input file you can avoid the pipe from cat entirely and just go with:



                  <file xargs -n1 command





                  share|improve this answer















                  The accepted answer has the right idea, but the key is to pass xargs the -n1 switch, which means "Execute the command once per line of output:"



                  cat file... | xargs -n1 command


                  Or, for a single input file you can avoid the pipe from cat entirely and just go with:



                  <file xargs -n1 command






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Apr 13 '17 at 12:37









                  Community

                  1




                  1










                  answered Nov 18 '14 at 15:44









                  Michael GoldshteynMichael Goldshteyn

                  1,572196




                  1,572196








                  • 1





                    Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.

                    – Ronan Jouchet
                    Oct 24 '15 at 19:58








                  • 4





                    How dyou access the line inside command ?

                    – B T
                    Apr 14 '16 at 1:21











                  • This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.

                    – masterxilo
                    Mar 15 '18 at 9:43






                  • 1





                    printf "foo barnbaz bat" | xargs -n1 echo whee splits by words and not by lines

                    – Gismo Ranas
                    Oct 24 '18 at 8:32














                  • 1





                    Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.

                    – Ronan Jouchet
                    Oct 24 '15 at 19:58








                  • 4





                    How dyou access the line inside command ?

                    – B T
                    Apr 14 '16 at 1:21











                  • This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.

                    – masterxilo
                    Mar 15 '18 at 9:43






                  • 1





                    printf "foo barnbaz bat" | xargs -n1 echo whee splits by words and not by lines

                    – Gismo Ranas
                    Oct 24 '18 at 8:32








                  1




                  1





                  Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.

                  – Ronan Jouchet
                  Oct 24 '15 at 19:58







                  Also of interest is the ability of xargs to not run if stdin is empty: --no-run-if-empty -r : If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.

                  – Ronan Jouchet
                  Oct 24 '15 at 19:58






                  4




                  4





                  How dyou access the line inside command ?

                  – B T
                  Apr 14 '16 at 1:21





                  How dyou access the line inside command ?

                  – B T
                  Apr 14 '16 at 1:21













                  This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.

                  – masterxilo
                  Mar 15 '18 at 9:43





                  This is the correct use of xargs. Without -n1, it only works on commands that treat lists of parameters as multiple invocations which not all do.

                  – masterxilo
                  Mar 15 '18 at 9:43




                  1




                  1





                  printf "foo barnbaz bat" | xargs -n1 echo whee splits by words and not by lines

                  – Gismo Ranas
                  Oct 24 '18 at 8:32





                  printf "foo barnbaz bat" | xargs -n1 echo whee splits by words and not by lines

                  – Gismo Ranas
                  Oct 24 '18 at 8:32











                  104














                  In Bash or any other Bourne-style shell (ash, ksh, zsh, …):



                  while read -r line; do command "$line"; done


                  read -r reads a single line from standard input (read without -r interprets backslashes, you don't want that). Thus you can do either of the following:



                  $ command | while read -r line; do command "$line"; done  

                  $ while read -r line; do command "$line"; done <file





                  share|improve this answer





















                  • 5





                    When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work

                    – user16468
                    Mar 10 '12 at 20:39













                  • This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done

                    – Den
                    Sep 10 '18 at 9:27


















                  104














                  In Bash or any other Bourne-style shell (ash, ksh, zsh, …):



                  while read -r line; do command "$line"; done


                  read -r reads a single line from standard input (read without -r interprets backslashes, you don't want that). Thus you can do either of the following:



                  $ command | while read -r line; do command "$line"; done  

                  $ while read -r line; do command "$line"; done <file





                  share|improve this answer





















                  • 5





                    When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work

                    – user16468
                    Mar 10 '12 at 20:39













                  • This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done

                    – Den
                    Sep 10 '18 at 9:27
















                  104












                  104








                  104







                  In Bash or any other Bourne-style shell (ash, ksh, zsh, …):



                  while read -r line; do command "$line"; done


                  read -r reads a single line from standard input (read without -r interprets backslashes, you don't want that). Thus you can do either of the following:



                  $ command | while read -r line; do command "$line"; done  

                  $ while read -r line; do command "$line"; done <file





                  share|improve this answer















                  In Bash or any other Bourne-style shell (ash, ksh, zsh, …):



                  while read -r line; do command "$line"; done


                  read -r reads a single line from standard input (read without -r interprets backslashes, you don't want that). Thus you can do either of the following:



                  $ command | while read -r line; do command "$line"; done  

                  $ while read -r line; do command "$line"; done <file






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Feb 17 '11 at 7:35









                  Gilles

                  547k13011131629




                  547k13011131629










                  answered Feb 17 '11 at 5:11









                  Steven DSteven D

                  32.9k898108




                  32.9k898108








                  • 5





                    When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work

                    – user16468
                    Mar 10 '12 at 20:39













                  • This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done

                    – Den
                    Sep 10 '18 at 9:27
















                  • 5





                    When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work

                    – user16468
                    Mar 10 '12 at 20:39













                  • This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done

                    – Den
                    Sep 10 '18 at 9:27










                  5




                  5





                  When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work

                  – user16468
                  Mar 10 '12 at 20:39







                  When I tried tail -f syslog | grep -e something -e somethingelse| while read line; do echo $line; done it didn't work. It worked with a file piped into the while loop, worked with just the tail -f, worked with just grep, but not with both pipes. Giving the grep the --line-buffered option made it work

                  – user16468
                  Mar 10 '12 at 20:39















                  This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done

                  – Den
                  Sep 10 '18 at 9:27







                  This works also when each line needs to be sent to stdin: command | while read -r line; do echo "$line" | command ; done

                  – Den
                  Sep 10 '18 at 9:27













                  19














                  I agree with Keith, xargs is the most general tool for the job.



                  I usually use a 3 step approach.




                  • do the basic stuff until you have something you would like to work with

                  • prepare the line with awk so it gets the correct syntax

                  • then let xargs execute it, maybe with the help of bash.


                  There are smaller and faster ways, but this ways almost always works.



                  A simple example:



                  ls | 
                  grep xls |
                  awk '{print "MyJavaProg --arg1 42 --arg2 "$1""}' |
                  xargs -0 bash -c


                  the 2 first lines selects some files to work with,
                  then awk prepares a nice string with a command to execute and some arguments and $1 is the first column input from the pipe.
                  And finally I make sure that xargs sends this string to bash that just execute it.



                  It is a little bit overkill, but this recipe has helped me in a lot of places since it is very flexible.






                  share|improve this answer





















                  • 5





                    Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args "%s"",$1)

                    – glenn jackman
                    Feb 17 '11 at 14:30













                  • @glenn: Missed the null char, will update the answer

                    – Johan
                    Feb 18 '11 at 12:43











                  • @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print...

                    – Eric Renouf
                    May 31 '16 at 15:19
















                  19














                  I agree with Keith, xargs is the most general tool for the job.



                  I usually use a 3 step approach.




                  • do the basic stuff until you have something you would like to work with

                  • prepare the line with awk so it gets the correct syntax

                  • then let xargs execute it, maybe with the help of bash.


                  There are smaller and faster ways, but this ways almost always works.



                  A simple example:



                  ls | 
                  grep xls |
                  awk '{print "MyJavaProg --arg1 42 --arg2 "$1""}' |
                  xargs -0 bash -c


                  the 2 first lines selects some files to work with,
                  then awk prepares a nice string with a command to execute and some arguments and $1 is the first column input from the pipe.
                  And finally I make sure that xargs sends this string to bash that just execute it.



                  It is a little bit overkill, but this recipe has helped me in a lot of places since it is very flexible.






                  share|improve this answer





















                  • 5





                    Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args "%s"",$1)

                    – glenn jackman
                    Feb 17 '11 at 14:30













                  • @glenn: Missed the null char, will update the answer

                    – Johan
                    Feb 18 '11 at 12:43











                  • @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print...

                    – Eric Renouf
                    May 31 '16 at 15:19














                  19












                  19








                  19







                  I agree with Keith, xargs is the most general tool for the job.



                  I usually use a 3 step approach.




                  • do the basic stuff until you have something you would like to work with

                  • prepare the line with awk so it gets the correct syntax

                  • then let xargs execute it, maybe with the help of bash.


                  There are smaller and faster ways, but this ways almost always works.



                  A simple example:



                  ls | 
                  grep xls |
                  awk '{print "MyJavaProg --arg1 42 --arg2 "$1""}' |
                  xargs -0 bash -c


                  the 2 first lines selects some files to work with,
                  then awk prepares a nice string with a command to execute and some arguments and $1 is the first column input from the pipe.
                  And finally I make sure that xargs sends this string to bash that just execute it.



                  It is a little bit overkill, but this recipe has helped me in a lot of places since it is very flexible.






                  share|improve this answer















                  I agree with Keith, xargs is the most general tool for the job.



                  I usually use a 3 step approach.




                  • do the basic stuff until you have something you would like to work with

                  • prepare the line with awk so it gets the correct syntax

                  • then let xargs execute it, maybe with the help of bash.


                  There are smaller and faster ways, but this ways almost always works.



                  A simple example:



                  ls | 
                  grep xls |
                  awk '{print "MyJavaProg --arg1 42 --arg2 "$1""}' |
                  xargs -0 bash -c


                  the 2 first lines selects some files to work with,
                  then awk prepares a nice string with a command to execute and some arguments and $1 is the first column input from the pipe.
                  And finally I make sure that xargs sends this string to bash that just execute it.



                  It is a little bit overkill, but this recipe has helped me in a lot of places since it is very flexible.







                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Feb 18 '11 at 12:44

























                  answered Feb 17 '11 at 11:49









                  JohanJohan

                  3,34911829




                  3,34911829








                  • 5





                    Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args "%s"",$1)

                    – glenn jackman
                    Feb 17 '11 at 14:30













                  • @glenn: Missed the null char, will update the answer

                    – Johan
                    Feb 18 '11 at 12:43











                  • @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print...

                    – Eric Renouf
                    May 31 '16 at 15:19














                  • 5





                    Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args "%s"",$1)

                    – glenn jackman
                    Feb 17 '11 at 14:30













                  • @glenn: Missed the null char, will update the answer

                    – Johan
                    Feb 18 '11 at 12:43











                  • @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print...

                    – Eric Renouf
                    May 31 '16 at 15:19








                  5




                  5





                  Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args "%s"",$1)

                  – glenn jackman
                  Feb 17 '11 at 14:30







                  Note, xargs -0 uses the null byte as a record separator, so your awk print statement should be printf("MyJavaProg --args "%s"",$1)

                  – glenn jackman
                  Feb 17 '11 at 14:30















                  @glenn: Missed the null char, will update the answer

                  – Johan
                  Feb 18 '11 at 12:43





                  @glenn: Missed the null char, will update the answer

                  – Johan
                  Feb 18 '11 at 12:43













                  @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print...

                  – Eric Renouf
                  May 31 '16 at 15:19





                  @Johan not a big deal, but if you're using awk you can have it do the pattern match and skip the grep e.g., ls | awk '/xls/ {print...

                  – Eric Renouf
                  May 31 '16 at 15:19











                  15














                  GNU Parallel is made for that kind of tasks. The simplest usage is:



                  cat stuff | grep pattern | parallel java MyProg


                  Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ






                  share|improve this answer



















                  • 1





                    No real need for the cat here since grep can directly read the file

                    – Eric Renouf
                    May 31 '16 at 15:20






                  • 3





                    @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html

                    – Ole Tange
                    May 31 '16 at 20:29






                  • 1





                    Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you

                    – Eric Renouf
                    Jun 1 '16 at 0:32
















                  15














                  GNU Parallel is made for that kind of tasks. The simplest usage is:



                  cat stuff | grep pattern | parallel java MyProg


                  Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ






                  share|improve this answer



















                  • 1





                    No real need for the cat here since grep can directly read the file

                    – Eric Renouf
                    May 31 '16 at 15:20






                  • 3





                    @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html

                    – Ole Tange
                    May 31 '16 at 20:29






                  • 1





                    Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you

                    – Eric Renouf
                    Jun 1 '16 at 0:32














                  15












                  15








                  15







                  GNU Parallel is made for that kind of tasks. The simplest usage is:



                  cat stuff | grep pattern | parallel java MyProg


                  Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ






                  share|improve this answer













                  GNU Parallel is made for that kind of tasks. The simplest usage is:



                  cat stuff | grep pattern | parallel java MyProg


                  Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered Feb 22 '11 at 13:55









                  Ole TangeOle Tange

                  13k1457107




                  13k1457107








                  • 1





                    No real need for the cat here since grep can directly read the file

                    – Eric Renouf
                    May 31 '16 at 15:20






                  • 3





                    @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html

                    – Ole Tange
                    May 31 '16 at 20:29






                  • 1





                    Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you

                    – Eric Renouf
                    Jun 1 '16 at 0:32














                  • 1





                    No real need for the cat here since grep can directly read the file

                    – Eric Renouf
                    May 31 '16 at 15:20






                  • 3





                    @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html

                    – Ole Tange
                    May 31 '16 at 20:29






                  • 1





                    Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you

                    – Eric Renouf
                    Jun 1 '16 at 0:32








                  1




                  1





                  No real need for the cat here since grep can directly read the file

                  – Eric Renouf
                  May 31 '16 at 15:20





                  No real need for the cat here since grep can directly read the file

                  – Eric Renouf
                  May 31 '16 at 15:20




                  3




                  3





                  @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html

                  – Ole Tange
                  May 31 '16 at 20:29





                  @EricRenouf oletange.blogspot.dk/2013/10/useless-use-of-cat.html

                  – Ole Tange
                  May 31 '16 at 20:29




                  1




                  1





                  Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you

                  – Eric Renouf
                  Jun 1 '16 at 0:32





                  Thanks for the link, I don't necessarily agree that it's easier to read, but nice to know it was considered regardless. I would only now slightly quibble that the link doesn't actually apply here since the alternative is not really < stuff grep pattern but is grep pattern stuff with no redirection or cat required at all. Still, that doesn't materially change your argument and if you think it's clearer to always use things in a pipe that starts with cat, then power to you

                  – Eric Renouf
                  Jun 1 '16 at 0:32











                  8














                  Also, while read loop in fish shell (I assume you want fish shell, considering you used fish tag).



                  command | while read line
                  command $line
                  end


                  Few points to note.





                  • read doesn't take -r argument, and it doesn't interpret your backslashes, in order to make most common use case easy.

                  • You don't need to quote $line, as unlike bash, fish doesn't separate variables by spaces.


                  • command by itself is a syntax error (to catch such use of placeholder arguments). Replace it with the real command.






                  share|improve this answer
























                  • Doesn't while need to be paired with do & done instead of end?

                    – aff
                    Nov 19 '18 at 7:55











                  • @aff This is specifically about fish shell, which has different syntax.

                    – Konrad Borowski
                    Nov 19 '18 at 14:29











                  • Ah, so that's what the fish means.

                    – aff
                    Nov 20 '18 at 6:33
















                  8














                  Also, while read loop in fish shell (I assume you want fish shell, considering you used fish tag).



                  command | while read line
                  command $line
                  end


                  Few points to note.





                  • read doesn't take -r argument, and it doesn't interpret your backslashes, in order to make most common use case easy.

                  • You don't need to quote $line, as unlike bash, fish doesn't separate variables by spaces.


                  • command by itself is a syntax error (to catch such use of placeholder arguments). Replace it with the real command.






                  share|improve this answer
























                  • Doesn't while need to be paired with do & done instead of end?

                    – aff
                    Nov 19 '18 at 7:55











                  • @aff This is specifically about fish shell, which has different syntax.

                    – Konrad Borowski
                    Nov 19 '18 at 14:29











                  • Ah, so that's what the fish means.

                    – aff
                    Nov 20 '18 at 6:33














                  8












                  8








                  8







                  Also, while read loop in fish shell (I assume you want fish shell, considering you used fish tag).



                  command | while read line
                  command $line
                  end


                  Few points to note.





                  • read doesn't take -r argument, and it doesn't interpret your backslashes, in order to make most common use case easy.

                  • You don't need to quote $line, as unlike bash, fish doesn't separate variables by spaces.


                  • command by itself is a syntax error (to catch such use of placeholder arguments). Replace it with the real command.






                  share|improve this answer













                  Also, while read loop in fish shell (I assume you want fish shell, considering you used fish tag).



                  command | while read line
                  command $line
                  end


                  Few points to note.





                  • read doesn't take -r argument, and it doesn't interpret your backslashes, in order to make most common use case easy.

                  • You don't need to quote $line, as unlike bash, fish doesn't separate variables by spaces.


                  • command by itself is a syntax error (to catch such use of placeholder arguments). Replace it with the real command.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered Aug 29 '13 at 11:53









                  Konrad BorowskiKonrad Borowski

                  316310




                  316310













                  • Doesn't while need to be paired with do & done instead of end?

                    – aff
                    Nov 19 '18 at 7:55











                  • @aff This is specifically about fish shell, which has different syntax.

                    – Konrad Borowski
                    Nov 19 '18 at 14:29











                  • Ah, so that's what the fish means.

                    – aff
                    Nov 20 '18 at 6:33



















                  • Doesn't while need to be paired with do & done instead of end?

                    – aff
                    Nov 19 '18 at 7:55











                  • @aff This is specifically about fish shell, which has different syntax.

                    – Konrad Borowski
                    Nov 19 '18 at 14:29











                  • Ah, so that's what the fish means.

                    – aff
                    Nov 20 '18 at 6:33

















                  Doesn't while need to be paired with do & done instead of end?

                  – aff
                  Nov 19 '18 at 7:55





                  Doesn't while need to be paired with do & done instead of end?

                  – aff
                  Nov 19 '18 at 7:55













                  @aff This is specifically about fish shell, which has different syntax.

                  – Konrad Borowski
                  Nov 19 '18 at 14:29





                  @aff This is specifically about fish shell, which has different syntax.

                  – Konrad Borowski
                  Nov 19 '18 at 14:29













                  Ah, so that's what the fish means.

                  – aff
                  Nov 20 '18 at 6:33





                  Ah, so that's what the fish means.

                  – aff
                  Nov 20 '18 at 6:33











                  5














                  If you need to control where exactly the input argument is inserted into your command line or if you need to repeat it several times then you need to use xargs -I{}.



                  EXAMPLE #1

                  Create an empty folder structure in another_folder that mirrors the subfolders in the current directory:





                      ls -1d ./*/ | xargs -I{} mkdir another_folder/{}


                  EXAMPLE #2

                  Apply an operation on a file list coming from stdin, in this case make a copy of each .html file by appending a .bak extension:



                      find . -iname "*.html" | xargs -I{} cp {} {}.bak


                  From the xargs man page for MacOS/BSD:



                   -I replstr
                  Execute utility for each input line, replacing one or more occurrences of
                  replstr in up to replacements (or 5 if no -R flag is specified) arguments
                  to utility with the entire line of input. The resulting arguments, after
                  replacement is done, will not be allowed to grow beyond 255 bytes; this is
                  implemented by concatenating as much of the argument containing replstr as
                  possible, to the constructed arguments to utility, up to 255 bytes. The
                  255 byte limit does not apply to arguments to utility which do not contain
                  replstr, and furthermore, no replacement will be done on utility itself.
                  Implies -x.


                  Linux xargs man page:



                     -I replace-str
                  Replace occurrences of replace-str in the initial-
                  arguments with names read from standard input. Al‐
                  so, unquoted blanks do not terminate input items;
                  instead the separator is the newline character.
                  Implies -x and -L 1.





                  share|improve this answer






























                    5














                    If you need to control where exactly the input argument is inserted into your command line or if you need to repeat it several times then you need to use xargs -I{}.



                    EXAMPLE #1

                    Create an empty folder structure in another_folder that mirrors the subfolders in the current directory:





                        ls -1d ./*/ | xargs -I{} mkdir another_folder/{}


                    EXAMPLE #2

                    Apply an operation on a file list coming from stdin, in this case make a copy of each .html file by appending a .bak extension:



                        find . -iname "*.html" | xargs -I{} cp {} {}.bak


                    From the xargs man page for MacOS/BSD:



                     -I replstr
                    Execute utility for each input line, replacing one or more occurrences of
                    replstr in up to replacements (or 5 if no -R flag is specified) arguments
                    to utility with the entire line of input. The resulting arguments, after
                    replacement is done, will not be allowed to grow beyond 255 bytes; this is
                    implemented by concatenating as much of the argument containing replstr as
                    possible, to the constructed arguments to utility, up to 255 bytes. The
                    255 byte limit does not apply to arguments to utility which do not contain
                    replstr, and furthermore, no replacement will be done on utility itself.
                    Implies -x.


                    Linux xargs man page:



                       -I replace-str
                    Replace occurrences of replace-str in the initial-
                    arguments with names read from standard input. Al‐
                    so, unquoted blanks do not terminate input items;
                    instead the separator is the newline character.
                    Implies -x and -L 1.





                    share|improve this answer




























                      5












                      5








                      5







                      If you need to control where exactly the input argument is inserted into your command line or if you need to repeat it several times then you need to use xargs -I{}.



                      EXAMPLE #1

                      Create an empty folder structure in another_folder that mirrors the subfolders in the current directory:





                          ls -1d ./*/ | xargs -I{} mkdir another_folder/{}


                      EXAMPLE #2

                      Apply an operation on a file list coming from stdin, in this case make a copy of each .html file by appending a .bak extension:



                          find . -iname "*.html" | xargs -I{} cp {} {}.bak


                      From the xargs man page for MacOS/BSD:



                       -I replstr
                      Execute utility for each input line, replacing one or more occurrences of
                      replstr in up to replacements (or 5 if no -R flag is specified) arguments
                      to utility with the entire line of input. The resulting arguments, after
                      replacement is done, will not be allowed to grow beyond 255 bytes; this is
                      implemented by concatenating as much of the argument containing replstr as
                      possible, to the constructed arguments to utility, up to 255 bytes. The
                      255 byte limit does not apply to arguments to utility which do not contain
                      replstr, and furthermore, no replacement will be done on utility itself.
                      Implies -x.


                      Linux xargs man page:



                         -I replace-str
                      Replace occurrences of replace-str in the initial-
                      arguments with names read from standard input. Al‐
                      so, unquoted blanks do not terminate input items;
                      instead the separator is the newline character.
                      Implies -x and -L 1.





                      share|improve this answer















                      If you need to control where exactly the input argument is inserted into your command line or if you need to repeat it several times then you need to use xargs -I{}.



                      EXAMPLE #1

                      Create an empty folder structure in another_folder that mirrors the subfolders in the current directory:





                          ls -1d ./*/ | xargs -I{} mkdir another_folder/{}


                      EXAMPLE #2

                      Apply an operation on a file list coming from stdin, in this case make a copy of each .html file by appending a .bak extension:



                          find . -iname "*.html" | xargs -I{} cp {} {}.bak


                      From the xargs man page for MacOS/BSD:



                       -I replstr
                      Execute utility for each input line, replacing one or more occurrences of
                      replstr in up to replacements (or 5 if no -R flag is specified) arguments
                      to utility with the entire line of input. The resulting arguments, after
                      replacement is done, will not be allowed to grow beyond 255 bytes; this is
                      implemented by concatenating as much of the argument containing replstr as
                      possible, to the constructed arguments to utility, up to 255 bytes. The
                      255 byte limit does not apply to arguments to utility which do not contain
                      replstr, and furthermore, no replacement will be done on utility itself.
                      Implies -x.


                      Linux xargs man page:



                         -I replace-str
                      Replace occurrences of replace-str in the initial-
                      arguments with names read from standard input. Al‐
                      so, unquoted blanks do not terminate input items;
                      instead the separator is the newline character.
                      Implies -x and -L 1.






                      share|improve this answer














                      share|improve this answer



                      share|improve this answer








                      edited 14 hours ago

























                      answered Apr 1 '18 at 8:56









                      ccpizzaccpizza

                      636911




                      636911























                          1














                          When dealing with potentially unsanitized inputs, I like to see the entire job 'spelled out' line by line for visual inspection before I run it (especially when it's something destructive like cleaning people's mailbox's).



                          So what I do is generate a list of parameters (ie. usernames), feed it to a file in one-record-per-line fashion, like this:



                          johndoe  
                          jamessmith
                          janebrown


                          Then I open the list in vim, and mangle it with search and replace expressions until I get a list of full commands that need to get executed, like this:



                          /bin/rm -fr /home/johndoe  
                          /bin/rm -fr /home/jamessmith


                          This way if your regex is incomplete, you will see in what command will have potential problems (ie. /bin/rm -fr johnnyo connor). This way you can undo your regex, and try it again with a more reliable version of it. Name mangling is notorious for this, because it's hard to take care of all the edge cases like Van Gogh, O'Connors, St. Clair, Smith-Wesson.



                          Having set hlsearch is useful for doing this in vim, as it will highlight all the matches, so you can easily spot if it doesn't match, or matches in an unintended way.



                          Once your regex is perfect and it catches all the cases you can test for/think of, then I usually convert it to a sed expression so it can be fully automated for another run.



                          For cases where the number of lines of input prevents you from doing a visual inspection, I highly recommend echoing the command to the screen (or better yet, a log) before it executes, so if it errors out, you know exactly which command caused it to fail. Then you can go back to your original regex and adjust once more.






                          share|improve this answer






























                            1














                            When dealing with potentially unsanitized inputs, I like to see the entire job 'spelled out' line by line for visual inspection before I run it (especially when it's something destructive like cleaning people's mailbox's).



                            So what I do is generate a list of parameters (ie. usernames), feed it to a file in one-record-per-line fashion, like this:



                            johndoe  
                            jamessmith
                            janebrown


                            Then I open the list in vim, and mangle it with search and replace expressions until I get a list of full commands that need to get executed, like this:



                            /bin/rm -fr /home/johndoe  
                            /bin/rm -fr /home/jamessmith


                            This way if your regex is incomplete, you will see in what command will have potential problems (ie. /bin/rm -fr johnnyo connor). This way you can undo your regex, and try it again with a more reliable version of it. Name mangling is notorious for this, because it's hard to take care of all the edge cases like Van Gogh, O'Connors, St. Clair, Smith-Wesson.



                            Having set hlsearch is useful for doing this in vim, as it will highlight all the matches, so you can easily spot if it doesn't match, or matches in an unintended way.



                            Once your regex is perfect and it catches all the cases you can test for/think of, then I usually convert it to a sed expression so it can be fully automated for another run.



                            For cases where the number of lines of input prevents you from doing a visual inspection, I highly recommend echoing the command to the screen (or better yet, a log) before it executes, so if it errors out, you know exactly which command caused it to fail. Then you can go back to your original regex and adjust once more.






                            share|improve this answer




























                              1












                              1








                              1







                              When dealing with potentially unsanitized inputs, I like to see the entire job 'spelled out' line by line for visual inspection before I run it (especially when it's something destructive like cleaning people's mailbox's).



                              So what I do is generate a list of parameters (ie. usernames), feed it to a file in one-record-per-line fashion, like this:



                              johndoe  
                              jamessmith
                              janebrown


                              Then I open the list in vim, and mangle it with search and replace expressions until I get a list of full commands that need to get executed, like this:



                              /bin/rm -fr /home/johndoe  
                              /bin/rm -fr /home/jamessmith


                              This way if your regex is incomplete, you will see in what command will have potential problems (ie. /bin/rm -fr johnnyo connor). This way you can undo your regex, and try it again with a more reliable version of it. Name mangling is notorious for this, because it's hard to take care of all the edge cases like Van Gogh, O'Connors, St. Clair, Smith-Wesson.



                              Having set hlsearch is useful for doing this in vim, as it will highlight all the matches, so you can easily spot if it doesn't match, or matches in an unintended way.



                              Once your regex is perfect and it catches all the cases you can test for/think of, then I usually convert it to a sed expression so it can be fully automated for another run.



                              For cases where the number of lines of input prevents you from doing a visual inspection, I highly recommend echoing the command to the screen (or better yet, a log) before it executes, so if it errors out, you know exactly which command caused it to fail. Then you can go back to your original regex and adjust once more.






                              share|improve this answer















                              When dealing with potentially unsanitized inputs, I like to see the entire job 'spelled out' line by line for visual inspection before I run it (especially when it's something destructive like cleaning people's mailbox's).



                              So what I do is generate a list of parameters (ie. usernames), feed it to a file in one-record-per-line fashion, like this:



                              johndoe  
                              jamessmith
                              janebrown


                              Then I open the list in vim, and mangle it with search and replace expressions until I get a list of full commands that need to get executed, like this:



                              /bin/rm -fr /home/johndoe  
                              /bin/rm -fr /home/jamessmith


                              This way if your regex is incomplete, you will see in what command will have potential problems (ie. /bin/rm -fr johnnyo connor). This way you can undo your regex, and try it again with a more reliable version of it. Name mangling is notorious for this, because it's hard to take care of all the edge cases like Van Gogh, O'Connors, St. Clair, Smith-Wesson.



                              Having set hlsearch is useful for doing this in vim, as it will highlight all the matches, so you can easily spot if it doesn't match, or matches in an unintended way.



                              Once your regex is perfect and it catches all the cases you can test for/think of, then I usually convert it to a sed expression so it can be fully automated for another run.



                              For cases where the number of lines of input prevents you from doing a visual inspection, I highly recommend echoing the command to the screen (or better yet, a log) before it executes, so if it errors out, you know exactly which command caused it to fail. Then you can go back to your original regex and adjust once more.







                              share|improve this answer














                              share|improve this answer



                              share|improve this answer








                              edited Feb 19 '11 at 22:12









                              Steven D

                              32.9k898108




                              32.9k898108










                              answered Feb 19 '11 at 17:59









                              MarcinMarcin

                              1,30187




                              1,30187























                                  0














                                  If a program ignores the pipe but accepts files as arguments, then you can just point it to the special file /dev/stdin.



                                  I am not familiar with java, but here is an example of how you would do it for bash:



                                  $ echo $'pwd n cd / n pwd' |bash /dev/stdin
                                  /home/rolf
                                  /


                                  The $ is necessary for bash to translate n into newlines. I'm not sure why.






                                  share|improve this answer






























                                    0














                                    If a program ignores the pipe but accepts files as arguments, then you can just point it to the special file /dev/stdin.



                                    I am not familiar with java, but here is an example of how you would do it for bash:



                                    $ echo $'pwd n cd / n pwd' |bash /dev/stdin
                                    /home/rolf
                                    /


                                    The $ is necessary for bash to translate n into newlines. I'm not sure why.






                                    share|improve this answer




























                                      0












                                      0








                                      0







                                      If a program ignores the pipe but accepts files as arguments, then you can just point it to the special file /dev/stdin.



                                      I am not familiar with java, but here is an example of how you would do it for bash:



                                      $ echo $'pwd n cd / n pwd' |bash /dev/stdin
                                      /home/rolf
                                      /


                                      The $ is necessary for bash to translate n into newlines. I'm not sure why.






                                      share|improve this answer















                                      If a program ignores the pipe but accepts files as arguments, then you can just point it to the special file /dev/stdin.



                                      I am not familiar with java, but here is an example of how you would do it for bash:



                                      $ echo $'pwd n cd / n pwd' |bash /dev/stdin
                                      /home/rolf
                                      /


                                      The $ is necessary for bash to translate n into newlines. I'm not sure why.







                                      share|improve this answer














                                      share|improve this answer



                                      share|improve this answer








                                      edited May 5 '17 at 1:16

























                                      answered May 5 '17 at 1:10









                                      RolfRolf

                                      176113




                                      176113























                                          0














                                          I prefer this - allowing multi-line commands and clear code



                                          find -type f -name filenam-pattern* | while read -r F
                                          do
                                          echo $F
                                          cat $F | grep 'some text'
                                          done


                                          ref https://stackoverflow.com/a/3891678/248616






                                          share|improve this answer






























                                            0














                                            I prefer this - allowing multi-line commands and clear code



                                            find -type f -name filenam-pattern* | while read -r F
                                            do
                                            echo $F
                                            cat $F | grep 'some text'
                                            done


                                            ref https://stackoverflow.com/a/3891678/248616






                                            share|improve this answer




























                                              0












                                              0








                                              0







                                              I prefer this - allowing multi-line commands and clear code



                                              find -type f -name filenam-pattern* | while read -r F
                                              do
                                              echo $F
                                              cat $F | grep 'some text'
                                              done


                                              ref https://stackoverflow.com/a/3891678/248616






                                              share|improve this answer















                                              I prefer this - allowing multi-line commands and clear code



                                              find -type f -name filenam-pattern* | while read -r F
                                              do
                                              echo $F
                                              cat $F | grep 'some text'
                                              done


                                              ref https://stackoverflow.com/a/3891678/248616







                                              share|improve this answer














                                              share|improve this answer



                                              share|improve this answer








                                              edited Aug 30 '18 at 11:22

























                                              answered Aug 14 '18 at 14:50









                                              Nam G VUNam G VU

                                              207313




                                              207313






























                                                  draft saved

                                                  draft discarded




















































                                                  Thanks for contributing an answer to Unix & Linux Stack Exchange!


                                                  • Please be sure to answer the question. Provide details and share your research!

                                                  But avoid



                                                  • Asking for help, clarification, or responding to other answers.

                                                  • Making statements based on opinion; back them up with references or personal experience.


                                                  To learn more, see our tips on writing great answers.




                                                  draft saved


                                                  draft discarded














                                                  StackExchange.ready(
                                                  function () {
                                                  StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f7558%2fexecute-a-command-once-per-line-of-piped-input%23new-answer', 'question_page');
                                                  }
                                                  );

                                                  Post as a guest















                                                  Required, but never shown





















































                                                  Required, but never shown














                                                  Required, but never shown












                                                  Required, but never shown







                                                  Required, but never shown

































                                                  Required, but never shown














                                                  Required, but never shown












                                                  Required, but never shown







                                                  Required, but never shown







                                                  Popular posts from this blog

                                                  Hudson River Historic District Contents Geography History The district today Aesthetics Cultural...

                                                  The number designs the writing. Feandra Aversely Definition: The act of ingrafting a sprig or shoot of one...

                                                  Ayherre Geografie Demografie Externe links Navigatiemenu43° 23′ NB, 1° 15′ WL43° 23′ NB, 1°...