Mdthbs

For kicks, here's Ruby

ruby -e '

    data = File.readlines(ARGV.shift)

               .map {|line| line.split.map(&:to_i)}

               .transpose

               .map {|(a,b)| (a==0 && b==0) ? [9,9] : [a,b]}

               .transpose

               .each {|row| puts row.join(" ")}

' file

1 9 9 9 9 9 9 9 9 9 1 2 1

0 9 9 9 9 9 9 9 9 9 0 0 0

To replace all the files:

ruby -e '

    require "tempfile"

    require "pathname"

    Pathname.new("/path/to/your/files/").each_child do |pathname|

        next unless pathname.file?

        temp = Tempfile.new(pathname.basename.to_s)

        filename = pathname.to_s

        File.readlines(filename)

            .map {|line| line.split.map(&:to_i)}

            .transpose

            .map {|(a,b)| (a==0 && b==0) ? [9,9] : [a,b]}

            .transpose

            .each {|row| temp.puts row.join(" ")}

        temp.close

        File.link filename, filename+".bak"

        File.rename temp.path, filename

    end

'

This is an alternative approach, which might be slow for million of files compared to pure awk solutions.

Using something like this, you can transpose rows to columns:

$ cat file1

1   0   0   0   0   0   0   0   0   0   1   2   1   

0   0   0   0   0   0   0   0   0   0   0   0   0



$ paste -d'-' <(head -n1 file1 |tr -s ' ' 'n') <(tail -n1 file1 |tr -s ' ' 'n')

1-0

0-0

0-0

0-0

0-0

0-0

0-0

0-0

0-0

0-0

1-0

2-0

1-0

You can then replace all 0-0 occurences with 9-9 with a simple sed, and you can store the output to a temp variable:

$ f1=$(sed 's/0-0/9-9/g' <(paste -d'-' <(head -n1 file1|tr -s ' ' 'n') <(tail -n1 file1 |tr -s ' ' 'n')))

$ echo "$f1"

1-0

9-9

9-9

9-9

9-9

9-9

9-9

9-9

9-9

9-9

1-0

2-0

1-0

You can now revert back from columns to rows like:

$ awk -F'-' 'NR==FNR{printf "%s ",$1;p=1;next}p{printf "n";p=0}{printf "%s ",$2}END{printf "n"}' <(echo "$f1") <(echo "$f1")

1 9 9 9 9 9 9 9 9 9 1 2 1  

0 9 9 9 9 9 9 9 9 9 0 0 0  

And you can also append >file1 at the end of last awk command to overwrite the file1 with the new contents.

Only thing left is to loop over all files. Can be done with a kind of bash loop:

for f in ./wa_filtering_DP15_good_pops_snps_file_*;do

  f1=$(sed 's/0-0/9-9/g' <(paste -d'-' <(head -n1 "$f"|tr -s ' ' 'n') <(tail -n1 "$f" |tr -s ' ' 'n')))

  awk -F'-' 'NR==FNR{printf "%s ",$1;p=1;next}p{printf "n";p=0}{printf "%s ",$2}END{printf "n"}' <(echo "$f1") <(echo "$f1") #>"$f" #uncomment >"$f" to overwrite the files...

done

With awk:

NR == 1 {   # save the values from 1st line in array t

            split($0, t, FS);

        }



NR == 2 {   # compare values from second line with those stored in array t

            for ( i = 1; i <= NF; ++i ) {

                # build l1 and l2 (line 1 and line 2) based on comparison

                if ($i == 0 && t[i] == 0) {

                    l1 = (i == 1 ? 9    : l1 OFS 9    );

                    l2 = (i == 1 ? 9    : l2 OFS 9    );

                } else {

                    l1 = (i == 1 ? t[i] : l1 OFS t[i] );

                    l2 = (i == 1 ? $i   : l2 OFS $i   );

                }

            }

        }



END     {   # output the two constructed lines

            print l1;

            print l2;

        }

Running it on the example file:

$ awk -f script.awk file

1 9 9 9 9 9 9 9 9 9 1 2 1

0 9 9 9 9 9 9 9 9 9 0 0 0

Running on all files matching wa_filtering_DP15_good_pops_snps_file_* in current directory:

mkdir modified



for name in wa_filtering_DP15_good_pops_snps_file_*; do

    awk -f script.awk "$name" >"modified/$name.new"

done

This will create a new file for each input file, with the name of the original file and an extra .new suffix. The new files will be placed in the modified folder in the current directory.

• I opted for creating new files so that the originals are left unmodified.


• I opted to put the new files in a new directory, as having 22 million files in a single directory could make the filesystem be a bit awkward to work with.

In general, try not to create millions of files in a single directory. Instead either

1. create many subdirectories and distribute the files in them, maybe based on a binning algorithm working on that last integer of the filename, or a hash, or


2. create a single output file that aggregates all data, possibly with extra lines of text identifying what the following two lines refer to.

The following variant will be more efficiently run on millions of files:

FNR == 1    {   # save the values from 1st line in array t

                split($0, t, FS);

            }



FNR == 2    {   # compare values from second line with those stored in array t

                for ( i = 1; i <= NF; ++i ) {

                    # build l1 and l2 (line 1 and line 2) based on comparison

                    if ($i == 0 && t[i] == 0) {

                        l1 = (i == 1 ? 9    : l1 OFS 9    );

                        l2 = (i == 1 ? 9    : l2 OFS 9    );

                    } else {

                        l1 = (i == 1 ? t[i] : l1 OFS t[i] );

                        l2 = (i == 1 ? $i   : l2 OFS $i   );

                    }

                }



                # create output filename based on input filename

                # and output the two lines

                f = "modified/" FILENAME ".new";

                print l1 >f;

                print l2 >f;

            }

To run it:

mkdir modified



find . -maxdepth 1 -type f -name 'wa_filtering_DP15_good_pops_snps_file_*' 

    -exec awk -f script.awk {} +

The new files will be generated in the modified folder as before, but this time only a fraction of awk processes will be started and the speed of processing will be greatly increased.

First variant:

For single file:

datamash -W transpose < input.txt | sed 's/0t0/9t9/' | datamash transpose

For many files do the same in the loop:

for i in *; do datamash -W transpose < "$i" |

sed 's/0t0/9t9/' |

datamash transpose > "new_$i"; done

This loop will create the new, changed file for the each file, with the prefix "new_" added. Then you can remove all old files and remove prefix "new_" from filenames.

Second variant:

This is a solution for the single file, for multiple files use loop, as in the previous variant.

tr 'n' 't' < input.txt |

awk '{

    num = NF / 2;

    for(up = 1; up <= NF; up++) {

        if(up <= num) {

            low = num + up;

            if(!$up && !$low) {

                $up = 9;    

                $low = 9;

            }

        }



        printf "%st", $up;



        if(up % num == 0) 

            print "";

    }

}'

Explanation

1. 
   tr 'n' 't' < input.txt - join two lines together.


2. 
   awk
   
   
   
   
   • checks the one element from the first line and the adjacent element from the second line simultaneously, like: 1 and 316, 2 and 317, 3 and 318, so on.
   
   
   • if both elements are 0, it changes them to 9.
   
   
   • print fields by the order - 1, 2, 3, 4 ... 628, 629, 630.
   
   
   • Each time the element number is a multiple of the number of elements in the line, adds a new line.
   
   
   
   

Input

1   0   0   0   0   0   0   0   0   0   1   2   1

0   0   0   0   0   0   0   0   0   0   0   0   0

Output

1   9   9   9   9   9   9   9   9   9   1   2   1

0   9   9   9   9   9   9   9   9   9   0   0   0

Probably not efficient enough for 11 million files but it's a different approach on the substitution. Takes one argument on the command line; the name of the directory where all of the files are stored. The name for the directory could be hard coded instead (see notes in code). The base name for the file is already hard coded without the number at the end (not required).

#!/bin/bash



# compare two rows in a file

# when both are 0, change both to 9

# otherwise keep original value



ProgName=${0##*/}

Pid=$$

DBG_FNAME=""

scriptUsage() {

cat <<ENDUSE



  $ProgName </path/to/directory> [ [-d|--debug] || [-f|--filename] ]



  path/to/directory:    Path to directory (NO trailing '/')

  -f|--filename:        Print the each file name to stdout after complete

  -d|--debug:           Run in debug mode (Implies filename option - SEE NOTE*)

  -h|--help:            Print this help message



  NOTE:  USING [-d|--debug] AUTOMATICALLY SETS [-f|--filename]

         You DO NOT need both together!



ENDUSE

}



# check args

#!# NOTE: you can delete from here to #!!# above 'WorkDir="$1"'

[[ -z $1 ]] && { >&2 echo "MISSING file source directory!"; scriptUsage; exit 1; }

[[ $1 == "-h" || $1 == "--help" ]] && { scriptUsage; exit 0; }

[[ -d $1 ]] || { >&2 echo "Unable to locate directory [$1]"; exit 1; }

if (( $# > 2 ))

  then

    DBG_FNAME=1

    >&2 echo "Running in debug mode from using ${2} & ${3} together!"

    echo "PID is: $Pid"

    sleep 2

    set -x

  else

    [[ $2 == "-f" || $2 == "--filename" ]] && DBG_FNAME=1

    [[ $2 == "-d" || $2 == "--debug" ]] && { echo "PID is: $Pid"; set -x; }

fi

#!!# to here #!!#

# directory as arg[1] or change to hardcoded

  WorkDir="$1"



# check for/remove trailing slash

[[ ${WorkDir:(-1)} == / ]] && WorkDir=${WorkDir:0:((${#WorkDir}-1))}



# given file root withOUT number ending

  WorkFile="${WorkDir}/wa_filtering_DP15_good_pops_snps_file_"





##== MAIN LOOP

for file in ${WorkFile}*

  do

    # reset these after each file

    TopRow=""

    BotRow=""

    NewTop=""

    NewBot=""

    SKIPME=""



    # get top row of file

    TopRow=$(sed -n '1{p;q}' $file)

    # get bottom row of file

    BotRow=$(sed -n '2{p;q}' $file)



    ##-- EACH FILE LOOP

    for (( f=0; f<${#TopRow}; f++ ))

      do

        if [[ -n $SKIPME ]]

          then

            # SKIPME is -z by default so

            # this runs every other time through

            NewTop="${NewTop} "

            NewBot="${NewBot} "

            SKIPME=""

        elif (( $((${TopRow:${f}:1}+${BotRow:${f}:1})) == 0 ))

          then

            # 0+0=0 so change to 9

            NewTop="${NewTop}9"

            NewBot="${NewBot}9"

            SKIPME=1

        else

            # (1+0 or 0+1)!=0 so keep originals

            NewTop="${NewTop}${TopRow:${f}:1}"

            NewBot="${NewBot}${BotRow:${f}:1}"

            SKIPME=1

        fi

    done

    ##--



    # overwrite original file

    printf "%sn%s" "$NewTop" "$NewBot" > $file



    # if -f|--filename given print file name

    [[ -n $DBG_FNAME ]] && echo "$file is complete"

done

##==

DOES EDIT FILES IN PLACE. Wouldn't be hard to have it make backups as it runs. Returns files exactly the way requested above.