perl 按第一列合并多个TSV文件

xcitsw88  于 2022-11-15  发布在  Perl
关注(0)|答案(3)|浏览(297)

我在一个只有两列的目录中有很多(几十个)TSV文件,我想根据第一列的值合并所有的TSV文件(两列都有我需要维护的标题);如果这个值存在,那么它必须加上相应的第二列的值,依此类推(见示例)。2文件可能有不同的行数,并且不按第一列排序,尽管这可以很容易地通过排序来完成。
我试过join,但是只对两个文件有效。join可以扩展到一个目录中的所有文件吗?我认为awk可能是一个更好的解决方案,但是我对awk的了解非常非常有限。有什么想法吗?
下面是三个文件的示例:

S01.tsv

Accesion    S01  
AJ863320    1  
AM930424    1  
AY664038    2

S02.tsv

Accesion    S02  
AJ863320    2  
AM930424    1  
EU236327    1  
EU434346    2 

S03.tsv

Accesion    S03  
AJ863320    5  
EU236327    2  
EU434346    2

输出文件应为:

Accesion    S01   S02   S03  
    AJ863320    1     2     5  
    AM930424    1     1
    AY664038    2  
    EU236327          1     2  
    EU434346          2     2

好的,感谢James Brown,我让这段代码正常工作(我将其命名为compile.awk),但有一些小故障:

BEGIN { OFS="\t" }                            # tab separated columns
FNR==1 { f++ }                                # counter of files
{
    a[0][$1]=$1                               # reset the key for every record 
    for(i=2;i<=NF;i++)                        # for each non-key element
        a[f][$1]=a[f][$1] $i ( i==NF?"":OFS ) # combine them to array element
}
END {                                         # in the end
    for(i in a[0])                            # go thru every key
        for(j=0;j<=f;j++)                     # and all related array elements
            printf "%s%s", a[j][i], (j==f?ORS:OFS)
}                                             # output them, nonexistent will output empty

当我用实际文件运行它时

awk -f compile.awk 01.tsv 02.tsv 03.tsv

输出如下:

LN854586.1.1236         1
JF128382.1.1303     1   
Accesion    S01 S02 S03
JN233077.1.1420 1       
HQ836180.1.1388     1   
KP718814.1.1338         1
JQ781640.1.1200         2

前两行不属于那里,因为文件应该以所有文件的标题(第三行)开始。有什么想法如何解决这个问题吗?

pftdvrlh

pftdvrlh1#

我可能会这样处理它:

#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;

my @header; 
my %all_rows;
my %seen_cols;

#read STDIN or files specified as args. 
while ( <> ) {
   #detect a header row by keyword. 
   #can probably do this after 'open' but this way
   #means we can use <> and an arbitrary file list. 
   if ( m/^Accesion/ ) { 
      @header = split;       
      shift @header; #drop "accession" off the list so it's just S01,02,03 etc. 
      $seen_cols{$_}++ for @header; #keep track of uniques. 
   }
   else {
      #not a header row - split the row on whitespace.
      #can do /\t/ if that's not good enough, but it looks like it should be. 
      my ( $ID, @fields ) = split; 
      #use has slice to populate row.

      my %this_row;
      @this_row{@header} = @fields;

      #debugging
      print Dumper \%this_row; 

      #push each field onto the all rows hash. 
      foreach my $column ( @header ) {
         #append current to field, in case there's duplicates (no overwriting)
         $all_rows{$ID}{$column} .= $this_row{$column}; 
      }
   }
}

#print for debugging
print Dumper \%all_rows;
print Dumper \%seen_cols;

#grab list of column headings we've seen, and order them. 
my @cols_to_print = sort keys %seen_cols;

#print header row. 
print join "\t", "Accesion", @cols_to_print,"\n";
#iteate keys, and splice. 
foreach my $key ( sort keys %all_rows ) { 
    #print one row at a time.
    #map iterates all the columns, and gives the value or an empty string
    #if it's undefined. (prevents errors)
    print join "\t", $key, (map { $all_rows{$key}{$_} // '' } @cols_to_print),"\n"
}

给定您的输入(排除调试),将打印:

Accesion    S01 S02 S03 
AJ863320    1   2   5   
AM930424    1   1       
AY664038    2           
EU236327        1   2   
EU434346        2   2
sq1bmfud

sq1bmfud2#

这里有一个更简单的解决方案,使用eBay的tsv-utils,特别是tsv-join命令,如下所示:

FILES="S01.tsv S02.tsv S03.tsv"

tsv-select -H -f Accesion $FILES | tsv-uniq >out.tsv

for infile in $FILES
do
        tsv-join -H --filter-file $infile --key-fields 1 \
                    --append-fields 2  --write-all '' \
                    out.tsv >tmp.tsv;
        mv tmp.tsv out.tsv
done

只需要非常基本的攻击技巧。

bmp9r5qi

bmp9r5qi3#

我使用csvtool来完成这个任务和许多csv/tsv任务(请参阅doc)。

COMPARED=1 # Controls which columns are compared 
COPIED=2-3 # Controls which columns are copied into the new file
TSV_FILES=( a.tsv b.tsv c.tsv )

csvtool join $COMPARED $COPIED "${TSV_FILES[@]}" -u TAB -t TAB

COMPAREDCOPIED参数可以是单个数字、数字范围或逗号分隔的数字列表。
-u TAB-t TAB参数分别告诉csvtool使用TAB作为输入和输出的分隔符。

相关问题