shell命令处理两个文件中的相同部分

比眉伴天荒 2023-02-17 03:55 1阅读 0赞

### 行完全相同 ###

*  [输出两个文件中相同的行][Link 1]  grep -f $file1.txt $file2.txt

root@ae02635267dc:/workspace/autohome/test/shell/awk# cat file2.txt |shuf|head -n 2 >file2_part.txt
    root@ae02635267dc:/workspace/autohome/test/shell/awk# cat file2.txt 
    name1 data1 dada2 data3 data4 dada5
    name2 dada6 data7 dada8
    name3 data9 dada10 data11 dada12
    name4 data13 dada14
    name5 data15 dada16
    name6 data17 data18
    root@ae02635267dc:/workspace/autohome/test/shell/awk# cat file2_part.txt 
    name5 data15 dada16
    name3 data9 dada10 data11 dada12
    root@ae02635267dc:/workspace/autohome/test/shell/awk# grep -f file2.txt file2_part.txt 
    name5 data15 dada16
    name3 data9 dada10 data11 dada12

### 行不完全相同 ###

*  行中部分列对应相同。如下所示，file1.txt的第二列与file2.txt的第一列对应相等。表格的第三列为待输出内容

root@ae02635267dc:/workspace/autohome/test/shell/awk# awk -F " " 'NR==FNR {a[$1]=$0} NR>FNR {print $1 " " a[$2]}' file2.txt file1.txt 
    no1 name1 data1 dada2 data3 data4 dada5
    no2 name2 dada6 data7 dada8
    no3 name2 dada6 data7 dada8
    no4 name3 data9 dada10 data11 dada12
    no5 name4 data13 dada14
    no6 name4 data13 dada14
    no7 name4 data13 dada14
    no8 name5 data15 dada16
    no9 name6 data17 data18
    no10 name6 data17 data18
    root@ae02635267dc:/workspace/autohome/test/shell/awk# awk -F " " 'NR==FNR {a[$1]=$0;next} {print $1 " " a[$2]}' file2.txt file1.txt 
    no1 name1 data1 dada2 data3 data4 dada5
    no2 name2 dada6 data7 dada8
    no3 name2 dada6 data7 dada8
    no4 name3 data9 dada10 data11 dada12
    no5 name4 data13 dada14
    no6 name4 data13 dada14
    no7 name4 data13 dada14
    no8 name5 data15 dada16
    no9 name6 data17 data18
    no10 name6 data17 data18

*  命令行分析

<table style="width:500px;"> 
 <tbody> 
  <tr> 
   <td style="width:140px;">字段</td> 
   <td style="width:359px;">意义</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">NR</td> 
   <td style="width:359px;">已经读出的记录数，就是行号，从1开始</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">FNR</td> 
   <td style="width:359px;">各文件分别计数的行号</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">NR==FNR</td> 
   <td style="width:359px;">表示读第一个文件时的条件分支。&nbsp; 因为行数相等</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">NR&gt;FNR</td> 
   <td style="width:359px;">表示读第二个文件时的条件分支</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">a</td> 
   <td style="width:359px;">存储数据的字典型变量，所以字段值应该保持唯一，否则会被覆盖；</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">$0</td> 
   <td style="width:359px;">当前行的所有内容</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">$1</td> 
   <td style="width:359px;">当前行的第一列</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">a[$1]=$0</td> 
   <td style="width:359px;">以第一列作为key值，存储当前行的内容$0</td> 
  </tr> 
  <tr> 
   <td style="width:140px;">next</td> 
   <td style="width:359px;">忽略之后的语句，进行下一行的匹配</td> 
  </tr> 
 </tbody> 
</table>

参考文献：

1.  [输出两个文件中相同的行][Link 1] 
2.  [Linux awk 命令][Linux awk]
3.  [AWK 技巧（取倒列，过滤行，匹配，不匹配，内置变量等）][AWK]
4.  [awk之NR==FNR问题][awk_NR_FNR]

[Link 1]: https://stackoverflow.com/questions/22100384/awk-to-compare-two-files
[Linux awk]: https://www.runoob.com/linux/linux-comm-awk.html
[AWK]: https://www.cnblogs.com/kevingrace/p/8481965.html
[awk_NR_FNR]: https://www.cnblogs.com/irockcode/p/7044722.html