awk命令示例详解

今天药忘吃喽~ 2022-05-15 14:50 150阅读 0赞

> **awk options program file**

> 一种用于文本处理的编程语言工具

参数**options**通常可以有以下选项

*  F fs：指定文件分隔符
 *  f file：指定awk脚本文件
 *  v var=value：定义变量

### 使用变量 ###

*  $0：表示整行
 *  $1：表示第一个数据字段
 *  $2：表示第二个数据字段
 *  $n：表示第N个数据字段

假设我们有**myfile**定义如下：

icbc@ubuntu:~$ cat myfile
    this is a test
    this is the second test.
    icbc@ubuntu:~$

执行命令：

awk '{print $1}' myfile

得到如下执行结果：

icbc@ubuntu:~$ awk '{print $1}' myfile
    this
    this
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

如果某些文件中分隔符不是空格或者Tab键，我们可以通过-**F**参数来指定文件分隔符：

awk -F: '{print $1}' /etc/passwd

得到如下结果：

icbc@ubuntu:~$ awk -F: '{print $1}' /etc/passwd
    root
    daemon
    bin
    sys
    sync
    games
    man
    lp
    mail
    news
    uucp
    proxy
    www-data
    backup
    list
    irc
    gnats
    nobody
    systemd-timesync
    systemd-network
    systemd-resolve
    syslog
    _apt
    messagebus
    uuidd
    lightdm
    whoopsie
    avahi-autoipd
    avahi
    dnsmasq
    colord
    speech-dispatcher
    hplip
    kernoops
    pulse
    rtkit
    saned
    usbmux
    icbc
    mysql
    cups-pk-helper
    geoclue
    gdm
    gnome-initial-setup
    redis
    jenkins
    sshd
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

### 使用多个命令 ###

echo "Hello Tom" | awk '{$2="Adam"; print $0}'

上述命令会将$2的值设置为**Adam**，打印整行得到如下结果：

icbc@ubuntu:~$ echo "Hello Tom" | awk '{$2="Adam"; print $0}'
    Hello Adam
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

### 从文件中读取脚本文件 ###

我们有**testfile**内容如下：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ cat testfile
    {print $1 " home at " $6}
    
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

调用脚本文件得到如下内容：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk -F: -f testfile /etc/passwd
    root home at /root
    daemon home at /usr/sbin
    bin home at /bin
    sys home at /dev
    sync home at /bin
    games home at /usr/games
    man home at /var/cache/man
    lp home at /var/spool/lpd
    mail home at /var/mail
    news home at /var/spool/news
    uucp home at /var/spool/uucp
    proxy home at /bin
    www-data home at /var/www
    backup home at /var/backups
    list home at /var/list
    irc home at /var/run/ircd
    gnats home at /var/lib/gnats
    nobody home at /nonexistent
    systemd-timesync home at /run/systemd
    systemd-network home at /run/systemd/netif
    systemd-resolve home at /run/systemd/resolve
    syslog home at /home/syslog
    _apt home at /nonexistent
    messagebus home at /var/run/dbus
    uuidd home at /run/uuidd
    lightdm home at /var/lib/lightdm
    whoopsie home at /nonexistent
    avahi-autoipd home at /var/lib/avahi-autoipd
    avahi home at /var/run/avahi-daemon
    dnsmasq home at /var/lib/misc
    colord home at /var/lib/colord
    speech-dispatcher home at /var/run/speech-dispatcher
    hplip home at /var/run/hplip
    kernoops home at /
    pulse home at /var/run/pulse
    rtkit home at /proc
    saned home at /var/lib/saned
    usbmux home at /var/lib/usbmux
    icbc home at /home/icbc
    mysql home at /nonexistent
    cups-pk-helper home at /home/cups-pk-helper
    geoclue home at /var/lib/geoclue
    gdm home at /var/lib/gdm3
    gnome-initial-setup home at /run/gnome-initial-setup/
    redis home at /var/lib/redis
    jenkins home at /var/lib/jenkins
    sshd home at /run/sshd
    icbc@ubuntu:~$

我们将脚本文件内容修改如下：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ cat testfile2
    {
    text = " home at "
    print $1 $6
    }
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

再次调用得到如下结果：

icbc@ubuntu:~$ awk -F: -f testfile2 /etc/passwd
    root/root
    daemon/usr/sbin
    bin/bin
    sys/dev
    sync/bin
    games/usr/games
    man/var/cache/man
    lp/var/spool/lpd
    mail/var/mail
    news/var/spool/news
    uucp/var/spool/uucp
    proxy/bin
    www-data/var/www
    backup/var/backups
    list/var/list
    irc/var/run/ircd
    gnats/var/lib/gnats
    nobody/nonexistent
    systemd-timesync/run/systemd
    systemd-network/run/systemd/netif
    systemd-resolve/run/systemd/resolve
    syslog/home/syslog
    _apt/nonexistent
    messagebus/var/run/dbus
    uuidd/run/uuidd
    lightdm/var/lib/lightdm
    whoopsie/nonexistent
    avahi-autoipd/var/lib/avahi-autoipd
    avahi/var/run/avahi-daemon
    dnsmasq/var/lib/misc
    colord/var/lib/colord
    speech-dispatcher/var/run/speech-dispatcher
    hplip/var/run/hplip
    kernoops/
    pulse/var/run/pulse
    rtkit/proc
    saned/var/lib/saned
    usbmux/var/lib/usbmux
    icbc/home/icbc
    mysql/nonexistent
    cups-pk-helper/home/cups-pk-helper
    geoclue/var/lib/geoclue
    gdm/var/lib/gdm3
    gnome-initial-setup/run/gnome-initial-setup/
    redis/var/lib/redis
    jenkins/var/lib/jenkins
    sshd/run/sshd
    icbc@ubuntu:~$

### awk预处理 ###

如果我们需要对我们的处理结果添加标题或者抬头，那么我们就可以使用**BEGIN**关键字来实现，**BEGIN**关键字会确保在数据处理前执行。

awk 'BEGIN {print "The File Contents:"}
     
    {print $0}' myfile

得到执行结果如下：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk 'BEGIN {print "The File Contents:"}
    >  
    > {print $0}' myfile
    The File Contents:
    this is a test
    this is the second test.
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

\#\#\#awk后处理  
使用**END**关键字

awk 'BEGIN {print "The File Contents:"}
     
    {print $0}
     
    END {print "File footer"}' myfile

得到执行结果如下：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk 'BEGIN {print "The File Contents:"}
    >  
    > {print $0}
    >  
    > END {print "File footer"}' myfile
    The File Contents:
    this is a test
    this is the second test.
    File footer
    icbc@ubuntu:~$

组合起来一起使用

icbc@ubuntu:~$ cat testfile3
    BEGIN{
    print "USERS and their corresponding home"
    print "UserName \t HomePath"
    print "---\t---"
    FS=":"
    }
    {
    print $1 "\t" $6
    }
    END {
    print "The end"
    }
    icbc@ubuntu:~$

调用执行得到如下结果

icbc@ubuntu:~$ awk -f testfile3  /etc/passwd
    USERS and their corresponding home
    UserName 	 HomePath
    ---	---
    root	/root
    daemon	/usr/sbin
    bin	/bin
    sys	/dev
    sync	/bin
    games	/usr/games
    man	/var/cache/man
    lp	/var/spool/lpd
    mail	/var/mail
    news	/var/spool/news
    uucp	/var/spool/uucp
    proxy	/bin
    www-data	/var/www
    backup	/var/backups
    list	/var/list
    irc	/var/run/ircd
    gnats	/var/lib/gnats
    nobody	/nonexistent
    systemd-timesync	/run/systemd
    systemd-network	/run/systemd/netif
    systemd-resolve	/run/systemd/resolve
    syslog	/home/syslog
    _apt	/nonexistent
    messagebus	/var/run/dbus
    uuidd	/run/uuidd
    lightdm	/var/lib/lightdm
    whoopsie	/nonexistent
    avahi-autoipd	/var/lib/avahi-autoipd
    avahi	/var/run/avahi-daemon
    dnsmasq	/var/lib/misc
    colord	/var/lib/colord
    speech-dispatcher	/var/run/speech-dispatcher
    hplip	/var/run/hplip
    kernoops	/
    pulse	/var/run/pulse
    rtkit	/proc
    saned	/var/lib/saned
    usbmux	/var/lib/usbmux
    icbc	/home/icbc
    mysql	/nonexistent
    cups-pk-helper	/home/cups-pk-helper
    geoclue	/var/lib/geoclue
    gdm	/var/lib/gdm3
    gnome-initial-setup	/run/gnome-initial-setup/
    redis	/var/lib/redis
    jenkins	/var/lib/jenkins
    sshd	/run/sshd
    The end
    icbc@ubuntu:~$

### 使用内嵌变量 ###

除了我们之前所提及的$1,$2等内嵌变量，awk还支持一些其他内嵌变量

*  FIELDWIDTHS ：指定字段宽度
 *  RS：指定记录分隔符
 *  FS：指定字段分隔符
 *  OFS：指定输出分隔符
 *  ORS：指定输出分隔符

**OFS**默认为空格，也可以指定别的字符。

icbc@ubuntu:~$ awk 'BEGIN{FS=":"; OFS="-"} {print $1,$6,$7}' /etc/passwd
    root-/root-/bin/bash
    daemon-/usr/sbin-/usr/sbin/nologin
    bin-/bin-/usr/sbin/nologin
    sys-/dev-/usr/sbin/nologin
    sync-/bin-/bin/sync
    games-/usr/games-/usr/sbin/nologin
    man-/var/cache/man-/usr/sbin/nologin
    lp-/var/spool/lpd-/usr/sbin/nologin
    mail-/var/mail-/usr/sbin/nologin
    news-/var/spool/news-/usr/sbin/nologin
    uucp-/var/spool/uucp-/usr/sbin/nologin
    proxy-/bin-/usr/sbin/nologin
    www-data-/var/www-/usr/sbin/nologin
    backup-/var/backups-/usr/sbin/nologin
    list-/var/list-/usr/sbin/nologin
    irc-/var/run/ircd-/usr/sbin/nologin
    gnats-/var/lib/gnats-/usr/sbin/nologin
    nobody-/nonexistent-/usr/sbin/nologin
    systemd-timesync-/run/systemd-/bin/false
    systemd-network-/run/systemd/netif-/bin/false
    systemd-resolve-/run/systemd/resolve-/bin/false
    syslog-/home/syslog-/bin/false
    _apt-/nonexistent-/bin/false
    messagebus-/var/run/dbus-/bin/false
    uuidd-/run/uuidd-/bin/false
    lightdm-/var/lib/lightdm-/bin/false
    whoopsie-/nonexistent-/bin/false
    avahi-autoipd-/var/lib/avahi-autoipd-/bin/false
    avahi-/var/run/avahi-daemon-/bin/false
    dnsmasq-/var/lib/misc-/bin/false
    colord-/var/lib/colord-/bin/false
    speech-dispatcher-/var/run/speech-dispatcher-/bin/false
    hplip-/var/run/hplip-/bin/false
    kernoops-/-/bin/false
    pulse-/var/run/pulse-/bin/false
    rtkit-/proc-/bin/false
    saned-/var/lib/saned-/bin/false
    usbmux-/var/lib/usbmux-/bin/false
    icbc-/home/icbc-/bin/bash
    mysql-/nonexistent-/bin/false
    cups-pk-helper-/home/cups-pk-helper-/usr/sbin/nologin
    geoclue-/var/lib/geoclue-/usr/sbin/nologin
    gdm-/var/lib/gdm3-/bin/false
    gnome-initial-setup-/run/gnome-initial-setup/-/bin/false
    redis-/var/lib/redis-/usr/sbin/nologin
    jenkins-/var/lib/jenkins-/bin/bash
    sshd-/run/sshd-/usr/sbin/nologin
    icbc@ubuntu:~$

假设我们有文件内容如下：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ cat cash
    1235.96521
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

使用**FIELDWIDTHS**关键字得到如下结果：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk 'BEGIN{FIELDWIDTHS="3 4 3"}{print $1,$2,$3}' cash
    123 5.96 521
    icbc@ubuntu:~$

假设我们有内容如下：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ cat person
    Person Name
    123 High Street
    (222)466-1234
    
    Another person
    487 High Street
    (523)643-8754
    
    icbc@ubuntu:~$

数据通过换行符区分，此时我们需要通过FS指定分隔符为换行符，RS为空

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk 'BEGIN{FS="\n"; RS=""} {print $1,$3}' person
    Person Name (222)466-1234
    Another person (523)643-8754
    icbc@ubuntu:~$

### 其他变量 ###

*  ARGC ：传参个数
 *  ARGV ：命令行参数
 *  ENVIRON ：环境变量
 *  FILENAME ：awk处理目标文件
 *  NF ：正在处理行的记录数
 *  NR ：已处理记录数
 *  FNR ：被处理记录
 *  IGNORECASE：忽略大小写

简单测试一下：

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk 'BEGIN{print ARGC,ARGV[1]}' myfile
    2 myfile
    icbc@ubuntu:~$

使用环境变量

icbc@ubuntu:~$  awk '
    >  
    > BEGIN{
    >  
    > print ENVIRON["PATH"]
    >  
    > }'
    /usr/lib/jvm/java-8-openjdk-amd64/bin:/usr/lib/jvm/java-8-openjdk-amd64/jre/bin:/home/icbc/software/go/bin:/home/icbc/software/node-v9.4.0-linux-x64/bin:/home/icbc/bin:/home/icbc/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
    icbc@ubuntu:~$

也可以直接使用bash变量

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ echo | awk -v home=$HOME '{print "My home is " home}'
    My home is /home/icbc
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

**NF**变量表示的是记录中的最后一个字段

icbc@ubuntu:~$ awk 'BEGIN{FS=":"; OFS=":"} {print $1,$NF}' /etc/passwd
    root:/bin/bash
    daemon:/usr/sbin/nologin
    bin:/usr/sbin/nologin
    sys:/usr/sbin/nologin
    sync:/bin/sync
    games:/usr/sbin/nologin
    man:/usr/sbin/nologin
    lp:/usr/sbin/nologin
    mail:/usr/sbin/nologin
    news:/usr/sbin/nologin
    uucp:/usr/sbin/nologin
    proxy:/usr/sbin/nologin
    www-data:/usr/sbin/nologin
    backup:/usr/sbin/nologin
    list:/usr/sbin/nologin
    irc:/usr/sbin/nologin
    gnats:/usr/sbin/nologin
    nobody:/usr/sbin/nologin
    systemd-timesync:/bin/false
    systemd-network:/bin/false
    systemd-resolve:/bin/false
    syslog:/bin/false
    _apt:/bin/false
    messagebus:/bin/false
    uuidd:/bin/false
    lightdm:/bin/false
    whoopsie:/bin/false
    avahi-autoipd:/bin/false
    avahi:/bin/false
    dnsmasq:/bin/false
    colord:/bin/false
    speech-dispatcher:/bin/false
    hplip:/bin/false
    kernoops:/bin/false
    pulse:/bin/false
    rtkit:/bin/false
    saned:/bin/false
    usbmux:/bin/false
    icbc:/bin/bash
    mysql:/bin/false
    cups-pk-helper:/us

我们看下**NR**与**FNR**的区别

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk 'BEGIN{FS=","}{print $1,"FNR="FNR}' myfile myfile
    this is a test FNR=1
    this is the second test. FNR=2
    this is a test FNR=1
    this is the second test. FNR=2
    icbc@ubuntu:~$

上面的例子中，我们定义了两个输入文件，同一个文件处理两次。输出是第一个字段值和**FNR**。看下**NR**与**FNR**的区别：处理到新文件时，**FNR**会重新变成1，而**NR**会继续累加。

icbc@ubuntu:~$ awk '
    >  
    > BEGIN {FS=","}
    >  
    > {print $1,"FNR="FNR,"NR="NR}
    >  
    > END{print "Total",NR,"processed lines"}' myfile myfile
    this is a test FNR=1 NR=1
    this is the second test. FNR=2 NR=2
    this is a test FNR=1 NR=3
    this is the second test. FNR=2 NR=4
    Total 4 processed lines
    icbc@ubuntu:~$

### 使用自定义变量 ###

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk '
    >  
    > BEGIN{
    >  
    > test="Welcome to LikeGeeks website"
    >  
    > print test
    >  
    > }'
    Welcome to LikeGeeks website
    icbc@ubuntu:~$

### 结构化命令 ###

假设有文件内容如下：

icbc@ubuntu:~$ cat numbers
    10
    15
    6
    33
    45
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

执行**IF**条件判断

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk '{if ($1 > 30) print $1}' numbers
    33
    45
    icbc@ubuntu:~$

或者使用大括号执行多条语句

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk '{
    >  
    > if ($1 > 30)
    >  
    > {
    >  
    > x = $1 * 3
    >  
    > print x
    >  
    > }
    >  
    > }' numbers
    99
    135
    icbc@ubuntu:~$

也可以使用**ELSE**语句

icbc@ubuntu:~$ awk '{
    >  
    > if ($1 > 30)
    >  
    > {
    >  
    > x = $1 * 3
    >  
    > print x
    >  
    > } else
    >  
    > {
    >  
    > x = $1 / 2
    >  
    > print x
    >  
    > }}' numbers
    5
    7.5
    3
    99
    135
    icbc@ubuntu:~$

也可以使用分号将**ELSE**写在一行上

icbc@ubuntu:~$ awk '{if ($1 > 20) print $1 *2;else print $1 /2 }' numbers
    5
    7.5
    3
    66
    90
    icbc@ubuntu:~$

### While循环 ###

有文件内容如下：

icbc@ubuntu:~$ cat numbers2
    124 127 130
    112 142 135
    175 158 245
    118 231 147
    icbc@ubuntu:~$ 
    icbc@ubuntu:~$

循环求平均

icbc@ubuntu:~$ awk '{
    >  
    > sum = 0
    >  
    > i = 1
    >  
    > while (i < 5)
    >  
    > {
    >  
    > sum += $i
    >  
    > i++
    >  
    > }
    >  
    > average = sum / 3
    >  
    > print "Average:",average
    >  
    > }' numbers2
    Average: 127
    Average: 129.667
    Average: 192.667
    Average: 165.333
    icbc@ubuntu:~$

使用**break**中断循环

icbc@ubuntu:~$ awk '{
     
    tot = 0
     
    i = 1
     
    while (i < 5)
     
    {
     
    tot += $i
     
    if (i == 3)
     
    break
     
    i++
     
    }
     
    average = tot / 3
     
    print "Average is:",average
     
    }' numbers2
    Average is: 127
    Average is: 129.667
    Average is: 192.667
    Average is: 165.333
    icbc@ubuntu:~$

### For循环 ###

icbc@ubuntu:~$ awk '{
     
    total = 0
     
    for (var = 1; var < 5; var++)
     
    {
     
    total += $var
     
    }
     
    avg = total / 3
     
    print "Average:",avg
     
    }' numbers2
    Average: 127
    Average: 129.667
    Average: 192.667
    Average: 165.333
    icbc@ubuntu:~$

### 格式化打印 ###

> %\[modifier\]control-letter

*  c：将数字作为字串打印
 *  d：打印整型数据
 *  e：科学计数
 *  f：浮点数
 *  o：八进制
 *  s：字符串

icbc@ubuntu:~$  awk 'BEGIN{
    >  
    > x = 100 * 100
    >  
    > printf "The result is: %e\n", x
    >  
    > }'
    The result is: 1.000000e+04
    icbc@ubuntu:~$

### 内嵌函数 ###

#### 数学函数 ####

icbc@ubuntu:~$ awk 'BEGIN{x=exp(5); print x}'
    148.413
    icbc@ubuntu:~$

#### 字符串函数 ####

icbc@ubuntu:~$ 
    icbc@ubuntu:~$ awk 'BEGIN{x = "likegeeks"; print toupper(x)}'
    LIKEGEEKS
    icbc@ubuntu:~$

### 使用自定义函数 ###

icbc@ubuntu:~$ awk '
    >  
    > function myfunc()
    >  
    > {
    >  
    > printf "The user %s has home path at %s\n", $1,$6
    >  
    > }
    >  
    > BEGIN{FS=":"}
    >  
    > {
    >  
    > myfunc()
    >  
    > }' /etc/passwd
    The user root has home path at /root
    The user daemon has home path at /usr/sbin
    The user bin has home path at /bin
    The user sys has home path at /dev
    The user sync has home path at /bin
    The user games has home path at /usr/games
    The user man has home path at /var/cache/man
    The user lp has home path at /var/spool/lpd
    The user mail has home path at /var/mail
    The user news has home path at /var/spool/news
    The user uucp has home path at /var/spool/uucp
    The user proxy has home path at /bin
    The user www-data has home path at /var/www
    The user backup has home path at /var/backups
    The user list has home path at /var/list
    The user irc has home path at /var/run/ircd
    The user gnats has home path at /var/lib/gnats
    The user nobody has home path at /nonexistent
    The user systemd-timesync has home path at /run/systemd
    The user systemd-network has home path at /run/systemd/netif
    The user systemd-resolve has home path at /run/systemd/resolve
    The user syslog has home path at /home/syslog
    The user _apt has home path at /nonexistent
    The user messagebus has home path at /var/run/dbus
    The user uuidd has home path at /run/uuidd
    The user lightdm has home path at /var/lib/lightdm
    The user whoopsie has home path at /nonexistent
    The user avahi-autoipd has home path at /var/lib/avahi-autoipd
    The user avahi has home path at /var/run/avahi-daemon
    The user dnsmasq has home path at /var/lib/misc
    The user colord has home path at /var/lib/colord
    The user speech-dispatcher has home path at /var/run/speech-dispatcher
    The user hplip has home path at /var/run/hplip
    The user kernoops has home path at /
    The user pulse has home path at /var/run/pulse
    The user rtkit has home path at /proc
    The user saned has home path at /var/lib/saned
    The user usbmux has home path at /var/lib/usbmux
    The user icbc has home path at /home/icbc
    The user mysql has home path at /nonexistent
    The user cups-pk-helper has home path at /home/cups-pk-helper
    The user geoclue has home path at /var/lib/geoclue
    The user gdm has home path at /var/lib/gdm3
    The user gnome-initial-setup has home path at /run/gnome-initial-setup/
    The user redis has home path at /var/lib/redis
    The user jenkins has home path at /var/lib/jenkins
    The user sshd has home path at /run/sshd
    icbc@ubuntu:~$

*原文链接*  
[https://likegeeks.com/awk-command/][https_likegeeks.com_awk-command]

[https_likegeeks.com_awk-command]: https://likegeeks.com/awk-command/