Linux:Awk跟Sed总结

sed 和awk都是Linux常见的命令,也是处理文件的常用的命令。

Posted by zhangshun on September 11, 2019

Linux 系统的两个神级工具:sed 和 awk,算是运维平时工作中最常用的工具了,所以有必要更加深入的学习一下了。

Awk

用例:log.txt

1
2
3
4
2 this is a test
3 Are you like awk
This's a test
10 There are orange,apple,mongo

Ⅰ、普通输出

1
2
3
4
5
[root@node2 awk]# awk '{print $1,$4}' log.txt
2 a
3 like
This's
10 orange,apple,mongo

Ⅱ、格式化输出

1
2
3
4
5
[root@node2 awk]# awk '{printf "%s|%s\n",$1,$4}' log.txt
2|a
3|like
This's|
10|orange,apple,mongo
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@node2 awk]# awk '{printf "%02s %-2s %-20s %-20s %s\n",NR,$1,$2,$3,"1"}' log.txt
1 2  this                 is                   1
2 3  Are                  you                  1
3 This's a                    test                 1
4 10 There                are                  1


%[flags][width][.precision]conversion 
% -     5       .2         f

- %是必须的, 任何格式符都由百分号开始
- flags 是可选的,详见下表
- width 是可选的, 表示输出的宽度
- precision 可选的, precision 依赖于 conversion, 详见下表.
- conversion 是必须的, 表示如何格式化参数, 详见下表

conversion:%s——>字符串,%d——>十进制整数
flags:0——>数字前面补0

Ⅲ、-F,按照指定字符分割

1
2
3
4
5
[root@node2 awk]# awk -F, '{print $1,$2}'   log.txt
2 this is a test
3 Are you like awk
This's a test
10 There are orange apple

Ⅳ、变量NR,表示当前处理的是第几行

1
2
3
4
5
[root@node2 awk]# awk '{print NR")" $1}' log.txt
1)2
2)3
3)This's
4)10

Ⅴ、内置函数tolower()、toupper()、length()

1
2
3
4
5
[root@node2 awk]# awk '{print tolower($0)}' log.txt 
2 this is a test
3 are you like awk
this's a test
10 there are orange,apple,mongo
1
2
3
4
5
[root@node2 awk]# awk '{print toupper($0)}' log.txt 
2 THIS IS A TEST
3 ARE YOU LIKE AWK
THIS'S A TEST
10 THERE ARE ORANGE,APPLE,MONGO
1
2
3
[root@node2 awk]# awk '{if (length($1)==1) print $0}' log.txt 
2 this is a test
3 Are you like awk

Ⅵ、awk ‘条件 动作’ 文件名,允许指定输出条件,只输出符合条件的行

1
2
3
[root@node2 awk]# awk '/is/ {print $1,$3}' log.txt
2 is
This's test

上面代码中,print命令前面是一个正则表达式,只输出包含is的行
下面的例子只输出奇数行,以及输出第二行以后的行

1
2
3
4
5
6
[root@node2 awk]# awk 'NR%2==1 {printf "%s\t%s\n",$1,$2}' log.txt
2    this
This's    a
[root@node2 awk]# awk 'NR>2 {printf "%s=====%s\n",tolower($1),toupper($2)}' log.txt
this's=====A
10=====THERE

Ⅶ、if语句
格式:awk '{if (条件) print $1,$2}' $filename

1
2
3
4
5
6
7
[root@node2 awk]# awk '{if ($1==2) printf "%s====%s\n",$1,$2}' log.txt
2====this
[root@node2 awk]# awk '{if (NR>2) printf "%s~~%s\n",$1,$2}' log.txt
This's~~a
10~~There
[root@node2 awk]# awk '{if ($NF=="awk") printf "%s==%s==%s==%s\n",$1,$2,$3,$4}' log.txt
3==Are==you==like

格式:awk '{if (条件) print $1,$2;else print "----"}' $filename

1
2
3
4
5
6
7
8
9
10
[root@node2 awk]# awk '{if ($1==2) print $0;else print "---"}' log.txt
2 this is a test
---
---
---
[root@node2 awk]# awk '{if ($NF=="awk") print $0;else print "---"}' log.txt
---
3 Are you like awk
---
---

Ⅷ、awk的运算符
关系运算符:<、<=、>、>=、!=、==
逻辑或:||
逻辑与:&&
赋值语句:=、+=、-=、*=、/=、%=、^=、**=
匹配正则表达式和不匹配正则表达式:~、!~

Ⅸ、字符串匹配

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@node2 awk]# awk '$1~/This/ {print $0}' log.txt
This's a test
[root@node2 awk]# awk '$NF~/est/ {print $0}' log.txt
2 this is a test
This's a test
[root@node2 awk]# awk '$NF!~/apple/ {print $0}' log.txt
2 this is a test
3 Are you like awk
This's a test

#同时匹配两个
[root@node2 awk]# awk '$NF~/mongo|awk/ {print $0}' log.txt
3 Are you like awk
10 There are orange,apple,mongo

Ⅹ、BEGIN{}、{}、END{},意味着执行前、执行中和执行后的意思

1
2
3
4
5
6
$ cat score.txt
Marry   2143 78 84 77
Jack    2321 66 78 45
Tom     2122 48 77 71
Mike    2537 87 97 95
Bob     2415 40 57 62

awk脚本如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ cat cal.awk
#!/bin/awk -f
#运行前
BEGIN {
    math = 0
    english = 0
    computer = 0
 
    printf "NAME    NO.   MATH  ENGLISH  COMPUTER   TOTAL\n"
    printf "---------------------------------------------\n"
}
#运行中
{
    math+=$3
    english+=$4
    computer+=$5
    printf "%-6s %-6s %4d %8d %8d %8d\n", $1, $2, $3,$4,$5, $3+$4+$5
}
#运行后
END {
    printf "---------------------------------------------\n"
    printf "  TOTAL:%10d %8d %8d \n", math, english, computer
    printf "AVERAGE:%10.2f %8.2f %8.2f\n", math/NR, english/NR, computer/NR
}

来看一下执行结果:(也可以这样运行 ./cal.awk score.txt)

1
2
3
4
5
6
7
8
9
10
11
$ awk -f cal.awk score.txt
NAME    NO.   MATH  ENGLISH  COMPUTER   TOTAL
---------------------------------------------
Marry  2143     78       84       77      239
Jack   2321     66       78       45      189
Tom    2122     48       77       71      196
Mike   2537     87       97       95      279
Bob    2415     40       57       62      159
---------------------------------------------
  TOTAL:       319      393      350
AVERAGE:     63.80    78.60    70.00

参考

AWK 简明教程