系统优化-案例分析-网络丢包分析

Posted on | Edited on | In ,

优化思路

见图：

avator

应用层：应用程序错误
Socket：读写缓冲区溢出。排查方案，查看socket参数，抓包
TCP：资源或者链接跟踪超限。排查方案，netstat -s
IP：MTU获取路由错误。排查方案，netstat -i
网络链路层：QoSor校验错误。排查方案，tc
网卡：网络拥塞，DMA的ringBuffer溢出。排查方案，netstat -i，抓包

案例排查链路

看网络包收发状态

1
2
3
4
5
root@nginx:/# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       100      106      0     32 0            43      0      0      0 BMRU
lo       65536        0      0      0 0             0      0      0      0 LRU

查看QoS策略

如下有一个丢包30%的策略删除之

1
2
3
4
5
6
7
root@nginx:/# tc -s qdisc show dev eth0
qdisc netem 8001: root refcnt 2 limit 1000 loss 30%
 Sent 2450 bytes 43 pkt (dropped 18, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

##删除策略
root@nginx:/# tc qdisc del dev eth0 root netem loss 30%

看tcp收发情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

root@nginx:/# netstat -s
Ip:
    Forwarding: 1          //开启转发
    31 total packets received    //总收包数
    0 forwarded            //转发包数
    0 incoming packets discarded  //接收丢包数
    25 incoming packets delivered  //接收的数据包数
    15 requests sent out      //发出的数据包数
Icmp:
    0 ICMP messages received    //收到的ICMP包数
    0 input ICMP message failed    //收到ICMP失败数
    ICMP input histogram:
    0 ICMP messages sent      //ICMP发送数
    0 ICMP messages failed      //ICMP失败数
    ICMP output histogram:
Tcp:
    0 active connection openings  //主动连接数
    0 passive connection openings  //被动连接数
    11 failed connection attempts  //失败连接尝试数
    0 connection resets received  //接收的连接重置数
    0 connections established    //建立连接数
    25 segments received      //已接收报文数
    21 segments sent out      //已发送报文数
    4 segments retransmitted    //重传报文数
    0 bad segments received      //错误报文数
    0 resets sent          //发出的连接重置数
Udp:
    0 packets received
    ...
TcpExt:
    11 resets received for embryonic SYN_RECV sockets  //半连接重置数
    0 packet headers predicted
    TCPTimeouts: 7    //超时数
    TCPSynRetrans: 4  //SYN重传数
  ...

查看iptables

查看各种链的策略

1
2
3
4
iptables -t filter -nvL  ip是数字 verbose list [--line-number-]

## 删除策略
iptables -t filter -D [INPUT or OUTPUT 等链名] [line-number]

注意MTU的大小

hping3 -S通过，但是http访问依然，原因可能和MTU有关，排查后调整

curl: (28) Operation timed out after 3005 milliseconds with 0 bytes received

1
2
3
4
5
6
7
8
root@nginx:/# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       100      106      0     32 0            43      0      0      0 BMRU
lo       65536        0      0      0 0             0      0      0      0 LRU

##调整
root@nginx:/# ifconfig eth0 mtu 1500