Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

连接频繁被reset，无法建立连接 #381

Open
icarus-777 opened this issue Sep 21, 2020 · 3 comments
Open

连接频繁被reset，无法建立连接 #381

icarus-777 opened this issue Sep 21, 2020 · 3 comments

Comments

@icarus-777
Copy link

10.36.196.16 是线上服务器， 10.36.84.26是测试服务器，intercept部署在测试服务器。
线上服务器运行命令
./tcpcopy -x 7352-10.36.84.26:7352 -s 10.36.84.26 -C 4 -n 1 -l /var/log/tiger/tcopy_server_test.log -c 10.36.84.x
测试服务器运行命令
./intercept -i bond0 -F tcp and src port 7352 -l /var/log/tiger/tcopy_intercept.log

测试服务器tcpdump对应端口显示，能够收到第一次握手包，在第二次握手时发送reset，只后发送一部分包
netstat查看发现无连接建立，SYN状态的连接也没有

15:59:56.193935 IP n36-084-010.byted.org.55869 > n36-084-026.byted.org.7352: Flags [S], seq 582262404, win 29200, options [mss 1460,sackOK,TS val 848898529 ecr 0,nop,wscale 10], length 0
15:59:56.194023 IP n36-084-010.byted.org.55869 > n36-084-026.byted.org.7352: Flags [.], ack 3436793995, win 29, options [nop,nop,TS val 848898529 ecr 3348241], length 0
15:59:56.194037 IP n36-084-010.byted.org.55869 > n36-084-026.byted.org.7352: Flags [P.], seq 0:157, ack 1, win 29, options [nop,nop,TS val 848898529 ecr 3348241], length 157
15:59:56.194049 IP n36-084-010.byted.org.55869 > n36-084-026.byted.org.7352: Flags [R], seq 582262405, win 0, length 0
15:59:56.194074 IP n36-084-010.byted.org.55869 > n36-084-026.byted.org.7352: Flags [R], seq 582262562, win 0, length 0
15:59:56.208190 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [S], seq 3881535137, win 28960, options [mss 1460,sackOK,TS val 288542307 ecr 671969558,nop,wscale 10], length 0
15:59:56.208270 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [.], ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 0
15:59:56.208304 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [R], seq 3881535138, win 0, length 0
15:59:56.208425 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 76489:77937, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1448
15:59:56.208440 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 77937:79385, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1448
15:59:56.208455 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 80585:82033, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1448
15:59:56.208465 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 79385:80585, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1200
15:59:56.208476 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 82033:83481, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1448
15:59:56.208487 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 83481:84929, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1448
15:59:56.208496 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 84929:86377, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1448
15:59:56.208506 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 86377:87825, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 1448
15:59:56.208518 IP n36-084-006.byted.org.12755 > n36-084-026.byted.org.7352: Flags [P.], seq 87825:87850, ack 299869118, win 29, options [nop,nop,TS val 288542307 ecr 3348244], length 25
15:59:56.211219 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [S], seq 1104873878, win 28960, options [mss 1460,sackOK,TS val 288541241 ecr 1347877801,nop,wscale 10], length 0
15:59:56.211259 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [R], seq 1104873879, win 0, length 0
15:59:56.211293 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [.], ack 719098419, win 29, options [nop,nop,TS val 288541241 ecr 3348245], length 0
15:59:56.211372 IP n36-084-013.byted.org.41888 > n36-084-026.byted.org.7352: Flags [S], seq 632415756, win 29200, options [mss 1460,sackOK,TS val 848761173 ecr 0,nop,wscale 10], length 0
15:59:56.211432 IP n36-084-013.byted.org.41888 > n36-084-026.byted.org.7352: Flags [.], ack 3094827976, win 29, options [nop,nop,TS val 848761173 ecr 3348245], length 0
15:59:56.211465 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [P.], seq 383087:384535, ack 719098419, win 29, options [nop,nop,TS val 288541241 ecr 3348245], length 1448
15:59:56.211482 IP n36-084-013.byted.org.41888 > n36-084-026.byted.org.7352: Flags [R], seq 632415757, win 0, length 0
15:59:56.211490 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [P.], seq 384535:385983, ack 719098419, win 29, options [nop,nop,TS val 288541241 ecr 3348245], length 1448
15:59:56.211501 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [P.], seq 385983:387183, ack 719098419, win 29, options [nop,nop,TS val 288541241 ecr 3348245], length 1200
15:59:56.211510 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [P.], seq 387183:388631, ack 719098419, win 29, options [nop,nop,TS val 288541241 ecr 3348245], length 1448
15:59:56.211520 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [P.], seq 388631:390079, ack 719098419, win 29, options [nop,nop,TS val 288541241 ecr 3348245], length 1448
15:59:56.211531 IP n36-084-002.byted.org.64529 > n36-084-026.byted.org.7352: Flags [P.], seq 390079:391527, ack 719098419, win 29, options [nop,nop,TS val 288541241 ecr 3348245], length 1448

初步想到是intercept无法起到黑洞作用，请求被返回在线服务器，排查了几个影响转发的参数rp_filter，ip_forword，iptables，发现设置都正确

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.bond0.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0

net.ipv4.ip_forward = 0

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

辅助服务器日志如下：

2020/09/21 23:39:14 +072 [notice] intercept version:1.0.0
2020/09/21 23:39:14 +072 [notice] intercept internal version:6
2020/09/21 23:39:14 +072 [notice] TC_COMBINED mode
2020/09/21 23:39:14 +072 [notice] pcap version:libpcap version 1.6.2
2020/09/21 23:39:14 +072 [notice] HAVE_SET_IMMEDIATE_MODE is true
2020/09/21 23:39:14 +072 [notice] TC_ADVANCED mode
2020/09/21 23:39:14 +072 [notice] epoll mode
2020/09/21 23:39:14 +072 [notice] device:bond0
2020/09/21 23:39:14 +072 [notice] user filter:tcp and src port 7352
2020/09/21 23:39:14 +072 [notice] msg listen socket:5
2020/09/21 23:39:14 +072 [notice] pcap open,device:bond0
2020/09/21 23:39:14 +072 [notice] pcap_set_buffer_size:16777216
2020/09/21 23:39:17 +437 [notice] it adds fd:7
2020/09/21 23:39:17 +437 [notice] it adds fd:8
2020/09/21 23:39:17 +438 [notice] it adds fd:9
2020/09/21 23:39:17 +438 [notice] it adds fd:10
2020/09/21 23:39:17 +453 [info] malloc memory for fd:7
2020/09/21 23:39:17 +688 [info] malloc memory for fd:8
2020/09/21 23:39:17 +795 [info] malloc memory for fd:9
2020/09/21 23:39:17 +809 [info] malloc memory for fd:10
2020/09/21 23:39:44 +072 [notice] total resp packs:2989, all:2989, route:531, sock write cnt:1711
2020/09/21 23:39:44 +072 [notice] cache hit:2783,missed:0,lost:206
2020/09/21 23:39:44 +072 [notice] search:2989,extra compared:1,all sessions:531
2020/09/21 23:40:14 +072 [notice] total resp packs:6377, all:6377, route:1139, sock write cnt:3676
2020/09/21 23:40:14 +072 [notice] cache hit:6040,missed:2,lost:335
2020/09/21 23:40:14 +072 [notice] search:6377,extra compared:6,all sessions:1139
2020/09/21 23:40:44 +072 [notice] total resp packs:9653, all:9653, route:1741, sock write cnt:5595
2020/09/21 23:40:44 +072 [notice] cache hit:9185,missed:4,lost:464
2020/09/21 23:40:44 +072 [notice] search:9653,extra compared:10,all sessions:1741
2020/09/21 23:41:14 +072 [notice] total resp packs:12946, all:12946, route:2340, sock write cnt:7528
2020/09/21 23:41:14 +072 [notice] cache hit:12349,missed:5,lost:592
2020/09/21 23:41:14 +072 [notice] search:12946,extra compared:14,all sessions:2340
2020/09/21 23:41:44 +072 [notice] total resp packs:16143, all:16143, route:2912, sock write cnt:9397
2020/09/21 23:41:44 +072 [notice] cache hit:15403,missed:6,lost:734
2020/09/21 23:41:44 +072 [notice] search:16143,extra compared:21,all sessions:2912
2020/09/21 23:42:14 +072 [notice] total resp packs:19228, all:19228, route:3457, sock write cnt:11177
2020/09/21 23:42:14 +072 [notice] cache hit:18416,missed:7,lost:805
2020/09/21 23:42:14 +072 [notice] search:19228,extra compared:25,all sessions:3457
2020/09/21 23:42:44 +072 [notice] total resp packs:22243, all:22243, route:4002, sock write cnt:12889
2020/09/21 23:42:44 +072 [notice] cache hit:21403,missed:7,lost:833
2020/09/21 23:42:44 +072 [notice] search:22243,extra compared:26,all sessions:4002
2020/09/21 23:43:14 +072 [notice] total resp packs:25413, all:25413, route:4571, sock write cnt:14711
2020/09/21 23:43:14 +072 [notice] cache hit:24538,missed:7,lost:868
2020/09/21 23:43:14 +072 [notice] search:25413,extra compared:29,all sessions:4571
2020/09/21 23:43:44 +072 [notice] total resp packs:28348, all:28348, route:5102, sock write cnt:16425
2020/09/21 23:43:44 +072 [notice] cache hit:27457,missed:7,lost:884
2020/09/21 23:43:44 +072 [notice] search:28348,extra compared:29,all sessions:5102
2020/09/21 23:44:14 +072 [notice] total resp packs:31307, all:31307, route:5643, sock write cnt:18133
2020/09/21 23:44:14 +072 [notice] cache hit:30377,missed:7,lost:923
2020/09/21 23:44:14 +072 [notice] search:31307,extra compared:30,all sessions:5643
2020/09/21 23:44:44 +072 [notice] total resp packs:34262, all:34262, route:6179, sock write cnt:19815
2020/09/21 23:44:44 +072 [notice] cache hit:33300,missed:7,lost:955
2020/09/21 23:44:44 +072 [notice] search:34262,extra compared:33,all sessions:6179
2020/09/21 23:45:14 +072 [notice] total resp packs:37220, all:37220, route:6707, sock write cnt:21495

在线服务器日志如下：

2020/09/21 23:39:12 +196 [notice] tcpcopy version:1.3.0
2020/09/21 23:39:12 +196 [notice] tcpcopy internal version:6
2020/09/21 23:39:12 +196 [notice] target:7352-10.36.84.26:7352
2020/09/21 23:39:12 +196 [notice] TC_COMBINED mode
2020/09/21 23:39:12 +196 [notice] epoll mode
2020/09/21 23:39:12 +196 [notice] session timeout:600
2020/09/21 23:39:12 +196 [notice] keepalive timeout:720
2020/09/21 23:39:12 +196 [notice] min sess pool size:608
2020/09/21 23:39:12 +196 [notice] sess pool size:1024
2020/09/21 23:39:12 +196 [notice] set global port for tcpcopy
2020/09/21 23:39:12 +196 [notice] raw_clt_tf_ip:10.36.84.x
2020/09/21 23:39:12 +196 [notice] parallel connections per target:4
2020/09/21 23:39:12 +196 [notice] s parameter:10.36.84.26
2020/09/21 23:39:12 +196 [notice] set only ip for tcpcopy
2020/09/21 23:39:12 +196 [error] Can not connect to remote server(10.36.84.26:36524) (Connection refused)
2020/09/21 23:39:12 +196 [warn] sig 0 received
2020/09/21 23:39:12 +196 [info] session table, size:65536, total:0
2020/09/21 23:39:12 +196 [notice] tc_event_loop_finish over
2020/09/21 23:39:12 +196 [notice] user time used:0
2020/09/21 23:39:12 +196 [notice] sys  time used:0
2020/09/21 23:39:12 +196 [notice] max memory size:5192
2020/09/21 23:39:12 +196 [notice] Total allocated space (bytes): 464
2020/09/21 23:39:12 +196 [notice] Total free space (bytes): 134704
2020/09/21 23:39:12 +196 [notice] Top-most, releasable space (bytes): 130128
2020/09/21 23:39:17 +433 [notice] tcpcopy version:1.3.0
2020/09/21 23:39:17 +433 [notice] tcpcopy internal version:6
2020/09/21 23:39:17 +433 [notice] target:7352-10.36.84.26:7352
2020/09/21 23:39:17 +433 [notice] TC_COMBINED mode
2020/09/21 23:39:17 +433 [notice] epoll mode
2020/09/21 23:39:17 +433 [notice] session timeout:600
2020/09/21 23:39:17 +433 [notice] keepalive timeout:720
2020/09/21 23:39:17 +433 [notice] min sess pool size:608
2020/09/21 23:39:17 +433 [notice] sess pool size:1024
2020/09/21 23:39:17 +433 [notice] set global port for tcpcopy
2020/09/21 23:39:17 +433 [notice] raw_clt_tf_ip:10.36.84.x
2020/09/21 23:39:17 +433 [notice] parallel connections per target:4
2020/09/21 23:39:17 +433 [notice] s parameter:10.36.84.26
2020/09/21 23:39:17 +433 [notice] set only ip for tcpcopy
2020/09/21 23:39:17 +433 [info] connect to remote server(10.36.84.26:36524)
2020/09/21 23:39:17 +433 [info] connect to remote server(10.36.84.26:36524)
2020/09/21 23:39:17 +433 [info] connect to remote server(10.36.84.26:36524)
2020/09/21 23:39:17 +433 [info] connect to remote server(10.36.84.26:36524)
2020/09/21 23:39:47 +433 [notice] active:153,rel:451,obs del:0,tw:104
2020/09/21 23:39:47 +433 [notice] conns:604,resp:1272,c-resp:29
2020/09/21 23:39:47 +433 [notice] resp fin:0,resp rst:555
2020/09/21 23:39:47 +433 [notice] send:3643,send content:2399
2020/09/21 23:39:47 +433 [notice] send syn:604, fin:7,reset:0
2020/09/21 23:39:47 +433 [notice] reconnect:0,for no syn:38
2020/09/21 23:39:47 +433 [notice] retransmit:0
2020/09/21 23:39:47 +433 [notice] recv packs after retransmission:0
2020/09/21 23:39:47 +433 [notice] syn cnt:566,all clt:2528,clt cont:830
2020/09/21 23:39:47 +433 [notice] total cont retransmit:0, frag:0
2020/09/21 23:39:47 +433 [notice] total captured packets:28591
2020/09/21 23:40:17 +433 [notice] user time used:0
2020/09/21 23:40:17 +433 [notice] sys  time used:0
2020/09/21 23:40:17 +433 [notice] max memory size:6384
2020/09/21 23:40:17 +433 [notice] Total allocated space (bytes): 1209168
2020/09/21 23:40:17 +433 [notice] Total free space (bytes): 142512
2020/09/21 23:40:17 +433 [notice] Top-most, releasable space (bytes): 13280
2020/09/21 23:40:17 +433 [notice] active:197,rel:1003,obs del:0,tw:85
2020/09/21 23:40:17 +433 [notice] conns:1200,resp:2510,c-resp:52
2020/09/21 23:40:17 +433 [notice] resp fin:0,resp rst:1088
2020/09/21 23:40:17 +433 [notice] send:7253,send content:4788
2020/09/21 23:40:17 +433 [notice] send syn:1200, fin:13,reset:0
2020/09/21 23:40:17 +433 [notice] reconnect:0,for no syn:89
2020/09/21 23:40:17 +433 [notice] retransmit:0
2020/09/21 23:40:17 +433 [notice] recv packs after retransmission:0
2020/09/21 23:40:17 +433 [notice] syn cnt:1116,all clt:4990,clt cont:1647
2020/09/21 23:40:17 +433 [notice] total cont retransmit:0, frag:0
2020/09/21 23:40:17 +433 [notice] total captured packets:56739
2020/09/21 23:40:47 +433 [notice] active:259,rel:1548,obs del:0,tw:82
2020/09/21 23:40:47 +433 [notice] conns:1807,resp:3763,c-resp:71
2020/09/21 23:40:47 +433 [notice] resp fin:1,resp rst:1629
2020/09/21 23:40:47 +433 [notice] send:10841,send content:7133
2020/09/21 23:40:47 +433 [notice] send syn:1807, fin:22,reset:0
2020/09/21 23:40:47 +433 [notice] reconnect:0,for no syn:141
2020/09/21 23:40:47 +433 [notice] retransmit:1
2020/09/21 23:40:47 +433 [notice] recv packs after retransmission:0
2020/09/21 23:40:47 +433 [notice] syn cnt:1674,all clt:7478,clt cont:2464
2020/09/21 23:40:47 +433 [notice] total cont retransmit:0, frag:0
2020/09/21 23:40:47 +433 [notice] total captured packets:84891
2020/09/21 23:41:17 +433 [notice] user time used:0
2020/09/21 23:41:17 +433 [notice] sys  time used:1
2020/09/21 23:41:17 +433 [notice] max memory size:6688
2020/09/21 23:41:17 +433 [notice] Total allocated space (bytes): 1457728
2020/09/21 23:41:17 +433 [notice] Total free space (bytes): 238016

可以看到在线服务器没有发送reset，测试服务器的tcpdump结果却收到了，请问可能是什么问题呢，或者还有什么手段可以排查（是否需要在在线服务器tcpdump，查看是否有reset发出），并且这些服务都没有运行在docker中
@wangbin579
Copy link
Member

问题是出在：
1）-c 10.36.84.x
2）路由设置

由于路由没有设置或者没有生效，测试服务器的响应包会直接回到这些客户端ip的机器（同一个网段），从而带来了reset数据包，干扰了测试进行。

@icarus-777
Copy link
Author

问题是出在：
1）-c 10.36.84.x
2）路由设置

由于路由没有设置或者没有生效，测试服务器的响应包会直接回到这些客户端ip的机器（同一个网段），从而带来了reset数据包，干扰了测试进行。

intercept部署在测试服务器上，应该会捕捉到返回包吧，还需要进行路由设置吗，目前路由是默认设置

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         n36-084-001.byt 0.0.0.0         UG    0      0        0 bond0
localnet        *               255.255.255.0           U     0      0        0 bond0

如果需要的话，应该设置为什么样呢

@wangbin579
Copy link
Member

路由设置的目的是为了让响应包路由到辅助服务器，从而达到黑洞响应包的目的。

路由可以参考文档设置。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wangbin579 @icarus-777