java 单机接口限流处理方案
592
2022-06-07
自己写的项目里,为了保证连接不中断,我起一个线程专门发送心跳包保持连接,那这个线程在send发送数据时,可能会与主线程中的send冲突,因此我就想探讨一下socket api是否具有线程安全性。网上很多说法,但多是推测,于是我结合man pages、StackOverflow和大佬们的博客等资料,做了简单的实验测试一下,用事实说话。
以下问题是主要关注Linux tcp,所有结论都是Linux环境下的, 但是也会有UDP, Windows C++,C++ boost库和java语言等StackOverflow上相同问题下的资料和链接。
当两个线程或进程同时对同一socket进行send(write)操作时,会不会出问题,例如两个线程的内容会不会交错, tcp和udp是否情况相同
send或write是否线程安全
关于socket的线程安全, 也有人认为是具备的,他们的意思是:可以一个线程发送,另一个线程同时接收,就叫线程安全,这个我是知道的,但是这与我想讨论的情况不一样,所以需要明确我说的socket线程安全的定义:两个线程同时发送时的线程安全性。
测试思路很简单,一端起两个线程同时循环发送数据(一个线程发一串aaaa,另一个发一串bbbb),另一端接收并打印,看看结果会不会有交错的。
关于内容交错,我并不关心数据达到的顺序,只关心一个大的数据包会不会穿插着另一个数据包,具体来说:两个线程Ta,Tb,有一个socket S,Ta和Tb同时向S发送数据,其中Ta的要发送数据很大(需要分片),由Pa1和Pa2两部分组成;Tb发送数据较小,为Pb1,那么如果对端收到的是:
Pa1 Pa2 Pb1
或者Pb1 Pa1 Pa2
:则认为没有交错(Ta和Tb的数据各自都连续到达,只是顺序未知而已)
Pa1 Pb1 Pa2
:则认为有交错。(因为Ta数据包被隔断不连续了)
注意:测试时候某个线程发送的数据包一定要足够大,否则不会出现交错的现象,我之前就是发送太小,误以为不会交错,原因在第三节的分析中会讲到。
测试代码地址:https://github.com/whuwzp/linuxc/tree/master/bug/twothread_send
server.cpp, 以下简要展示关键代码:
typedef struct {//为了更好地展示,我发送的是结构体,用index标记是第几个a或者b
char buf;
int index;
} myint;
//Ta
void * athread(void *) {
for (i = 0; i < 4 * 1000; i++) {
buf[i].buf = 'a';
buf[i].index = i;//标记是第多少个'a'
}
//循环发送, 每次发送4000个结构体数据
while (true) ret = write(fd_client, buf, 1000 * 4 * sizeof(myint));
}
//Tb
void *bthread(void *) {
for (i = 0; i < 100; i++) {
buf[i].buf = 'b';
buf[i].index = i;//标记是第多少个'b'
}
while (true) {
ret = 0;
for (int i = 0; i < 10; ++i) {//循环发送10次100个结构体
ret += write(fd_client, buf, 100*sizeof(myint));
}
sleep(1);
}
}
client.cpp
while (true) {
memset(buf, 0, 4 * 1000 * sizeof(myint));
ret = (int)read(fd_client, buf, 4 * 1000 * sizeof(myint));
for (i = 0; i < ret/sizeof(myint); i++) {
//因为4000个a打印起来很多,所以只打印b,或者接收数据不完全的情况
if (buf[i].buf == 'b' || ret != 4 * 1000 * sizeof(myint)) {
printf("ret=%d\n", ret);
for (i = 0; i < ret/sizeof(myint); i++){
printf("%c, %d\n", buf[i].buf, buf[i].index);//打印其索引
break;
}
}
}
}
运行:
./server
./client > result.txt
输出结果result.txt
# 此处省略a0~a3391
a, 3389
a, 3390
a, 3391
# 此处是重点, 内容交错的起点
b, 0
b, 1
# 此处省略b0~b99, 10次循环
b, 98
b, 99
a, 3392
a, 3393
# 此处省略a3394~a3999,以及b的后续
通过结果可以看出:
也就是说, Ta与Tb的数据包发生了交错。
首先这两个概念是有区别的(详见stackoverflow上的讨论):
socket不是c/c++语言标准, 所以它的线程安全依赖具体的实现. 每个系统不同, 有的系统实现socket时用了锁来保护内部数据结构, 有的系统可能没有, 所以得看系统具体实现.
Sockets are not part of C++ Standard so it depends on implementation. Generally they are not thread safe since send is not an atomic operation. Check this discussion for additional information.
EDIT: OS could have or couldn't have internal lock for protecting internal structures. It depends on implementation. So you should not count on it.
摘自: https://stackoverflow.com/questions/2354417/c-socket-api-is-thread-safe
以下内容参考:https://quark.tistory.com/m/235 (这个链接不是原文章的链接,原文链接已经失效了,这个是韩国网站上转载的),如果访问慢,可以看我转载的:https://cnblogs.com/whuwzp/articles/thread_safe.html。
简单解释一下:
这个解释非常符合实验结果。但是这只是一种可能,我们只能判断一定是释放了锁,且被别的线程竞争到了锁,是不是只有内存分配失败才会导致这样呢?我也不太确定,希望读者一起读第四节,一起探讨。
其实到现在我仍然不能确定send、write是否具备原子性,所以这里暂时只是分享一些观点,我个人稍微倾向于相信它不是原子性的,但是也希望大佬多指教:
以下摘自IBM文章
the POSIX/SUSv3 standard developers indicate that atomicity for socket I/O is "unspecified".
Why not atomic?
Using Linux kernel 2.6.11 as a reference, because it's the latest kernel processed for easy web cross-referencing at LXR
When a connected TCP send(), sendto(), or sendmsg() arrives in the Linux kernel, it eventually comes through tcp_sendmsg(). tcp_sendmsg() protects itself by acquiring a lock at invocation by calling lock_sock(). tcp_sendmsg() then loops over the buffers in the iovec, allocating associated sk_buff's and cache pages for use in the actual send. As it does so, it pushes the data out to tcp for actual transmission. However, if one of those allocation fails (because a large number of large sends is being processed, for example), it must wait for memory to become available. It does so by jumping to wait_for_sndbuf or wait_for_memory, both of which eventually cause a call to sk_stream_wait_memory(). sk_stream_wait_memory() contains a code path that calls sk_wait_event(). Finally, sk_wait_event() contains the call to release_sock().
At this point, any one of the threads that were heretofore serialized at the initial call to lock_sock() in tcp_sendmsg() can proceed. Memory may either become available, or a small enough send may not require enough memory to block and may proceed immediately, thus intermixing data from one call to send() with another.
but in the definition of read() in the IEEE Std 1003.1-2001 System Interfaces volume, it gives the following rationale: "The standard developers considered adding atomicity requirements..."
以下认为send是系统调用, 是原子操作, 不会在内核中有竞争. 但是这个和IBM文章就有矛盾了.IBM文章的分析应该是有竞争的(当多个线程同时send). 虽然他认为是原子的, 但也认为多个线程同时发送也要保证同步问题, 以免内容交错.
A lock is not required; send() is a syscall, it is an atomic operation with no race conditions in the kernel.For stream sockets (TCP) too, the send() function is atomic; but there is no concept of distinct messages or packets, the data treated as a single stream of bytes. So even though send() is thread-safe, synchronisation is required to ensure that the bytes from different send calls are merged into the byte stream in a predictable manner.
摘自: https://stackoverflow.com/questions/1981372/are-parallel-calls-to-send-recv-on-the-same-socket-valid/61246416#61246416
以下的观点也认为是原子的, 但是他说send的调用总是完全成功, 或者完全失败, 但是实际上send是可以返回小于请求值的(也就是小于待发送的数据长度).
Sorry, you're wrong. Send is 100% an atomic operation. If the data is too large and won't fit in the output buffer, then the call will block. If the call to send is interrupted, then no data is sent and an error is returned. Calls to send always completely succeed, or completely fail.
摘自: https://stackoverflow.com/questions/3235424/c-sockets-send-thread-safety
以下摘自:Linux man pages http://man7.org/linux/man-pages/man2/send.2.html
If space is not available at the sending socket to hold the message to be transmitted, and the socket file descriptor does not have O_NONBLOCK set, send() shall block until space is available.
When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in nonblocking I/O mode. In nonblocking mode it would fail with the error EAGAIN or EWOULDBLOCK in this case. The select(2) call may be used to determine when it is possible to send more data.
意思是如果socket缓冲区不能装下待发送的数据,则send函数会阻塞直到内存可用,然后发送。
上一节ibm文章依据实验结果,认为并没有阻塞,从而判断man pages的这一观点是错的。
我的观点上节也说了:但是这只是一种可能,我们只能判断一定是释放了锁,且被别的线程竞争到了锁,是不是只有内存分配失败才会导致这样呢?我也不太确定
据所有收集的资料的说法:不会。
因为udp是数据报大小受限(UDP不具备分片能力,或者说不具备分片后,对端正确组包的能力),因此不像tcp发送这么大的数据,他们的操作会很快发送(不会像前面分析的,还会等待内存分配而阻塞),因此“看上去”udp的操作是原子的。
以下是支持这个观点的资料:
Note that datagram sockets (UDP) are connection-less, so the the datagram packets may be delivered in a sequence that is different from the sequence in which they were sent (even if all the send calls were from within a single thread).
摘自:https://stackoverflow.com/questions/1981372/are-parallel-calls-to-send-recv-on-the-same-socket-valid/61246416#61246416
Q: can I concurrently have two threads at the same time write (sendto) on a same file descriptor? How about readfrom and sendt on the same file descriptor at the same time?
A: Yes, but assume two threads A (writes a one message containing "aaaa")
and B which (writes two messages "b" and "c").You can be assured that B's data won't show up inside A's message:
ie. never aabaa
but you can not assume anything about the order they will be received.摘自: https://linux-newbie.vger.kernel.narkive.com/wdAJ3Kzt/is-sendto-and-recvfrom-thread-safe#post4
对于 UDP,多线程读写同一个 socket 不用加锁,不过更好的做法是每个线程有自己的 socket,避免 contention,可以用 SO_REUSEPORT 来实现这一点。
对于 TCP,通常多线程读写同一个 socket 是错误的设计,因为有 short write 的可能。假如你加锁,而又发生 short write,你是不是要一直等到整条消息发送完才解锁(无论阻塞IO还是非阻塞IO)?如果这样,你的临界区长度由对方什么时候接收数据来决定,一个慢的 peer 就把你的程序搞死了。
总结:对于 UDP,加锁是多余的;对于 TCP,加锁是错误的。
摘自知乎陈硕大佬的回答: https://zhihu.com/question/56899596/answer/150926723
fwrite是c/c++语言库实现的,它最终是调用write系统调用,但是做了一些封装,关于FILE*等可以看我之前的内容:https://cnblogs.com/whuwzp/p/stdin_stdout_fflush_file.html。
另外这篇文章用实验证明了fwrite的线程安全性:https://cloud.tencent.comhttps://p.download-x.com/developer/article/1412015 (但是我和他关于write的线程安全性的实验的结果是不一样的,我测试是也具备线程安全性的,即使不加APPEND模式)
在同一个进程内, 针对同一个FILE*的操作(比如fwrite), 是线程安全的. 当然这只在POSIX兼容的系统上成立, Windows上的FILE*的操作并不是线程安全的.
http://gcc.gnu.org/onlinedocs/libstdc++/manual/using_concurrency.htmlAs a n example, the POSIX standard requires that C stdio FILE* operations are atomic. POSIX-conforming C libraries (e.g, on Solaris and GNU/Linux) have an internal mutex to serialize operations on FILE*s. However, you still need to not do stupid things like calling fclose(fs) in one thread followed by an access of fs in another.
摘自知乎大神egmkang wang的回答: https://zhihu.com/question/40472431/answer/87077691
It depends upon which primitives you're using to submit data to the socket.
If you're using write(2), send(2), sendto(2), or sendmsg(2) and the size of your message is small enough to fit entirely within the kernel buffers for the socket, then that entire write will be sent as a block without other data being interspersed.
摘自:https://stackoverflow.com/questions/7942595/linux-c-c-socket-send-in-multi-thread-code
无论什么平台,都不要尝试多线程send。
版权声明:本文内容由网络用户投稿,版权归原作者所有,本站不拥有其著作权,亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容,请联系我们jiasou666@gmail.com 处理,核实后本网站将在24小时内删除侵权内容。
发表评论
暂时没有评论,来抢沙发吧~