Run ./compile.sh to compile send.c and generate the shared files.
Run sudo python3 tests/send.py.
test_send_c's Introduction
test_send_c's People
Contributors
Watchers
test_send_c's Issues
Segmentation fault occurs when the size of tensor array achieves 10000000.
Environment
- Branch: use-cpython
- File: send.c
Problem
When modifing the size of gradient array to test the send_gardients(), some errors happened:
- buffer_size == 1000000, ERROR: Failed to call sendto(): No buffer space available.
- buffer_size == 10000000, Segmentation Fault.
- When send_gradients() is called multiple times in the same process, ERROR: Failed to call sendto(): No buffer space available.
Solution
- it seems that we can increase the value of send_buff_size.
- No solutions. Avoid this error by pass small gradient.
- No solutions. Try to avoid by testing the multi-processes send.
socket移出线程时间比较
线程中创建socekt
Test data 0.003725290298461914 GB
arrat len: 1000000,thread num: 20, dstip: 172.16.210.33, worker id: 0, agg idx: 0, tensor idx: 0
Time consumed: 0.071722 s
Time consumed: 0.075023 s
Time consumed: 0.077465 s
Time consumed: 0.078640 s
Time consumed: 0.083199 s
Time consumed: 0.077696 s
Time consumed: 0.065982 s
Time consumed: 0.058636 s
Time consumed: 0.059009 s
Time consumed: 0.060869 s
Time consumed: 0.032201 s
Time consumed: 0.051958 s
Time consumed: 0.056413 s
Time consumed: 0.060162 s
Time consumed: 0.045847 s
Time consumed: 0.060843 s
Time consumed: 0.062563 s
Time consumed: 0.065439 s
Time consumed: 0.045755 s
Time consumed: 0.053917 s
Total cost: 0.008537054061889648 sec; Throuthput 0.43636719077275393 GBps
线程外创建socket
Test data 0.003725290298461914 GB
arrat len: 1000000,thread num: 20, dstip: 172.16.210.33, worker id: 0, agg idx: 0, tensor idx: 0
Time consumed: 0.078823 s
Time consumed: 0.077935 s
Time consumed: 0.205034 s
Time consumed: 0.284954 s
Time consumed: 0.561072 s
Time consumed: 0.634509 s
Time consumed: 0.536135 s
Time consumed: 0.813445 s
Time consumed: 0.714760 s
Time consumed: 0.938576 s
Time consumed: 1.020436 s
Time consumed: 1.098183 s
Time consumed: 0.984910 s
Time consumed: 1.081839 s
Time consumed: 1.299061 s
Time consumed: 1.393215 s
Time consumed: 1.280993 s
Time consumed: 1.553009 s
Time consumed: 1.604607 s
Time consumed: 1.613474 s
Total cost: 0.037398576736450195 sec; Throuthput 0.09961048316662523 GBps
编译send.cc时error: no matching function for call to ‘std::thread::thread(<unresolved overloaded function type>, int&, int&, __u32*&, int&, int&, int&, int&)’
thread_pool[i] = new std::thread(_thread_send_gradients, i, thread_num, tensor_array, array_len, str_dst_ip, worker_id, aggregator_index, tensor_index);行增加 str_dst_ip, worker_id, aggregator_index, tensor_index几个参数报错
sdn@server08:~/bc/test_send$ ./compile.sh
send.cc: In function ‘void multiple_threads_send_gradient(__u32*, int, int, unsigned int, int, int, int)’:
send.cc:88:171: error: no matching function for call to ‘std::thread::thread(<unresolved overloaded function type>, int&, int&, __u32*&, int&, int&, int&, int&)’
thread_pool[i] = new std::thread(_thread_send_gradients, i, thread_num, tensor_array, array_len,/* str_dst_ip.c_str(),*/ worker_id, aggregator_index, tensor_index);
^
In file included from send.h:10:0,
from send.cc:1:
/usr/include/c++/7/thread:118:7: note: candidate: template<class _Callable, class ... _Args> std::thread::thread(_Callable&&, _Args&& ...)
thread(_Callable&& __f, _Args&&... __args)
^~~~~~
/usr/include/c++/7/thread:118:7: note: template argument deduction/substitution failed:
send.cc:88:171: note: couldn't deduce template parameter ‘_Callable’
thread_pool[i] = new std::thread(_thread_send_gradients, i, thread_num, tensor_array, array_len,/* str_dst_ip.c_str(),*/ worker_id, aggregator_index, tensor_index);
^
In file included from send.h:10:0,
from send.cc:1:
/usr/include/c++/7/thread:113:5: note: candidate: std::thread::thread(std::thread&&)
thread(thread&& __t) noexcept
^~~~~~
/usr/include/c++/7/thread:113:5: note: candidate expects 1 argument, 8 provided
/usr/include/c++/7/thread:106:5: note: candidate: std::thread::thread()
thread() noexcept = default;
^~~~~~
/usr/include/c++/7/thread:106:5: note: candidate expects 0 arguments, 8 provided
Pool performance in multi-processes
Background
Communication with raw sockets from server 11 (xxx.xxx.210.xxx) to server 12.
Results
| No.of processes | Throughput (Gbps) |
|---|---|
| 1 | 1.6 |
| 4 | 1.76 |
| 10 | 2.24 |
| 20 | 2.4 |
Problem
Implement multi-process via package multiprocessing.Pool. Test with htop, proving multiple cores are working.
但是多进程并没有实现吞吐率的线性提升。我们的运行结果显示，使用10个进程发送300MB左右数据，每个进程大概花0.1s左右，但是总时间是1s左右。
我们希望总时间和单进程时间差不多（因为理论上来说是并行发送的）。
可能的原因
- 目前直接通过multiprocessing模块的Pool实现并行化的，比较底层。可以试一下用concurrent.futures模块实现。
- 进程间通信花费时间太多，numpy数组实现底层是可以支持共享内存的，现有方案可能没有用到shared memory，可以考虑debug一下。
Error: Failed to call sendto()
Background
当调用线程数过多或者发送数据量过大时，会报sendto()出错
Solutions
调用函数前在终端输入sudo iptables -t raw -A OUTPUT -p udp -j NOTRACK
Segmentation fault occurs when the size of tensor array achieves 10000000.
Environment
- Branch: use-cpython
- File: send.c
Problem
When modifing the size of gradient array to test the send_gardients(), some errors happened:
- buffer_size == 1000000, ERROR: Failed to call sendto(): No buffer space available.
- buffer_size == 10000000, Segmentation Fault.
- When send_gradients() is called multiple times in the same process, ERROR: Failed to call sendto(): No buffer space available.
Solution
- it seems that we can increase the value of send_buff_size.
- No solutions. Avoid this error by pass small gradient.
- No solutions. Try to avoid by testing the multi-processes send.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.