 |
YARP
Yet Another Robot Platform
|
|
|
YARP
Yet Another Robot Platform
|
|
Along with TCP, YARP supports UDP and multicast communication on a local network.
These are unreliable network protocols, but can be quite efficient. We discuss here steps that are useful for making your network friendly to UDP and multicast. But if you find yourself spending a lot of time on this, you may need to reconsider whether you have made the right choice of protocol.
NOTE: if you are just getting started with YARP, you don't need to look at any of this yet! It only matters when you are working with a cluster of computers using unreliable protocols, or run into bandwidth problems. But it is important to remember that this page is here for when you need it.
If you are doing any non-trivial networking, you need to learn how to test your network and debug throughput issues. A key resource in this battle is the "iperf" program.
Please make sure you have iperf available on all your machines before going any further.
TCP is the simplest protocol to work with, so let's start with it. Let's check the throughput between two of your machines. Pick two machines, and choose one (arbitrarily) to be the iperf server. Determine its IP address of domain name – let's say it is 10.0.0.26. Run the following commands:
[On server, e.g. 10.0.0.26] iperf -s [On client, e.g. 10.0.0.27] iperf -c 10.0.0.26
This will run for a while and then describe your TCP throughput, for example:
------------------------------------------------------------ Client connecting to 10.0.0.26, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.25 port 34160 connected with 10.0.0.26 port 5001 [ 3] 0.0-10.0 sec 1.08 GBytes 924 Mbits/sec ...
UDP is an unreliable protocol, and there are lots of possible pitfalls in using it. YARP will try to send messages broken into rather large UDP packets (around 60K), and your network may or may not actually pass those packets through.
To test UDP behavior on your network with iperf, add the "-u" flag to the client and server. To the server, we can add "-i 1" to get reports at regular intervals. To the client, we can specify target bandwidth, e.g. "-b 50m" for a target of 50 MBits/sec.
[On server, 10.0.0.26] iperf -s -u -i 1 [On client, 10.0.0.27] iperf -c 10.0.0.26 -b 50m
To test a gigabit network:
[On server, 10.0.0.26] iperf -s -u -i 1 [On client, 10.0.0.27] iperf -c 10.0.0.26 -b 1000m
On a healthy gigabit network you should get a bandwidth of ~950 Mbits/sec.
Example result on the client side:
------------------------------------------------------------ Client connecting to 10.0.0.26, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 122 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.221 port 52990 connected with 10.0.0.222 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 6.1 sec 700 MBytes 956 Mbits/sec [ 3] Sent 499273 datagrams [ 3] Server Report: [ 3] 0.0- 6.1 sec 699 MBytes 955 Mbits/sec 0.014 ms 575/499272 (0.12%) [ 3] 0.0- 6.1 sec 1 datagrams received out-of-order
Result on the server side:
------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 122 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.26 port 5001 connected with 10.0.0.27 port 52990 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 114 MBytes 954 Mbits/sec 0.012 ms 88/81245 (0.11%) [ 3] 1.0- 2.0 sec 114 MBytes 955 Mbits/sec 0.011 ms 92/81263 (0.11%) [ 3] 2.0- 3.0 sec 114 MBytes 955 Mbits/sec 0.014 ms 90/81287 (0.11%) [ 3] 3.0- 4.0 sec 114 MBytes 955 Mbits/sec 0.012 ms 98/81273 (0.12%) [ 3] 4.0- 5.0 sec 114 MBytes 954 Mbits/sec 0.012 ms 99/81263 (0.12%) [ 3] 5.0- 6.0 sec 114 MBytes 955 Mbits/sec 0.012 ms 95/81288 (0.12%) [ 3] 0.0- 6.1 sec 699 MBytes 955 Mbits/sec 0.014 ms 575/499272 (0.12%) [ 3] 0.0- 6.1 sec 1 datagrams received out-of-order
A key test for YARP is that you can send big datagrams okay. This is necessary for efficient transmission of large images using UDP or MCAST. To check 63KB datagrams, do:
[On server, 10.0.0.26] iperf -s -u -l 63K -w 200K -i 1 [On client, 10.0.0.27] iperf -c 10.0.0.26 -u -l 63K -w 200K -b 1000m
Make sure you try this in both directions! For example, we've seen an asymmetric problems going from a Linux machine to a Windows machine on a gigabit network with a particular switch. If you see problems, reduce the "-l" size until things work just to understand your system (but if you want to send large images you'll need to tweak your system until large packets work).
It could also be worth checking several machines since different network cards can have different performance.
For testing multicast, make sure you have the most recent version of iperf, and try this:
iperf -s -u -B 224.0.55.55 -i 1 -T 32 -l 60k iperf -c 224.0.55.55 -u -T 32 -t 100 -i 1 -l 60k -b 1000m
Note the IP addresses are multicast-style, they don't correspond to individual machines.
On the client ("-c") side, you should see something like:
------------------------------------------------------------ Client connecting to 224.0.55.55, UDP port 5001 Sending 61440 byte datagrams Setting multicast TTL to 32 UDP buffer size: 122 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.221 port 60992 connected with 224.0.55.55 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 114 MBytes 958 Mbits/sec [ 3] 1.0- 2.0 sec 114 MBytes 956 Mbits/sec [ 3] 2.0- 3.0 sec 114 MBytes 956 Mbits/sec [ 3] 3.0- 4.0 sec 114 MBytes 956 Mbits/sec
Server side:
------------------------------------------------------------ Server listening on UDP port 5001 Binding to local address 224.0.55.55 Joining multicast group 224.0.55.55 Receiving 61440 byte datagrams UDP buffer size: 122 KByte (default) ------------------------------------------------------------ [ 3] local 224.0.55.55 port 5001 connected with 10.0.0.221 port 60992 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 111 MBytes 931 Mbits/sec 0.380 ms 50/ 1945 (2.6%) [ 3] 1.0- 2.0 sec 111 MBytes 934 Mbits/sec 0.442 ms 46/ 1947 (2.4%) [ 3] 2.0- 3.0 sec 112 MBytes 941 Mbits/sec 0.280 ms 31/ 1946 (1.6%) [ 3] 3.0- 4.0 sec 112 MBytes 935 Mbits/sec 0.362 ms 43/ 1946 (2.2%) [ 3] 4.0- 5.0 sec 110 MBytes 920 Mbits/sec 0.459 ms 75/ 1947 (3.9%) [ 3] 5.0- 6.0 sec 111 MBytes 934 Mbits/sec 0.418 ms 45/ 1946 (2.3%) [ 3] 6.0- 7.0 sec 111 MBytes 933 Mbits/sec 0.380 ms 48/ 1947 (2.5%) [ 3] 7.0- 8.0 sec 111 MBytes 933 Mbits/sec 0.365 ms 49/ 1947 (2.5%) [ 3] 8.0- 9.0 sec 112 MBytes 936 Mbits/sec 0.319 ms 41/ 1946 (2.1%)
If MCAST/UDP packets drop when using YARP, YARP may suggest this:
The UDP/MCAST system buffer limit on your system is low. You may get packet loss under heavy conditions. To change the buffer limit on linux: sysctl -w net.core.rmem_max=8388608 (Might be something like: sudo /sbin/sysctl -w net.core.rmem_max=8388608) To change the limit use: sysctl for Linux/FreeBSD, ndd for Solaris, no for AIX
This controls the maximum allowed size for a RECEIVER buffer, so that a number of packets adding up to a large amount of data can be received and stored while your program is doing something else. YARP will try to get a large buffer, and your operating system may not comply unless you do the above. This configuration step is not necessary on MS-Windows.
There are a lot more suggestions for Linux here:
If you are confident of your network, but are having trouble with making a YARP connection of some kind, read about that kind of connection in Configuring YARP Connections and see if there's a tweak that can be made. There are some parameters you can control for the udp carrier and mcast (multicast) carrier.
In case you need to fix something, the relevant source code in YARP is:
src/libYARP_os/src/yarp/os/impl/DgramTwoWayStream.hsrc/libYARP_os/src/yarp/os/impl/DgramTwoWayStream.cppWe appreciate patches to these files. We only have a few networks to test on, and this topic can be quite network-dependent and subtle.
1.8.20