[Battlemesh] tests for WBMv3

Luca Tavanti luca.tavanti at iet.unipi.it
Mon May 31 16:53:07 CEST 2010


Hi all,

I have tried to summarise all comments and to merge them with the first "document" I wrote. I have also included the test suggested by Benjamin.
Hope this is useful :-)

Cheers,
Luca


(I also made a PDF with nicer figures, but it was too big for the list. If someone is interested, I'll mail it directly to him/her, just let me know)


===


Tests for WBMv3
 
Network topology
 
a) Linear: nodes are placed in a line; each node can communicate also with its two/three/etc hop neighbours, not just its one hop neighbour.

O---O---O---O---O---O
 
b) Regular (single/multi-channel): place nodes in order to obtain something like the topology in the figure. If nodes with two interfaces are available, set the non-overlapping channels (say 1,2,3) as in the figure (in practice: 1=ch.1, 2=ch.11/13, 3=ch.6/7).
              1
           O-----O
        1/|\ 3 /|\1
        / | \ / | \
        S  |  X  |  D
        \ | / \ | / 
        2\|/ 3 \|/2 
           O-----O
              2

c) Large: use all deployed nodes, without pre-defined links.
 
d) Simple dual channel: place nodes and set channels as in the figure( | = ch1, + = ch13, - = ethernet).
        A
        |
        B-C
        | +
        D-E

 
Serial / Parallel (S/P) execution:
Some tests can be run in either “parallel mode”, i.e. two protocols running at the same time over the networks using two different subnets, or in “serial” way (run proto A first, then B, then C, etc.).
The parallel method can be used to test the protocols under exactly the same radio environment (to avoid the radio fluctuations), but it cannot be employed with high traffic loads (the protocols are going to disturb each other).
Conversely, when the goal is to find the maximum network capacity, the serial approach must be used (note that over long runs and/or over several tests, the fluctuations on the environment might somewhat “cancel out” on average if the environment is rather stable - should we measure that?). 
 
Reference networks: 
Topologies (b) + multiple radios, and (d) can be used as reference scenarios (i.e. to set static routes as suggested by Aaron).
 
Tests
 
1) Efficiency
Purpose: Measure the efficiency of the protocol and the metric in a simple configuration. When multiple links with different quality and data rates are available on the same route, which ones are chosen by the protocol?
How: Set up the linear topology (a). Run the network with a single data flow (either UDP or TCP, UDP is preferable) from one end node to the other; increase the data rate and measure the received throughput; if possible, record the chosen route (i.e. which nodes are taking active part in relaying). The most efficient protocol is the one that gets the highest throuput.
Can also be repeated with topology (b) with multiple channels to verify whether the protocol chooses the “best” routes, i.e. those with non overapping channels.
S/P: S-mode should be used to find the most efficient protocol. P-mode can be used only to detect the route with unloaded network.
 
2) Route Stability
Purpose: Verify the stability of the routes (frequent route changes may lead to performance degradation, especially if not due to a change in the radio environment).
How: Set up the network with topology (b) – single channel. Run the protocol with a single data flow from S to D (either UDP or TCP or ICMP, ICMP/UDP is preferable), then register the chosen route(s) and the actual metric (whenever and wherever possible); register statistics data (e.g throughput, end-to-end delay, losses) as well; repeat with multi-channel. 
With some difficulty, it can be repeated over topo (c).
S/P: Both P and S modes can be used as long as the network is unloaded.
 
3) Recovery
Purpose: Verify the capability to recover after a sudden node failure (node failures may occur for several reasons, e.g. hardware failure, software bugs, device turned off by user; in such a case the network should find a backup route as quickly as possible).
How: Set-up the network with topo (b) or (c). Run the protocol with a single data flow from S to D (UDP, to avoid TCP timers) and wait for a stable working condition; then switch off one of the intermadiate nodes and register the service interruption time (e.g. by monitoring the received throughput, packet losses, etc). 
Repeat with different source/dest pairs and/or intermediate nodes.
Repeat with more load on the network (slightly less than the maximum).
Optionally, turn the failed node on again and measure whether the traffic is rerouted through it and how long it takes (note that switching back on the old route is not a “bonus” feature; the new route may also be a well-performing one). 
S/P: Both P and S modes can be used as long as the network is unloaded. Yet, to stress the protocols at the maximum, the network should be heavily loaded, hence the S mode is preferable.
 
4) Capacity and load balancing
Purpose: Measure the maximum capacity of the network. This is often (but not necessarily) associated to a good load balancing strategy (due to inter- and intra-flow interferences, disjoint routes can lead to higher overall capacity).
How: Set-up the network with topology (c) (perhaps (b) as well?). Run the protocol with as many flows as possible (either UDP or TCP). Start with one flow, wait for a stable situation, then start a second flow, wait for a stable situation, start a third flow, and so on. Measure the received throughput at the destination nodes. Stop when the throughput no longer grows. Select source and destination nodes at the edge of the network first. 
To assess load balancing, measure the traffic over all links and compute a balancing metric (e.g. standard deviation of throughput).
S/P: S-mode should be used.
 
5) Quality of Service (QoS)
Purpose: Measure the quality perceived by a user when using real-time services (e.g. voice over IP); measure the capacity in terms of supported flows with sufficient quality.  
How: Set-up the network with topology (b) and/or (c). Run the protocol with a single UDP flow (e.g. that mimics a voice codec). Then measure the typical QoS parameters: paket loss, end-to-end delay, jitter (at the receiving node). Possibly calculate a unique performance metric, such as MOS (e.g. through the E-model).
To assess the capacity, follow the same procedure of (4), i.e. gradually increase the number of flows until the worst metric (MOS) value among of all flows drops below a given threshold. That is the capacity of the network.
S/P: S-mode should be used.
 
6) Simple dual channel (suggested by Benjamin Henrion)
Purpose: Test whether the protocol picks the best route in a pre-designed topology.  
How: Set-up the network with topology (d). Run the protocol with a single flow from A to E. Verify that the chosen route is             A-B-C-D-E. This is the “best” route because it has no interference between wireless channels.
S/P: Either P or S modes can be used (given that the goal is only to register the chosen route).
 
 



More information about the Battlemesh mailing list