[Battlemesh] tests for WBMv3

[Luca Tavanti]

Hi everyone.

 From the talks I had with some of you, I understood that currently there are no well-defined scenarios for testing the protocols.
Therefore I have prepared a brief list of tests it would be nice to perform, as I believe that having a common set of tests for comparing the various protocols would be very valuable.
Let me know what you think (e.g. if they are feasible), or any comments you have.

Luca

===

Tests for WBMv3
 
* Network topology *

a) Linear: nodes are placed on a line; each node can communicate also with its two/three hop neighbours, not just its one hop neighbour.
O---O---O---O---O---O
 
b) Regular (single/multi-channel): place nodes in order to obtain something like the topology in the figure. If nodes with two interfaces are available, set the channels (say 1,2,3) as in the figure.
      1
   O-----O
 1/|\ 3 /|\1
 / | \ / | \
O  |  X  |  O
 \ | / \ | / 
 2\|/ 3 \|/2 
   O-----O
      2
 
c) Large: use all deployed nodes, without pre-defined links.
 
* Tests * 

1) Efficiency
-Purpose: Measure the efficiency of the protocol and the metric in a simple configuration. When multiple links with different quality and data rates are available on the same route, which ones are chosen by the protocol?
-How: Set up the linear topology (a). Run the network with a single data flow (either UDP or TCP, UDP is preferable) from one end node to the other; measure the received throughput; if possible, record the chosen route (i.e. which nodes are taking active part in relaying).
 
2) Route Stability
-Purpose: Verify the stability of the routes (frequent route changes may lead to performance degradation, especially if not due to a change in the radio environment).
-How: Set up the network with topology (b). Run the protocol with a single data flow (either UDP or TCP, UDP is preferable), then register the chosen route(s) and the actual metric (whenever and wherever possible); register statistics data (e.g throughput, end-to-end delay, losses) as well; repeat with different source/dest pairs and with single/multi-channel.
 
3) Recovery
-Purpose: Verify the capability to recover after a sudden node failure (node failures may occur for several reasons, e.g. hardware failure, software bugs, device turned off by user; in such a case the network should find a backup route as quickly as possible).
-How: Set-up the network with topo (b). Run the protocol with a single data flow (UDP, to avoid TCP timers) and wait for a stable working condition; then switch off one of the intermadiate nodes and register the service interruption time (e.g. by monitoring the received throughput, packet losses, etc). 
Repeat with different source/dest pairs and/or intermediate nodes.
Repeat with more load on the network (slightly less than the maximum).
Optionally, turn the failed node on again and measure whether the traffic is rerouted through it and how long it takes (note that switching back on the old route is not a “bonus” feature; the new route may also be a well-performing one). 
 
4) Capacity and load balancing
-Purpose: Measure the maximum capacity of the network. This is often (but not necessarily) associated to a good load balancing strategy (due to inter- and intra-flow interferences, disjoint routes can lead to higher overall capacity).
-How: Set-up the network with topology (b) ad/or (c). Run the protocol with as many flows as possible (either UDP or TCP). Start with one flow, wait for a stable situation, then start a second flow, wait for a stable situation, start a third flow, and so on. Measure the received throughput at the destination nodes. Stop when the throughput no longer grows. Select source and destination nodes at the edge of the network first. 
To assess load balancing, measure the traffic over all links and compute a balancing metric (e.g. standard deviation of throughput).
 
5) Quality of Service (QoS)
-Purpose: Measure the quality perceived by a user when using real-time services (e.g. voice over IP); measure the capacity in terms of supported flows with sufficient quality.  
-How: Set-up the network with topology (b) ad/or (c). Run the protocol with a single UDP flow (e.g. that mimics a voice codec). Then measure the typical QoS parameters: paket loss, end-to-end delay, jitter (at the receiving node). Possibly calculate a unique performance metric, such as MOS (e.g. through the E-model).
To assess the capacity, follow the same procedure of (4), i.e. gradually increase the number of flows until the worst metric (MOS) value among of all flows drops below a given threshold. That is the capacity of the network.