[Battlemesh] The Flent testing tool

[Toke Høiland-Jørgensen]

Nemesis <nemesis at ninux.org> writes:

> The first batch of graphs is:
>
> rrul-box.pdf 12-Aug-2015 12:12 111773
> rrul-box.svg 12-Aug-2015 12:12 121160
> rrul-cdf.pdf 12-Aug-2015 12:12 125117
> rrul-cdf.svg 12-Aug-2015 12:12 120621
>
> The second batch:
>
> rrul_be-box.pdf 12-Aug-2015 12:12 111472
> rrul_be-box.svg 12-Aug-2015 12:12 120830
> rrul_be-cdf.pdf 12-Aug-2015 12:12 123218
> rrul_be-cdf.svg 12-Aug-2015 12:12 114186

Yup, those are two different graphs from each of the two RRUL variant
tests.

> Few questions:
>
> * I guess there are no other graphs, is this correct? 

I haven't generated any other graphs, but you can do so yourself from
the Flent data files.

> * I guess the two batches correspond to two tests, is this correct?

Yes, each of those graphs is from a single test, repeated for all the
routing protocols.

> * each test offers two visualization stylet, correct?

No, they offer several more. But I have only generated those two types
of graphs for each of the tests.

> * what was done exactly in each tests? 

Quoting my email with the original test data:

> The RRUL test consists of running four simultaneous bulk TCP streams in
> both the upstream and downstream directions (so eight streams total),
> while simultaneously running UDP and ICMP latency measurements. The test
> runs for 60 seconds, to make sure the network is fully saturated. There
> are two variants of the test: A best-effort only (called RRUL_BE) in
> which no diffserv marking is employed. And the straight RRUL test, which
> marks each of the four TCP streams with different diffserv markings
> matching the mapping into hardware queues in the WiFi stack.

See the rest of that mail for the description of the other tests.

> * how to interpret the box graph?

The data points in the box graph are the individual samples over the
duration of the test. The download and upload speeds are totals for all
four data streams, while the latency is an average of the different
latency measurement flows. The boxes show the lower to upper quartile
range, the red line is the median value, the whiskers are the 1.5
inter-quartile range, and the dots are outliers.

Note that since the throughput data points are correlated (it's TCP),
be wary of drawing statistical conclusions from the box plots. For that,
several independent repetitions of the test is needed.

> * I think the CDF plot is easier to grasp and explain to viewers 

Definitely agree. Only drawback is that it doesn't show the throughput;
you can get very good latency by cutting throughput to a trickle. Flent
also does an ellipsis type plot that is nice for showing (for an
example, see figure 1 of
https://flent.org/flent-the-flexible-network-tester.pdf) -- however,
it doesn't really work with data as noisy as what we have here, so I
haven't included that.

> * I really like having all the results on a single graph, it makes it really
>   easy to glance and compare - @Mathieu: would it be possible to do this for
>   some of the other graphs too (eg: CDF plots only)? 

Yup, it's really the only good way to compare CDFs, IMO. :)

> Thank you Toke, excellent work! :-)

You're welcome. Just glad that others find it useful :)

-Toke