[Battlemesh] battlemesh v6 test data postprocessing

[Nicolás Echániz]

El 17/05/14 12:37, Juliusz Chroboczek escribió:
>>> The pair of nodes was chosen randomly at each single test.  And, we
>>> didn't collect enough measurements to obtain the same set of nodes
>>> for all the protocols.
> 
>> Oh, so nothing to worry about.  This data is clearly invalid.
> 
> Clauz,
> 
> Sorry if that was strongly worded, but please realise that this is not
> a useful test setup.
> 
> The tests that we used to do at Battlemesh in the olden times served
> two purposes:
> 
>   (1) find out how the different routing protocols are doing in
>       real-world scenarios;
>   (2) uncover issues and help the protocol developers work them out.
> 
> Unfortunately, the last time I got any useful data from Battlemesh was
> in Catalogna.
> 
> Your test fails to achieve either of the points above.  Actually, they
> are worse than useless, since they might be misleading and cause us to
> spend time chasing nonexistent issues.
> 
>   (1) random tests on such a small sample do not provide any statistically
>       meaningful results;
>   (2) without a clear description of the scenarios, there is no way to
>       work out what is going on.
> 
> I'd really like to encourage everyone present at WBM this year to work
> on providing useful results, both to users and to protocol developers.

As far as I remember not all tests done during WBMv6 were done with this
setup of changing random nodes at each run.
In particular, the longest path tests used the same endpoints for every
protocol and data collected should be useful.
Also, the moving node/convergence test was done with all protocols in
parallel.

By the way, your wording still comes strong.


Cheers,
Nico