[Battlemesh] tests for WBMv3

[Luca Tavanti]

At 26/05/2010 16.19, L. Aaron Kaplan wrote:

>However when you quickly change the topology (how? ;-) ) then my reasoning would say that OLSR and / or Babel would "win" it.

The way I usually change the topology is to suddenly switch off one of the nodes that is relaying the traffic ;-).



>> What I mean is that perhaps you set up a network which is too much "black/white", with no grays, so you don't give the protocols much degrees of freedom in chosing the routes.
>
>Yes, acknowledged. 
>
>> So, ok you get the repeatability, but miss much other info. 
>> 
>But my experience from the first Battlemeshes was that a) it took a very long time to set things up (and without Nico's cools scripts we would not have enough routers finished on the final day even) b) once we set up the network we were just hunting bugs and tried to understand ("wtf") is actually happening. 
>This was pretty hard and distressing. Maybe this also explains a bit why I am so much insisting on reproducibility of test results ;-)
>
>Now, if you have a known setup that you can revert to (f.ex.: static routing) then you can compare what went wrong at least.
>And in once case the bug might be some software bug or in another case it might be that you find a weakness in a protocol or metric. 
>I expect this also to be a very valuable testcase for the new metric in Babel (the one where you try to avoid using the same channel)


I'm a bit confused...
What is the goal of the tests? Debugging? Comparing the protocols? Both?

If it's debugging, then your procedure might be ok (and, since you've been to previou WBMs and I have not, I trust you fully ;-) ).

But if it's getting some comparison and performance results, then I think you're going to loose much more than you gain. 
Btw, you can also get some confidence in the results by repeating the tests a few times (not necessarily tens or 100s). For example, if you have 3 protocols, you can run them alternatively: A, B, C, A, B, C, and so on... If the results of the various runs of each protocols are comparable, then you can assume that the environment has not changed much, and that the other protocols worked under (more or less) the same conditions.

I don't know how much it takes to switch from a protocol to the other, but I have devised some tests that shouldn't take much to execute (again in my first post).

Cheers,
Luca