[Battlemesh] tests for WBMv3

[L. Aaron Kaplan]

On May 26, 2010, at 4:04 PM, Luca Tavanti wrote:

> Hi Aaron,
> 
> At 26/05/2010 11.47, you wrote:
>> Hm, maybe I was not clear (I hinted it at in the parentheses):
>> by designing the test network so that the other links are worse (of course you need to confirm this by measurement).
>> Then as a result you have a network where your desinged links are optimum (optimum according to some criterium of course).
>> Then you do static routes there.
>> 
>> So - as a result you have a benchmark against which you can compare different protocols in a *repeatable* way.
>> The problem that I see with many of the WBM tests: it is nice to meet and to discover bugs while testing.
>> But it is really hard to thoroughly convince oneself or somebody else that this measurement result is really true and repeatable.
>> (changes in SNR, interference, ... you name it).
>> 
>> Therefore I suggested to try to aim at _some kind of_ reference against which we should compare any protocol.
>> 
>> See my point?
> 
> Ok, perhaps I got your point.
> So... let's assume we set up a sort of reference topology using your method. You design good and bad links, set static routes and then... how do you use them? 
> Run the mesh protocols and see if they choose exactly the same links?
> Hmm... what I expect is that, if you have only very good and very bad links, any "decent" protocol will use only the good links, and all protocols will behave (roughly) the same. At most you can detect if a proto selects the bad links... 

Exactly! This would be the test.
My gut feeling says that Batman will "win" on this test (after some initial time).
However when you quickly change the topology (how? ;-) ) then my reasoning would say that OLSR and / or Babel would "win" it.

> What I mean is that perhaps you set up a network which is too much "black/white", with no grays, so you don't give the protocols much degrees of freedom in chosing the routes.

Yes, acknowledged. 

> So, ok you get the repeatability, but miss much other info. 
> 
But my experience from the first Battlemeshes was that a) it took a very long time to set things up (and without Nico's cools scripts we would not have enough routers finished on the final day even) b) once we set up the network we were just hunting bugs and tried to understand ("wtf") is actually happening. 
This was pretty hard and distressing. Maybe this also explains a bit why I am so much insisting on reproducibility of test results ;-)


Now, if you have a known setup that you can revert to (f.ex.: static routing) then you can compare what went wrong at least.
And in once case the bug might be some software bug or in another case it might be that you find a weakness in a protocol or metric. 
I expect this also to be a very valuable testcase for the new metric in Babel (the one where you try to avoid using the same channel)


Concerning mobile nodes:
hmmm... you might want to use one or the other laptop walking around in the mesh and measure how quick the handover works (as seen from the internet uplink  or as seen from every other node).

In any case, this is my opinion and what I learned from the first WBMs.

Best,
a.