[Battlemesh] WBMv8 -- a personal perspective

[nemesis]

 On Sun, 09 Aug 2015 22:01:33 +0200, Juliusz Chroboczek 
 <jch at pps.univ-paris-diderot.fr> wrote:
>
 [...]
>
> The venue and organisation
> ==========================
>
> Wlan-SI has a large production network (over 400 nodes, if memory 
> serves),
> and covers much of Slovenia and some of the bordering areas (if you
> haven't seen their website yet, do so now -- they make a serious 
> effort at
> translating everything into English).  Because of that, they have a 
> large
> number of technically competent people, and discussions here are
> particularly interesting.  (There are too many competent people to 
> list
> here, so I'll only mention my good friends Mitar and Jernej, who gave 
> me
> a really hard time.)
>
> Maribor is the second largest city in Slovenia, in a large valley 
> between
> the mountains.  One side is covered in vineyards, the other side has 
> ski
> slopes.  WBMv8 was held in a large, convenient building at the foot 
> of the
> ski slopes, some 7km away from the city center.  We were lodged in 
> two
> hotels and one camping a kilometre or so downhill from the vanue.  
> (That
> means we needed to go uphill in the heat -- a strong incentive to get 
> up
> early.)
>
> Hacking and discussions happened in a large hall with good lighting 
> (which
> unfortunately made it uncomfortably hot at times, you cannot have
> everything).  The talks were in a luxurious, large lecture hall, and 
> there
> was a small classroom with a blackboard for more private discussions.
> Toilets, kitchen, fridge, large lawn for naps and informal 
> discussions,
> what more could you wish for?
>
> (A shower?)
>
> I was impressed by the food -- a late lunch was served daily, solid 
> and
> good countryside food.  I was particularly impressed by the mushroom 
> soup
> (juha), made with a variety of real mushrooms.  The most exotic thing 
> we
> had was something made of cooked flour that had the consistence of
> a potato gratin but completely different taste.  (What's it called?  
> Ras
> told me, but I forgot.)  We weren't served burek, surprisingly, I 
> guess
> it's difficult to organise for that number of people.
>
> I had never been to Slovenia before, and I was pleasantly surprised.  
> The
> Slovenians are friendly and helpful, everything works (trains, 
> busses,
> hotels, etc.), and the cities are suspiciously clean (although you 
> can see
> the effect of the economic crisis when you go away from the main 
> streets).
> If you come here for holidays, I would tend to recommend it, but be 
> aware
> that it is not exactly cheap -- cheaper than France, certainly, but 
> way
> more expensive than Poland, let alone the Czech republic.
>
> Musti remained reasonably calm and managed to keep smiling throughout 
> the
> event.

 I agree 100%

> The technical achievements and otherwise
> ========================================
>
> We quickly split into two teams, one that was preparing configuration
> files for the various protocols (Batman-Advanced, grr) and one that 
> was
> setting up the mysterious WiBed firmware.  The configuration team was
> flashing routers, obviously, since you cannot verify a config file 
> without
> working routers.
>
> After a couple of days, the configuration team was ready, so we 
> decided to
> flash 10 routers.  Amadeus, Federico, Goran and Tomislava were busily
> flashing, getting it wrong, reflashing, tweaking something, 
> reflashing
> (have you recovered yet, Fede?), while Henning and Thijs were 
> scripting
> the tests and Matthieu was scripting the data analysis, and Toke was
> preparing his favourite analysis tool (flent), while Amadeus was 
> trying to
> bring order to this chaos.  (Yes, Amadeus appears twice in this list,
> that's deliberate.)  It was a lot of fun.

 Thanks for the credit, don't forget Riccardo Bloise and Dario Stelitano 
 who helped me quite a lot during the whole process.

> We were planning two topologies (full mesh and ring) and two mobility
> tests (router crash and mobile node).  The good news is that we 
> managed to
> run both analysis scripts.  The bad news is that we only did one 
> topology
> (full mesh), and none of the mobility tests.
>
> The horrible news is that the official firmware team never finished.  
> This
> is not the first time that this happens, and we must understand why 
> this
> is and ensure it never happens again.  Should WiBed be abandoned in 
> favour
> of the scripts written by Henning and Thijs?

 Manos was telling me how he thinks we should have united force on one 
 testbed.

 I'm sorry if someone felt we were splitting forces,
 let me explain that me and the other people contributing to the second 
 (manual) testbed
 were quite useless with helping out the wibed team, infact my friend 
 and colleague Riccardo
 offered help with wibed firmware from the first day, but he was told it 
 was not necessary
 and we'd get the firmware shortly afterwards.

 We started to setup a manual test as a plan B, in the beginning I was 
 quite sure it would just be a temporary replacement
 to start doing something, while the automated testbed would be ready. 
 Now I'm glad we did that, because if we didn't
 we would have been of much help with wibed, being a system we didn't 
 know anything about (we also had problems getting to maribor on the 
 first day, so we missed the wibed presentation) - so we would not have 
 had any result.

 Regarding the question:

 "Should WiBed be abandoned in favour of the scripts written by Henning 
 and Thijs?"

 according to the people I talked to, we'd prefer to have a system like 
 wibed to avoid the repetitive manual stuff,
 but we also need a system that doesn't get in our way.
 I think the test team felt more comfortable in running scripts via 
 shell.
 If we want to use wibed in the next edition, there should be a quick 
 start guide to allow volunteers to become comfortable with it and be 
 productive.
 
> Results
> =======
>
> As pointed out above, we only have results for the full mesh case.  
> I've
> looked carefully at the results, and the only conclusion I can draw 
> is
> that OLSRv2 works very well in this particular topology.  This is 
> great
> news, since we've been waiting for OLSRv2 for a long time, and 
> Henning has
> beein doing some great work both in the standardisation forums and on 
> the
> implementation front.
>
> I'll be back home on Wednesday, we'll look at the results real hard
> together with Matthieu, Toke and Dave (anyone else?), and try to put
> something comprehensible on the web.

 Me and Riccardo met Mathiew Boutier (in charge of generating graphs 
 from the raw data) and we checked out the rescaled graphs.

 The results are quite interesting, we can see that under heavy load the 
 protocols behave differently.

 Let's wait for Mathiew to send the updated graphs.
 
> Suggested improvements
> ======================
>
> No event is so perfect that it cannot be improved upon, and WBMv8 is 
> no
> exception.
>
> I thought many talks were too long -- everyone got a one hour slot, 
> while
> many of the talks could have been done just as well in 30 minutes or 
> so.
> (I originally asked for 40 minutes for my Babel talk, but got one 
> hour
> anyway.)  I also thought there were many talks that were not clearly 
> mesh
> related -- I think we should have a rule that says "if it's not mesh
> related, you get 15 minutes, and we'll interrupt you rudely if you go 
> over
> your allotted time".  (And if you intend to sit down and scroll a 
> text
> file rather than preparing a proper set of overhead slides and 
> speaking to
> the audience, you get 0 minutes.  Sorry.)

 I quite agree.

 It must be also said that those who work on the testbed cannot attend 
 all the talks.

 This was my 4th battlemesh, and the first time I helped out with the 
 tests.
 After doing this, I think I'll do that in future editions too, because 
 it makes
 so much sense rather than coming there to do what in italian we call 
 "cazzeggio".

 I would push in a direction which priorities tests. I would encourage 
 more people in volunteering in the
 test process, this year we saw that if we clearly separate 
 responsabilities we can achieve a lot in parallel
 and glue the results together nicely.
 This team work  we have been doing last week has been beautiful, fun, 
 inspiring, educational and social.

 If I would have to organize a battlemesh, I would prepare a projector 
 in the
 room where the test preparation happen, and I would allocate some time 
 for daily recaps,
 as we were doing this every day in the morning and in the evening but 
 amid confusion, noise and without a projector.

 I would also encourage more participation of ALL the routing protocol 
 developers, because in my opinion the batman-adv team did not follow the 
 process as much as the other routing protocol developers.

 And when Julius said "Sorry if we did sleep", is quite true, I don't 
 think this event can rely on a few heroes that pull up a huge amount of 
 work 24/hours a day for a week (not what happened this time because we 
 were at least 8-10 people working in parallel, but happened a few times 
 in the past). Being able to work remotely would be cool although we HAVE 
 TO avoid human single point of failures. I think we need to focus the 
 event on the test battle, find more volunteers, clearly separate 
 responsibilities so people can work in parallel.

 The battlemesh is unique because of the testing process, if we don't 
 prioritise this we should rename the event in meshconf and run talks 
 only, but would it be the same?

> There were a small number of talks from commercial entities 
> advertising
> their wares, which I think is great, but we should respect the right 
> of
> other folks to avoid such talks.  Hence, they should be clearly 
> described
> as marketing talks.  For example, one talk was described as "Talk 
> about
> products from $COMPANY", where it should have said "This is a talk 
> given
> by the marketing people from $COMPANY.  Full disclosure: $COMPANY has
> donated $NUMBER antennas and a small goat to WLAN-SI."

 It would be great to have even more of these talks (as long as they 
 talk about hardware that is relevant to us, EG: OpenWRT compatible),
 although 1 hour is quite overkill.
 And yes it would be good to be transparent and clear about 
 sponsorships,
 although for me it was clear that it would have been a moslty marketing 
 talk.
 
> Somebody bullied Federico into adding Gnunet to the firmwares at the 
> last
> moment.  Folks, that's not how it is done -- last minute additions 
> are not
> acceptable, no matter how good your sales pitch.  Thanks to the cjdns 
> and
> MaidSafe people for their restraint.

 I wouldn't say that Daniel bullied me, he was not rude or bully, I 
 think there was a misunderstanding because we didn't know we were meant 
 to test more protocols than those we prepared.

 I'm glad we froze the number of protocols in the manual (and I want to 
 underline manual) testbed because that could have resulted in a 
 disaster.

 More suggestions for next editions
 ==================================

 * live streaming is great, let's do more of that
 * I think we all agree to avoid august in next editions, I need 
 moderate temperature to perform well
 * I (and many others) would be grateful if we select a location which 
 is easy to reach by plane
 * as agreed during discussion, let's freeze the openwrt revision in 
 advance, no last minute compilations
 * all the work that can be done before the event should be done before 
 the event
 * the devs that want their protocol to be tested have to participate 
 and contribute
 * I think that if we get creative (and probably smart) we can achieve a 
 configuration that is flexible enough to be prepared before the event 
 and tweaked there, what do you think?
 * social events are a great chance to get to know each other, I think 
 we should have a couple of them but not start them too early, I would 
 prefer a few short social events than a very long one

 About publishing the results
 ============================

 The idea is to publish a documentation which comprises everything:

 * topology drawing
 * description of what, why and how
 * graphs with interpretation hints
 * link to raw data
 * source code of R scripts for graph generation
 * source code of test scripts
 * uci configurations

 It will take me a while though, also because i'll be on vacation 
 (finally) till the 20th of August.

 Federico