[Ninux-Wireless] Strumenti per benchmark di rete

[Germano Massullo]

Il 18/07/2016 12:25, Germano Massullo ha scritto:
> Il 18/07/2016 10:10, Stefano De Carlo ha scritto:
>>> hardware per la gestione dei pacchetti, soffrono di rallentamenti in
>>> caso di installazione di firmware non stock, come Lede/OpenWRT.
>>> Tale dubbio è venuto fuori ricordando che alcuni dispositivi Tp-Link
>>> hanno un NAT hardware che non viene sfruttato se non si usa il firmware
>>> originale[3]
>> Se è questo che devi verificare, puoi anche evitare la trafila. Nessun Network Accelerator "prosumer" ha supporto open source perché usano tutti custom-hack di netfilter, e i dev OpenWrt (presumo anche quelli LEDE), che li giudicano ignobili come qualità del codice, non si muoveranno finché non ci sarà un approccio general purpose nel kernel. Tutto questo in aggiunta ai soliti problemi sulla documentazione e reverse engineering dei chip.
>>
>> Nei benchmark otterresti gli stessi risultati che se il chip non ci fosse, non è predisposto nessun offloading.
> Sì conosco la storia, comunque ho avuto una interessante discussione con
> gli sviluppatori Lede circa i processori Cavium Octeon che equipaggiano
> questi router. Stasera vi incollo tutto
>
<Germano> Ubiquiti EdgeRouter(s) claims to have hardware acceleration
for packets processing. Their CPU is a Cavium Octeon (example of general
datasheet here http://www.cavium.com/pdfFiles/CN50XX_PB_Rev1.pdf ). What
I would like to know is: does Ubiquiti enables the hardware acceleration
for packets processing using nasty netfilter hacks like Tp-Link does?
(example https://dev.openwrt.org/ticket/11779 ). I checked into Linux
kernel tree and there
<Germano> are many references to Octeon CPU, so it could it be possible
that the hardware acceleration is already managed by Linux kernel
https://github.com/torvalds/linux/search?utf8=%E2%9C%93&q=cavium+octeon&type=Code       
So Lede could benefit of such accelration without having to care about
nasty code hacks. What do you think about?
<lede-bot> Title: #11779 (WDR4300 - hardware nat feature) – OpenWrt (at
dev.openwrt.org)
<lede-bot> Title: Search Results · GitHub (at github.com)
<Germano> perhaps I should also send an e-mail to mailing list
<stintel> Germano: what you are seeing is just the support for that CPU
and ethernet driver
<stintel> the Linux kernel does not support the offloading like in EdgeOS
<Germano> stintel: mmh, so did Cavium keep such "magic" secret to use it
in commercial agreements with networking devices producers?
<stintel> either that, or the code is so crappy that it will never be
accepted upstream in that form
<Germano> stintel: is there a chance that Octeon hardware acceleration
works without having to be supported by operating system?
<stintel> afaik nope
<stintel> there are some crypto modules for octeon:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/mips/cavium-octeon/crypto
<lede-bot> Title: kernel/git/torvalds/linux.git - Linux kernel source
tree (at git.kernel.org)
<stintel> but I never measured the impact of using those over generic
<stintel> (I have 2 ERLs)
<Germano> stintel: what is an ERLs?
<Germano> ahh
<stintel> EdgeRouter Lite :)
<Germano> EgeRouter Lite
<Germano> stintel: what worries me is that Cavium provides an SDK for
Octeon, as I can read from http://www.cavium.com/pdfFiles/CN50XX_PB_Rev1.pdf
<stintel> well most manufacturers are the same, the provided ancient
kernels with crappy modifications and can't be bothered with trying to
get stuff upstream
<Germano> stintel: so perhaps the only way to see the differencies is to
run benchmarks like tcpdive - https://github.com/fastos/tcpdive   
and    Apache benchmarks as written in
http://arstechnica.com/gadgets/2016/01/numbers-dont-lie-its-time-to-build-your-own-router/

<lede-bot> Title: GitHub - fastos/tcpdive: A TCP performance profiling
tool. (at github.com)
<Germano> stintel: here
http://phx.corporate-ir.net/phoenix.zhtml?c=209126&p=irol-newsArticle_print&ID=1052223  
it is written that hardware acceleration concerns encryption, so perhaps
even Linux upstream kernel supports such accelerations as we saw before
<lede-bot> Title: Cavium Networks - Investors > Press Release (at
phx.corporate-ir.net)
<plntyk2> Germano: there are probably some datasheets/NDA that cover
"other" acceleration areas on Cavium - and i dont think that ERL/ER do
provide 16 MIPS64 cores for example
<Germano> what does mean ù
<Germano> " and i dont think that ERL/ER do provide 16 MIPS64 cores for
example"
<plntyk2> " All accelerators support the OCTEON Plus CN58XX processors
with up to 16 MIPS64 cores" from press release
<stintel> ERL is CN5020
<Germano> ah ok you wanted to say that the press release is about 
OCTEON Plus CN58XX and Ubiquiti does not use such accelerator
<plntyk2> yeah- i dont know what ERL /ER is using but you cannot get
that price tag with 16 cores
<Germano> plntyk2: Ubiquiti uses dual core Octeon
<plntyk2> and even now dual core processors on x86 are not that often
used in network servers with 10Gbps connections
<Germano> plntyk2: it's a MIPS CPU
<plntyk2> so that should mean to examine possible throughput claims
regarding "how" more carefully
<plntyk2> ASM optimized DSPs or so?
<plntyk2> afaik mips does have some dsp subsets
<Germano> plntyk2: we don't know
<Germano> plntyk2: the fastest way for me is to run benchmark on both
stock firmware and Lede, and see how much difference there is
<stintel> well, LEDE doesn't support the IP offloading stuff EdgeOS
does, so with offloading enabled, EdgeOS will be faster
<stintel> but LEDE is much faster than EdgeOS with offloading disabled,
due to LEDE having more recent kernel with other optimizations and fixes
et c
<Germano> stintel: I think that all EdgeOS IP offloading is coming from
Cavium direct support to Ubiquiti
<plntyk2> https://wiki.openwrt.org/doc/howto/benchmark.openssl - the
edge router results are not that impressive there compared to other
multi core ARM or ramips units
<plntyk2> but they are old - might need some refreshing
<Germano> plntyk2: lol Nokia N900 is very fast
<Germano> in such list, I can see 0.9.8o w/o hw crypto
<Germano> how did they find out that was without hw crypto?
<plntyk2> afaik openssl has modules that handle hw crypto stuff - OCF
(removed recently) and/or cryptolinux (support via Kernel driver afaik)
<Ntemis> am trying to add support for a chinese router and i need some
help identify gpios
<Ntemis> how to i get the correct gpios?
<Germano> plntyk2: why OCF has been removed?
<stintel> Ntemis: either look at the manufacturers source or try
instructions on https://wiki.openwrt.org/doc/devel/add.new.device#gpios
<Ntemis> stintel: no source
<Ntemis> looks tedious
<matteo> Ntemis: put a led on it, and raise it one by one via a register
write
<Ntemis> matteo: in english? :D
<Hauke> plntyk2: nice SoC comparison, so you know more such documents?
<plntyk2> sorry no - I just came across that document via reddit and
thought it might be quite an interesting read since Open Hardware is a
"hot topic" (like those "libre" forks)
<matteo> Hauke: I have a google sheet with benchmarks run on sone hardware
<matteo>
https://docs.google.com/spreadsheets/d/1xUSe7d46mop7UMhMMsB-EFukLQEHi40LAYoujoj3MsI/edit#gid=0
<lede-bot> Title: dhrystones - Google Sheets (at docs.google.com)
<plntyk2> Germano: commit says only: "kernel: remove ocf support,
cryptodev-linux should be used instead" - maybe maintainability  since
its out of tree and so it will regularly break with new kernels
<stintel> | r1005 | UBNT_E100 (CN5020p1.1-500-SCP) | Unknown | Cavium
Octeon+ V0.1 | 1000.00 | Cavium Octeon+ V0.1 | 1000.00 | 1.0.2h |
34687660 | 12121770 | 8691370 | 14499500 | 5619370 | 1980420 | 9586010 |
8449020 | 7528270 | 6.5 | 234.7 23.2 | 19.1 |
<matteo> I have found a bug in the menuconfig, can someone try to
reproduce it?
<Germano> stintel: have you just runned the bench on your ERL with Lede?
<stintel> Germano: yes, that's the above output
<matteo> run menuconfig and show the help for a package, like ipset in
networking
<matteo> you have a dependency like PACKAGE_libmnl [=n]
<matteo> select it, and libmnl still remains n
<matteo> type / for search, search for mnl and it's y
<matteo> show help again, mnl is y now
<Hauke> matteo: thanks for the document
<matteo> you're welcome
<stintel> Germano: I am now building an image with kmod-cryptodev and
cryptodev enabled in openssl
<Germano> stintel: interesting. What will be the benefits in managing
encryption?
<matteo> Germano: on geode I get a 100x speed improvement
<Germano> matteo: omg
<stintel> well it might improve md5/sha*
<matteo> it depends on the supported chiphers
<stintel> yeah, no aes support for octeon in the kernel :(
<Germano> stintel: but didn't we found some Octeon crypto stuff?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/mips/cavium-octeon/crypto
<lede-bot> Title: kernel/git/torvalds/linux.git - Linux kernel source
tree (at git.kernel.org)
<stintel> Germano: yes, that's what I sent you. there is no aes in
there, just md5 and sha*
<stintel> and it looks like openssl with cryptodev is much much slower
on my ERL :P
<Germano> stintel: :-( perhaps Octeon does not have AES acceleration at all
<Germano> stintel: omg
<scelestic> mh, even with gcc 5 something is leaking memory like crazy
<stintel> well the  cryptodev readme mentions openssl cryptodev support
is outdated
<nbd> on octeon it would be possible to do hwaccel completely in user
space without kernel driver involvement
<nbd> that would be much faster than any kernel assisted hwaccel
<nbd> but it needs somebody to actually write the code for it ;)
<matteo> and easier to maintain I guess
<nbd> well, i wouldn't say easier to maintain
<nbd> because the crypto stuff has to be implemented in the ssl/crypto
libs directly
<nbd> and there's more than one
<nbd> ;)
<nbd> crypto accel on octeon works through cpu registers, without dma or
memory mapped devices
<nbd> that's why it can be used easily from user space
<Germano> can I have your permission to quote pieces of this chat into
Ninux wireless community mailing list?
<nbd> sure
<Germano> nbd: cool, you are very well informed about Octeon
<nbd> if somebody starts working on this, i can provide some more help
and guidance for it
<nbd> so please get back to me when this is getting more specific
<Germano> nbd: just for my curiosity: do you think that Octeon
datasheets are publicly available?
<Germano> I mean the documentation of registers, etc.
<nbd> no idea
<nbd> didn't check
<Germano> nbd: where did you find out that they manage encryption in
registers? Is it a consideration coming from the fact that such
acceleration is available also in userspace?
<stintel> Germano: actually, md5 is faster but sha1 is slower with
cryptodev. the rest is the same, unpatched openssl cryptodev doesn't
support sha2
<nbd> Germano: looked through some sources somewhere, had some
discussion with sebastian from dd-wrt who also worked on this
<nbd> i don't remember the exact details, it was years ago
<nbd> it would probably make sense to write a small user space octeon hw
crypto library
<nbd> and hook that into polarssl/mbedtls/openssl