[ninux-dev] Fwd: [WLANware] Freifunk Google Summer of Code proposal

[Nemesis]

Hi Matthias,

On 03/06/2016 10:41 PM, Matthias Schiffer wrote:
> Hi,
> I've attached my proposal for the Google Summer of Code. After your last
> mail, I've decided to shift the focus of my project towards the
> configuration. Feedback is welcome.

I'm glad my email was a motivating input.

I'll try to provide some useful feedback, as well as telling you what
I'm doing with OpenWISP, NetJSON and experimenting in Ninux.

I'm adding ninux-dev in CC because I believe it's good to share our
thoughts with more people and because your proposal is very interesting.

On 03/06/2016 10:31 PM, Matthias Schiffer wrote:
> Hi,
> I could probably spare my introduction, as I'm not exactly a new face in
> Freifunk and OpenWrt development, but as I'm interested in partaking in
> this year's Google Summer of Code as a student, I'll give some more
> background about myself. Also, I'm writing in English, as I know of a few
> people in other, non-German community network projects that are interested
> in this proposal.
>
> I'm currently working on my Masters degree in Computer Science at the
> University of Lübeck, Germany. I've been involved with the Freifunk
> Lübeck project for a few years now and have become an active contributor
> to OpenWrt since then. I consider myself a very expecienced C programmer
> (but also have learned many other languages over the years). In the past
> years I've developed the VPN daemon fastd [1] (which was also the subject
> of my Bachelor thesis), and helped start the Gluon project [2], which are
> both used by many community network projects like German Freifunk
> communities now.
>
>
> The TL;DR one-sentence version: I'd like to create an alternative to the
> UCI config system, providing schema-based, structured configuration with
> saner upgrade behaviour.
>
>
> This proposal is concerned with one of the basic utilities of OpenWrt: the
> UCI config system. Over the years, we've come to the conclusion that the
> current state of UCI is not really what we need in Freifunk firmwares like
> Gluon. The main issue is the upgrade behaviour:
>
> * When setting new default configurations in a new firmware release, we
>   don't know if the old setting came from an older firmware's defaults, or
>   if a user set the value herself
> * Old settings not used anymore need to be removed explicitly or will
>   accumulate; these explicit removal commands in upgrade script also
>   accumulate if skipping versions on upgrades should be supported
> * UCI config files are usually installed by packages, but because of the
>   upgrade concept of OpenWrt, there is no sane way to define a pre/post-rm
>   script cleaning up and removing the config files when a package is
>   removed during a firmware upgrade

A way to migrate configuration settings is indeed missing from OpenWRT
and would be welcome.

Consider the possibility of implementing a layer that keeps
compatibility with UCI to speed up adoption.

> I've posted an UCI extension proposal to the openwrt-devel ML last year
> [3], but didn't get much positive feedback (but enough to know that other
> people have similar issues). Since then, I've moved away from the idea of
> creating an API-compatible extension to UCI, and think creating an
> alternative to UCI with mostly different concepts would be preferable.
> Package maintainers could decide themselves if UCI or this New
> Configuration system ("NC", working title) should be used for some
> specific application; as proposed in some replies to [3], generating UCI
> config from NC would also be a possible approach.

The type of response you got from the OpenWRT Mailing List is pretty
standard.
I have ever hardly seen proposals for radical change welcomed and cherished.
I suggest to not let that discourage you and keep bringing up the
discussion in the
near future whenever you have interesting ideas, work to show and so on.

> My current ideas are inspired by GNOME's gconf/gsettings systems; I'm also
> borrowing some vocabulary from the systemd people. I'll use the following
> terms to describe different types of config:

You might attract negative attention just because you're mentioning
systemd, so beware of this.

> Schema
>     The schema defines which config keys exist and which type the
>     corresponding values have. It can also define default values and
>     presets for these keys. Schema files lie somewhere below /lib and are
>     installed by packages which need configuration.
>
> Default values
>     Config keys not set by the user may have default values defined by the
>     schema.
>
> User configuration
>     User configuration is stored in a database file below /etc. Per-user
>     configuration in home directories is not part of this proposal (as
>     in OpenWrt most config is system-wide), but it would be a possible
>     extension.
>
> Presets
>     Presets define the default values for the user configuration. This
>     differs from the default values: changing the default values on
>     upgrades will change the effective value, but changing a preset won't
>     (as on initialization/upgrades, non-existing values in the user
>     configuration will be initialised from the presets, but existing values
>     will be retained.)
>
> Besides upgrades, there are some more issues of UCI I'd like to solve with
> NC:
>
> * UCI config files are designed to be both human- and machine-readable/
>   writable (also, parsable by shell scripts.) NC would rather use a binary
>   format that can be read, written (regarding performance) and stored
>   (regarding space) more efficiently, combined with an intuitive
>   configuration tool (maybe optionally curses-based)

Are you sure this is a real advantage when compared with the costs of
developing all the necessary tools
for storing and reading the configuration on the device?

> * UCI configuration has a flat section/key/value structure, NC would allow
>   (JSON-like?) structured configuration

JSON or even NetJSON could be good options (even YAML using a 1 to 1
conversion with the NetJSON schema would be interesting).

I've developed a library that I'm using to generate UCI configurations
from a remote web application, it's called netjsonconfig and published
several releases already:
http://netjsonconfig.openwisp.org/

What I learnt from this project is that converting NetJSON to UCI is
doable and less hard than what I thought (except some annoying corner
cases). But another thing I've learnt, which might be useful to you, is
that a lot of people love UCI because its structure is simple, and after
years of working with it I believe the design choice was wise in this
regard.
Complexity brings all sorts of problems and bugs.

If you want to find out more about the motivations and goals behind
netjsonconfig and the other softwares related to it, I wrote a blog post
here:
http://nemesisdesign.net/blog/coding/netjsonconfig-convert-netjson-to-openwrt-uci/

I'm experimenting these ideas in ninux too and I feel optimistic about
the outcome.

> * NC would use only a single database for storage of the user configuration
>   (JFFS2 doesn't seem to handle lots of small files very efficiently; this
>   needs more research)

What do you mean for "database" here? Filesystem? Or anything else?

> * UCI only stores strings. This has led to some funny inconsistencies (for
>   storing booleans, any of the following values are understood as true:
>   1/on/true/yes/enabled.) NC should natively support at least all primitive
>   data types of JSON.

Good. I also dislike the inconsitencies of UCI.

> * UCI separates the "save" and "commit" stages when changing configuration.
>   Changes that have been saved but not committed will already be effective,
>   but aren't stored to non-volatile memory till they are committed.
>   This can not only be confusing to users, but also leads to bad
>   programming practices like making configuration changes in scripts that
>   are never supposed to be committed, which is obviously fragile as other
>   scripts or the user may commit the config.
>   NC would not separate these steps: Configuration should become effective
>   only when it is stored permanently.

Do you mean "add", "set" or "import" rather than "save"?

I think this behaviour of UCI is wise, the concept is widely used stuff
from Juniper and Cisco.

Try to read the NETCONF RFC in your spare time:
https://tools.ietf.org/html/rfc6241

Look for concepts like "Running Configuration", "Candidate
Configuration", and "Testing new Configuration".

> * Comments in UCI files are possible, but get lost when the file is
>   changed using libuci or the uci CLI. NC could support documentation of
>   config keys in the schema.

No strong opinions on this point.

> This project would involve designing the configuration formats (schema
> files and user configuration), and implementing:
> * C library
> * CLI configuration tool
> * Lua binding
> * Some kind of shell support (for accessing NC from initscripts etc.)
> * Documentation for all of this
>
> I'm also thinking about adding "active defaults", i.e. default values that
> are not statically defined in the schema, but computed by plugin modules.
> This would e.g. allow Gluon packages to install complex configuration for
> other packages without the need to store this in the user config in /etc.
>
> I'm not totally sure about the time required to implement all of this, but
> I suspect I might be finished in less than the 3 months of GSoC coding
> time. Therefore, and to give NC some more practical testing, I'm thinking
> about implementing a minimal version of [4] (which was also proposed by me)
> based on NC. [4] is obviously a very complex project which I believe is too
> big to complete during the time of the GSoC, but I think minimal version to
> show the soundness of NC would be a nice addition to this project (for
> example only supporting static address/route configuration and network
> namespaces.)
>
> Of course, if another student wants to work on [4] (and someone wants to
> mentor it), I'd rather not take [4] away from them and think of something
> else to add to my project in case it gets finished too quickly.
>
> Regarding mentoring, I'd love to get a mentor who is not involved with
> Gluon yet, so he or she can make sure what I create is not only useful for
> Gluon, but also for the broader OpenWrt community (or even for other Linux
> distributions.)

Very good idea.

> Please let me know what you think of this proposal.
>
> Regards,
> Matthias
>
>
> [1] https://projects.universe-factory.net/projects/fastd/wiki
> [2] https://github.com/freifunk-gluon/gluon
> [3] https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html
> [4]
> https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt

One thing it see you did not touch with this email is your initial
"Configuration Overlay prooposal".

I am encountering the need to apply different layers of configurations,
for example:

  * local layer, the user can change this and it won't be overwritten
  * remote/auto layer, managed remotely or automatically in some
    fashion, the user should not change these settings locally because
    his changes will be overwritten

Have you also encountered this need?

Federico

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.ninux.org/pipermail/ninux-dev/attachments/20160308/74028d04/attachment-0001.html>