[ninux-dev] Fwd: [WLANware] Freifunk Google Summer of Code proposal

[Nemesis]

Hi Matthias,

sorry for my late response, I would have liked to answer earlier but
didn't have a chance to do it properly.

On 03/09/2016 05:54 AM, Matthias Schiffer wrote:
> On 03/08/2016 12:23 PM, Nemesis wrote:
>> Hi Matthias,
>>
>> On 03/06/2016 10:41 PM, Matthias Schiffer wrote:
>>> Hi,
>>> I've attached my proposal for the Google Summer of Code. After your last
>>> mail, I've decided to shift the focus of my project towards the
>>> configuration. Feedback is welcome.
>> I'm glad my email was a motivating input.
>>
>> I'll try to provide some useful feedback, as well as telling you what
>> I'm doing with OpenWISP, NetJSON and experimenting in Ninux.
>>
>> I'm adding ninux-dev in CC because I believe it's good to share our
>> thoughts with more people and because your proposal is very interesting.
>>
>> On 03/06/2016 10:31 PM, Matthias Schiffer wrote:
>>> Hi,
>>> I could probably spare my introduction, as I'm not exactly a new face in
>>> Freifunk and OpenWrt development, but as I'm interested in partaking in
>>> this year's Google Summer of Code as a student, I'll give some more
>>> background about myself. Also, I'm writing in English, as I know of a few
>>> people in other, non-German community network projects that are interested
>>> in this proposal.
>>>
>>> I'm currently working on my Masters degree in Computer Science at the
>>> University of Lübeck, Germany. I've been involved with the Freifunk
>>> Lübeck project for a few years now and have become an active contributor
>>> to OpenWrt since then. I consider myself a very expecienced C programmer
>>> (but also have learned many other languages over the years). In the past
>>> years I've developed the VPN daemon fastd [1] (which was also the subject
>>> of my Bachelor thesis), and helped start the Gluon project [2], which are
>>> both used by many community network projects like German Freifunk
>>> communities now.
>>>
>>>
>>> The TL;DR one-sentence version: I'd like to create an alternative to the
>>> UCI config system, providing schema-based, structured configuration with
>>> saner upgrade behaviour.
>>>
>>>
>>> This proposal is concerned with one of the basic utilities of OpenWrt: the
>>> UCI config system. Over the years, we've come to the conclusion that the
>>> current state of UCI is not really what we need in Freifunk firmwares like
>>> Gluon. The main issue is the upgrade behaviour:
>>>
>>> * When setting new default configurations in a new firmware release, we
>>>   don't know if the old setting came from an older firmware's defaults, or
>>>   if a user set the value herself
>>> * Old settings not used anymore need to be removed explicitly or will
>>>   accumulate; these explicit removal commands in upgrade script also
>>>   accumulate if skipping versions on upgrades should be supported
>>> * UCI config files are usually installed by packages, but because of the
>>>   upgrade concept of OpenWrt, there is no sane way to define a pre/post-rm
>>>   script cleaning up and removing the config files when a package is
>>>   removed during a firmware upgrade
>> A way to migrate configuration settings is indeed missing from OpenWRT
>> and would be welcome.
>>
>> Consider the possibility of implementing a layer that keeps
>> compatibility with UCI to speed up adoption.
> As far as my current plan goes, this would be a one-way compatiblity layer
> (generating UCI from NC). Also, a schema would need to be created for all
> packages that should be supported by such a layer (as I explicitly don't
> want to allow creating configuration without a matching schema; I think
> gconf/gsettings work the same way.)

It's a good idea. Have you thought about the format to use?

JSON-Schema is quite good (although hopefully improvable), and YAML is
quite popular nowadays.
Maybe yaml and json-schema can be use together by converting YAML to
JSON before validating the content against JSON-Schema.

>>> I've posted an UCI extension proposal to the openwrt-devel ML last year
>>> [3], but didn't get much positive feedback (but enough to know that other
>>> people have similar issues). Since then, I've moved away from the idea of
>>> creating an API-compatible extension to UCI, and think creating an
>>> alternative to UCI with mostly different concepts would be preferable.
>>> Package maintainers could decide themselves if UCI or this New
>>> Configuration system ("NC", working title) should be used for some
>>> specific application; as proposed in some replies to [3], generating UCI
>>> config from NC would also be a possible approach.
>> The type of response you got from the OpenWRT Mailing List is pretty
>> standard.
>> I have ever hardly seen proposals for radical change welcomed and cherished.
>> I suggest to not let that discourage you and keep bringing up the
>> discussion in the
>> near future whenever you have interesting ideas, work to show and so on.
>>
>>> My current ideas are inspired by GNOME's gconf/gsettings systems; I'm also
>>> borrowing some vocabulary from the systemd people. I'll use the following
>>> terms to describe different types of config:
>> You might attract negative attention just because you're mentioning
>> systemd, so beware of this.
>>
>>> Schema
>>>     The schema defines which config keys exist and which type the
>>>     corresponding values have. It can also define default values and
>>>     presets for these keys. Schema files lie somewhere below /lib and are
>>>     installed by packages which need configuration.
>>>
>>> Default values
>>>     Config keys not set by the user may have default values defined by the
>>>     schema.
>>>
>>> User configuration
>>>     User configuration is stored in a database file below /etc. Per-user
>>>     configuration in home directories is not part of this proposal (as
>>>     in OpenWrt most config is system-wide), but it would be a possible
>>>     extension.
>>>
>>> Presets
>>>     Presets define the default values for the user configuration. This
>>>     differs from the default values: changing the default values on
>>>     upgrades will change the effective value, but changing a preset won't
>>>     (as on initialization/upgrades, non-existing values in the user
>>>     configuration will be initialised from the presets, but existing values
>>>     will be retained.)
>>>
>>> Besides upgrades, there are some more issues of UCI I'd like to solve with
>>> NC:
>>>
>>> * UCI config files are designed to be both human- and machine-readable/
>>>   writable (also, parsable by shell scripts.) NC would rather use a binary
>>>   format that can be read, written (regarding performance) and stored
>>>   (regarding space) more efficiently, combined with an intuitive
>>>   configuration tool (maybe optionally curses-based)
>> Are you sure this is a real advantage when compared with the costs of
>> developing all the necessary tools
>> for storing and reading the configuration on the device?
> Well, I definitely don't want the whole default configuration to be held in
> /etc, but only a diff to the defaults. Currently even the simplest changes
> causes the whole file to be copied to the overlay on OpenWrt; this can
> easily fill the whole overlay even though most of the information is redundant.
>
> Example: A fresh install of Gluon will use about 290KB of overlay space;
> the minimum size of the overlay is 320KB (5 erase blocks), and on some
> devices we're getting dangerously close to this minimum. Having all
> configuration in a single file with nothing but differences from the
> defaults, and in binary instead of ASCII, might reduct the overhead and
> redundancy significantly (obviusly, exact numbers aren't available yet.)
>
> Working with a diff without proper tooling would be very complicated, so
> proper tools need to developed anyways.
>

I see.

If I'm not wrong, systemd is currently doing something similar. I think
as long as there are tools to work easily with the binary data,
everything will be eventually ok.

>>
>>> * UCI configuration has a flat section/key/value structure, NC would allow
>>>   (JSON-like?) structured configuration
>> JSON or even NetJSON could be good options (even YAML using a 1 to 1
>> conversion with the NetJSON schema would be interesting).
>>
>> I've developed a library that I'm using to generate UCI configurations
>> from a remote web application, it's called netjsonconfig and published
>> several releases already:
>> http://netjsonconfig.openwisp.org/
>>
>> What I learnt from this project is that converting NetJSON to UCI is
>> doable and less hard than what I thought (except some annoying corner
>> cases). But another thing I've learnt, which might be useful to you, is
>> that a lot of people love UCI because its structure is simple, and after
>> years of working with it I believe the design choice was wise in this
>> regard.
>> Complexity brings all sorts of problems and bugs.
> JSON-like structures are also well-known and are used for various usecases.
> I find some parts of the flat UCI config quite awkward (like having
> "singleton" section types for which there's always exactly one section with
> the type.)
>
> Using JSON as a basis would also have the advantage that there are already
> tons of tools and models available for JSON schemas, that could either be
> used directly, or at least give some inspiration for NC.

Completely agree.

>
>> If you want to find out more about the motivations and goals behind
>> netjsonconfig and the other softwares related to it, I wrote a blog post
>> here:
>> http://nemesisdesign.net/blog/coding/netjsonconfig-convert-netjson-to-openwrt-uci/
> Thanks, I'll take a look.
>
>> I'm experimenting these ideas in ninux too and I feel optimistic about
>> the outcome.
>>
>>> * NC would use only a single database for storage of the user configuration
>>>   (JFFS2 doesn't seem to handle lots of small files very efficiently; this
>>>   needs more research)
>> What do you mean for "database" here? Filesystem? Or anything else?
> The aforementioned binary file used to store the diff between defaults and
> user configuration.

Ok.

>>
>>> * UCI only stores strings. This has led to some funny inconsistencies (for
>>>   storing booleans, any of the following values are understood as true:
>>>   1/on/true/yes/enabled.) NC should natively support at least all primitive
>>>   data types of JSON.
>> Good. I also dislike the inconsitencies of UCI.
>>
>>> * UCI separates the "save" and "commit" stages when changing configuration.
>>>   Changes that have been saved but not committed will already be effective,
>>>   but aren't stored to non-volatile memory till they are committed.
>>>   This can not only be confusing to users, but also leads to bad
>>>   programming practices like making configuration changes in scripts that
>>>   are never supposed to be committed, which is obviously fragile as other
>>>   scripts or the user may commit the config.
>>>   NC would not separate these steps: Configuration should become effective
>>>   only when it is stored permanently.
>> Do you mean "add", "set" or "import" rather than "save"?
> "save" is the term used by the C and Lua APIs. "add", "set", "import" etc.
> all imply a "save".
>
>> I think this behaviour of UCI is wise, the concept is widely used stuff
>> from Juniper and Cisco.
>>
>> Try to read the NETCONF RFC in your spare time:
>> https://tools.ietf.org/html/rfc6241
>>
>> Look for concepts like "Running Configuration", "Candidate
>> Configuration", and "Testing new Configuration".
> I'll have a look at that as well. At a glance, there doesn't seem to be a
> nice way to support such a behaviour when there is not a single
> configurator (i.e. administrator), but both human and automatic (scripted)
> entities change configuration (transient changes may become unapplicable
> when the underlying configuration changes.)
>
> Outside the world of professional routers, I haven't really seen similar
> concepts. For most applications, transient configuration doesn't make much
> sense and mostly causes confusion. Network configuration *might* be special
> in this regard, as it may be preferable to get back to an old state of
> config after breaking something, but I'd rather handle this on a higher
> level and not in the configuration system. This also keeps the config
> system simpler.

I noticed also AirOS does something similar with its tool cfgmtd:

Usage: cfgmtd [options]
    -t <type>            - Configuration type to use
[1(active)|2(backup)]. (Default: 1(active))
    -f <config file>        - Configuration file to use. (Default:
/tmp/system.cfg)
    -p <persistent directory>    - Directory to persistent dir.
(Default: none)
    -w                - Write to flash action.
    -r                - Read from flash action.
    -u                - Update flash action. (Default: update
configuration ONLY)
    -o <mtd|file name>        - Use mtd or file name. (Default: cfg)
    -h                - This message.

In openwisp-config <https://github.com/openwisp/openwisp-config>, an
agent that interacts with django-netjsonconfig
<https://github.com/openwisp/django-netjsonconfig> (the successor of
OpenWISP Manager, currently under active development together with
netjsonconfig), there's a "test configuration" feature, that is
implemented in nothing more than shell (ash more precisely), which tests
the new configuration before making it final and does rollback in case
of failure.

While implementing that type of feature I would really have preferred to
have a configuration system that would have assisted me more in doing it.

Leaving the door open for a similar feature to be introduced even tually
in the future may be a good compromise.

>>> * Comments in UCI files are possible, but get lost when the file is
>>>   changed using libuci or the uci CLI. NC could support documentation of
>>>   config keys in the schema.
>> No strong opinions on this point.
>>
>>> This project would involve designing the configuration formats (schema
>>> files and user configuration), and implementing:
>>> * C library
>>> * CLI configuration tool
>>> * Lua binding
>>> * Some kind of shell support (for accessing NC from initscripts etc.)
>>> * Documentation for all of this
>>>
>>> I'm also thinking about adding "active defaults", i.e. default values that
>>> are not statically defined in the schema, but computed by plugin modules.
>>> This would e.g. allow Gluon packages to install complex configuration for
>>> other packages without the need to store this in the user config in /etc.
>>>
>>> I'm not totally sure about the time required to implement all of this, but
>>> I suspect I might be finished in less than the 3 months of GSoC coding
>>> time. Therefore, and to give NC some more practical testing, I'm thinking
>>> about implementing a minimal version of [4] (which was also proposed by me)
>>> based on NC. [4] is obviously a very complex project which I believe is too
>>> big to complete during the time of the GSoC, but I think minimal version to
>>> show the soundness of NC would be a nice addition to this project (for
>>> example only supporting static address/route configuration and network
>>> namespaces.)
>>>
>>> Of course, if another student wants to work on [4] (and someone wants to
>>> mentor it), I'd rather not take [4] away from them and think of something
>>> else to add to my project in case it gets finished too quickly.
>>>
>>> Regarding mentoring, I'd love to get a mentor who is not involved with
>>> Gluon yet, so he or she can make sure what I create is not only useful for
>>> Gluon, but also for the broader OpenWrt community (or even for other Linux
>>> distributions.)
>> Very good idea.
>>
>>> Please let me know what you think of this proposal.
>>>
>>> Regards,
>>> Matthias
>>>
>>>
>>> [1] https://projects.universe-factory.net/projects/fastd/wiki
>>> [2] https://github.com/freifunk-gluon/gluon
>>> [3] https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html
>>> [4]
>>> https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt
>> One thing it see you did not touch with this email is your initial
>> "Configuration Overlay prooposal".
>>
>> I am encountering the need to apply different layers of configurations,
>> for example:
>>
>>   * local layer, the user can change this and it won't be overwritten
>>   * remote/auto layer, managed remotely or automatically in some
>>     fashion, the user should not change these settings locally because
>>     his changes will be overwritten
>>
>> Have you also encountered this need?
> Yes, this is actually one of the principal points I'd like to address with
> NC, although that got a bit lost when writing my proposal.
>
> I'm not sure if it really makes sense to try to *prevent* the user from
> changing a specific setting; in my experience, users would just try harder
> and break things a layer deeper when they couldn't get their way on the
> surface ;)
>
> Regarding automatically managed configuration:
>
> It should be possible for packages to install defaults for other, already
> installed schemata. The automatic/remote config system would just replace
> the defaults on updates; if the user didn't change any options, the user
> config diff will be empty and the new defaults will become active.
>
> If the user wants to change things, this is possible, and the new defaults
> won't have an effect then.

Sounds very good.

> Thanks for your feedback,
>
> Matthias

You are very welcome.

Federico

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ml.ninux.org/pipermail/ninux-dev/attachments/20160403/2d5065d0/attachment-0001.html>