[ninux-dev] [WLANware] Fwd: Freifunk Google Summer of Code proposal
Matthias Schiffer
mschiffer at universe-factory.net
Fri Mar 11 12:28:40 CET 2016
So... is there any more feedback? Anyone who's interested in mentoring such
a project?
The student registration phase is getting closer, and if this proposal
doesn't have much chance of being accepted, I'd like to know as soon as
possible to I have some time left to think of something different...
Regards,
Matthias
On 03/09/2016 05:54 AM, Matthias Schiffer wrote:
> On 03/08/2016 12:23 PM, Nemesis wrote:
>> Hi Matthias,
>>
>> On 03/06/2016 10:41 PM, Matthias Schiffer wrote:
>>> Hi,
>>> I've attached my proposal for the Google Summer of Code. After your last
>>> mail, I've decided to shift the focus of my project towards the
>>> configuration. Feedback is welcome.
>>
>> I'm glad my email was a motivating input.
>>
>> I'll try to provide some useful feedback, as well as telling you what
>> I'm doing with OpenWISP, NetJSON and experimenting in Ninux.
>>
>> I'm adding ninux-dev in CC because I believe it's good to share our
>> thoughts with more people and because your proposal is very interesting.
>>
>> On 03/06/2016 10:31 PM, Matthias Schiffer wrote:
>>> Hi,
>>> I could probably spare my introduction, as I'm not exactly a new face in
>>> Freifunk and OpenWrt development, but as I'm interested in partaking in
>>> this year's Google Summer of Code as a student, I'll give some more
>>> background about myself. Also, I'm writing in English, as I know of a few
>>> people in other, non-German community network projects that are interested
>>> in this proposal.
>>>
>>> I'm currently working on my Masters degree in Computer Science at the
>>> University of Lübeck, Germany. I've been involved with the Freifunk
>>> Lübeck project for a few years now and have become an active contributor
>>> to OpenWrt since then. I consider myself a very expecienced C programmer
>>> (but also have learned many other languages over the years). In the past
>>> years I've developed the VPN daemon fastd [1] (which was also the subject
>>> of my Bachelor thesis), and helped start the Gluon project [2], which are
>>> both used by many community network projects like German Freifunk
>>> communities now.
>>>
>>>
>>> The TL;DR one-sentence version: I'd like to create an alternative to the
>>> UCI config system, providing schema-based, structured configuration with
>>> saner upgrade behaviour.
>>>
>>>
>>> This proposal is concerned with one of the basic utilities of OpenWrt: the
>>> UCI config system. Over the years, we've come to the conclusion that the
>>> current state of UCI is not really what we need in Freifunk firmwares like
>>> Gluon. The main issue is the upgrade behaviour:
>>>
>>> * When setting new default configurations in a new firmware release, we
>>> don't know if the old setting came from an older firmware's defaults, or
>>> if a user set the value herself
>>> * Old settings not used anymore need to be removed explicitly or will
>>> accumulate; these explicit removal commands in upgrade script also
>>> accumulate if skipping versions on upgrades should be supported
>>> * UCI config files are usually installed by packages, but because of the
>>> upgrade concept of OpenWrt, there is no sane way to define a pre/post-rm
>>> script cleaning up and removing the config files when a package is
>>> removed during a firmware upgrade
>>
>> A way to migrate configuration settings is indeed missing from OpenWRT
>> and would be welcome.
>>
>> Consider the possibility of implementing a layer that keeps
>> compatibility with UCI to speed up adoption.
>
> As far as my current plan goes, this would be a one-way compatiblity layer
> (generating UCI from NC). Also, a schema would need to be created for all
> packages that should be supported by such a layer (as I explicitly don't
> want to allow creating configuration without a matching schema; I think
> gconf/gsettings work the same way.)
>
>>
>>> I've posted an UCI extension proposal to the openwrt-devel ML last year
>>> [3], but didn't get much positive feedback (but enough to know that other
>>> people have similar issues). Since then, I've moved away from the idea of
>>> creating an API-compatible extension to UCI, and think creating an
>>> alternative to UCI with mostly different concepts would be preferable.
>>> Package maintainers could decide themselves if UCI or this New
>>> Configuration system ("NC", working title) should be used for some
>>> specific application; as proposed in some replies to [3], generating UCI
>>> config from NC would also be a possible approach.
>>
>> The type of response you got from the OpenWRT Mailing List is pretty
>> standard.
>> I have ever hardly seen proposals for radical change welcomed and cherished.
>> I suggest to not let that discourage you and keep bringing up the
>> discussion in the
>> near future whenever you have interesting ideas, work to show and so on.
>>
>>> My current ideas are inspired by GNOME's gconf/gsettings systems; I'm also
>>> borrowing some vocabulary from the systemd people. I'll use the following
>>> terms to describe different types of config:
>>
>> You might attract negative attention just because you're mentioning
>> systemd, so beware of this.
>>
>>> Schema
>>> The schema defines which config keys exist and which type the
>>> corresponding values have. It can also define default values and
>>> presets for these keys. Schema files lie somewhere below /lib and are
>>> installed by packages which need configuration.
>>>
>>> Default values
>>> Config keys not set by the user may have default values defined by the
>>> schema.
>>>
>>> User configuration
>>> User configuration is stored in a database file below /etc. Per-user
>>> configuration in home directories is not part of this proposal (as
>>> in OpenWrt most config is system-wide), but it would be a possible
>>> extension.
>>>
>>> Presets
>>> Presets define the default values for the user configuration. This
>>> differs from the default values: changing the default values on
>>> upgrades will change the effective value, but changing a preset won't
>>> (as on initialization/upgrades, non-existing values in the user
>>> configuration will be initialised from the presets, but existing values
>>> will be retained.)
>>>
>>> Besides upgrades, there are some more issues of UCI I'd like to solve with
>>> NC:
>>>
>>> * UCI config files are designed to be both human- and machine-readable/
>>> writable (also, parsable by shell scripts.) NC would rather use a binary
>>> format that can be read, written (regarding performance) and stored
>>> (regarding space) more efficiently, combined with an intuitive
>>> configuration tool (maybe optionally curses-based)
>>
>> Are you sure this is a real advantage when compared with the costs of
>> developing all the necessary tools
>> for storing and reading the configuration on the device?
>
> Well, I definitely don't want the whole default configuration to be held in
> /etc, but only a diff to the defaults. Currently even the simplest changes
> causes the whole file to be copied to the overlay on OpenWrt; this can
> easily fill the whole overlay even though most of the information is redundant.
>
> Example: A fresh install of Gluon will use about 290KB of overlay space;
> the minimum size of the overlay is 320KB (5 erase blocks), and on some
> devices we're getting dangerously close to this minimum. Having all
> configuration in a single file with nothing but differences from the
> defaults, and in binary instead of ASCII, might reduct the overhead and
> redundancy significantly (obviusly, exact numbers aren't available yet.)
>
> Working with a diff without proper tooling would be very complicated, so
> proper tools need to developed anyways.
>
>>
>>> * UCI configuration has a flat section/key/value structure, NC would allow
>>> (JSON-like?) structured configuration
>>
>> JSON or even NetJSON could be good options (even YAML using a 1 to 1
>> conversion with the NetJSON schema would be interesting).
>>
>> I've developed a library that I'm using to generate UCI configurations
>> from a remote web application, it's called netjsonconfig and published
>> several releases already:
>> http://netjsonconfig.openwisp.org/
>>
>> What I learnt from this project is that converting NetJSON to UCI is
>> doable and less hard than what I thought (except some annoying corner
>> cases). But another thing I've learnt, which might be useful to you, is
>> that a lot of people love UCI because its structure is simple, and after
>> years of working with it I believe the design choice was wise in this
>> regard.
>> Complexity brings all sorts of problems and bugs.
>
> JSON-like structures are also well-known and are used for various usecases.
> I find some parts of the flat UCI config quite awkward (like having
> "singleton" section types for which there's always exactly one section with
> the type.)
>
> Using JSON as a basis would also have the advantage that there are already
> tons of tools and models available for JSON schemas, that could either be
> used directly, or at least give some inspiration for NC.
>
>>
>> If you want to find out more about the motivations and goals behind
>> netjsonconfig and the other softwares related to it, I wrote a blog post
>> here:
>> http://nemesisdesign.net/blog/coding/netjsonconfig-convert-netjson-to-openwrt-uci/
>
> Thanks, I'll take a look.
>
>>
>> I'm experimenting these ideas in ninux too and I feel optimistic about
>> the outcome.
>>
>>> * NC would use only a single database for storage of the user configuration
>>> (JFFS2 doesn't seem to handle lots of small files very efficiently; this
>>> needs more research)
>>
>> What do you mean for "database" here? Filesystem? Or anything else?
>
> The aforementioned binary file used to store the diff between defaults and
> user configuration.
>
>>
>>> * UCI only stores strings. This has led to some funny inconsistencies (for
>>> storing booleans, any of the following values are understood as true:
>>> 1/on/true/yes/enabled.) NC should natively support at least all primitive
>>> data types of JSON.
>>
>> Good. I also dislike the inconsitencies of UCI.
>>
>>> * UCI separates the "save" and "commit" stages when changing configuration.
>>> Changes that have been saved but not committed will already be effective,
>>> but aren't stored to non-volatile memory till they are committed.
>>> This can not only be confusing to users, but also leads to bad
>>> programming practices like making configuration changes in scripts that
>>> are never supposed to be committed, which is obviously fragile as other
>>> scripts or the user may commit the config.
>>> NC would not separate these steps: Configuration should become effective
>>> only when it is stored permanently.
>>
>> Do you mean "add", "set" or "import" rather than "save"?
>
> "save" is the term used by the C and Lua APIs. "add", "set", "import" etc.
> all imply a "save".
>
>>
>> I think this behaviour of UCI is wise, the concept is widely used stuff
>> from Juniper and Cisco.
>>
>> Try to read the NETCONF RFC in your spare time:
>> https://tools.ietf.org/html/rfc6241
>>
>> Look for concepts like "Running Configuration", "Candidate
>> Configuration", and "Testing new Configuration".
>
> I'll have a look at that as well. At a glance, there doesn't seem to be a
> nice way to support such a behaviour when there is not a single
> configurator (i.e. administrator), but both human and automatic (scripted)
> entities change configuration (transient changes may become unapplicable
> when the underlying configuration changes.)
>
> Outside the world of professional routers, I haven't really seen similar
> concepts. For most applications, transient configuration doesn't make much
> sense and mostly causes confusion. Network configuration *might* be special
> in this regard, as it may be preferable to get back to an old state of
> config after breaking something, but I'd rather handle this on a higher
> level and not in the configuration system. This also keeps the config
> system simpler.
>
>>
>>> * Comments in UCI files are possible, but get lost when the file is
>>> changed using libuci or the uci CLI. NC could support documentation of
>>> config keys in the schema.
>>
>> No strong opinions on this point.
>>
>>> This project would involve designing the configuration formats (schema
>>> files and user configuration), and implementing:
>>> * C library
>>> * CLI configuration tool
>>> * Lua binding
>>> * Some kind of shell support (for accessing NC from initscripts etc.)
>>> * Documentation for all of this
>>>
>>> I'm also thinking about adding "active defaults", i.e. default values that
>>> are not statically defined in the schema, but computed by plugin modules.
>>> This would e.g. allow Gluon packages to install complex configuration for
>>> other packages without the need to store this in the user config in /etc.
>>>
>>> I'm not totally sure about the time required to implement all of this, but
>>> I suspect I might be finished in less than the 3 months of GSoC coding
>>> time. Therefore, and to give NC some more practical testing, I'm thinking
>>> about implementing a minimal version of [4] (which was also proposed by me)
>>> based on NC. [4] is obviously a very complex project which I believe is too
>>> big to complete during the time of the GSoC, but I think minimal version to
>>> show the soundness of NC would be a nice addition to this project (for
>>> example only supporting static address/route configuration and network
>>> namespaces.)
>>>
>>> Of course, if another student wants to work on [4] (and someone wants to
>>> mentor it), I'd rather not take [4] away from them and think of something
>>> else to add to my project in case it gets finished too quickly.
>>>
>>> Regarding mentoring, I'd love to get a mentor who is not involved with
>>> Gluon yet, so he or she can make sure what I create is not only useful for
>>> Gluon, but also for the broader OpenWrt community (or even for other Linux
>>> distributions.)
>>
>> Very good idea.
>>
>>> Please let me know what you think of this proposal.
>>>
>>> Regards,
>>> Matthias
>>>
>>>
>>> [1] https://projects.universe-factory.net/projects/fastd/wiki
>>> [2] https://github.com/freifunk-gluon/gluon
>>> [3] https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html
>>> [4]
>>> https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt
>>
>> One thing it see you did not touch with this email is your initial
>> "Configuration Overlay prooposal".
>>
>> I am encountering the need to apply different layers of configurations,
>> for example:
>>
>> * local layer, the user can change this and it won't be overwritten
>> * remote/auto layer, managed remotely or automatically in some
>> fashion, the user should not change these settings locally because
>> his changes will be overwritten
>>
>> Have you also encountered this need?
>
> Yes, this is actually one of the principal points I'd like to address with
> NC, although that got a bit lost when writing my proposal.
>
> I'm not sure if it really makes sense to try to *prevent* the user from
> changing a specific setting; in my experience, users would just try harder
> and break things a layer deeper when they couldn't get their way on the
> surface ;)
>
> Regarding automatically managed configuration:
>
> It should be possible for packages to install defaults for other, already
> installed schemata. The automatic/remote config system would just replace
> the defaults on updates; if the user didn't change any options, the user
> config diff will be empty and the new defaults will become active.
>
> If the user wants to change things, this is possible, and the new defaults
> won't have an effect then.
>
> Thanks for your feedback,
>
> Matthias
>
>
>>
>> Federico
>
>
>
>
>
>
> _______________________________________________
> WLANware mailing list
> WLANware at freifunk.net
> Abonnement abbestellen? -> http://lists.freifunk.net/mailman/listinfo/wlanware-freifunk.net
>
> Weitere Infos zu den freifunk.net Mailinglisten und zur An- und Abmeldung unter http://freifunk.net/mailinglisten
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://ml.ninux.org/pipermail/ninux-dev/attachments/20160311/e0c89dc7/attachment-0001.sig>
More information about the ninux-dev
mailing list