<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Matthias,<br>
<br>
On 03/06/2016 10:41 PM, Matthias Schiffer wrote:<br>
</div>
<blockquote cite="mid:56DCA40F.50001@universe-factory.net"
type="cite">
<pre wrap="">Hi,
I've attached my proposal for the Google Summer of Code. After your last
mail, I've decided to shift the focus of my project towards the
configuration. Feedback is welcome.
</pre>
</blockquote>
<br>
I'm glad my email was a motivating input.<br>
<br>
I'll try to provide some useful feedback, as well as telling you
what I'm doing with OpenWISP, NetJSON and experimenting in Ninux.<br>
<br>
I'm adding ninux-dev in CC because I believe it's good to share our
thoughts with more people and because your proposal is very
interesting.<br>
<br>
<div class="moz-cite-prefix">On 03/06/2016 10:31 PM, Matthias
Schiffer wrote:<br>
</div>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">Hi,
I could probably spare my introduction, as I'm not exactly a new face in
Freifunk and OpenWrt development, but as I'm interested in partaking in
this year's Google Summer of Code as a student, I'll give some more
background about myself. Also, I'm writing in English, as I know of a few
people in other, non-German community network projects that are interested
in this proposal.
I'm currently working on my Masters degree in Computer Science at the
University of Lübeck, Germany. I've been involved with the Freifunk
Lübeck project for a few years now and have become an active contributor
to OpenWrt since then. I consider myself a very expecienced C programmer
(but also have learned many other languages over the years). In the past
years I've developed the VPN daemon fastd [1] (which was also the subject
of my Bachelor thesis), and helped start the Gluon project [2], which are
both used by many community network projects like German Freifunk
communities now.
The TL;DR one-sentence version: I'd like to create an alternative to the
UCI config system, providing schema-based, structured configuration with
saner upgrade behaviour.
This proposal is concerned with one of the basic utilities of OpenWrt: the
UCI config system. Over the years, we've come to the conclusion that the
current state of UCI is not really what we need in Freifunk firmwares like
Gluon. The main issue is the upgrade behaviour:
* When setting new default configurations in a new firmware release, we
don't know if the old setting came from an older firmware's defaults, or
if a user set the value herself
* Old settings not used anymore need to be removed explicitly or will
accumulate; these explicit removal commands in upgrade script also
accumulate if skipping versions on upgrades should be supported
* UCI config files are usually installed by packages, but because of the
upgrade concept of OpenWrt, there is no sane way to define a pre/post-rm
script cleaning up and removing the config files when a package is
removed during a firmware upgrade
</pre>
</blockquote>
<br>
A way to migrate configuration settings is indeed missing from
OpenWRT and would be welcome.<br>
<br>
Consider the possibility of implementing a layer that keeps
compatibility with UCI to speed up adoption.<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">I've posted an UCI extension proposal to the openwrt-devel ML last year
[3], but didn't get much positive feedback (but enough to know that other
people have similar issues). Since then, I've moved away from the idea of
creating an API-compatible extension to UCI, and think creating an
alternative to UCI with mostly different concepts would be preferable.
Package maintainers could decide themselves if UCI or this New
Configuration system ("NC", working title) should be used for some
specific application; as proposed in some replies to [3], generating UCI
config from NC would also be a possible approach.
</pre>
</blockquote>
<br>
The type of response you got from the OpenWRT Mailing List is pretty
standard.<br>
I have ever hardly seen proposals for radical change welcomed and
cherished.<br>
I suggest to not let that discourage you and keep bringing up the
discussion in the <br>
near future whenever you have interesting ideas, work to show and so
on.<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">My current ideas are inspired by GNOME's gconf/gsettings systems; I'm also
borrowing some vocabulary from the systemd people. I'll use the following
terms to describe different types of config:
</pre>
</blockquote>
<br>
You might attract negative attention just because you're mentioning
systemd, so beware of this.<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">Schema
The schema defines which config keys exist and which type the
corresponding values have. It can also define default values and
presets for these keys. Schema files lie somewhere below /lib and are
installed by packages which need configuration.
Default values
Config keys not set by the user may have default values defined by the
schema.
User configuration
User configuration is stored in a database file below /etc. Per-user
configuration in home directories is not part of this proposal (as
in OpenWrt most config is system-wide), but it would be a possible
extension.
Presets
Presets define the default values for the user configuration. This
differs from the default values: changing the default values on
upgrades will change the effective value, but changing a preset won't
(as on initialization/upgrades, non-existing values in the user
configuration will be initialised from the presets, but existing values
will be retained.)
Besides upgrades, there are some more issues of UCI I'd like to solve with
NC:
* UCI config files are designed to be both human- and machine-readable/
writable (also, parsable by shell scripts.) NC would rather use a binary
format that can be read, written (regarding performance) and stored
(regarding space) more efficiently, combined with an intuitive
configuration tool (maybe optionally curses-based)</pre>
</blockquote>
<br>
Are you sure this is a real advantage when compared with the costs
of developing all the necessary tools <br>
for storing and reading the configuration on the device?<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">* UCI configuration has a flat section/key/value structure, NC would allow
(JSON-like?) structured configuration
</pre>
</blockquote>
<br>
JSON or even NetJSON could be good options (even YAML using a 1 to 1
conversion with the NetJSON schema would be interesting).<br>
<br>
I've developed a library that I'm using to generate UCI
configurations from a remote web application, it's called
netjsonconfig and published several releases already:<br>
<a class="moz-txt-link-freetext" href="http://netjsonconfig.openwisp.org/">http://netjsonconfig.openwisp.org/</a><br>
<br>
What I learnt from this project is that converting NetJSON to UCI is
doable and less hard than what I thought (except some annoying
corner cases). But another thing I've learnt, which might be useful
to you, is that a lot of people love UCI because its structure is
simple, and after years of working with it I believe the design
choice was wise in this regard.<br>
Complexity brings all sorts of problems and bugs.<br>
<br>
If you want to find out more about the motivations and goals behind
netjsonconfig and the other softwares related to it, I wrote a blog
post here:<br>
<a class="moz-txt-link-freetext" href="http://nemesisdesign.net/blog/coding/netjsonconfig-convert-netjson-to-openwrt-uci/">http://nemesisdesign.net/blog/coding/netjsonconfig-convert-netjson-to-openwrt-uci/</a><br>
<br>
I'm experimenting these ideas in ninux too and I feel optimistic
about the outcome.<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">* NC would use only a single database for storage of the user configuration
(JFFS2 doesn't seem to handle lots of small files very efficiently; this
needs more research)</pre>
</blockquote>
<br>
What do you mean for "database" here? Filesystem? Or anything else?<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">* UCI only stores strings. This has led to some funny inconsistencies (for
storing booleans, any of the following values are understood as true:
1/on/true/yes/enabled.) NC should natively support at least all primitive
data types of JSON.</pre>
</blockquote>
<br>
Good. I also dislike the inconsitencies of UCI.<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">* UCI separates the "save" and "commit" stages when changing configuration.
Changes that have been saved but not committed will already be effective,
but aren't stored to non-volatile memory till they are committed.
This can not only be confusing to users, but also leads to bad
programming practices like making configuration changes in scripts that
are never supposed to be committed, which is obviously fragile as other
scripts or the user may commit the config.
NC would not separate these steps: Configuration should become effective
only when it is stored permanently.</pre>
</blockquote>
<br>
Do you mean "add", "set" or "import" rather than "save"?<br>
<br>
I think this behaviour of UCI is wise, the concept is widely used
stuff from Juniper and Cisco.<br>
<br>
Try to read the NETCONF RFC in your spare time:
<a class="moz-txt-link-freetext" href="https://tools.ietf.org/html/rfc6241">https://tools.ietf.org/html/rfc6241</a><br>
<br>
Look for concepts like "Running Configuration", "Candidate
Configuration", and "Testing new Configuration".<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">* Comments in UCI files are possible, but get lost when the file is
changed using libuci or the uci CLI. NC could support documentation of
config keys in the schema.
</pre>
</blockquote>
<br>
No strong opinions on this point.<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">
This project would involve designing the configuration formats (schema
files and user configuration), and implementing:
* C library
* CLI configuration tool
* Lua binding
* Some kind of shell support (for accessing NC from initscripts etc.)
* Documentation for all of this
I'm also thinking about adding "active defaults", i.e. default values that
are not statically defined in the schema, but computed by plugin modules.
This would e.g. allow Gluon packages to install complex configuration for
other packages without the need to store this in the user config in /etc.
I'm not totally sure about the time required to implement all of this, but
I suspect I might be finished in less than the 3 months of GSoC coding
time. Therefore, and to give NC some more practical testing, I'm thinking
about implementing a minimal version of [4] (which was also proposed by me)
based on NC. [4] is obviously a very complex project which I believe is too
big to complete during the time of the GSoC, but I think minimal version to
show the soundness of NC would be a nice addition to this project (for
example only supporting static address/route configuration and network
namespaces.)
Of course, if another student wants to work on [4] (and someone wants to
mentor it), I'd rather not take [4] away from them and think of something
else to add to my project in case it gets finished too quickly.
Regarding mentoring, I'd love to get a mentor who is not involved with
Gluon yet, so he or she can make sure what I create is not only useful for
Gluon, but also for the broader OpenWrt community (or even for other Linux
distributions.)</pre>
</blockquote>
<br>
Very good idea.<br>
<br>
<blockquote cite="mid:%3C56DCA19B.6030908@universe-factory.net%3E"
type="cite">
<pre wrap="">Please let me know what you think of this proposal.
Regards,
Matthias
[1] <a class="moz-txt-link-freetext" href="https://projects.universe-factory.net/projects/fastd/wiki">https://projects.universe-factory.net/projects/fastd/wiki</a>
[2] <a class="moz-txt-link-freetext" href="https://github.com/freifunk-gluon/gluon">https://github.com/freifunk-gluon/gluon</a>
[3] <a class="moz-txt-link-freetext" href="https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html">https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html</a>
[4]
<a class="moz-txt-link-freetext" href="https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt">https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt</a>
</pre>
</blockquote>
<br>
One thing it see you did not touch with this email is your initial
"Configuration Overlay prooposal".<br>
<br>
I am encountering the need to apply different layers of
configurations, for example:<br>
<ul>
<li>local layer, the user can change this and it won't be
overwritten</li>
<li>remote/auto layer, managed remotely or automatically in some
fashion, the user should not change these settings locally
because his changes will be overwritten</li>
</ul>
<p>Have you also encountered this need?<br>
</p>
<p>Federico<br>
</p>
</body>
</html>