<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Matthias,<br>
<br>
sorry for my late response, I would have liked to answer earlier
but didn't have a chance to do it properly.<br>
<br>
On 03/09/2016 05:54 AM, Matthias Schiffer wrote:<br>
</div>
<blockquote cite="mid:56DFAC8F.5010602@universe-factory.net"
type="cite">
<pre wrap="">On 03/08/2016 12:23 PM, Nemesis wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Matthias,
On 03/06/2016 10:41 PM, Matthias Schiffer wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
I've attached my proposal for the Google Summer of Code. After your last
mail, I've decided to shift the focus of my project towards the
configuration. Feedback is welcome.
</pre>
</blockquote>
<pre wrap="">
I'm glad my email was a motivating input.
I'll try to provide some useful feedback, as well as telling you what
I'm doing with OpenWISP, NetJSON and experimenting in Ninux.
I'm adding ninux-dev in CC because I believe it's good to share our
thoughts with more people and because your proposal is very interesting.
On 03/06/2016 10:31 PM, Matthias Schiffer wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
I could probably spare my introduction, as I'm not exactly a new face in
Freifunk and OpenWrt development, but as I'm interested in partaking in
this year's Google Summer of Code as a student, I'll give some more
background about myself. Also, I'm writing in English, as I know of a few
people in other, non-German community network projects that are interested
in this proposal.
I'm currently working on my Masters degree in Computer Science at the
University of Lübeck, Germany. I've been involved with the Freifunk
Lübeck project for a few years now and have become an active contributor
to OpenWrt since then. I consider myself a very expecienced C programmer
(but also have learned many other languages over the years). In the past
years I've developed the VPN daemon fastd [1] (which was also the subject
of my Bachelor thesis), and helped start the Gluon project [2], which are
both used by many community network projects like German Freifunk
communities now.
The TL;DR one-sentence version: I'd like to create an alternative to the
UCI config system, providing schema-based, structured configuration with
saner upgrade behaviour.
This proposal is concerned with one of the basic utilities of OpenWrt: the
UCI config system. Over the years, we've come to the conclusion that the
current state of UCI is not really what we need in Freifunk firmwares like
Gluon. The main issue is the upgrade behaviour:
* When setting new default configurations in a new firmware release, we
don't know if the old setting came from an older firmware's defaults, or
if a user set the value herself
* Old settings not used anymore need to be removed explicitly or will
accumulate; these explicit removal commands in upgrade script also
accumulate if skipping versions on upgrades should be supported
* UCI config files are usually installed by packages, but because of the
upgrade concept of OpenWrt, there is no sane way to define a pre/post-rm
script cleaning up and removing the config files when a package is
removed during a firmware upgrade
</pre>
</blockquote>
<pre wrap="">
A way to migrate configuration settings is indeed missing from OpenWRT
and would be welcome.
Consider the possibility of implementing a layer that keeps
compatibility with UCI to speed up adoption.
</pre>
</blockquote>
<pre wrap="">
As far as my current plan goes, this would be a one-way compatiblity layer
(generating UCI from NC). Also, a schema would need to be created for all
packages that should be supported by such a layer (as I explicitly don't
want to allow creating configuration without a matching schema; I think
gconf/gsettings work the same way.)
</pre>
</blockquote>
<br>
It's a good idea. Have you thought about the format to use?<br>
<br>
JSON-Schema is quite good (although hopefully improvable), and YAML
is quite popular nowadays.<br>
Maybe yaml and json-schema can be use together by converting YAML to
JSON before validating the content against JSON-Schema.<br>
<br>
<blockquote cite="mid:56DFAC8F.5010602@universe-factory.net"
type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">I've posted an UCI extension proposal to the openwrt-devel ML last year
[3], but didn't get much positive feedback (but enough to know that other
people have similar issues). Since then, I've moved away from the idea of
creating an API-compatible extension to UCI, and think creating an
alternative to UCI with mostly different concepts would be preferable.
Package maintainers could decide themselves if UCI or this New
Configuration system ("NC", working title) should be used for some
specific application; as proposed in some replies to [3], generating UCI
config from NC would also be a possible approach.
</pre>
</blockquote>
<pre wrap="">
The type of response you got from the OpenWRT Mailing List is pretty
standard.
I have ever hardly seen proposals for radical change welcomed and cherished.
I suggest to not let that discourage you and keep bringing up the
discussion in the
near future whenever you have interesting ideas, work to show and so on.
</pre>
<blockquote type="cite">
<pre wrap="">My current ideas are inspired by GNOME's gconf/gsettings systems; I'm also
borrowing some vocabulary from the systemd people. I'll use the following
terms to describe different types of config:
</pre>
</blockquote>
<pre wrap="">
You might attract negative attention just because you're mentioning
systemd, so beware of this.
</pre>
<blockquote type="cite">
<pre wrap="">Schema
The schema defines which config keys exist and which type the
corresponding values have. It can also define default values and
presets for these keys. Schema files lie somewhere below /lib and are
installed by packages which need configuration.
Default values
Config keys not set by the user may have default values defined by the
schema.
User configuration
User configuration is stored in a database file below /etc. Per-user
configuration in home directories is not part of this proposal (as
in OpenWrt most config is system-wide), but it would be a possible
extension.
Presets
Presets define the default values for the user configuration. This
differs from the default values: changing the default values on
upgrades will change the effective value, but changing a preset won't
(as on initialization/upgrades, non-existing values in the user
configuration will be initialised from the presets, but existing values
will be retained.)
Besides upgrades, there are some more issues of UCI I'd like to solve with
NC:
* UCI config files are designed to be both human- and machine-readable/
writable (also, parsable by shell scripts.) NC would rather use a binary
format that can be read, written (regarding performance) and stored
(regarding space) more efficiently, combined with an intuitive
configuration tool (maybe optionally curses-based)
</pre>
</blockquote>
<pre wrap="">
Are you sure this is a real advantage when compared with the costs of
developing all the necessary tools
for storing and reading the configuration on the device?
</pre>
</blockquote>
<pre wrap="">
Well, I definitely don't want the whole default configuration to be held in
/etc, but only a diff to the defaults. Currently even the simplest changes
causes the whole file to be copied to the overlay on OpenWrt; this can
easily fill the whole overlay even though most of the information is redundant.
Example: A fresh install of Gluon will use about 290KB of overlay space;
the minimum size of the overlay is 320KB (5 erase blocks), and on some
devices we're getting dangerously close to this minimum. Having all
configuration in a single file with nothing but differences from the
defaults, and in binary instead of ASCII, might reduct the overhead and
redundancy significantly (obviusly, exact numbers aren't available yet.)
Working with a diff without proper tooling would be very complicated, so
proper tools need to developed anyways.
</pre>
</blockquote>
<br>
I see.<br>
<br>
If I'm not wrong, systemd is currently doing something similar. I
think as long as there are tools to work easily with the binary
data, everything will be eventually ok.<br>
<br>
<blockquote cite="mid:56DFAC8F.5010602@universe-factory.net"
type="cite">
<blockquote type="cite"><br>
<blockquote type="cite">
<pre wrap="">* UCI configuration has a flat section/key/value structure, NC would allow
(JSON-like?) structured configuration
</pre>
</blockquote>
<pre wrap="">
JSON or even NetJSON could be good options (even YAML using a 1 to 1
conversion with the NetJSON schema would be interesting).
I've developed a library that I'm using to generate UCI configurations
from a remote web application, it's called netjsonconfig and published
several releases already:
<a class="moz-txt-link-freetext" href="http://netjsonconfig.openwisp.org/">http://netjsonconfig.openwisp.org/</a>
What I learnt from this project is that converting NetJSON to UCI is
doable and less hard than what I thought (except some annoying corner
cases). But another thing I've learnt, which might be useful to you, is
that a lot of people love UCI because its structure is simple, and after
years of working with it I believe the design choice was wise in this
regard.
Complexity brings all sorts of problems and bugs.
</pre>
</blockquote>
<pre wrap="">
JSON-like structures are also well-known and are used for various usecases.
I find some parts of the flat UCI config quite awkward (like having
"singleton" section types for which there's always exactly one section with
the type.)
Using JSON as a basis would also have the advantage that there are already
tons of tools and models available for JSON schemas, that could either be
used directly, or at least give some inspiration for NC.
</pre>
</blockquote>
<br>
Completely agree.<br>
<br>
<blockquote cite="mid:56DFAC8F.5010602@universe-factory.net"
type="cite"><br>
<blockquote type="cite">
<pre wrap="">If you want to find out more about the motivations and goals behind
netjsonconfig and the other softwares related to it, I wrote a blog post
here:
<a class="moz-txt-link-freetext" href="http://nemesisdesign.net/blog/coding/netjsonconfig-convert-netjson-to-openwrt-uci/">http://nemesisdesign.net/blog/coding/netjsonconfig-convert-netjson-to-openwrt-uci/</a>
</pre>
</blockquote>
<pre wrap="">
Thanks, I'll take a look.
</pre>
<blockquote type="cite">
<pre wrap="">
I'm experimenting these ideas in ninux too and I feel optimistic about
the outcome.
</pre>
<blockquote type="cite">
<pre wrap="">* NC would use only a single database for storage of the user configuration
(JFFS2 doesn't seem to handle lots of small files very efficiently; this
needs more research)
</pre>
</blockquote>
<pre wrap="">
What do you mean for "database" here? Filesystem? Or anything else?
</pre>
</blockquote>
<pre wrap="">
The aforementioned binary file used to store the diff between defaults and
user configuration.
</pre>
</blockquote>
<br>
Ok.<br>
<br>
<blockquote cite="mid:56DFAC8F.5010602@universe-factory.net"
type="cite">
<blockquote type="cite"><br>
<blockquote type="cite">
<pre wrap="">* UCI only stores strings. This has led to some funny inconsistencies (for
storing booleans, any of the following values are understood as true:
1/on/true/yes/enabled.) NC should natively support at least all primitive
data types of JSON.
</pre>
</blockquote>
<pre wrap="">
Good. I also dislike the inconsitencies of UCI.
</pre>
<blockquote type="cite">
<pre wrap="">* UCI separates the "save" and "commit" stages when changing configuration.
Changes that have been saved but not committed will already be effective,
but aren't stored to non-volatile memory till they are committed.
This can not only be confusing to users, but also leads to bad
programming practices like making configuration changes in scripts that
are never supposed to be committed, which is obviously fragile as other
scripts or the user may commit the config.
NC would not separate these steps: Configuration should become effective
only when it is stored permanently.
</pre>
</blockquote>
<pre wrap="">
Do you mean "add", "set" or "import" rather than "save"?
</pre>
</blockquote>
<pre wrap="">
"save" is the term used by the C and Lua APIs. "add", "set", "import" etc.
all imply a "save".
</pre>
<blockquote type="cite">
<pre wrap="">
I think this behaviour of UCI is wise, the concept is widely used stuff
from Juniper and Cisco.
Try to read the NETCONF RFC in your spare time:
<a class="moz-txt-link-freetext" href="https://tools.ietf.org/html/rfc6241">https://tools.ietf.org/html/rfc6241</a>
Look for concepts like "Running Configuration", "Candidate
Configuration", and "Testing new Configuration".
</pre>
</blockquote>
<pre wrap="">
I'll have a look at that as well. At a glance, there doesn't seem to be a
nice way to support such a behaviour when there is not a single
configurator (i.e. administrator), but both human and automatic (scripted)
entities change configuration (transient changes may become unapplicable
when the underlying configuration changes.)
Outside the world of professional routers, I haven't really seen similar
concepts. For most applications, transient configuration doesn't make much
sense and mostly causes confusion. Network configuration *might* be special
in this regard, as it may be preferable to get back to an old state of
config after breaking something, but I'd rather handle this on a higher
level and not in the configuration system. This also keeps the config
system simpler.</pre>
</blockquote>
<br>
I noticed also AirOS does something similar with its tool cfgmtd:<br>
<br>
Usage: cfgmtd [options]<br>
-t <type> - Configuration type to use
[1(active)|2(backup)]. (Default: 1(active))<br>
-f <config file> - Configuration file to use.
(Default: /tmp/system.cfg)<br>
-p <persistent directory> - Directory to persistent
dir. (Default: none)<br>
-w - Write to flash action.<br>
-r - Read from flash action.<br>
-u - Update flash action. (Default: update
configuration ONLY)<br>
-o <mtd|file name> - Use mtd or file name.
(Default: cfg)<br>
-h - This message.<br>
<br>
In <a href="https://github.com/openwisp/openwisp-config">openwisp-config</a>,
an agent that interacts with <a
href="https://github.com/openwisp/django-netjsonconfig">django-netjsonconfig</a>
(the successor of OpenWISP Manager, currently under active
development together with netjsonconfig), there's a "test
configuration" feature, that is implemented in nothing more than
shell (ash more precisely), which tests the new configuration before
making it final and does rollback in case of failure.<br>
<br>
While implementing that type of feature I would really have
preferred to have a configuration system that would have assisted me
more in doing it.<br>
<br>
Leaving the door open for a similar feature to be introduced even
tually in the future may be a good compromise.<br>
<br>
<blockquote cite="mid:56DFAC8F.5010602@universe-factory.net"
type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">* Comments in UCI files are possible, but get lost when the file is
changed using libuci or the uci CLI. NC could support documentation of
config keys in the schema.
</pre>
</blockquote>
<pre wrap="">
No strong opinions on this point.
</pre>
<blockquote type="cite">
<pre wrap="">This project would involve designing the configuration formats (schema
files and user configuration), and implementing:
* C library
* CLI configuration tool
* Lua binding
* Some kind of shell support (for accessing NC from initscripts etc.)
* Documentation for all of this
I'm also thinking about adding "active defaults", i.e. default values that
are not statically defined in the schema, but computed by plugin modules.
This would e.g. allow Gluon packages to install complex configuration for
other packages without the need to store this in the user config in /etc.
I'm not totally sure about the time required to implement all of this, but
I suspect I might be finished in less than the 3 months of GSoC coding
time. Therefore, and to give NC some more practical testing, I'm thinking
about implementing a minimal version of [4] (which was also proposed by me)
based on NC. [4] is obviously a very complex project which I believe is too
big to complete during the time of the GSoC, but I think minimal version to
show the soundness of NC would be a nice addition to this project (for
example only supporting static address/route configuration and network
namespaces.)
Of course, if another student wants to work on [4] (and someone wants to
mentor it), I'd rather not take [4] away from them and think of something
else to add to my project in case it gets finished too quickly.
Regarding mentoring, I'd love to get a mentor who is not involved with
Gluon yet, so he or she can make sure what I create is not only useful for
Gluon, but also for the broader OpenWrt community (or even for other Linux
distributions.)
</pre>
</blockquote>
<pre wrap="">
Very good idea.
</pre>
<blockquote type="cite">
<pre wrap="">Please let me know what you think of this proposal.
Regards,
Matthias
[1] <a class="moz-txt-link-freetext" href="https://projects.universe-factory.net/projects/fastd/wiki">https://projects.universe-factory.net/projects/fastd/wiki</a>
[2] <a class="moz-txt-link-freetext" href="https://github.com/freifunk-gluon/gluon">https://github.com/freifunk-gluon/gluon</a>
[3] <a class="moz-txt-link-freetext" href="https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html">https://lists.openwrt.org/pipermail/openwrt-devel/2015-March/031628.html</a>
[4]
<a class="moz-txt-link-freetext" href="https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt">https://wiki.freifunk.net/Ideas#Profile-based_network_configuration_for_OpenWrt</a>
</pre>
</blockquote>
<pre wrap="">
One thing it see you did not touch with this email is your initial
"Configuration Overlay prooposal".
I am encountering the need to apply different layers of configurations,
for example:
* local layer, the user can change this and it won't be overwritten
* remote/auto layer, managed remotely or automatically in some
fashion, the user should not change these settings locally because
his changes will be overwritten
Have you also encountered this need?
</pre>
</blockquote>
<pre wrap="">
Yes, this is actually one of the principal points I'd like to address with
NC, although that got a bit lost when writing my proposal.
I'm not sure if it really makes sense to try to *prevent* the user from
changing a specific setting; in my experience, users would just try harder
and break things a layer deeper when they couldn't get their way on the
surface ;)
Regarding automatically managed configuration:
It should be possible for packages to install defaults for other, already
installed schemata. The automatic/remote config system would just replace
the defaults on updates; if the user didn't change any options, the user
config diff will be empty and the new defaults will become active.
If the user wants to change things, this is possible, and the new defaults
won't have an effect then.
</pre>
</blockquote>
<br>
Sounds very good.<br>
<br>
<blockquote cite="mid:56DFAC8F.5010602@universe-factory.net"
type="cite">
<pre wrap="">
Thanks for your feedback,
Matthias
</pre>
</blockquote>
<br>
You are very welcome.<br>
<br>
Federico<br>
<br>
</body>
</html>