Joshua, I read the systemd article - very interesting. Would you envision a stateless DISTRO_FEATURE would enforce using systemd to take advantage of that work, or would you still allow the user to select either systemd or classic sysV init? Seems like the later would require more effort to make stateless.

(In reply to comment #1)
> Joshua, I read the systemd article - very interesting. Would you envision a
> stateless DISTRO_FEATURE would enforce using systemd to take advantage of
> that work, or would you still allow the user to select either systemd or
> classic sysV init? Seems like the later would require more effort to make
> stateless.

Certainly in the short term, to help bound this initial work, I'd assumed a systemd dependency.

For the benefit of anyone reading this issue who hasn't read the article; systemd provides mechanisms to ensure the OS remains usable when it is stateless i.e automatically populating /var, creating required users and groups when /etc/passwd and /etc/group are missing (or just missing entries) etc. which would need implementing (or at least packaging separately and init scripts written to call them) for sysvinit.

That being said, much (most?) of the work should be init manager independent; for example teaching software to load its default configuration from outside of /etc (or better yet, have sane defaults without a configuration file), so we should be able to provide this functionality on sysvinit with a smaller amount of effort once it's available for systemd.

I finally got around to looking at this in a bit more detail today. For initial stateless support I think we'll need to do at least the following:

* We still need useradd to create postinst scripts to add users and groups in /etc/[passwd|groups] to support online package management, however we should also extend the useradd mechanism for stateless systems to drop appropriate files in ${libdir}/sysusers.d, see sysusers.d(5)[1], which will then be included in the generated package's contents and run at boot by systemd-sysusers

* We should switch to installing configuration files to somewhere in /usr (i.e. set sysconfdir="${datadir}/etc" or sysconfdir="${libdir}/etc") on stateless systems and create a tmpfiles snippet, see tmpfiles.d(5)[2], to copy those files into /etc on boot

* prevent base_files from creating any files/directories in /var and /etc for stateless systems

* any files base_files create in /etc should be installed to the new sysconfdir and any that *need* to be in /etc should be copied at boot time by a tmpfiles snippet

* create tmpfiles snippet to populate the base /var

* add a QA check to ensure the /etc and /var in a constructed rootfs match expectations

* And more… The above are just the obvious changes required to boot a core-image-minimal with empty /var and /etc, likely other issues will crop up during development and testing.

This will very much be a phase 0 for stateless support, patches exist for many upstreams (either upstream or in distros who are also looking at stateless support) to correctly handle operation on a stateless system including creating required files in /var, treating /etc as user customisations and looking for default configuration elsewhere (or better yet, handle the defaults entirely in code) and we should be looking to start adopting those patches, or enabling configuration flags, etc, on a recipe-by-recipe basis.

1. https://www.freedesktop.org/software/systemd/man/sysusers.d.html
2. https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

(In reply to comment #3)
> * We should switch to installing configuration files to somewhere in /usr
> (i.e. set sysconfdir="${datadir}/etc" or sysconfdir="${libdir}/etc") on
> stateless systems and create a tmpfiles snippet, see tmpfiles.d(5)[2], to
> copy those files into /etc on boot

Just to expand a bit on this, for the phase 0 we'll probably need to do this as a postinstall step where we copy all of the files a recipe installs in /etc to a ${PN} named subdirectory of whichever OSV configuration location we pick (either {datadir}/etc or ${libdir}/etc, being consistent) and then create a tmpfiles snippet to perform the copy at boot.

This is because for the phase 0 we're not teaching any software the distinction between vendor and admin/local configurations and can't assume that software we're shipping knows to use the files we copy to /etc if we've configured that software to install into the OSV configuration location by changing the sysconfdir setting globally.

I had hoped to be able to easily support a stateless OS with a shortcut class which would:
* move a recipe's files in /etc to /usr/share/factory/etc
* install a tmpfiles.conf that would copy those files from /usr/share/factory/etc to /etc when they weren't present
* override useradd's behaviour to install sysuser config fragments rather than writing to /etc/group and /etc/passwd

However when prototyping this class I ran into a couple of issues that I haven't quite figured out how to resolve cleanly:
1) use of update-alternatives on config files (in /etc) requires that the file exist in ${PKGD}${sysconfdir} at the time update-alternatives is invoked
2) kernel module package splitting tries to write files into /etc/modprobe.d which doesn't exist at the time the splitting happens if we've moved the files to /usr/share/factory/etc

Sorry for being late to the design party - why not use a filesystem with snapshots/rollback support and sidestep the whole 'million things to fix' problem altogether?

Adding Patrick to CC as he's made some steps towards stateless in Ostro OS and may have useful input here.

One aspect of a "stateless" distro is that the /etc is still allowed to contain modifications made by the local admin. Those modifications must be supported in a way that updating the distro does not wipe out the admin's changes. The proposed "phase 0" does not achieve that because /etc remain entirely under the control of the distro. It does support the "factory reset" use case, but not seamless updating of the distro. Not sure how useful that would be.

As Joshua said, I had started with stateless support and got much of it working (as in "/etc had considerably less files") without loss of functionality, but /etc still wasn't completely empty.

My proposal is to implement "stateless" gradually by adapting more and more packages such that they are "stateless", meaning that they ship and read read-only system defaults outside of /etc and in addition, check for admin changes in /etc. Packages that do not support that can still have their config in /etc, but then admins shouldn't be allowed to modify those and a system update will be allowed to update the config files.

The alternative to "stateless" is the traditional Linux distro model: admins and packages are both allowed to modify files in /etc, which implies that some kind of three-way-merge is necessary when both admin and package have modified a file.

Just for the record, I myself don't believe that "stateless" is the silver bullet that solves that problem. It's still possible to run into problems where an admin created file in /etc becomes incompatible with an updated package, and then the admin would be the only one who can resolve that problem, because by design, the package wouldn't be allowed to modify the files in /etc. Such a situation has to be avoided by keeping the "configuration API" of a package stable.

"stateless" also doesn't work well for cases where there's simply no sane default and writing complex per-user configuration files are needed (for example, I think getting audio and a full desktop working with Clear OS requires more work from a user than in other distros - but that's based on hearsay).

This may need breaking up into smaller tasks so retitling this bug to reflect the role of performing initial investigation into stateless support. Further bugs should be filed for any required work.

Reassigning to Jussi as this work is destined for the content team.

Mariano and I are doing a proof-of-concept based on the Ostro code. See https://wiki.yoctoproject.org/wiki/Stateless

(In reply to comment #10)
> Mariano and I are doing a proof-of-concept based on the Ostro code. See
> https://wiki.yoctoproject.org/wiki/Stateless

This is cool. If you'd like to take this bug (even if only to hash out details), feel free: I just ended up with it, and mentioned back then that likely won't get to it in this release. Should have mentioned that in the bug as well, sorry about that.

My plan is to propose some of the technology shortly after 2.3 is done, but not before -> moving to 2.4 M1.

I've written about the suggested "stateless" distro feature in http://lists.openembedded.org/pipermail/openembedded-architecture/2017-July/000652.html and there is a Refkit PR with actual code here: https://github.com/intel/intel-iot-refkit/pull/233

I'm undecided whether I should consider this issue here "resolved". If further work is needed, then we probably need a new assignee and a specific plan for next steps.

(In reply to comment #13)
> I'm undecided whether I should consider this issue here "resolved". If
> further work is needed, then we probably need a new assignee and a specific
> plan for next steps.

Let's consider this done.