Bug 12234

Summary:	GIDs in /etc/group changing
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	Jonathan Liu <net147>
Component:	core	Assignee:	Ross Burton <ross.burton>
Status:	RESOLVED NOTABUG	QA Contact:
Severity:	normal
Priority:	Undecided	CC:	juro.bystricky, meta.mr.watcher, meta.watcher
Version:	5.99
Target Milestone:	---
Hardware:	All
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	Regression (Used to work)
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Jonathan Liu 2017-10-18 12:32:13 UTC

When updating and building a systemd image multiple times, I notice in buildhistory that the group ids in /etc/group are changing for systemd-journal, lock, messagebus and netdev.

We should have fixed group ids for these groups to make builds reproducible.

--- a/etc/group
+++ b/etc/group
@@ -38,8 +38,8 @@ staff:x:50:
 games:x:60:
 shutdown:x:70:
 users:x:100:
-messagebus:x:996:
-netdev:x:997:
-systemd-journal:x:998:
-lock:x:999:
+systemd-journal:x:996:
+lock:x:997:
+messagebus:x:998:
+netdev:x:999:
 nogroup:x:65534:

Comment 1 Juro Bystricky 2017-10-18 22:51:09 UTC

Not really a bug but a feature.
There is a way to do this, using static IDs, something like:

USERADDEXTENSION = "useradd-staticids"
USERADD_ERROR_DYNAMIC ??= "warn"
USERADD_UID_TABLES += "my-passwd"
USERADD_GID_TABLES += "my-group"

Comment 2 Jonathan Liu 2017-10-18 23:17:34 UTC

Also noticed the following:
--- a/etc/passwd
+++ b/etc/passwd
@@ -15,5 +15,5 @@ backup:x:34:34:backup:/var/backups:/bin/sh
 list:x:38:38:Mailing List Manager:/var/list:/bin/sh
 irc:x:39:39:ircd:/var/run/ircd:/bin/sh
 gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
-messagebus:x:999:996::/var/lib/dbus:/bin/false
+messagebus:x:999:998::/var/lib/dbus:/bin/false
 nobody:x:65534:65534:nobody:/nonexistent:/bin/sh

These GID changes didn't happen when I was using the pyro branch so it seems like a regression to me...

Comment 3 Juro Bystricky 2017-10-18 23:48:32 UTC

I don't believe this is a regression. 
The manual says:
The default behavior of the OpenEmbedded build system for assigning uid and gid values when packages add users and groups during package install time is to add them dynamically. This works fine for programs that do not care what the values of the resulting users and groups become. In these cases, the order of the installation determines the final uid and gid values

Comment 4 Jonathan Liu 2017-10-18 23:58:29 UTC

(In reply to comment #3)
> I don't believe this is a regression. 
> The manual says:
> The default behavior of the OpenEmbedded build system for assigning uid and
> gid values when packages add users and groups during package install time is
> to add them dynamically. This works fine for programs that do not care what
> the values of the resulting users and groups become. In these cases, the
> order of the installation determines the final uid and gid values

So in rocko the order of package installation when generating images is not the same even when there are no changes to the list of packages installed?

Comment 5 Juro Bystricky 2017-10-19 14:12:59 UTC

(In reply to comment #4)
> (In reply to comment #3)
> > I don't believe this is a regression. 
> > The manual says:
> > The default behavior of the OpenEmbedded build system for assigning uid and
> > gid values when packages add users and groups during package install time is
> > to add them dynamically. This works fine for programs that do not care what
> > the values of the resulting users and groups become. In these cases, the
> > order of the installation determines the final uid and gid values
> 
> So in rocko the order of package installation when generating images is not
> the same even when there are no changes to the list of packages installed?

It is the same. Try it a few times and you will see you will eventually get the same (random order) results.

Comment 6 Ross Burton 2017-10-19 14:31:21 UTC

As Juro says this is by design, if you want static IDs then there's a class to do this.

Comment 7 Jonathan Liu 2017-10-24 23:02:17 UTC

(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > I don't believe this is a regression. 
> > > The manual says:
> > > The default behavior of the OpenEmbedded build system for assigning uid and
> > > gid values when packages add users and groups during package install time is
> > > to add them dynamically. This works fine for programs that do not care what
> > > the values of the resulting users and groups become. In these cases, the
> > > order of the installation determines the final uid and gid values
> > 
> > So in rocko the order of package installation when generating images is not
> > the same even when there are no changes to the list of packages installed?
> 
> It is the same. Try it a few times and you will see you will eventually get
> the same (random order) results.

I checked again on daisy branch with buildhistory. Over the last 695 builds (spanning about 11 months), /etc/group and /etc/passwd did not change from the initial generated files. Does that count as a regression from daisy?

Comment 8 Juro Bystricky 2017-10-24 23:27:58 UTC

While you may have gotten the same result, this was never guaranteed.

There have been a lot of work done since Daisy to improve parallel processing, which may have contributed to this as well. Another major change was switching to python3, with different sorting defaults. Having more CPUs available also exacerbates the problem...

Anyway, no matter how annoying this is, it is not a bug. If determinism  is desirable, you need to use userad-staticids.bblass. You should be able to use 
your previous /etc/group and /etc/passwd as "templates" for reproducible values. If that does not work properly, then that **is** a bug.