Patchwork [RFC,0/2] RFC: Implement deterministic uid/gid

login
register
mail settings
Submitter Mark Hatle
Date Dec. 10, 2013, 6:31 p.m.
Message ID <cover.1386700055.git.mark.hatle@windriver.com>
Download mbox
Permalink /patch/63165/
State New
Headers show

Pull-request

git://git.yoctoproject.org/poky-contrib mhatle/uidgid

Comments

Mark Hatle - Dec. 10, 2013, 6:31 p.m.
The following series implements the deterministic uid/gid setting for a
distribution.  Currently when a filesystem is generated the uid/gid values
are generally set at install time, so the install order determines what
the actual uid/gid values become.  In order to create a deterministic uid/gid
set, that still dynamically constructs the passwd/group file, we add an
option to read a special passwd/group file to allow the system to determine
the values.

It uses the existing parameters, and the values from the special passwd/group
files to reconstruct the parameter set to ensure these items are fully
defined with static values.

The first patch (01/02) is generally applicable.  It fixes a real bug in
the way the user/group adds occur today within the system.

Patch 02/02 implements the new functionality.


The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:

  libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mhatle/uidgid
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid

Mark Hatle (2):
  useradd.bbclass: Fix build time install issues
  useradd.bbclass: Add ability to select a static uid/gid automatically

 meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
 meta/conf/local.conf.sample.extended |  24 ++++
 2 files changed, 265 insertions(+), 6 deletions(-)
Mark Hatle - Jan. 9, 2014, 7:49 p.m.
I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the 
latest oe-core master.

I haven't seen any comments on this RFC yet.  Does anyone have any opinion 
either way on the code referenced here?

I'm confident patch 01/02 should be added to master.

The patch 02/02 works properly in all of my testing and does implement support 
for 'deterministic uid/gid' with dynamic passwd/group file construction during 
package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to 
'1', so it's low risk -- but it does add a fairly large chunk of code to be 
maintained over time.

--Mark

On 12/10/13, 12:31 PM, Mark Hatle wrote:
> The following series implements the deterministic uid/gid setting for a
> distribution.  Currently when a filesystem is generated the uid/gid values
> are generally set at install time, so the install order determines what
> the actual uid/gid values become.  In order to create a deterministic uid/gid
> set, that still dynamically constructs the passwd/group file, we add an
> option to read a special passwd/group file to allow the system to determine
> the values.
>
> It uses the existing parameters, and the values from the special passwd/group
> files to reconstruct the parameter set to ensure these items are fully
> defined with static values.
>
> The first patch (01/02) is generally applicable.  It fixes a real bug in
> the way the user/group adds occur today within the system.
>
> Patch 02/02 implements the new functionality.
>
>
> The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
>
>    libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib mhatle/uidgid
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
>
> Mark Hatle (2):
>    useradd.bbclass: Fix build time install issues
>    useradd.bbclass: Add ability to select a static uid/gid automatically
>
>   meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
>   meta/conf/local.conf.sample.extended |  24 ++++
>   2 files changed, 265 insertions(+), 6 deletions(-)
>
Martin Jansa - Jan. 9, 2014, 11:19 p.m.
On Thu, Jan 09, 2014 at 01:49:28PM -0600, Mark Hatle wrote:
> I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the 
> latest oe-core master.
> 
> I haven't seen any comments on this RFC yet.  Does anyone have any opinion 
> either way on the code referenced here?

I wanted to do more tests and find some smaller reproducer, but I'm
interested in this series.

What I'm seeing in our builds is that when you compare
files-in-image.txt report from independent (not reusing sstate-cache)
builds with exactly the same metadata, group owners aren't
deterministic.

Random files in image getting owned by "messagebus" group etc., these
issues went away when useradd class creating these ad-hoc groups had
parameter for fixed gid or when I've added these groups directly to
base-passwd group.master.

Is this the same problem you're addressing in this patchset?

> I'm confident patch 01/02 should be added to master.
> 
> The patch 02/02 works properly in all of my testing and does implement support 
> for 'deterministic uid/gid' with dynamic passwd/group file construction during 
> package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to 
> '1', so it's low risk -- but it does add a fairly large chunk of code to be 
> maintained over time.
> 
> --Mark
> 
> On 12/10/13, 12:31 PM, Mark Hatle wrote:
> > The following series implements the deterministic uid/gid setting for a
> > distribution.  Currently when a filesystem is generated the uid/gid values
> > are generally set at install time, so the install order determines what
> > the actual uid/gid values become.  In order to create a deterministic uid/gid
> > set, that still dynamically constructs the passwd/group file, we add an
> > option to read a special passwd/group file to allow the system to determine
> > the values.
> >
> > It uses the existing parameters, and the values from the special passwd/group
> > files to reconstruct the parameter set to ensure these items are fully
> > defined with static values.
> >
> > The first patch (01/02) is generally applicable.  It fixes a real bug in
> > the way the user/group adds occur today within the system.
> >
> > Patch 02/02 implements the new functionality.
> >
> >
> > The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
> >
> >    libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
> >
> > are available in the git repository at:
> >
> >    git://git.yoctoproject.org/poky-contrib mhatle/uidgid
> >    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
> >
> > Mark Hatle (2):
> >    useradd.bbclass: Fix build time install issues
> >    useradd.bbclass: Add ability to select a static uid/gid automatically
> >
> >   meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
> >   meta/conf/local.conf.sample.extended |  24 ++++
> >   2 files changed, 265 insertions(+), 6 deletions(-)
> >
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
Mark Hatle - Jan. 13, 2014, 7:58 p.m.
On 1/9/14, 5:19 PM, Martin Jansa wrote:
> On Thu, Jan 09, 2014 at 01:49:28PM -0600, Mark Hatle wrote:
>> I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the
>> latest oe-core master.
>>
>> I haven't seen any comments on this RFC yet.  Does anyone have any opinion
>> either way on the code referenced here?
>
> I wanted to do more tests and find some smaller reproducer, but I'm
> interested in this series.
>
> What I'm seeing in our builds is that when you compare
> files-in-image.txt report from independent (not reusing sstate-cache)
> builds with exactly the same metadata, group owners aren't
> deterministic.

Yes, this is partially to address that.  The issue is that the package install 
order itself is not deterministic.  The only thing that is, is the resulting set 
of files, modes, owner and group -names-.

Since the order the scripts are run is determines the effective uid/gid, the 
non-deterministic ordering causes these issues.

This patch set deals with that by allowing the user to specify -the- one and 
true deterministic set which is then reused.

> Random files in image getting owned by "messagebus" group etc., these
> issues went away when useradd class creating these ad-hoc groups had
> parameter for fixed gid or when I've added these groups directly to
> base-passwd group.master.
>
> Is this the same problem you're addressing in this patchset?

I have never seen a problem where specific files uname/gname are wrong.  It 
might be a related problem, but that shouldn't be happening in any case.

The -names- should always be right compared to the contents of the passwd/group 
files located within the constructed image.

I know there is an open bug where there were a few cases (in the sysroot) where 
things used a uname/gname from the default passwd/group file, but it had not yet 
been written.  So it fell back to the host's passwd/group file which had 
different numbers.  This is certainly incorrect behavior, and I think it's fixed 
by either inheriting the adduser class in those cases (even if they don't add 
users) or requiring the base-passwd in those specific package depends... but 
I've been looking for a reproducer to work though that.

--Mark

>> I'm confident patch 01/02 should be added to master.
>>
>> The patch 02/02 works properly in all of my testing and does implement support
>> for 'deterministic uid/gid' with dynamic passwd/group file construction during
>> package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to
>> '1', so it's low risk -- but it does add a fairly large chunk of code to be
>> maintained over time.
>>
>> --Mark
>>
>> On 12/10/13, 12:31 PM, Mark Hatle wrote:
>>> The following series implements the deterministic uid/gid setting for a
>>> distribution.  Currently when a filesystem is generated the uid/gid values
>>> are generally set at install time, so the install order determines what
>>> the actual uid/gid values become.  In order to create a deterministic uid/gid
>>> set, that still dynamically constructs the passwd/group file, we add an
>>> option to read a special passwd/group file to allow the system to determine
>>> the values.
>>>
>>> It uses the existing parameters, and the values from the special passwd/group
>>> files to reconstruct the parameter set to ensure these items are fully
>>> defined with static values.
>>>
>>> The first patch (01/02) is generally applicable.  It fixes a real bug in
>>> the way the user/group adds occur today within the system.
>>>
>>> Patch 02/02 implements the new functionality.
>>>
>>>
>>> The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
>>>
>>>     libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
>>>
>>> are available in the git repository at:
>>>
>>>     git://git.yoctoproject.org/poky-contrib mhatle/uidgid
>>>     http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
>>>
>>> Mark Hatle (2):
>>>     useradd.bbclass: Fix build time install issues
>>>     useradd.bbclass: Add ability to select a static uid/gid automatically
>>>
>>>    meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
>>>    meta/conf/local.conf.sample.extended |  24 ++++
>>>    2 files changed, 265 insertions(+), 6 deletions(-)
>>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
Mark Hatle - Jan. 16, 2014, 4:37 p.m.
On 1/9/14, 1:49 PM, Mark Hatle wrote:
> I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the
> latest oe-core master.
>
> I haven't seen any comments on this RFC yet.  Does anyone have any opinion
> either way on the code referenced here?
>
> I'm confident patch 01/02 should be added to master.
>
> The patch 02/02 works properly in all of my testing and does implement support
> for 'deterministic uid/gid' with dynamic passwd/group file construction during
> package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to
> '1', so it's low risk -- but it does add a fairly large chunk of code to be
> maintained over time.

I'm replying to myself on behalf of an external reviewer who is not on this 
mailing list.  They identified one item that they would like to see:

> Looking at how to override the passwd fields, it seems like it's an
> 'all-or-nothing' scenario for each user?
> E.g., if you specify a passwd line for a user, all fields has to be set
> and will be overridden?
> Look at lines like 227 and 248-252 in the patched file (I looked at
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/classes/useradd.bbclass?h=mhatle/uidgid&id=21a7d772b9aee15d16ee8e29cb00a22e5fe21aac
>
> ).
> We've discussed the possibility to have this kind of scenario where we
> only want to e.g. override the default shell for the user (that is, not
> override uid, comment, etc). Is it maybe possible to e.g. add some kind
> of special marker to fields that should be discarded in the overridden
> file (like username:#:#:#:#:#:/bin/tcsh). That way we can automatically
> track upstream changes of the user but still override the shell.

It should be pretty easy for me to generate a third commit that implements this 
type of functional.  Does anyone have any opinions on a proper marker to 
indicate re-use the dynamic value?  (I'd like to stay away from a blank, as 
there are valid reasons to set fields to blank.  They're suggesting in the above 
a single '#' statement -- does this seem reasonable?

--Mark

> --Mark
>
> On 12/10/13, 12:31 PM, Mark Hatle wrote:
>> The following series implements the deterministic uid/gid setting for a
>> distribution.  Currently when a filesystem is generated the uid/gid values
>> are generally set at install time, so the install order determines what
>> the actual uid/gid values become.  In order to create a deterministic uid/gid
>> set, that still dynamically constructs the passwd/group file, we add an
>> option to read a special passwd/group file to allow the system to determine
>> the values.
>>
>> It uses the existing parameters, and the values from the special passwd/group
>> files to reconstruct the parameter set to ensure these items are fully
>> defined with static values.
>>
>> The first patch (01/02) is generally applicable.  It fixes a real bug in
>> the way the user/group adds occur today within the system.
>>
>> Patch 02/02 implements the new functionality.
>>
>>
>> The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
>>
>>     libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
>>
>> are available in the git repository at:
>>
>>     git://git.yoctoproject.org/poky-contrib mhatle/uidgid
>>     http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
>>
>> Mark Hatle (2):
>>     useradd.bbclass: Fix build time install issues
>>     useradd.bbclass: Add ability to select a static uid/gid automatically
>>
>>    meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
>>    meta/conf/local.conf.sample.extended |  24 ++++
>>    2 files changed, 265 insertions(+), 6 deletions(-)
>>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>