Patchwork issue with file-ownership in images

login
register
mail settings
Submitter Andreas Müller
Date April 17, 2014, 9:07 a.m.
Message ID <CALbNGRSqnLj6Hr+6dL0rcyacz1pXSf+XySbYs_tx_gnq5+QfEg@mail.gmail.com>
Download mbox | patch
Permalink /patch/70631/
State New
Headers show

Comments

Andreas Müller - April 17, 2014, 9:07 a.m.
Hi,

I know it is release-panic but at least for me this is an issue
causing lots of headaches.

Symptom:
In case a recipe changes ownership of files during install, file UIDs
& GIDs are found with incorrect values in image - at least for opkg
(this is what I have tested so far).

To investigate, I created an image containing polkit with the patch
attached. Polkit was chosen because it is not working: no rules are
analyzed due to missing access rights caused by wrong ownership.

log.do_rootfs shows:
| Installing polkit (0.112-r0) to root...
| Downloading file:/home/a.mueller/tmp/oe-core-eglibc/deploy/ipk/armv7at2hf-vfp-neon/polkit_0.112-r0_armv7at2hf-vfp-neon.ipk.
...
| update_unamecache returned false for polkitd UID: 995
...
| update_unamecache returned false for polkitd UID: 995
...
| Running useradd commands...
| NOTE: Performing useradd with [--root
/home/a.mueller/tmp/oe-core-eglibc/work/overo-angstrom-linux-gnueabi/xfce4-tiny-image/1.0-r0/rootfs
--system --no-create-home --user-group --home-dir /etc/polkit-1
polkitd] and 10 times of retry
| update_unamecache returned true for  UID: 0
...

The code creating this log is:
    if (update_unamecache(tar.formated.uname))
    {
        tar_entry->uid = uid_cache;
        opkg_msg(NOTICE,"update_unamecache returned true for %s UID:
%i\n", tar.formated.uname, tar_entry->uid);
    }
    else
    {
        tar_entry->uid = strtol(tar.formated.uid, NULL, 8);
        opkg_msg(NOTICE,"update_unamecache returned false for %s UID:
%i\n", tar.formated.uname, tar_entry->uid);
    }

Conclusion:
opkg sets file ownership during unpacking ipk. At this time the
users/groups are not found in the image since preinst has not run yet.
So opkg decides to set the UID/GID which were found during install as
fallback. These IDs are coming from sysroot and are different form
those found in the image. I checked: polkit has 995 in sysroot but
image ends up with 997 for polkit.

I did some attempts to change order of operations in opkg but these
did not lead to some improvement yet - and I have no idea if other
package-managers are affected too.

So anybody around with more experience and willing to take care or give hints?

Andreas
Paul Barker - April 17, 2014, 9:46 a.m.
On 17 April 2014 10:07, Andreas Müller <schnitzeltony@googlemail.com> wrote:
> Hi,
>
> I know it is release-panic but at least for me this is an issue
> causing lots of headaches.
>
> Symptom:
> In case a recipe changes ownership of files during install, file UIDs
> & GIDs are found with incorrect values in image - at least for opkg
> (this is what I have tested so far).
>
> To investigate, I created an image containing polkit with the patch
> attached. Polkit was chosen because it is not working: no rules are
> analyzed due to missing access rights caused by wrong ownership.
>
> log.do_rootfs shows:
> | Installing polkit (0.112-r0) to root...
> | Downloading file:/home/a.mueller/tmp/oe-core-eglibc/deploy/ipk/armv7at2hf-vfp-neon/polkit_0.112-r0_armv7at2hf-vfp-neon.ipk.
> ...
> | update_unamecache returned false for polkitd UID: 995
> ...
> | update_unamecache returned false for polkitd UID: 995
> ...
> | Running useradd commands...
> | NOTE: Performing useradd with [--root
> /home/a.mueller/tmp/oe-core-eglibc/work/overo-angstrom-linux-gnueabi/xfce4-tiny-image/1.0-r0/rootfs
> --system --no-create-home --user-group --home-dir /etc/polkit-1
> polkitd] and 10 times of retry
> | update_unamecache returned true for  UID: 0
> ...
>
> The code creating this log is:
>     if (update_unamecache(tar.formated.uname))
>     {
>         tar_entry->uid = uid_cache;
>         opkg_msg(NOTICE,"update_unamecache returned true for %s UID:
> %i\n", tar.formated.uname, tar_entry->uid);
>     }
>     else
>     {
>         tar_entry->uid = strtol(tar.formated.uid, NULL, 8);
>         opkg_msg(NOTICE,"update_unamecache returned false for %s UID:
> %i\n", tar.formated.uname, tar_entry->uid);
>     }
>

As a slight aside to the immediate problem, the update_unamecache() in
opkg v0.2.x is fairly ugly and is removed in the current development
sources (which will become opkg v0.3.0). Instead, libarchive is used
for package extraction and it has a more flexible interface for uid
and gid lookup. It would certainly be possible to add options to the
next version of opkg to lookup uids & gids from a given passwd & group
file (which could be those from the sysroot). Is that something that
would be useful in the future?

> Conclusion:
> opkg sets file ownership during unpacking ipk. At this time the
> users/groups are not found in the image since preinst has not run yet.
> So opkg decides to set the UID/GID which were found during install as
> fallback. These IDs are coming from sysroot and are different form
> those found in the image. I checked: polkit has 995 in sysroot but
> image ends up with 997 for polkit.
>
> I did some attempts to change order of operations in opkg but these
> did not lead to some improvement yet - and I have no idea if other
> package-managers are affected too.
>
> So anybody around with more experience and willing to take care or give hints?

For the problem at hand, do we know if this a new issue or was it
present in the dora or dylan branches of OE?

>
> Andreas
>

Many thanks,
Andreas Müller - April 17, 2014, 11:53 a.m.
On Thu, Apr 17, 2014 at 11:46 AM, Paul Barker <paul@paulbarker.me.uk> wrote:
> On 17 April 2014 10:07, Andreas Müller <schnitzeltony@googlemail.com> wrote:
>> Hi,
>>
>> I know it is release-panic but at least for me this is an issue
>> causing lots of headaches.
>>
>> Symptom:
>> In case a recipe changes ownership of files during install, file UIDs
>> & GIDs are found with incorrect values in image - at least for opkg
>> (this is what I have tested so far).
>>
>> To investigate, I created an image containing polkit with the patch
>> attached. Polkit was chosen because it is not working: no rules are
>> analyzed due to missing access rights caused by wrong ownership.
>>
>> log.do_rootfs shows:
>> | Installing polkit (0.112-r0) to root...
>> | Downloading file:/home/a.mueller/tmp/oe-core-eglibc/deploy/ipk/armv7at2hf-vfp-neon/polkit_0.112-r0_armv7at2hf-vfp-neon.ipk.
>> ...
>> | update_unamecache returned false for polkitd UID: 995
>> ...
>> | update_unamecache returned false for polkitd UID: 995
>> ...
>> | Running useradd commands...
>> | NOTE: Performing useradd with [--root
>> /home/a.mueller/tmp/oe-core-eglibc/work/overo-angstrom-linux-gnueabi/xfce4-tiny-image/1.0-r0/rootfs
>> --system --no-create-home --user-group --home-dir /etc/polkit-1
>> polkitd] and 10 times of retry
>> | update_unamecache returned true for  UID: 0
>> ...
>>
>> The code creating this log is:
>>     if (update_unamecache(tar.formated.uname))
>>     {
>>         tar_entry->uid = uid_cache;
>>         opkg_msg(NOTICE,"update_unamecache returned true for %s UID:
>> %i\n", tar.formated.uname, tar_entry->uid);
>>     }
>>     else
>>     {
>>         tar_entry->uid = strtol(tar.formated.uid, NULL, 8);
>>         opkg_msg(NOTICE,"update_unamecache returned false for %s UID:
>> %i\n", tar.formated.uname, tar_entry->uid);
>>     }
>>
>
> As a slight aside to the immediate problem, the update_unamecache() in
> opkg v0.2.x is fairly ugly and is removed in the current development
> sources (which will become opkg v0.3.0). Instead, libarchive is used
> for package extraction and it has a more flexible interface for uid
> and gid lookup. It would certainly be possible to add options to the
> next version of opkg to lookup uids & gids from a given passwd & group
> file (which could be those from the sysroot). Is that something that
> would be useful in the future?
I think it would be useful but will not solve the problem here: For
some reason (checking upgraded files?) extracting ipk is performed
before running 'preinst' scripts. Extracting does also translation of
user/group names to GUIDs. But the IDs are not found since they were
not created yet which causes fallback to IDs at creation time
(=sysroot IDs).

As you mentioned, the extracting code (libbb) was replaced. So before
doing any further, I suggest I upgrade to current git HEAD, do some
investigation and let you know.

>
>> Conclusion:
>> opkg sets file ownership during unpacking ipk. At this time the
>> users/groups are not found in the image since preinst has not run yet.
>> So opkg decides to set the UID/GID which were found during install as
>> fallback. These IDs are coming from sysroot and are different form
>> those found in the image. I checked: polkit has 995 in sysroot but
>> image ends up with 997 for polkit.
>>
>> I did some attempts to change order of operations in opkg but these
>> did not lead to some improvement yet - and I have no idea if other
>> package-managers are affected too.
>>
>> So anybody around with more experience and willing to take care or give hints?
>
> For the problem at hand, do we know if this a new issue or was it
> present in the dora or dylan branches of OE?
I have written this here without investigating further [1] - there
were some changes which made me think the problem is fixed. Problem
with this issue: depending on the sequence in which recipes are
installed it might occure that sysroot and image share same IDs. => I
do not really have facts if releases are affected or not - sorry. But
creating images based on releases containing recipes doing chown at
install should show.

[1] http://lists.openembedded.org/pipermail/openembedded-core/2013-May/077949.html

Andreas
Andreas Müller - April 17, 2014, 5:08 p.m.
On Thu, Apr 17, 2014 at 1:53 PM, Andreas Müller
<schnitzeltony@googlemail.com> wrote:
OK I did a test with current git master.

Good news:

the sequence of running preinst-script and unpacking is correct now:

| NOTE: Performing useradd with [--root
/home/a.mueller/tmp/oe-core-eglibc/work/overo-angstrom-linux-gnueabi/xfce4-tiny-image/1.0-r0/rootfs
--system --no-create-home --user-group --home-dir /etc/polkit-1
polkitd] and 10 times of retry
...
...
| Installing polkit (0.112-r0) on root.
| Downloading file:/home/a.mueller/tmp/oe-core-eglibc/deploy/ipk/armv7at2hf-vfp-neon/polkit_0.112-r0_armv7at2hf-vfp-neon.ipk.
...

Bad news:

gid/uid are still identical to those with opkg 2.01 (based on same ipks created)

I think what happens is that opkg simply uses the ids which were found
during installation.

So now with the sequence looking solved, I understand better what you suggested:
> It would certainly be possible to add options to the
> next version of opkg to lookup uids & gids from a given passwd & group
> file (which could be those from the sysroot). Is that something that
> would be useful in the future?
Yes this would solve this issue (nit: replace 'sysroot' by rootfs or image).

enjoy Easter

Andreas
Paul Barker - April 21, 2014, 7:52 p.m.
On 21 April 2014 15:36, Paul Barker <paul@paulbarker.me.uk> wrote:
> On 21 April 2014 11:35, Andreas Müller <schnitzeltony@googlemail.com> wrote:
>> On Thu, Apr 17, 2014 at 7:08 PM, Andreas Müller
>> <schnitzeltony@googlemail.com> wrote:
>>> On Thu, Apr 17, 2014 at 1:53 PM, Andreas Müller
>>> <schnitzeltony@googlemail.com> wrote:
>>> OK I did a test with current git master.
>>>
>>> Good news:
>>>
>>> the sequence of running preinst-script and unpacking is correct now:
>>>
>>> | NOTE: Performing useradd with [--root
>>> /home/a.mueller/tmp/oe-core-eglibc/work/overo-angstrom-linux-gnueabi/xfce4-tiny-image/1.0-r0/rootfs
>>> --system --no-create-home --user-group --home-dir /etc/polkit-1
>>> polkitd] and 10 times of retry
>>> ...
>>> ...
>>> | Installing polkit (0.112-r0) on root.
>>> | Downloading file:/home/a.mueller/tmp/oe-core-eglibc/deploy/ipk/armv7at2hf-vfp-neon/polkit_0.112-r0_armv7at2hf-vfp-neon.ipk.
>>> ...
>>>
>>> Bad news:
>>>
>>> gid/uid are still identical to those with opkg 2.01 (based on same ipks created)
>>>
>>> I think what happens is that opkg simply uses the ids which were found
>>> during installation.
>>>
>>> So now with the sequence looking solved, I understand better what you suggested:
>>>> It would certainly be possible to add options to the
>>>> next version of opkg to lookup uids & gids from a given passwd & group
>>>> file (which could be those from the sysroot). Is that something that
>>>> would be useful in the future?
>>> Yes this would solve this issue (nit: replace 'sysroot' by rootfs or image).
>> I think it would be better to translate user/group names to IDs
>> generally and error out if that fails. As seen with polkit: using IDs
>> used during install/packing as fallback causes strange effects - or in
>> other words: If environment where a package is installed has not
>> prepared the required users/groups, something is wrong and the package
>> will not work as expected.
>
> Equally, non-OpenEmbedded users of opkg might want to use the
> uids/gids in packages without modification. I think the solution for
> opkg is to make things more configurable. I'll open an enhancement
> issue to that effect on the opkg bug tracker. Unsure how soon it will
> get solved as I'm very busy at the minute but I can at least write it
> down so it doesn't get forgotten.
>

http://code.google.com/p/opkg/issues/detail?id=129 has been opened to
track this for opkg.

Patch

From ab2d1c0873591104f5208ee56a0283f4594baad9 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Andreas=20M=C3=BCller?= <schnitzeltony@googlemail.com>
Date: Thu, 17 Apr 2014 10:07:14 +0200
Subject: [PATCH] opkg: add debug messages to analyse polkit's file ownership
 issue in rootfs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Signed-off-by: Andreas Müller <schnitzeltony@googlemail.com>
---
 .../opkg/0001-unarchive-c-add-debug-messages.patch | 34 ++++++++++++++++++++++
 meta/recipes-devtools/opkg/opkg_0.2.1.bb           |  2 ++
 2 files changed, 36 insertions(+)
 create mode 100644 meta/recipes-devtools/opkg/opkg/0001-unarchive-c-add-debug-messages.patch

diff --git a/meta/recipes-devtools/opkg/opkg/0001-unarchive-c-add-debug-messages.patch b/meta/recipes-devtools/opkg/opkg/0001-unarchive-c-add-debug-messages.patch
new file mode 100644
index 0000000..3625926
--- /dev/null
+++ b/meta/recipes-devtools/opkg/opkg/0001-unarchive-c-add-debug-messages.patch
@@ -0,0 +1,34 @@ 
+From 8033659f170b7f45162172cd3a761c313db809e5 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Andreas=20M=C3=BCller?= <schnitzeltony@googlemail.com>
+Date: Wed, 16 Apr 2014 17:30:31 +0200
+Subject: [PATCH] unarchive.c: add debug messages
+
+---
+ libbb/unarchive.c | 7 +++++++
+ 1 file changed, 7 insertions(+)
+
+diff --git a/libbb/unarchive.c b/libbb/unarchive.c
+index d583767..46587d5 100644
+--- a/libbb/unarchive.c
++++ b/libbb/unarchive.c
+@@ -554,10 +554,17 @@ get_header_tar(FILE *tar_stream)
+ */
+         tar_entry->mode = 07777 & strtol(tar.formated.mode, NULL, 8);
+ 
++	
+ 	if (update_unamecache(tar.formated.uname))
++	{
+ 		tar_entry->uid = uid_cache;
++		opkg_msg(NOTICE,"update_unamecache returned true for %s UID: %i\n", tar.formated.uname, tar_entry->uid);
++	}
+ 	else
++	{
+ 		tar_entry->uid = strtol(tar.formated.uid, NULL, 8);
++		opkg_msg(NOTICE,"update_unamecache returned false for %s UID: %i\n", tar.formated.uname, tar_entry->uid);
++	}
+ 	if (update_gnamecache(tar.formated.gname))
+ 		tar_entry->gid = gid_cache;
+ 	else
+-- 
+1.8.3.1
+
diff --git a/meta/recipes-devtools/opkg/opkg_0.2.1.bb b/meta/recipes-devtools/opkg/opkg_0.2.1.bb
index 09c0cca..a85acc6 100644
--- a/meta/recipes-devtools/opkg/opkg_0.2.1.bb
+++ b/meta/recipes-devtools/opkg/opkg_0.2.1.bb
@@ -6,6 +6,8 @@  SRC_URI = "http://downloads.yoctoproject.org/releases/${BPN}/${BPN}-${PV}.tar.gz
            file://opkg-configure.service \
 "
 
+SRC_URI_append_class-native = "file://0001-unarchive-c-add-debug-messages.patch"
+
 S = "${WORKDIR}/${BPN}-${PV}"
 
 SRC_URI[md5sum] = "1881d170b9dfbd7ecf0aa468cb9779c0"
-- 
1.8.3.1