| Message ID | 20221205045856.68662-1-Qi.Chen@windriver.com |
|---|---|
| State | Accepted, archived |
| Commit | b25cc45c9b39f79ba0a03c4556cb2e2431677b4e |
| Headers | show |
| Series | rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work | expand |
Hi Chen, Just for your information, This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines. I am investigating it more deeply and will come back with my findings. Jose Chen Qi <Qi.Chen@windriver.com> escreveu no dia segunda, 5/12/2022 à(s) 04:59: > For now, if we use rm_work and `bitbake core-image-minimal', some > recipes' WORKDIRs are not cleaned up, e.g., makedevs-native. > > Adjust the dependency to make do_rm_work_all depend on do_rm_work > to solve this problem. > > Below are the detailed explanation of why this would work. > > Without this patch, the dependency chain is like: > [other deps] -> [do_rm_work] -+-> [do_build] > | > [do_rm_work_all] -------------+ > > With this patch, the depedency chain is like: > [other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build] > > Such dependency chain adjustment fixes the issue because do_rm_work_all > now depends on [other deps] and thus the [depends] of these [other deps]. > Take core-image-minimal as an example. Before this adjustment, > do_rm_work_all does not have any relationship with do_rootfs, and we have > do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." > This essentially prevents 'recrdeptask' setting of do_rm_work_all extend > to makedevs-native. With this patch, the do_rm_work_all now depends > on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's > recrdeptask could have effect on makedevs-native. > > With this patch, all built recipes WORKDIR will be cleaned up with > a few expected exceptions such as kernel and qemu-helper-native. > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > --- > meta/classes/rm_work.bbclass | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass > index 4121a13279..1f28bc7187 100644 > --- a/meta/classes/rm_work.bbclass > +++ b/meta/classes/rm_work.bbclass > @@ -180,7 +180,7 @@ python inject_rm_work() { > # other recipes and thus will typically run much later than > completion of > # work in the recipe itself. > # In practice, addtask() here merely updates the dependencies. > - bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) > + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' > '.join(deps), d) > > # Always update do_build_without_rm_work dependencies. > bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d) > -- > 2.37.1 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#174276): > https://lists.openembedded.org/g/openembedded-core/message/174276 > Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Hi Chen, Jose Quaresma via lists.openembedded.org <quaresma.jose= gmail.com@lists.openembedded.org> escreveu no dia sexta, 13/01/2023 à(s) 15:39: > Hi Chen, > > Just for your information, > This patch introduces a regression that breaks the do_create_spdx tasks > when building with multiconfig machines. > > I am investigating it more deeply and will come back with my findings. > I don't understand the reason for this regression but I am thinking of sending a revert for this patch. Here are the steps to reproduce: git clone https://git.yoctoproject.org/git/poky cd poky git clone https://git.yoctoproject.org/git/meta-arm git clone https://git.yoctoproject.org/git/meta-ti source oe-init-build-env bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf MACHINE=am64xx-evm bitbake core-image-minimal MACHINE=am62xx-evm bitbake core-image-minimal At least these two machines are broken. Jose > > Jose > > > Chen Qi <Qi.Chen@windriver.com> escreveu no dia segunda, 5/12/2022 à(s) > 04:59: > >> For now, if we use rm_work and `bitbake core-image-minimal', some >> recipes' WORKDIRs are not cleaned up, e.g., makedevs-native. >> >> Adjust the dependency to make do_rm_work_all depend on do_rm_work >> to solve this problem. >> >> Below are the detailed explanation of why this would work. >> >> Without this patch, the dependency chain is like: >> [other deps] -> [do_rm_work] -+-> [do_build] >> | >> [do_rm_work_all] -------------+ >> >> With this patch, the depedency chain is like: >> [other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build] >> >> Such dependency chain adjustment fixes the issue because do_rm_work_all >> now depends on [other deps] and thus the [depends] of these [other deps]. >> Take core-image-minimal as an example. Before this adjustment, >> do_rm_work_all does not have any relationship with do_rootfs, and we have >> do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." >> This essentially prevents 'recrdeptask' setting of do_rm_work_all extend >> to makedevs-native. With this patch, the do_rm_work_all now depends >> on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's >> recrdeptask could have effect on makedevs-native. >> >> With this patch, all built recipes WORKDIR will be cleaned up with >> a few expected exceptions such as kernel and qemu-helper-native. >> >> Signed-off-by: Chen Qi <Qi.Chen@windriver.com> >> --- >> meta/classes/rm_work.bbclass | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass >> index 4121a13279..1f28bc7187 100644 >> --- a/meta/classes/rm_work.bbclass >> +++ b/meta/classes/rm_work.bbclass >> @@ -180,7 +180,7 @@ python inject_rm_work() { >> # other recipes and thus will typically run much later than >> completion of >> # work in the recipe itself. >> # In practice, addtask() here merely updates the dependencies. >> - bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) >> + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' >> '.join(deps), d) >> >> # Always update do_build_without_rm_work dependencies. >> bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d) >> -- >> 2.37.1 >> >> >> >> >> > > -- > Best regards, > > José Quaresma > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#175835): > https://lists.openembedded.org/g/openembedded-core/message/175835 > Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > quaresma.jose@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
You may need to check with the layer maintainers to find out why these two layers are not working. It does not feel right if you revert a patch but cannot prove it’s wrong. Regards, Qi From: Jose Quaresma <quaresma.jose@gmail.com> Sent: Wednesday, January 18, 2023 9:10 PM To: quaresma.jose@gmail.com Cc: Chen, Qi <Qi.Chen@windriver.com>; openembedded-core@lists.openembedded.org Subject: Re: [OE-core][PATCH] rm_work: adjust dependency to make do_rm_work_all depend on do_rm_work Hi Chen, Jose Quaresma via lists.openembedded.org<https://urldefense.com/v3/__http:/lists.openembedded.org__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7ewoiqb7_$> <quaresma.jose=gmail.com@lists.openembedded.org<mailto:gmail.com@lists.openembedded.org>> escreveu no dia sexta, 13/01/2023 à(s) 15:39: Hi Chen, Just for your information, This patch introduces a regression that breaks the do_create_spdx tasks when building with multiconfig machines. I am investigating it more deeply and will come back with my findings. I don't understand the reason for this regression but I am thinking of sending a revert for this patch. Here are the steps to reproduce: git clone https://git.yoctoproject.org/git/poky<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/poky__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-cya_bF$> cd poky git clone https://git.yoctoproject.org/git/meta-arm<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/meta-arm__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-3E8fSx$> git clone https://git.yoctoproject.org/git/meta-ti<https://urldefense.com/v3/__https:/git.yoctoproject.org/git/meta-ti__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e3bU6PC9$> source oe-init-build-env bitbake-layers add-layer ../meta-arm/meta-arm-toolchain bitbake-layers add-layer ../meta-arm/meta-arm bitbake-layers add-layer ../meta-ti/meta-ti-bsp bitbake-layers add-layer ../meta-ti/meta-ti-extras echo 'INHERIT += "rm_work"' >>conf/local.conf echo 'INHERIT += "create-spdx"' >>conf/local.conf MACHINE=am64xx-evm bitbake core-image-minimal MACHINE=am62xx-evm bitbake core-image-minimal At least these two machines are broken. Jose Jose Chen Qi <Qi.Chen@windriver.com<mailto:Qi.Chen@windriver.com>> escreveu no dia segunda, 5/12/2022 à(s) 04:59: For now, if we use rm_work and `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native. Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem. Below are the detailed explanation of why this would work. Without this patch, the dependency chain is like: [other deps] -> [do_rm_work] -+-> [do_build] | [do_rm_work_all] -------------+ With this patch, the depedency chain is like: [other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build] Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native. With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native. Signed-off-by: Chen Qi <Qi.Chen@windriver.com<mailto:Qi.Chen@windriver.com>> --- meta/classes/rm_work.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass index 4121a13279..1f28bc7187 100644 --- a/meta/classes/rm_work.bbclass +++ b/meta/classes/rm_work.bbclass @@ -180,7 +180,7 @@ python inject_rm_work() { # other recipes and thus will typically run much later than completion of # work in the recipe itself. # In practice, addtask() here merely updates the dependencies. - bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' '.join(deps), d) # Always update do_build_without_rm_work dependencies. bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d) -- 2.37.1 -- Best regards, José Quaresma -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#175835): https://lists.openembedded.org/g/openembedded-core/message/175835<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/message/175835__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e-J3aJAB$> Mute This Topic: https://lists.openembedded.org/mt/95462765/5052612<https://urldefense.com/v3/__https:/lists.openembedded.org/mt/95462765/5052612__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e4RCh9cz$> Group Owner: openembedded-core+owner@lists.openembedded.org<mailto:openembedded-core%2Bowner@lists.openembedded.org> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!csH392lnV0cJ63FSfojT8DZUmync7DpN8VDX94xTuMdnDMr6kyMTjJics14-P1kGSHtIabhQJXal8ub7e4wWPBlU$> [quaresma.jose@gmail.com<mailto:quaresma.jose@gmail.com>] -=-=-=-=-=-=-=-=-=-=-=- -- Best regards, José Quaresma
On Wed, 2023-01-18 at 13:10 +0000, Jose Quaresma wrote: > Hi Chen, > > Jose Quaresma via lists.openembedded.org > <quaresma.jose=gmail.com@lists.openembedded.org> escreveu no dia > sexta, 13/01/2023 à(s) 15:39: > > Hi Chen, > > > > Just for your information, > > This patch introduces a regression that breaks the do_create_spdx > > tasks when building with multiconfig machines. > > > > I am investigating it more deeply and will come back with my > > findings. > > > > > I don't understand the reason for this regression but I am thinking > of sending a revert for this patch. > > Here are the steps to reproduce: > > git clone https://git.yoctoproject.org/git/poky > cd poky > git clone https://git.yoctoproject.org/git/meta-arm > git clone https://git.yoctoproject.org/git/meta-ti > source oe-init-build-env > bitbake-layers add-layer ../meta-arm/meta-arm-toolchain > bitbake-layers add-layer ../meta-arm/meta-arm > bitbake-layers add-layer ../meta-ti/meta-ti-bsp > bitbake-layers add-layer ../meta-ti/meta-ti-extras > echo 'INHERIT += "rm_work"' >>conf/local.conf > echo 'INHERIT += "create-spdx"' >>conf/local.conf > > MACHINE=am64xx-evm bitbake core-image-minimal > MACHINE=am62xx-evm bitbake core-image-minimal > > At least these two machines are broken. > Do you have a pointer to the actual errors that result? I'm not sure I would take a revert as I did carefully check this patch and I think it does fix a number of problems. Cheers, Richard
Richard Purdie <richard.purdie@linuxfoundation.org> escreveu no dia quarta, 18/01/2023 à(s) 13:39: > On Wed, 2023-01-18 at 13:10 +0000, Jose Quaresma wrote: > > Hi Chen, > > > > Jose Quaresma via lists.openembedded.org > > <quaresma.jose=gmail.com@lists.openembedded.org> escreveu no dia > > sexta, 13/01/2023 à(s) 15:39: > > > Hi Chen, > > > > > > Just for your information, > > > This patch introduces a regression that breaks the do_create_spdx > > > tasks when building with multiconfig machines. > > > > > > I am investigating it more deeply and will come back with my > > > findings. > > > > > > > > > I don't understand the reason for this regression but I am thinking > > of sending a revert for this patch. > > > > Here are the steps to reproduce: > > > > git clone https://git.yoctoproject.org/git/poky > > cd poky > > git clone https://git.yoctoproject.org/git/meta-arm > > git clone https://git.yoctoproject.org/git/meta-ti > > source oe-init-build-env > > bitbake-layers add-layer ../meta-arm/meta-arm-toolchain > > bitbake-layers add-layer ../meta-arm/meta-arm > > bitbake-layers add-layer ../meta-ti/meta-ti-bsp > > bitbake-layers add-layer ../meta-ti/meta-ti-extras > > echo 'INHERIT += "rm_work"' >>conf/local.conf > > echo 'INHERIT += "create-spdx"' >>conf/local.conf > > > > MACHINE=am64xx-evm bitbake core-image-minimal > > MACHINE=am62xx-evm bitbake core-image-minimal > > > > At least these two machines are broken. > > > > Do you have a pointer to the actual errors that result? > The issue can be reproduced quite fast in less than 1 minute, the error is the following: ERROR: mc:k3r5-gp:quilt-native-0.67-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_create_spdx(d) 0003: File: '/home/jq/foundries/git/poky/meta/classes/create-spdx-2.2.bbclass', lineno: 525, function: do_create_spdx 0521: recipe.packageFileName = str(recipe_archive.name) 0522: 0523: dep_recipes = collect_dep_recipes(d, doc, recipe) 0524: *** 0525: doc_sha1 = oe.sbom.write_doc(d, doc, "recipes", indent=get_json_indent(d)) 0526: dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe)) 0527: 0528: recipe_ref = oe.spdx.SPDXExternalDocumentRef() 0529: recipe_ref.externalDocumentId = "DocumentRef-recipe-" + recipe.name File: '/home/jq/foundries/git/poky/meta/lib/oe/sbom.py', lineno: 50, function: write_doc 0046: doc_sha1 = spdx_doc.to_json(f, sort_keys=True, indent=indent) 0047: 0048: l = spdx_deploy / "by-namespace" / spdx_doc.documentNamespace.replace("/", "_") 0049: l.parent.mkdir(exist_ok=True, parents=True) *** 0050: l.symlink_to(os.path.relpath(dest, l.parent)) 0051: 0052: return doc_sha1 0053: 0054: File: '/usr/lib/python3.10/pathlib.py', lineno: 1255, function: symlink_to 1251: """ 1252: Make this path a symlink pointing to the target path. 1253: Note the order of arguments (link, target) is the reverse of os.symlink. 1254: """ *** 1255: self._accessor.symlink(target, self, target_is_directory) 1256: 1257: def hardlink_to(self, target): 1258: """ 1259: Make this path a hard link pointing to the same file as *target*. Exception: FileExistsError: [Errno 17] File exists: '../recipes/recipe-quilt-native.spdx.json' -> '/home/jq/foundries/git/poky/build/tmp/work/x86_64-linux/quilt-native/0.67-r0/spdx/deploy/by-namespace/http:__spdx.org_spdxdoc_recipe-quilt-native-16737c05-483f-569e-ab20-5a7a779c8317' ERROR: Logfile of failure stored in: /home/jq/foundries/git/poky/build/tmp/work/x86_64-linux/quilt-native/0.67-r0/temp/log.do_create_spdx.1386095 ERROR: Task (mc:k3r5-gp:/home/jq/foundries/git/poky/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_create_spdx) failed with exit code '1' > > I'm not sure I would take a revert as I did carefully check this patch > and I think it does fix a number of problems. > I understand and use this patch for a bunch of other machines and it works in all of them. When writing this last sentence I remembered that I use the patch as well in some other multiconfig machines but these ones are from Xilinx that has one TMP folder for each machine. Is not the first time I have an issue with multiconfig machines that uses the same TMP folder and looks like this can be the issue with the TI machines. Jose > > Cheers, > > Richard > >
diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass index 4121a13279..1f28bc7187 100644 --- a/meta/classes/rm_work.bbclass +++ b/meta/classes/rm_work.bbclass @@ -180,7 +180,7 @@ python inject_rm_work() { # other recipes and thus will typically run much later than completion of # work in the recipe itself. # In practice, addtask() here merely updates the dependencies. - bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d) + bb.build.addtask('do_rm_work', 'do_rm_work_all do_build', ' '.join(deps), d) # Always update do_build_without_rm_work dependencies. bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)
For now, if we use rm_work and `bitbake core-image-minimal', some recipes' WORKDIRs are not cleaned up, e.g., makedevs-native. Adjust the dependency to make do_rm_work_all depend on do_rm_work to solve this problem. Below are the detailed explanation of why this would work. Without this patch, the dependency chain is like: [other deps] -> [do_rm_work] -+-> [do_build] | [do_rm_work_all] -------------+ With this patch, the depedency chain is like: [other deps] -> [do_rm_work] -> [do_rm_work_all] -> [do_build] Such dependency chain adjustment fixes the issue because do_rm_work_all now depends on [other deps] and thus the [depends] of these [other deps]. Take core-image-minimal as an example. Before this adjustment, do_rm_work_all does not have any relationship with do_rootfs, and we have do_rootfs[depends] += "makedevs-native:do_populate_sysroot ..." This essentially prevents 'recrdeptask' setting of do_rm_work_all extend to makedevs-native. With this patch, the do_rm_work_all now depends on do_rm_work which in turn depends on do_rootfs, and so do_rm_work_all's recrdeptask could have effect on makedevs-native. With this patch, all built recipes WORKDIR will be cleaned up with a few expected exceptions such as kernel and qemu-helper-native. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> --- meta/classes/rm_work.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)