| Message ID | 20230818174754.988128-1-charlie.johnston@ni.com |
|---|---|
| Headers | show |
| Series | Add new packagefeed recipe class | expand |
Hello,
This seems to cause a check-layer failure with meta-mingw:
https://autobuilder.yoctoproject.org/typhoon/#/builders/39/builds/7667/steps/15/logs/stdio
AssertionError: Adding layer meta-mingw changed signatures.
2 signatures changed, initial differences (first hash before, second after):
packagefeed-core-rpmtest:do_packagefeed: 8af5d3caf528efaeaa6469c7350fc323affe0ac7684e660b66a4b1d88a5bfb72 -> 289f357ebf8cb3ef7670a2857d63640381ce812de7abe333d3fb7fb9abecbc29
bitbake-diffsigs --task packagefeed-core-rpmtest do_packagefeed --signature 8af5d3caf528efaeaa6469c7350fc323affe0ac7684e660b66a4b1d88a5bfb72 289f357ebf8cb3ef7670a2857d63640381ce812de7abe333d3fb7fb9abecbc29
NOTE: Starting bitbake server...
Taint (by forced/invalidated task) changed from nostamp(uuid4):4a1257f0-298a-43ab-ac44-3808649481a1 to nostamp(uuid4):f6871f5b-5790-4e6b-8079-db9e13bce98d
On 18/08/2023 12:44:49-0500, Charlie Johnston wrote:
> Currently, the only way to build a feed natively in OE is to build all the desired packages
> and then manually run bitbake package-index. This approach has a few drawbacks:
> - The index creation methods ONLY work on the package deploy directory. If there are
> packages that are not meant to be in the feed in the deploy directory, they will be included
> in the package index. Also, multiple feeds cannot be built in one command due to this limitation.
> - If a package feed depends on another package feed being side-by-side to it (that is, if
> packages in Feed A depend on packages in Feed B and users of Feed B are required to use Feed A) a
> user would have to manually remove duplicate packages from the deploy directory before making
> Feed B's index.
>
> To address this, this patch set creates a new packagefeed.bbclass that enables the creation of
> feeds in a new deploy directory location for feeds. The patch set updates existing logic in the
> package_manager class that was previously used to create a temporary feed when building a rootfs.
> That logic is then used to create a feed and exclude any packages which might be included in any
> provided side-by-side feeds.
>
> Note that currently the class does not make use of sstate. Since the individual packages are
> cached, there did not seem to be anything to be gained from the extra effort required to cache the
> feed. Caching the whole feed was not space efficient, and reworking package index creation to be
> compatible with caching didn't seem worth the effort given that operation is fairly inexpensive.
>
> I've tested this in oe-core by creating a few sample recipes and running with each combination of
> the PACKAGE_CLASSES variable. The testimage cases for dnf were also updated to use the new packagefeed
> creation mechanism and several images were run with testimage to confirm proper behavior.
>
> Changes since v2:
> - Added Summary to packagefeed-core-rpmtest.bb for recipe_qa.
> - Updated do_recipe_qa to not require a maintainer on packagefeed-* recipes.
>
> Changes since v1:
> - Added example of implementation to bbclass comments.
> - Updated testimage to use do_packagefeed for dnf tests and ran testimage on several images with and
> without dnf tests.
>
> Changes since RFC v2:
> - maps used to look up configurations for each package class have been updated to use nested dicts
> to clarify what each item is.
> - DEPLOY_DIR_FEED_<pkg type> definitions now include the ${PN} in the paths instead of relying on
> adding it in the bbclass.
>
> Regards,
> Charlie Johnston
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186383): https://lists.openembedded.org/g/openembedded-core/message/186383
> Mute This Topic: https://lists.openembedded.org/mt/100825496/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
poky-tiny also fails: https://autobuilder.yoctoproject.org/typhoon/#/builders/15/builds/7945/steps/12/logs/stdio ERROR: Nothing RPROVIDES 'glibc' (but /home/pokybuild/yocto-worker/poky-tiny/build/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb RDEPENDS on or otherwise requires it) glibc was skipped: incompatible with host i686-poky-linux-musl (not in COMPATIBLE_HOST) NOTE: Runtime target 'glibc' is unbuildable, removing... Missing or unbuildable dependency chain was: ['glibc'] NOTE: Runtime target 'core-image-minimal' is unbuildable, removing... Missing or unbuildable dependency chain was: ['core-image-minimal', 'packagefeed-core-rpmtest', 'glibc'] NOTE: Runtime target 'packagefeed-core-rpmtest' is unbuildable, removing... Missing or unbuildable dependency chain was: ['packagefeed-core-rpmtest', 'glibc'] ERROR: Nothing RPROVIDES 'packagefeed-core-rpmtest-dev' (but /home/pokybuild/yocto-worker/poky-tiny/build/meta/recipes-core/packagefeeds/packagefeed-core-rpmtest.bb RDEPENDS on or otherwise requires it) No eligible RPROVIDERs exist for 'packagefeed-core-rpmtest-dev' NOTE: Runtime target 'packagefeed-core-rpmtest-dev' is unbuildable, removing... Missing or unbuildable dependency chain was: ['packagefeed-core-rpmtest-dev'] There are 6 oe-selftest failures, some are related, failing because glibc is not available: 2023-08-19 17:40:07,485 - oe-selftest - INFO - RESULTS - baremetal.BaremetalTest.test_baremetal: FAILED (138.06s) 2023-08-19 17:40:07,485 - oe-selftest - INFO - RESULTS - distrodata.Distrodata.test_maintainers: FAILED (296.69s) 2023-08-19 17:40:07,485 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_no_busybox_base_utils: FAILED (73.43s) 2023-08-19 17:40:07,485 - oe-selftest - INFO - RESULTS - multiconfig.MultiConfig.test_multiconfig: FAILED (131.44s) 2023-08-19 17:40:07,485 - oe-selftest - INFO - RESULTS - sstatetests.SStateHashSameSigs3.test_sstate_sametune_samesigs: FAILED (451.37s) 2023-08-19 17:40:07,485 - oe-selftest - INFO - RESULTS - sstatetests.SStateHashSameSigs4.test_sstate_noop_samesigs: FAILED (402.88s) https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5621/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1938/steps/15/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5633/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5655/steps/14/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5579/steps/14/logs/stdio And musl builds are also failing: https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/7633/steps/11/logs/stdio https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/7655/steps/11/logs/stdio I would think the series also causes this one but I have to confirm: https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/7672/steps/26/logs/stdio On 19/08/2023 21:41:27+0200, Alexandre Belloni wrote: > Hello, > > This seems to cause a check-layer failure with meta-mingw: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/39/builds/7667/steps/15/logs/stdio > > AssertionError: Adding layer meta-mingw changed signatures. > 2 signatures changed, initial differences (first hash before, second after): > packagefeed-core-rpmtest:do_packagefeed: 8af5d3caf528efaeaa6469c7350fc323affe0ac7684e660b66a4b1d88a5bfb72 -> 289f357ebf8cb3ef7670a2857d63640381ce812de7abe333d3fb7fb9abecbc29 > bitbake-diffsigs --task packagefeed-core-rpmtest do_packagefeed --signature 8af5d3caf528efaeaa6469c7350fc323affe0ac7684e660b66a4b1d88a5bfb72 289f357ebf8cb3ef7670a2857d63640381ce812de7abe333d3fb7fb9abecbc29 > NOTE: Starting bitbake server... > Taint (by forced/invalidated task) changed from nostamp(uuid4):4a1257f0-298a-43ab-ac44-3808649481a1 to nostamp(uuid4):f6871f5b-5790-4e6b-8079-db9e13bce98d > > On 18/08/2023 12:44:49-0500, Charlie Johnston wrote: > > Currently, the only way to build a feed natively in OE is to build all the desired packages > > and then manually run bitbake package-index. This approach has a few drawbacks: > > - The index creation methods ONLY work on the package deploy directory. If there are > > packages that are not meant to be in the feed in the deploy directory, they will be included > > in the package index. Also, multiple feeds cannot be built in one command due to this limitation. > > - If a package feed depends on another package feed being side-by-side to it (that is, if > > packages in Feed A depend on packages in Feed B and users of Feed B are required to use Feed A) a > > user would have to manually remove duplicate packages from the deploy directory before making > > Feed B's index. > > > > To address this, this patch set creates a new packagefeed.bbclass that enables the creation of > > feeds in a new deploy directory location for feeds. The patch set updates existing logic in the > > package_manager class that was previously used to create a temporary feed when building a rootfs. > > That logic is then used to create a feed and exclude any packages which might be included in any > > provided side-by-side feeds. > > > > Note that currently the class does not make use of sstate. Since the individual packages are > > cached, there did not seem to be anything to be gained from the extra effort required to cache the > > feed. Caching the whole feed was not space efficient, and reworking package index creation to be > > compatible with caching didn't seem worth the effort given that operation is fairly inexpensive. > > > > I've tested this in oe-core by creating a few sample recipes and running with each combination of > > the PACKAGE_CLASSES variable. The testimage cases for dnf were also updated to use the new packagefeed > > creation mechanism and several images were run with testimage to confirm proper behavior. > > > > Changes since v2: > > - Added Summary to packagefeed-core-rpmtest.bb for recipe_qa. > > - Updated do_recipe_qa to not require a maintainer on packagefeed-* recipes. > > > > Changes since v1: > > - Added example of implementation to bbclass comments. > > - Updated testimage to use do_packagefeed for dnf tests and ran testimage on several images with and > > without dnf tests. > > > > Changes since RFC v2: > > - maps used to look up configurations for each package class have been updated to use nested dicts > > to clarify what each item is. > > - DEPLOY_DIR_FEED_<pkg type> definitions now include the ${PN} in the paths instead of relying on > > adding it in the bbclass. > > > > Regards, > > Charlie Johnston > > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#186383): https://lists.openembedded.org/g/openembedded-core/message/186383 > > Mute This Topic: https://lists.openembedded.org/mt/100825496/3617179 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
Currently, the only way to build a feed natively in OE is to build all the desired packages and then manually run bitbake package-index. This approach has a few drawbacks: - The index creation methods ONLY work on the package deploy directory. If there are packages that are not meant to be in the feed in the deploy directory, they will be included in the package index. Also, multiple feeds cannot be built in one command due to this limitation. - If a package feed depends on another package feed being side-by-side to it (that is, if packages in Feed A depend on packages in Feed B and users of Feed B are required to use Feed A) a user would have to manually remove duplicate packages from the deploy directory before making Feed B's index. To address this, this patch set creates a new packagefeed.bbclass that enables the creation of feeds in a new deploy directory location for feeds. The patch set updates existing logic in the package_manager class that was previously used to create a temporary feed when building a rootfs. That logic is then used to create a feed and exclude any packages which might be included in any provided side-by-side feeds. Note that currently the class does not make use of sstate. Since the individual packages are cached, there did not seem to be anything to be gained from the extra effort required to cache the feed. Caching the whole feed was not space efficient, and reworking package index creation to be compatible with caching didn't seem worth the effort given that operation is fairly inexpensive. I've tested this in oe-core by creating a few sample recipes and running with each combination of the PACKAGE_CLASSES variable. The testimage cases for dnf were also updated to use the new packagefeed creation mechanism and several images were run with testimage to confirm proper behavior. Changes since v2: - Added Summary to packagefeed-core-rpmtest.bb for recipe_qa. - Updated do_recipe_qa to not require a maintainer on packagefeed-* recipes. Changes since v1: - Added example of implementation to bbclass comments. - Updated testimage to use do_packagefeed for dnf tests and ran testimage on several images with and without dnf tests. Changes since RFC v2: - maps used to look up configurations for each package class have been updated to use nested dicts to clarify what each item is. - DEPLOY_DIR_FEED_<pkg type> definitions now include the ${PN} in the paths instead of relying on adding it in the bbclass. Regards, Charlie Johnston