| Message ID | 20230802142432.2296716-1-jstephan@baylibre.com |
|---|---|
| Headers | show |
| Series | Add bblock helper scripts | expand |
When ' include conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default?
Regards,
Qi
-----Original Message-----
From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Julien Stephan
Sent: Wednesday, August 2, 2023 10:24 PM
To: openembedded-core@lists.openembedded.org
Cc: Julien Stephan <jstephan@baylibre.com>
Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scripts
Hi all,
This is v4 for bblock script.
Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
display a Note when recipe contains locked task(s)
Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist
Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the
following advantages:
* findSigInfo needs the task to be already built to get the siginfo
file, meaning we cannot lock a recipe on a fresh build
* we can now generate the signatures for all available task of a given
recipe
* Check if a given task is already locked. If so, don't duplicate
entry in bblock.conf
Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist
I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
Improvement from V1:
* Signatures are now package architecture specific meaning that if you
switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock
Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock
only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying
to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before
locking. Need to find a way to generate it if missing
V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584
My branch is available here [1]
Cheers
Julien
[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
bitbake.conf: include bblock.conf
bitbake: cooker: add a new function to retrieve task signatures
sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
scripts/bblock: add a script to lock/unlock recipes
oeqa/selftest/bblock: add self test for bblock tool
bitbake/lib/bb/command.py | 6 +
bitbake/lib/bb/cooker.py | 16 +++
bitbake/lib/bb/event.py | 8 ++
meta/conf/bitbake.conf | 1 +
meta/lib/oe/sstatesig.py | 13 +-
meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
scripts/bblock | 182 +++++++++++++++++++++++++
7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
create mode 100755 scripts/bblock
--
2.41.0
Le jeu. 3 août 2023 à 04:57, Chen, Qi <Qi.Chen@windriver.com> a écrit : > > When ' include conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default? > > Regards, > Qi > Hi Qi, This was a request from Richard on the first submission [1] : > To simplify things, I think it would be reasonable to add a bblock.inc > file unconditionally in our core configuration so that it is always > included if present, just to make the tool easier since I think this > will be a common use. Cheers Julien [1): https://lists.openembedded.org/g/openembedded-core/message/184586 > -----Original Message----- > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Julien Stephan > Sent: Wednesday, August 2, 2023 10:24 PM > To: openembedded-core@lists.openembedded.org > Cc: Julien Stephan <jstephan@baylibre.com> > Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scripts > > Hi all, > > This is v4 for bblock script. > > Improvement from v3: > * Add self test > * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to > display a Note when recipe contains locked task(s) > > Limitations: > * Silently does nothing if given task doesn't exist > * Silently does nothing when resetting a recipe that doesn't exist > > Improvement from v2: > * Add a function in bb.cooker to compute task signatures > * Replace the findSigInfo function by the new created one. This has the > following advantages: > * findSigInfo needs the task to be already built to get the siginfo > file, meaning we cannot lock a recipe on a fresh build > * we can now generate the signatures for all available task of a given > recipe > * Check if a given task is already locked. If so, don't duplicate > entry in bblock.conf > > Limitations: > * Needs to taint tasks that are locked, to display a warning > * I may be still missing some checks on user input > * Silently does nothing if given task doesn't exist > * Silently does nothing when resetting a recipe that doesn't exist > > I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases. > > Improvement from V1: > * Signatures are now package architecture specific meaning that if you > switch MACHINE, the lock sig will not be taken into account > * I added the -r option to unlock recipes > * I added a -d option to display the current bblock.conf > * Added an include directive for conf/bblock.conf inside bitbake.conf > * Added -t option to specify the tasks to lock/unlock > > Limitations: > * I may be still missing some checks on user input > * I need to find a way to get the list of tasks ( by default still lock > only the do_compile for now, unless -t is specified) > * Do not check if a particular recipe/task is already locked when trying > to add lock. So entries may appear multiple times > * We still need the signature of the tasks to be already computed before > locking. Need to find a way to generate it if missing > > V3: https://lists.openembedded.org/g/openembedded-core/message/184932 > V2: https://lists.openembedded.org/g/openembedded-core/message/184697 > V1: https://lists.openembedded.org/g/openembedded-core/message/184584 > > My branch is available here [1] > > Cheers > Julien > > [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock > Julien Stephan (5): > bitbake.conf: include bblock.conf > bitbake: cooker: add a new function to retrieve task signatures > sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK > scripts/bblock: add a script to lock/unlock recipes > oeqa/selftest/bblock: add self test for bblock tool > > bitbake/lib/bb/command.py | 6 + > bitbake/lib/bb/cooker.py | 16 +++ > bitbake/lib/bb/event.py | 8 ++ > meta/conf/bitbake.conf | 1 + > meta/lib/oe/sstatesig.py | 13 +- > meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ > scripts/bblock | 182 +++++++++++++++++++++++++ > 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py > create mode 100755 scripts/bblock > > -- > 2.41.0
Hello, this causes oe-selftest failures on the autobuilders:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
self.assertNotIn(info_message, result.output)
File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
self.fail(self._formatMessage(msg, standardMsg))
File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
raise self.failureException(msg)
AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
On 02/08/2023 16:24:27+0200, Julien Stephan wrote:
> Hi all,
>
> This is v4 for bblock script.
>
> Improvement from v3:
> * Add self test
> * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
> display a Note when recipe contains locked task(s)
>
> Limitations:
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
>
> Improvement from v2:
> * Add a function in bb.cooker to compute task signatures
> * Replace the findSigInfo function by the new created one. This has the
> following advantages:
> * findSigInfo needs the task to be already built to get the siginfo
> file, meaning we cannot lock a recipe on a fresh build
> * we can now generate the signatures for all available task of a given
> recipe
> * Check if a given task is already locked. If so, don't duplicate
> entry in bblock.conf
>
> Limitations:
> * Needs to taint tasks that are locked, to display a warning
> * I may be still missing some checks on user input
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
>
> I did some tests using qemux86-64 and qemuarm but I may be missing some
> corner cases.
>
> Improvement from V1:
> * Signatures are now package architecture specific meaning that if you
> switch MACHINE, the lock sig will not be taken into account
> * I added the -r option to unlock recipes
> * I added a -d option to display the current bblock.conf
> * Added an include directive for conf/bblock.conf inside bitbake.conf
> * Added -t option to specify the tasks to lock/unlock
>
> Limitations:
> * I may be still missing some checks on user input
> * I need to find a way to get the list of tasks ( by default still lock
> only the do_compile for now, unless -t is specified)
> * Do not check if a particular recipe/task is already locked when trying
> to add lock. So entries may appear multiple times
> * We still need the signature of the tasks to be already computed before
> locking. Need to find a way to generate it if missing
>
> V3: https://lists.openembedded.org/g/openembedded-core/message/184932
> V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> V1: https://lists.openembedded.org/g/openembedded-core/message/184584
>
> My branch is available here [1]
>
> Cheers
> Julien
>
> [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> Julien Stephan (5):
> bitbake.conf: include bblock.conf
> bitbake: cooker: add a new function to retrieve task signatures
> sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
> scripts/bblock: add a script to lock/unlock recipes
> oeqa/selftest/bblock: add self test for bblock tool
>
> bitbake/lib/bb/command.py | 6 +
> bitbake/lib/bb/cooker.py | 16 +++
> bitbake/lib/bb/event.py | 8 ++
> meta/conf/bitbake.conf | 1 +
> meta/lib/oe/sstatesig.py | 13 +-
> meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
> scripts/bblock | 182 +++++++++++++++++++++++++
> 7 files changed, 371 insertions(+), 1 deletion(-)
> create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
> create mode 100755 scripts/bblock
>
> --
> 2.41.0
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
> Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote: > Hello, this causes oe-selftest failures on the autobuilders: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) > 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL > Stderr: > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific > self.assertNotIn(info_message, result.output) > File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn > self.fail(self._formatMessage(msg, standardMsg)) > File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail > raise self.failureException(msg) > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server... > Hum, wait, I got this without the patch series. Let me investigate. > > > On 02/08/2023 16:24:27+0200, Julien Stephan wrote: > > Hi all, > > > > This is v4 for bblock script. > > > > Improvement from v3: > > * Add self test > > * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to > > display a Note when recipe contains locked task(s) > > > > Limitations: > > * Silently does nothing if given task doesn't exist > > * Silently does nothing when resetting a recipe that doesn't exist > > > > Improvement from v2: > > * Add a function in bb.cooker to compute task signatures > > * Replace the findSigInfo function by the new created one. This has the > > following advantages: > > * findSigInfo needs the task to be already built to get the siginfo > > file, meaning we cannot lock a recipe on a fresh build > > * we can now generate the signatures for all available task of a given > > recipe > > * Check if a given task is already locked. If so, don't duplicate > > entry in bblock.conf > > > > Limitations: > > * Needs to taint tasks that are locked, to display a warning > > * I may be still missing some checks on user input > > * Silently does nothing if given task doesn't exist > > * Silently does nothing when resetting a recipe that doesn't exist > > > > I did some tests using qemux86-64 and qemuarm but I may be missing some > > corner cases. > > > > Improvement from V1: > > * Signatures are now package architecture specific meaning that if you > > switch MACHINE, the lock sig will not be taken into account > > * I added the -r option to unlock recipes > > * I added a -d option to display the current bblock.conf > > * Added an include directive for conf/bblock.conf inside bitbake.conf > > * Added -t option to specify the tasks to lock/unlock > > > > Limitations: > > * I may be still missing some checks on user input > > * I need to find a way to get the list of tasks ( by default still lock > > only the do_compile for now, unless -t is specified) > > * Do not check if a particular recipe/task is already locked when trying > > to add lock. So entries may appear multiple times > > * We still need the signature of the tasks to be already computed before > > locking. Need to find a way to generate it if missing > > > > V3: https://lists.openembedded.org/g/openembedded-core/message/184932 > > V2: https://lists.openembedded.org/g/openembedded-core/message/184697 > > V1: https://lists.openembedded.org/g/openembedded-core/message/184584 > > > > My branch is available here [1] > > > > Cheers > > Julien > > > > [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock > > Julien Stephan (5): > > bitbake.conf: include bblock.conf > > bitbake: cooker: add a new function to retrieve task signatures > > sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK > > scripts/bblock: add a script to lock/unlock recipes > > oeqa/selftest/bblock: add self test for bblock tool > > > > bitbake/lib/bb/command.py | 6 + > > bitbake/lib/bb/cooker.py | 16 +++ > > bitbake/lib/bb/event.py | 8 ++ > > meta/conf/bitbake.conf | 1 + > > meta/lib/oe/sstatesig.py | 13 +- > > meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ > > scripts/bblock | 182 +++++++++++++++++++++++++ > > 7 files changed, 371 insertions(+), 1 deletion(-) > > create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py > > create mode 100755 scripts/bblock > > > > -- > > 2.41.0 > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403 > > Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179 > > Group Owner: openembedded-core+owner@lists.openembedded.org > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote: > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote: > > Hello, this causes oe-selftest failures on the autobuilders: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio > > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL > > Stderr: > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > > File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific > > self.assertNotIn(info_message, result.output) > > File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn > > self.fail(self._formatMessage(msg, standardMsg)) > > File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail > > raise self.failureException(msg) > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server... > > > > Hum, wait, I got this without the patch series. Let me investigate. Actually, I still had the series :) > > > > > > > On 02/08/2023 16:24:27+0200, Julien Stephan wrote: > > > Hi all, > > > > > > This is v4 for bblock script. > > > > > > Improvement from v3: > > > * Add self test > > > * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to > > > display a Note when recipe contains locked task(s) > > > > > > Limitations: > > > * Silently does nothing if given task doesn't exist > > > * Silently does nothing when resetting a recipe that doesn't exist > > > > > > Improvement from v2: > > > * Add a function in bb.cooker to compute task signatures > > > * Replace the findSigInfo function by the new created one. This has the > > > following advantages: > > > * findSigInfo needs the task to be already built to get the siginfo > > > file, meaning we cannot lock a recipe on a fresh build > > > * we can now generate the signatures for all available task of a given > > > recipe > > > * Check if a given task is already locked. If so, don't duplicate > > > entry in bblock.conf > > > > > > Limitations: > > > * Needs to taint tasks that are locked, to display a warning > > > * I may be still missing some checks on user input > > > * Silently does nothing if given task doesn't exist > > > * Silently does nothing when resetting a recipe that doesn't exist > > > > > > I did some tests using qemux86-64 and qemuarm but I may be missing some > > > corner cases. > > > > > > Improvement from V1: > > > * Signatures are now package architecture specific meaning that if you > > > switch MACHINE, the lock sig will not be taken into account > > > * I added the -r option to unlock recipes > > > * I added a -d option to display the current bblock.conf > > > * Added an include directive for conf/bblock.conf inside bitbake.conf > > > * Added -t option to specify the tasks to lock/unlock > > > > > > Limitations: > > > * I may be still missing some checks on user input > > > * I need to find a way to get the list of tasks ( by default still lock > > > only the do_compile for now, unless -t is specified) > > > * Do not check if a particular recipe/task is already locked when trying > > > to add lock. So entries may appear multiple times > > > * We still need the signature of the tasks to be already computed before > > > locking. Need to find a way to generate it if missing > > > > > > V3: https://lists.openembedded.org/g/openembedded-core/message/184932 > > > V2: https://lists.openembedded.org/g/openembedded-core/message/184697 > > > V1: https://lists.openembedded.org/g/openembedded-core/message/184584 > > > > > > My branch is available here [1] > > > > > > Cheers > > > Julien > > > > > > [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock > > > Julien Stephan (5): > > > bitbake.conf: include bblock.conf > > > bitbake: cooker: add a new function to retrieve task signatures > > > sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK > > > scripts/bblock: add a script to lock/unlock recipes > > > oeqa/selftest/bblock: add self test for bblock tool > > > > > > bitbake/lib/bb/command.py | 6 + > > > bitbake/lib/bb/cooker.py | 16 +++ > > > bitbake/lib/bb/event.py | 8 ++ > > > meta/conf/bitbake.conf | 1 + > > > meta/lib/oe/sstatesig.py | 13 +- > > > meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ > > > scripts/bblock | 182 +++++++++++++++++++++++++ > > > 7 files changed, 371 insertions(+), 1 deletion(-) > > > create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py > > > create mode 100755 scripts/bblock > > > > > > -- > > > 2.41.0 > > > > > > > > -=-=-=-=-=-=-=-=-=-=-=- > > > Links: You receive all messages sent to this group. > > > View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403 > > > Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179 > > > Group Owner: openembedded-core+owner@lists.openembedded.org > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com] > > > -=-=-=-=-=-=-=-=-=-=-=- > > > > > > > > > -- > > Alexandre Belloni, co-owner and COO, Bootlin > > Embedded Linux and Kernel engineering > > https://bootlin.com > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
Hi Alexandre, sorry for the late reply. I was not able to reproduce this issue on my local setup, but I did find another (similar??) bug, I fixed it, so hopefully it will also fix the one detected by the autobuilder. I updated my branch on poky-contrib (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) (with this fix and a few other ones), can you try it on autobuilder? I didn't send a newer version of the series in case the issue is still there. Please let me know if I can do anything else Cheers Julien Le ven. 11 août 2023 à 10:45, Alexandre Belloni <alexandre.belloni@bootlin.com> a écrit : > > On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote: > > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote: > > > Hello, this causes oe-selftest failures on the autobuilders: > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio > > > > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL > > > Stderr: > > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf > > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > > > File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific > > > self.assertNotIn(info_message, result.output) > > > File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn > > > self.fail(self._formatMessage(msg, standardMsg)) > > > File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail > > > raise self.failureException(msg) > > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server... > > > > > > > Hum, wait, I got this without the patch series. Let me investigate. > > Actually, I still had the series :) > Hi Alexandre, I am not able to reproduce it locally, so it is difficult to fix the issue. > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
Hello, On 20/09/2023 16:20:42+0200, Julien Stephan wrote: > Hi Alexandre, > sorry for the late reply. > > I was not able to reproduce this issue on my local setup, but I did > find another (similar??) bug, I fixed it, so hopefully it will also > fix the one detected by the autobuilder. > I updated my branch on poky-contrib > (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) > (with this fix and a few other ones), can you try it on autobuilder? I > didn't send a newer version of the series in case the issue is still > there. This was sent to the autobuilders: https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio > > Please let me know if I can do anything else > Cheers > Julien > > Le ven. 11 ao�t 2023 � 10:45, Alexandre Belloni > <alexandre.belloni@bootlin.com> a �crit : > > > > On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote: > > > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote: > > > > Hello, this causes oe-selftest failures on the autobuilders: > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio > > > > > > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL > > > > Stderr: > > > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf > > > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > > > > File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific > > > > self.assertNotIn(info_message, result.output) > > > > File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn > > > > self.fail(self._formatMessage(msg, standardMsg)) > > > > File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail > > > > raise self.failureException(msg) > > > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server... > > > > > > > > > > Hum, wait, I got this without the patch series. Let me investigate. > > > > Actually, I still had the series :) > > > > Hi Alexandre, > > I am not able to reproduce it locally, so it is difficult to fix the issue. > > > > > Alexandre Belloni, co-owner and COO, Bootlin > > Embedded Linux and Kernel engineering > > https://bootlin.com
Le jeu. 21 sept. 2023 à 15:11, Alexandre Belloni <alexandre.belloni@bootlin.com> a écrit : > > Hello, > > On 20/09/2023 16:20:42+0200, Julien Stephan wrote: > > Hi Alexandre, > > sorry for the late reply. > > > > I was not able to reproduce this issue on my local setup, but I did > > find another (similar??) bug, I fixed it, so hopefully it will also > > fix the one detected by the autobuilder. > > I updated my branch on poky-contrib > > (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) > > (with this fix and a few other ones), can you try it on autobuilder? I > > didn't send a newer version of the series in case the issue is still > > there. > > This was sent to the autobuilders: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio > Hi Alexandre, Thank you! Unfortunately I can see that it is still failing on the same test :( I may need your help here. Here is what my test is supposed to do: From the autobuilder log I can see that the machine is set to qemux86-64 so : $ bblock -d # unlock all recipes and remove build/conf/bblock.conf file $ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for qemuarm arch so block.conf file should look like this: # Generated by bblock SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info" SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}" SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>" # modify the quilt's do_compile task by adding an empty do_compile:append function $ bitbake -n quilt # for qemux86-64 machine Because quilt is supposed to be locked for qemuarm, I am not supposed to see any warning, but the log on the autobuilder says: "WARNING: The quilt:do_compile sig is computed to be d5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but the sig is locked to c9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fe in SIGGEN_LOCKEDSIGS_core2-64\"" meaning bblock.conf contains: SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig> So I can see two possible reasons for that: * MACHINE=qemuarm block quilt -t compile is not using the machine I give on the command line, but why?? * bblock.conf gets polluted by another test that also tries to lock quilt's do_compile function for qemux86-64 because of parallelism (I was not able to reproduce it locally) Do you have an idea why this test is failing? My next move would be to place all the tests in a single test to remove the parallelism issue (if that is the issue) but I would prefer to avoid wasting your time starting builds for me several times.. Also, I didn't manage to print log messages on my test, how can I do that? Cheers Julien > > > > Please let me know if I can do anything else > > Cheers > > Julien > > > > Le ven. 11 août 2023 à 10:45, Alexandre Belloni > > <alexandre.belloni@bootlin.com> a écrit : > > > > > > On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote: > > > > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote: > > > > > Hello, this causes oe-selftest failures on the autobuilders: > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio > > > > > > > > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL > > > > > Stderr: > > > > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf > > > > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > > > > > File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific > > > > > self.assertNotIn(info_message, result.output) > > > > > File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn > > > > > self.fail(self._formatMessage(msg, standardMsg)) > > > > > File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail > > > > > raise self.failureException(msg) > > > > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server... > > > > > > > > > > > > > Hum, wait, I got this without the patch series. Let me investigate. > > > > > > Actually, I still had the series :) > > > > > > > Hi Alexandre, > > > > I am not able to reproduce it locally, so it is difficult to fix the issue. > > > > > > > > > Alexandre Belloni, co-owner and COO, Bootlin > > > Embedded Linux and Kernel engineering > > > https://bootlin.com > > -- > Alexandre Belloni, co-owner and COO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
On 21/09/2023 16:24:30+0200, Julien Stephan wrote: > Le jeu. 21 sept. 2023 � 15:11, Alexandre Belloni > <alexandre.belloni@bootlin.com> a �crit : > > > > Hello, > > > > On 20/09/2023 16:20:42+0200, Julien Stephan wrote: > > > Hi Alexandre, > > > sorry for the late reply. > > > > > > I was not able to reproduce this issue on my local setup, but I did > > > find another (similar??) bug, I fixed it, so hopefully it will also > > > fix the one detected by the autobuilder. > > > I updated my branch on poky-contrib > > > (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) > > > (with this fix and a few other ones), can you try it on autobuilder? I > > > didn't send a newer version of the series in case the issue is still > > > there. > > > > This was sent to the autobuilders: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio > > > > Hi Alexandre, > > Thank you! Unfortunately I can see that it is still failing on the same test :( > I may need your help here. > > Here is what my test is supposed to do: > From the autobuilder log I can see that the machine is set to qemux86-64 so : > > $ bblock -d # unlock all recipes and remove build/conf/bblock.conf file > $ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for > qemuarm arch so block.conf file should look like this: > > # Generated by bblock > SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info" > SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}" > > SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>" > > # modify the quilt's do_compile task by adding an empty > do_compile:append function > $ bitbake -n quilt # for qemux86-64 machine > > Because quilt is supposed to be locked for qemuarm, I am not supposed > to see any warning, but the log on the autobuilder says: > "WARNING: The quilt:do_compile sig is computed to be > d5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but > the sig is locked to > c9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fe in > SIGGEN_LOCKEDSIGS_core2-64\"" > > meaning bblock.conf contains: > > SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig> > > > So I can see two possible reasons for that: > * MACHINE=qemuarm block quilt -t compile is not using the machine I > give on the command line, but why?? As I said in the call, look at this log from the build: https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is then used by the build so overriding it from the command line doesn't work. > * bblock.conf gets polluted by another test that also tries to lock > quilt's do_compile function for qemux86-64 because of parallelism (I > was not able to reproduce it locally) > > Do you have an idea why this test is failing? > My next move would be to place all the tests in a single test to > remove the parallelism issue (if that is the issue) but I would prefer > to avoid wasting your time starting builds for me several times.. > > Also, I didn't manage to print log messages on my test, how can I do that? > > Cheers > Julien > > > > > > Please let me know if I can do anything else > > > Cheers > > > Julien > > > > > > Le ven. 11 ao�t 2023 � 10:45, Alexandre Belloni > > > <alexandre.belloni@bootlin.com> a �crit : > > > > > > > > On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote: > > > > > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote: > > > > > > Hello, this causes oe-selftest failures on the autobuilders: > > > > > > > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio > > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio > > > > > > > > > > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase) > > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - ... FAIL > > > > > > Stderr: > > > > > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf > > > > > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf > > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific) > > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > > > > > > File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific > > > > > > self.assertNotIn(info_message, result.output) > > > > > > File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn > > > > > > self.fail(self._formatMessage(msg, standardMsg)) > > > > > > File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail > > > > > > raise self.failureException(msg) > > > > > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server... > > > > > > > > > > > > > > > > Hum, wait, I got this without the patch series. Let me investigate. > > > > > > > > Actually, I still had the series :) > > > > > > > > > > Hi Alexandre, > > > > > > I am not able to reproduce it locally, so it is difficult to fix the issue. > > > > > > > > > > > > > Alexandre Belloni, co-owner and COO, Bootlin > > > > Embedded Linux and Kernel engineering > > > > https://bootlin.com > > > > -- > > Alexandre Belloni, co-owner and COO, Bootlin > > Embedded Linux and Kernel engineering > > https://bootlin.com
> > > > So I can see two possible reasons for that: > > * MACHINE=qemuarm block quilt -t compile is not using the machine I > > give on the command line, but why?? > > As I said in the call, look at this log from the build: > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio > > It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is > then used by the build so overriding it from the command line doesn't > work. > Hi Alexandre, Thank you, I managed to reproduce it locally and fixed it properly. I just force pushed my poky-contrib branch (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can you start a build? If successful, I'll send the v5 today. Cheers Julien
This is started: https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5792 On 22/09/2023 10:54:56+0200, Julien Stephan wrote: > > > > > > So I can see two possible reasons for that: > > > * MACHINE=qemuarm block quilt -t compile is not using the machine I > > > give on the command line, but why?? > > > > As I said in the call, look at this log from the build: > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio > > > > It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is > > then used by the build so overriding it from the command line doesn't > > work. > > > > Hi Alexandre, > > Thank you, I managed to reproduce it locally and fixed it properly. I > just force pushed my poky-contrib branch > (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can > you start a build? If successful, I'll send the v5 today. > > Cheers > Julien
Hi all, This is v4 for bblock script. Improvement from v3: * Add self test * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to display a Note when recipe contains locked task(s) Limitations: * Silently does nothing if given task doesn't exist * Silently does nothing when resetting a recipe that doesn't exist Improvement from v2: * Add a function in bb.cooker to compute task signatures * Replace the findSigInfo function by the new created one. This has the following advantages: * findSigInfo needs the task to be already built to get the siginfo file, meaning we cannot lock a recipe on a fresh build * we can now generate the signatures for all available task of a given recipe * Check if a given task is already locked. If so, don't duplicate entry in bblock.conf Limitations: * Needs to taint tasks that are locked, to display a warning * I may be still missing some checks on user input * Silently does nothing if given task doesn't exist * Silently does nothing when resetting a recipe that doesn't exist I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases. Improvement from V1: * Signatures are now package architecture specific meaning that if you switch MACHINE, the lock sig will not be taken into account * I added the -r option to unlock recipes * I added a -d option to display the current bblock.conf * Added an include directive for conf/bblock.conf inside bitbake.conf * Added -t option to specify the tasks to lock/unlock Limitations: * I may be still missing some checks on user input * I need to find a way to get the list of tasks ( by default still lock only the do_compile for now, unless -t is specified) * Do not check if a particular recipe/task is already locked when trying to add lock. So entries may appear multiple times * We still need the signature of the tasks to be already computed before locking. Need to find a way to generate it if missing V3: https://lists.openembedded.org/g/openembedded-core/message/184932 V2: https://lists.openembedded.org/g/openembedded-core/message/184697 V1: https://lists.openembedded.org/g/openembedded-core/message/184584 My branch is available here [1] Cheers Julien [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock Julien Stephan (5): bitbake.conf: include bblock.conf bitbake: cooker: add a new function to retrieve task signatures sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK scripts/bblock: add a script to lock/unlock recipes oeqa/selftest/bblock: add self test for bblock tool bitbake/lib/bb/command.py | 6 + bitbake/lib/bb/cooker.py | 16 +++ bitbake/lib/bb/event.py | 8 ++ meta/conf/bitbake.conf | 1 + meta/lib/oe/sstatesig.py | 13 +- meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++ scripts/bblock | 182 +++++++++++++++++++++++++ 7 files changed, 371 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py create mode 100755 scripts/bblock -- 2.41.0