Message ID | 20221124091045.12018-1-Qi.Chen@windriver.com |
---|---|
State | Accepted, archived |
Commit | d386fa81848247a3d407debf889db8cbcce03359 |
Headers | show |
Series | command.py: cleanup bb.cache.parse_recipe | expand |
On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote: > The parse_recipe has been removed from bb.cache, replace it with > the databuilder._parse_recipe. > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > --- > lib/bb/command.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/lib/bb/command.py b/lib/bb/command.py > index fa9fd054c..c9571f63d 100644 > --- a/lib/bb/command.py > +++ b/lib/bb/command.py > @@ -564,7 +564,7 @@ class CommandsSync: > if config_data: > # We have to use a different function here if we're passing in a datastore > # NOTE: we took a copy above, so we don't do it here again > - envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)[''] > + envdata = command.cooker.databuilder._parse_recipe(config_data, fn, appendfiles, mc)[''] > else: > # Use the standard path > envdata = command.cooker.databuilder.parseRecipe(fn, appendfiles) How did you run into that? I'm a bit worried we're missing some test coverage here? Cheers, Richard
Hi Richard, I met the error on our layerindex. Error message is as below. NOTE: Starting bitbake server... ERROR: Unable to read /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call last): File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 567, in parseRecipeFile envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)[''] AttributeError: module 'bb.cache' has no attribute 'parse_recipe' Regards, Qi -----Original Message----- From: Richard Purdie <richard.purdie@linuxfoundation.org> Sent: Thursday, November 24, 2022 8:02 PM To: Chen, Qi <Qi.Chen@windriver.com>; bitbake-devel@lists.openembedded.org Subject: Re: [bitbake-devel] [PATCH] command.py: cleanup bb.cache.parse_recipe On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote: > The parse_recipe has been removed from bb.cache, replace it with the > databuilder._parse_recipe. > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > --- > lib/bb/command.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/lib/bb/command.py b/lib/bb/command.py index > fa9fd054c..c9571f63d 100644 > --- a/lib/bb/command.py > +++ b/lib/bb/command.py > @@ -564,7 +564,7 @@ class CommandsSync: > if config_data: > # We have to use a different function here if we're passing in a datastore > # NOTE: we took a copy above, so we don't do it here again > - envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)[''] > + envdata = > + command.cooker.databuilder._parse_recipe(config_data, fn, > + appendfiles, mc)[''] > else: > # Use the standard path > envdata = command.cooker.databuilder.parseRecipe(fn, > appendfiles) How did you run into that? I'm a bit worried we're missing some test coverage here? Cheers, Richard
Hi Qi, On Thu, 2022-11-24 at 12:17 +0000, Chen, Qi wrote: > I met the error on our layerindex. > > Error message is as below. > NOTE: Starting bitbake server... > ERROR: Unable to read /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted > -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call last): > File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 90, in runCommand > result = command_method(self, commandline) > File "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", line 567, in parseRecipeFile > envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)[''] > AttributeError: module 'bb.cache' has no attribute 'parse_recipe' Thanks. I had a look at what could call parseRecipeFile(). I can see some commented out code in layerindex but it doesn't pass a config_data. The only other calls are from tinfoil's parse_recipe_file() function and none of those I can see pass config_data either. Can you see where the config_data is coming from? It looks like we've broken this at least once before: https://git.yoctoproject.org/poky/commit/bitbake/lib/bb/command.py?id=70ee928f8f4152f198f4db61517c958d54f1c0a7 so we really need to figure out what is using it and add a test to ensure coverage. Is this some local customisation you have to layerindex? Cheers, Richard
On Thu, Nov 24, 2022 at 1:17 PM Chen Qi <Qi.Chen@windriver.com> wrote: > Hi Richard, > > I met the error on our layerindex. > "our" is https://layers.openembedded.org or some other instance in windriver? I'm asking because parsing on https://layers.openembedded.org seems to be broken for a while. I've checked randomly selected 3 layers and all show the same issue as "Updates" tab in: https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/ ERROR: fatal: Unable to create '/opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue. ERROR: Traceback (most recent call last): File "update_layer.py", line 384, in main (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath, nocheckout=options.nocheckout, logger=logger) File "/opt/layerindex/layerindex/recipeparse.py", line 36, in init_parser utils.checkout_repo(bitbakepath, bitbake_ref, logger=logger) File "/opt/layerindex/layerindex/utils.py", line 282, in checkout_repo runcmd(['git', 'reset', '--hard'], repodir, logger=logger) File "/opt/layerindex/layerindex/utils.py", line 369, in runcmd raise e subprocess.CalledProcessError: Command '['git', 'reset', '--hard']' returned non-zero exit status 128. > Error message is as below. > NOTE: Starting bitbake server... > ERROR: Unable to read > /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted > -firmware-m/trusted-firmware-m_1.6.0.bb: Traceback (most recent call > last): > File > "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", > line 90, in runCommand > result = command_method(self, commandline) > File > "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", > line 567, in parseRecipeFile > envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)[''] > AttributeError: module 'bb.cache' has no attribute 'parse_recipe' > > Regards, > Qi > > -----Original Message----- > From: Richard Purdie <richard.purdie@linuxfoundation.org> > Sent: Thursday, November 24, 2022 8:02 PM > To: Chen, Qi <Qi.Chen@windriver.com>; bitbake-devel@lists.openembedded.org > Subject: Re: [bitbake-devel] [PATCH] command.py: cleanup > bb.cache.parse_recipe > > On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote: > > The parse_recipe has been removed from bb.cache, replace it with the > > databuilder._parse_recipe. > > > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com> > > --- > > lib/bb/command.py | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/lib/bb/command.py b/lib/bb/command.py index > > fa9fd054c..c9571f63d 100644 > > --- a/lib/bb/command.py > > +++ b/lib/bb/command.py > > @@ -564,7 +564,7 @@ class CommandsSync: > > if config_data: > > # We have to use a different function here if we're passing > in a datastore > > # NOTE: we took a copy above, so we don't do it here again > > - envdata = bb.cache.parse_recipe(config_data, fn, > appendfiles, mc)[''] > > + envdata = > > + command.cooker.databuilder._parse_recipe(config_data, fn, > > + appendfiles, mc)[''] > > else: > > # Use the standard path > > envdata = command.cooker.databuilder.parseRecipe(fn, > > appendfiles) > > How did you run into that? I'm a bit worried we're missing some test > coverage here? > > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14107): > https://lists.openembedded.org/g/bitbake-devel/message/14107 > Mute This Topic: https://lists.openembedded.org/mt/95234638/3617156 > Group Owner: bitbake-devel+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ > Martin.Jansa@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > >
Hi Richard and Martin, I've setup a pure oe-core layerindex (No Wind River specific settings) locally, it works well. For Richard's question, where is the config_data from, it is from layerindex-web/layerindex/update_layer.py: envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data) The error will be gone if remove "config_date=data", so we can confirm that it is from the line. For Martin, maybe a git process had crashed so that index.lock was not cleaned up, you can try to remove the lock file (Or ask layerindex maintainer to do that): $ rm -f /opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock // Robert On 11/30/22 02:20, Martin Jansa wrote: > On Thu, Nov 24, 2022 at 1:17 PM Chen Qi <Qi.Chen@windriver.com > <mailto:Qi.Chen@windriver.com>> wrote: > > Hi Richard, > > I met the error on our layerindex. > > > "our" is > https://layers.openembedded.org > <https://urldefense.com/v3/__https://layers.openembedded.org__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcmIy7OIU$> > > or some other instance in windriver? > > I'm asking because parsing on https://layers.openembedded.org > <https://urldefense.com/v3/__https://layers.openembedded.org__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcmIy7OIU$> seems to be broken for a while. I've checked randomly selected 3 layers and all show the same issue as "Updates" tab in: > https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/ <https://urldefense.com/v3/__https://layers.openembedded.org/layerindex/branch/master/layer/openembedded-core/__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcM7oxI8A$> > > ERROR: fatal: Unable to create > '/opt/workdir/git___git_openembedded_org_bitbake/.git/index.lock': File exists. > > Another git process seems to be running in this repository, e.g. > an editor opened by 'git commit'. Please make sure all processes > are terminated then try again. If it still fails, a git process > may have crashed in this repository earlier: > remove the file manually to continue. > ERROR: Traceback (most recent call last): > File "update_layer.py", line 384, in main > (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath, > nocheckout=options.nocheckout, logger=logger) > File "/opt/layerindex/layerindex/recipeparse.py", line 36, in init_parser > utils.checkout_repo(bitbakepath, bitbake_ref, logger=logger) > File "/opt/layerindex/layerindex/utils.py", line 282, in checkout_repo > runcmd(['git', 'reset', '--hard'], repodir, logger=logger) > File "/opt/layerindex/layerindex/utils.py", line 369, in runcmd > raise e > subprocess.CalledProcessError: Command '['git', 'reset', '--hard']' returned > non-zero exit status 128. > > > > Error message is as below. > NOTE: Starting bitbake server... > ERROR: Unable to read > /home/oelayer/layerindex/git___lxgit_wrs_com_layers_meta-arm/meta-arm/recipes-bsp/trusted > -firmware-m/trusted-firmware-m_1.6.0.bb > <https://urldefense.com/v3/__http://trusted-firmware-m_1.6.0.bb__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcR8tYvdQ$>: Traceback (most recent call last): > File > "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", > line 90, in runCommand > result = command_method(self, commandline) > File > "/home/oelayer/layerindex/git___lxgit_wrs_com_bitbake/lib/bb/command.py", > line 567, in parseRecipeFile > envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)[''] > AttributeError: module 'bb.cache' has no attribute 'parse_recipe' > > Regards, > Qi > > -----Original Message----- > From: Richard Purdie <richard.purdie@linuxfoundation.org > <mailto:richard.purdie@linuxfoundation.org>> > Sent: Thursday, November 24, 2022 8:02 PM > To: Chen, Qi <Qi.Chen@windriver.com <mailto:Qi.Chen@windriver.com>>; > bitbake-devel@lists.openembedded.org > <mailto:bitbake-devel@lists.openembedded.org> > Subject: Re: [bitbake-devel] [PATCH] command.py > <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$>: cleanup bb.cache.parse_recipe > > On Thu, 2022-11-24 at 01:10 -0800, Chen Qi wrote: > > The parse_recipe has been removed from bb.cache, replace it with the > > databuilder._parse_recipe. > > > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com <mailto:Qi.Chen@windriver.com>> > > --- > > lib/bb/command.py > <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/lib/bb/command.py > <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> b/lib/bb/command.py <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> index > > fa9fd054c..c9571f63d 100644 > > --- a/lib/bb/command.py > <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> > > +++ b/lib/bb/command.py > <https://urldefense.com/v3/__http://command.py__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcJ_BcC5o$> > > @@ -564,7 +564,7 @@ class CommandsSync: > > if config_data: > > # We have to use a different function here if we're passing > in a datastore > > # NOTE: we took a copy above, so we don't do it here again > > - envdata = bb.cache.parse_recipe(config_data, fn, > appendfiles, mc)[''] > > + envdata = > > + command.cooker.databuilder._parse_recipe(config_data, fn, > > + appendfiles, mc)[''] > > else: > > # Use the standard path > > envdata = command.cooker.databuilder.parseRecipe(fn, > > appendfiles) > > How did you run into that? I'm a bit worried we're missing some test > coverage here? > > Cheers, > > Richard > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#14107): > https://lists.openembedded.org/g/bitbake-devel/message/14107 > <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/14107__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcFi8osuw$> > Mute This Topic: https://lists.openembedded.org/mt/95234638/3617156 > <https://urldefense.com/v3/__https://lists.openembedded.org/mt/95234638/3617156__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYcb2VAskM$> > Group Owner: bitbake-devel+owner@lists.openembedded.org > <mailto:bitbake-devel%2Bowner@lists.openembedded.org> > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub > <https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!AjveYdw8EvQ!dAyNXqU5oP_MdnjNgg7iOIT0fNgYfCx6IJx_5jvGByabqoJrG8NnQky1xP1KOAGSW5K8aoKC-Im1stueEvYc339AjuA$> [Martin.Jansa@gmail.com <mailto:Martin.Jansa@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=- >
On Thu, 2022-12-01 at 20:10 +0800, Robert Yang wrote: > Hi Richard and Martin, > > I've setup a pure oe-core layerindex (No Wind River specific settings) locally, > it works well. > > For Richard's question, where is the config_data from, it is from > layerindex-web/layerindex/update_layer.py: > > envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data) > > The error will be gone if remove "config_date=data", so we can confirm that it > is from the line. > Thanks, that helps. I've sent a patch to the tinfoil selftests in OE- Core so we will test this piece of API in future and try and prevent regressions. Cheers, Richard
diff --git a/lib/bb/command.py b/lib/bb/command.py index fa9fd054c..c9571f63d 100644 --- a/lib/bb/command.py +++ b/lib/bb/command.py @@ -564,7 +564,7 @@ class CommandsSync: if config_data: # We have to use a different function here if we're passing in a datastore # NOTE: we took a copy above, so we don't do it here again - envdata = bb.cache.parse_recipe(config_data, fn, appendfiles, mc)[''] + envdata = command.cooker.databuilder._parse_recipe(config_data, fn, appendfiles, mc)[''] else: # Use the standard path envdata = command.cooker.databuilder.parseRecipe(fn, appendfiles)
The parse_recipe has been removed from bb.cache, replace it with the databuilder._parse_recipe. Signed-off-by: Chen Qi <Qi.Chen@windriver.com> --- lib/bb/command.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)