[bitbake-devel,1/3] cooker.py: Save prioritized BBFILES

Submitted by Robert Yang on July 30, 2020, 9:23 a.m. | Patch ID: 174898

Details

Message ID 20200730092342.7420-1-liezhi.yang@windriver.com
State New
Headers show

Commit Message

Robert Yang July 30, 2020, 9:23 a.m.
The original code saved BBFILES back to BBFILES without changes, I think that
it was for avoiding undefined BBFILES. Now save prioritized BBFILES back which
is more usable for the one which requires it such as bb.utils.get_file_layer().

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 lib/bb/cooker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Patch hide | download patch | download mbox

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 3a58a3a33..5571e5312 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1738,10 +1738,10 @@  class CookerCollectFiles(object):
         collectlog.debug(1, "collecting .bb files")
 
         files = (config.getVar( "BBFILES") or "").split()
-        config.setVar("BBFILES", " ".join(files))
 
         # Sort files by priority
         files.sort( key=lambda fileitem: self.calc_bbfile_priority(fileitem)[0] )
+        config.setVar("BBFILES", " ".join(files))
 
         if not len(files):
             files = self.get_bbfiles()

Comments

Richard Purdie July 30, 2020, 10:37 a.m.
On Thu, 2020-07-30 at 02:23 -0700, Robert Yang wrote:
> The original code saved BBFILES back to BBFILES without changes, I think that
> it was for avoiding undefined BBFILES. Now save prioritized BBFILES back which
> is more usable for the one which requires it such as bb.utils.get_file_layer().
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  lib/bb/cooker.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 3a58a3a33..5571e5312 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1738,10 +1738,10 @@ class CookerCollectFiles(object):
>          collectlog.debug(1, "collecting .bb files")
>  
>          files = (config.getVar( "BBFILES") or "").split()
> -        config.setVar("BBFILES", " ".join(files))

I traced that back to 2007 but didn't go further. I don't think we
should be writing this value back at all. Its going to confuse anyone
trying to understand the history of the data in the variable.

>          # Sort files by priority
>          files.sort( key=lambda fileitem: self.calc_bbfile_priority(fileitem)[0] )
> +        config.setVar("BBFILES", " ".join(files))

If we want to make this information available to the wider system, we
should perhaps have bitbake API for querying these kinds of things?

Cheers,

Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11527): https://lists.openembedded.org/g/bitbake-devel/message/11527
Mute This Topic: https://lists.openembedded.org/mt/75883085/3617530
Group Owner: bitbake-devel+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Robert Yang July 30, 2020, 11:17 a.m.
On 7/30/20 6:37 PM, Richard Purdie wrote:
> On Thu, 2020-07-30 at 02:23 -0700, Robert Yang wrote:
>> The original code saved BBFILES back to BBFILES without changes, I think that
>> it was for avoiding undefined BBFILES. Now save prioritized BBFILES back which
>> is more usable for the one which requires it such as bb.utils.get_file_layer().
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   lib/bb/cooker.py | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index 3a58a3a33..5571e5312 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1738,10 +1738,10 @@ class CookerCollectFiles(object):
>>           collectlog.debug(1, "collecting .bb files")
>>   
>>           files = (config.getVar( "BBFILES") or "").split()
>> -        config.setVar("BBFILES", " ".join(files))
> 
> I traced that back to 2007 but didn't go further. I don't think we
> should be writing this value back at all. Its going to confuse anyone
> trying to understand the history of the data in the variable.
> 
>>           # Sort files by priority
>>           files.sort( key=lambda fileitem: self.calc_bbfile_priority(fileitem)[0] )
>> +        config.setVar("BBFILES", " ".join(files))
> 
> If we want to make this information available to the wider system, we
> should perhaps have bitbake API for querying these kinds of things?

This prioritized BBFILES can improve the performance, for example, if
BBFILES has no priority, the "bitbake-layers show-recipes"s' time would
increase from 40s to 110s in my testing.

Did you mean something like bb.utils.get_prioritized_bbfiles(), please?

// Robert


> 
> Cheers,
> 
> Richard
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11528): https://lists.openembedded.org/g/bitbake-devel/message/11528
Mute This Topic: https://lists.openembedded.org/mt/75883085/3617530
Group Owner: bitbake-devel+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Robert Yang July 30, 2020, 11:24 a.m.
It's strange that the cover letter was lost. I used "git send-email 000*" to
send all of the patches including cover letter, but I didn't find it in
the mailing list. So I sent it here:

Subject: [PATCH 0/3] Fix performance issues for bitbake-layers show-recipes

The ruunning time of "bitbake-layers show-recipes" increased from 40s to 14min
after the following patch:

tinfoil: Simplify remote datastore connections

The patch 2 and 3 retores the time from 14min to 40s, which saves about 95% time.

# Updated: Patch 1 also improves the performance.

// Robert

The following changes since commit 20e9df57217c5f37817653d2c3d492f2d4d37623:

   lib/oe/reproducible.py: Fix git HEAD check (2020-07-29 11:37:31 +0100)

are available in the Git repository at:

   git://git.yoctoproject.org/poky-contrib rbt/bblayers
   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rbt/bblayers

Robert Yang (3):
   cooker.py: Save prioritized BBFILES
   utils.py: get_file_layer(): Exit the loop when file is matched
   utils.py: get_file_layer(): Improve performance

  bitbake/lib/bb/cooker.py      |  2 +-
  bitbake/lib/bb/utils.py       | 15 +++++++++++----
  bitbake/lib/bblayers/query.py | 10 ++++++++--
  3 files changed, 20 insertions(+), 7 deletions(-)

// Robert

On 7/30/20 5:23 PM, Robert Yang wrote:
> This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
> 6min) in my build (more than 3000 recipes)
> 
> The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
> each recipe, and get_file_layer() compare the file with each item in BBFILES
> which makes it very time consuming when there are a lot of recipes and items in
> BBFILES. So exit when file is matched, it doesn't make sense to go on the loop
> when file is matched.
> 
> And use fnmatchcase to replace of fnmatch since the comparison should be
> case-sensitive.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   lib/bb/utils.py | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index 50032e50c..03568474d 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -1494,9 +1494,10 @@ def get_file_layer(filename, d):
>       bbfiles = (d.getVar('BBFILES') or '').split()
>       bbfilesmatch = False
>       for bbfilesentry in bbfiles:
> -        if fnmatch.fnmatch(filename, bbfilesentry):
> +        if fnmatch.fnmatchcase(filename, bbfilesentry):
>               bbfilesmatch = True
>               result = path_to_layer(bbfilesentry)
> +            break
>   
>       if not bbfilesmatch:
>           # Probably a bbclass
> 
> 
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11529): https://lists.openembedded.org/g/bitbake-devel/message/11529
Mute This Topic: https://lists.openembedded.org/mt/75884260/3617530
Group Owner: bitbake-devel+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-