Patchwork Could we build tar-replacement firstly and not parallel if tar-replacement is needed to build

login
register
mail settings
Submitter rongqing.li@windriver.com
Date Aug. 7, 2012, 5:59 a.m.
Message ID <5020AED3.1020406@windriver.com>
Download mbox | patch
Permalink /patch/34003/
State New
Headers show

Comments

rongqing.li@windriver.com - Aug. 7, 2012, 5:59 a.m.
Paste a RFC PATCH

[RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel

Compiling tar-replacement or not is decided by version of host tar,
if the host tar version is lower than 1.23, Compiling tar-replacement
is needed.

When doing popoluate tar-replacement sysroot to write the tar to
sysroot, but writing is not finished. other packages probably
use the being written tar to unzip file, which will lead to failure
and report the below error:
"bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy "

Now we compile tar-replacement firstly and not parallel to ensure
that a being written tar command will not be used.

Signed-off-by: Roy.Li <rongqing.li@windriver.com>
---
  scripts/bitbake |   14 +++++++++++++-
  1 files changed, 13 insertions(+), 1 deletions(-)

      ret=$?
      if [ "$ret" != "0" ]; then
          exit 1
Richard Purdie - Aug. 7, 2012, 7:41 a.m.
On Tue, 2012-08-07 at 13:59 +0800, Rongqing Li wrote:
> Paste a RFC PATCH
> 
> [RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
> 
> Compiling tar-replacement or not is decided by version of host tar,
> if the host tar version is lower than 1.23, Compiling tar-replacement
> is needed.
> 
> When doing popoluate tar-replacement sysroot to write the tar to
> sysroot, but writing is not finished. other packages probably
> use the being written tar to unzip file, which will lead to failure
> and report the below error:
> "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy "
> 
> Now we compile tar-replacement firstly and not parallel to ensure
> that a being written tar command will not be used.
> 
> Signed-off-by: Roy.Li <rongqing.li@windriver.com>
> ---
>   scripts/bitbake |   14 +++++++++++++-
>   1 files changed, 13 insertions(+), 1 deletions(-)
> 
> diff --git a/scripts/bitbake b/scripts/bitbake
> index 3772d82..eb6b144 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
>               fi
>           done
>       done
> -    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
> +
> +    if [ $needtar = "1" ]; then
> +       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf 
> |awk -F':' '{print $1}'`
> +	test -n "$NUM_THREAD_LINE" &&
> +	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
> +
> +	bitbake $TARTARGET -c populate_sysroot
> +
> +	test -n "$NUM_THREAD_LINE" &&
> +	sed -i 
> ''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' 
> conf/local.conf
> +    fi

We are *NOT* running sed over local.conf. What is the user used a
different configuration file for example?

If you're just building tar-native, is there a parallel race possible?
I'm not sure that there is?

For the record I hate this script and what it has become, its horrible.
It started as a workaround for pseudo-native, its becoming a dumping
ground for a whole set of nasty workarounds :(.

Cheers,

Richard
rongqing.li@windriver.com - Aug. 7, 2012, 8:02 a.m.
On 2012?08?07? 15:41, Richard Purdie wrote:
> On Tue, 2012-08-07 at 13:59 +0800, Rongqing Li wrote:
>> Paste a RFC PATCH
>>
>> [RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
>>
>> Compiling tar-replacement or not is decided by version of host tar,
>> if the host tar version is lower than 1.23, Compiling tar-replacement
>> is needed.
>>
>> When doing popoluate tar-replacement sysroot to write the tar to
>> sysroot, but writing is not finished. other packages probably
>> use the being written tar to unzip file, which will lead to failure
>> and report the below error:
>> "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
>>
>> Now we compile tar-replacement firstly and not parallel to ensure
>> that a being written tar command will not be used.
>>
>> Signed-off-by: Roy.Li <rongqing.li@windriver.com>
>> ---
>>    scripts/bitbake |   14 +++++++++++++-
>>    1 files changed, 13 insertions(+), 1 deletions(-)
>>
>> diff --git a/scripts/bitbake b/scripts/bitbake
>> index 3772d82..eb6b144 100755
>> --- a/scripts/bitbake
>> +++ b/scripts/bitbake
>> @@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
>>                fi
>>            done
>>        done
>> -    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
>> +
>> +    if [ $needtar = "1" ]; then
>> +       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
>> |awk -F':' '{print $1}'`
>> +	test -n "$NUM_THREAD_LINE" &&
>> +	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
>> +
>> +	bitbake $TARTARGET -c populate_sysroot
>> +
>> +	test -n "$NUM_THREAD_LINE" &&
>> +	sed -i
>> ''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g'
>> conf/local.conf
>> +    fi
>
> We are *NOT* running sed over local.conf. What is the user used a
> different configuration file for example?

I use the sed to disable NUM_THREAD_LINE in conf/local.conf if
NUM_THREAD_LINE has been enabled.

After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE
in conf/local.conf



> If you're just building tar-native, is there a parallel race possible?
> I'm not sure that there is?

Even if I just build tar-native, I still have several packages which
are needed to building.


$bitbake tar-replacement-native -g

$ cat pn-buildlist
autoconf-native
libtool-native
m4-native
gettext-minimal-native
tar-replacement-native
quilt-native
gnu-config-native
automake-native


Thanks for your reply.
-Roy
>
> For the record I hate this script and what it has become, its horrible.
> It started as a workaround for pseudo-native, its becoming a dumping
> ground for a whole set of nasty workarounds :(.
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
xin.ouyang@windriver.com - Aug. 7, 2012, 8:57 a.m.
? 2012?08?07? 16:02, Rongqing Li ??:
>
>
> On 2012?08?07? 15:41, Richard Purdie wrote:
>> On Tue, 2012-08-07 at 13:59 +0800, Rongqing Li wrote:
>>> Paste a RFC PATCH
>>>
>>> [RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not
>>> parallel
>>>
>>> Compiling tar-replacement or not is decided by version of host tar,
>>> if the host tar version is lower than 1.23, Compiling tar-replacement
>>> is needed.
>>>
>>> When doing popoluate tar-replacement sysroot to write the tar to
>>> sysroot, but writing is not finished. other packages probably
>>> use the being written tar to unzip file, which will lead to failure
>>> and report the below error:
>>> "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
>>>
>>> Now we compile tar-replacement firstly and not parallel to ensure
>>> that a being written tar command will not be used.
>>>
>>> Signed-off-by: Roy.Li <rongqing.li@windriver.com>
>>> ---
>>>    scripts/bitbake |   14 +++++++++++++-
>>>    1 files changed, 13 insertions(+), 1 deletions(-)
>>>
>>> diff --git a/scripts/bitbake b/scripts/bitbake
>>> index 3772d82..eb6b144 100755
>>> --- a/scripts/bitbake
>>> +++ b/scripts/bitbake
>>> @@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
>>>                fi
>>>            done
>>>        done
>>> -    bitbake pseudo-native $TARTARGET $additionalopts -c
>>> populate_sysroot
>>> +
>>> +    if [ $needtar = "1" ]; then
>>> +       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
>>> |awk -F':' '{print $1}'`
>>> +    test -n "$NUM_THREAD_LINE" &&
>>> +    sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g'
>>> conf/local.conf
>>> +
>>> +    bitbake $TARTARGET -c populate_sysroot
>>> +
>>> +    test -n "$NUM_THREAD_LINE" &&
>>> +    sed -i
>>> ''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g'
>>> conf/local.conf
>>> +    fi
>>
>> We are *NOT* running sed over local.conf. What is the user used a
>> different configuration file for example?
>
> I use the sed to disable NUM_THREAD_LINE in conf/local.conf if
> NUM_THREAD_LINE has been enabled.
>
> After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE
> in conf/local.conf
>
>
>
>> If you're just building tar-native, is there a parallel race possible?
>> I'm not sure that there is?
>
> Even if I just build tar-native, I still have several packages which
> are needed to building.
>
>
> $bitbake tar-replacement-native -g
>
> $ cat pn-buildlist
> autoconf-native
> libtool-native
> m4-native
> gettext-minimal-native
> tar-replacement-native
> quilt-native
> gnu-config-native
> automake-native
>

To avoid to use the incomplete tar binary, I think we could rename the 
binary name to "tar-native" before do_populate_sysroot, and then rename 
back to "tar" after do_populate_sysroot().

Thanks.

- Pascal

>
> Thanks for your reply.
> -Roy
>>
>> For the record I hate this script and what it has become, its horrible.
>> It started as a workaround for pseudo-native, its becoming a dumping
>> ground for a whole set of nasty workarounds :(.
>>
>> Cheers,
>>
>> Richard
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>>
>
Richard Purdie - Aug. 7, 2012, 9:03 a.m.
On Tue, 2012-08-07 at 16:02 +0800, Rongqing Li wrote:
> 
> On 2012?08?07? 15:41, Richard Purdie wrote:
> > On Tue, 2012-08-07 at 13:59 +0800, Rongqing Li wrote:
> >> Paste a RFC PATCH
> >>
> >> [RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
> >>
> >> Compiling tar-replacement or not is decided by version of host tar,
> >> if the host tar version is lower than 1.23, Compiling tar-replacement
> >> is needed.
> >>
> >> When doing popoluate tar-replacement sysroot to write the tar to
> >> sysroot, but writing is not finished. other packages probably
> >> use the being written tar to unzip file, which will lead to failure
> >> and report the below error:
> >> "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
> >>
> >> Now we compile tar-replacement firstly and not parallel to ensure
> >> that a being written tar command will not be used.
> >>
> >> Signed-off-by: Roy.Li <rongqing.li@windriver.com>
> >> ---
> >>    scripts/bitbake |   14 +++++++++++++-
> >>    1 files changed, 13 insertions(+), 1 deletions(-)
> >>
> >> diff --git a/scripts/bitbake b/scripts/bitbake
> >> index 3772d82..eb6b144 100755
> >> --- a/scripts/bitbake
> >> +++ b/scripts/bitbake
> >> @@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
> >>                fi
> >>            done
> >>        done
> >> -    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
> >> +
> >> +    if [ $needtar = "1" ]; then
> >> +       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
> >> |awk -F':' '{print $1}'`
> >> +	test -n "$NUM_THREAD_LINE" &&
> >> +	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
> >> +
> >> +	bitbake $TARTARGET -c populate_sysroot
> >> +
> >> +	test -n "$NUM_THREAD_LINE" &&
> >> +	sed -i
> >> ''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g'
> >> conf/local.conf
> >> +    fi
> >
> > We are *NOT* running sed over local.conf. What is the user used a
> > different configuration file for example?
> 
> I use the sed to disable NUM_THREAD_LINE in conf/local.conf if
> NUM_THREAD_LINE has been enabled.
> 
> After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE
> in conf/local.conf

I understand what you did, I just don't want to do this. Its horrible
and buggy.

> > If you're just building tar-native, is there a parallel race possible?
> > I'm not sure that there is?
> 
> Even if I just build tar-native, I still have several packages which
> are needed to building.
> 
> 
> $bitbake tar-replacement-native -g
> 
> $ cat pn-buildlist
> autoconf-native
> libtool-native
> m4-native
> gettext-minimal-native
> tar-replacement-native
> quilt-native
> gnu-config-native
> automake-native

but tar-replacement-native will not build until all the above have
executed their populate_sysroot tasks, correct?

So I don't see a task that would run at the same time as
tar-replacement:do_populate_sysroot which will be the last task to run?

Cheers,

Richard
rongqing.li@windriver.com - Aug. 7, 2012, 9:25 a.m.
On 2012?08?07? 17:03, Richard Purdie wrote:
> On Tue, 2012-08-07 at 16:02 +0800, Rongqing Li wrote:
>>
>> On 2012?08?07? 15:41, Richard Purdie wrote:
>>> On Tue, 2012-08-07 at 13:59 +0800, Rongqing Li wrote:
>>>> Paste a RFC PATCH
>>>>
>>>> [RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not parallel
>>>>
>>>> Compiling tar-replacement or not is decided by version of host tar,
>>>> if the host tar version is lower than 1.23, Compiling tar-replacement
>>>> is needed.
>>>>
>>>> When doing popoluate tar-replacement sysroot to write the tar to
>>>> sysroot, but writing is not finished. other packages probably
>>>> use the being written tar to unzip file, which will lead to failure
>>>> and report the below error:
>>>> "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
>>>>
>>>> Now we compile tar-replacement firstly and not parallel to ensure
>>>> that a being written tar command will not be used.
>>>>
>>>> Signed-off-by: Roy.Li <rongqing.li@windriver.com>
>>>> ---
>>>>     scripts/bitbake |   14 +++++++++++++-
>>>>     1 files changed, 13 insertions(+), 1 deletions(-)
>>>>
>>>> diff --git a/scripts/bitbake b/scripts/bitbake
>>>> index 3772d82..eb6b144 100755
>>>> --- a/scripts/bitbake
>>>> +++ b/scripts/bitbake
>>>> @@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
>>>>                 fi
>>>>             done
>>>>         done
>>>> -    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
>>>> +
>>>> +    if [ $needtar = "1" ]; then
>>>> +       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf
>>>> |awk -F':' '{print $1}'`
>>>> +	test -n "$NUM_THREAD_LINE" &&
>>>> +	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
>>>> +
>>>> +	bitbake $TARTARGET -c populate_sysroot
>>>> +
>>>> +	test -n "$NUM_THREAD_LINE" &&
>>>> +	sed -i
>>>> ''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g'
>>>> conf/local.conf
>>>> +    fi
>>>
>>> We are *NOT* running sed over local.conf. What is the user used a
>>> different configuration file for example?
>>
>> I use the sed to disable NUM_THREAD_LINE in conf/local.conf if
>> NUM_THREAD_LINE has been enabled.
>>
>> After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE
>> in conf/local.conf
>
> I understand what you did, I just don't want to do this. Its horrible
> and buggy.
>
>>> If you're just building tar-native, is there a parallel race possible?
>>> I'm not sure that there is?
>>
>> Even if I just build tar-native, I still have several packages which
>> are needed to building.
>>
>>
>> $bitbake tar-replacement-native -g
>>
>> $ cat pn-buildlist
>> autoconf-native
>> libtool-native
>> m4-native
>> gettext-minimal-native
>> tar-replacement-native
>> quilt-native
>> gnu-config-native
>> automake-native
>
> but tar-replacement-native will not build until all the above have
> executed their populate_sysroot tasks, correct?
>
> So I don't see a task that would run at the same time as
> tar-replacement:do_populate_sysroot which will be the last task to run?
>
> Cheers,
>
> Richard
>
>


You are right, seems I only need to build tar-replacement firstly, donot
need to disable the BB_NUMBER_THREADS.


-Roy



>
rongqing.li@windriver.com - Aug. 7, 2012, 9:27 a.m.
On 2012?08?07? 16:57, Pascal Ouyang wrote:
> ? 2012?08?07? 16:02, Rongqing Li ??:
>>
>>
>> On 2012?08?07? 15:41, Richard Purdie wrote:
>>> On Tue, 2012-08-07 at 13:59 +0800, Rongqing Li wrote:
>>>> Paste a RFC PATCH
>>>>
>>>> [RFC PATCH 1/1] bitbake: compile tar-replacement firstly, and not
>>>> parallel
>>>>
>>>> Compiling tar-replacement or not is decided by version of host tar,
>>>> if the host tar version is lower than 1.23, Compiling tar-replacement
>>>> is needed.
>>>>
>>>> When doing popoluate tar-replacement sysroot to write the tar to
>>>> sysroot, but writing is not finished. other packages probably
>>>> use the being written tar to unzip file, which will lead to failure
>>>> and report the below error:
>>>> "bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/tar: Text file busy"
>>>>
>>>> Now we compile tar-replacement firstly and not parallel to ensure
>>>> that a being written tar command will not be used.
>>>>
>>>> Signed-off-by: Roy.Li <rongqing.li@windriver.com>
>>>> ---
>>>>    scripts/bitbake |   14 +++++++++++++-
>>>>    1 files changed, 13 insertions(+), 1 deletions(-)
>>>>
>>>> diff --git a/scripts/bitbake b/scripts/bitbake
>>>> index 3772d82..eb6b144 100755
>>>> --- a/scripts/bitbake
>>>> +++ b/scripts/bitbake
>>>> @@ -134,7 +134,19 @@ if [ $buildpseudo -gt 0 ]; then
>>>>                fi
>>>>            done
>>>>        done
>>>> -    bitbake pseudo-native $TARTARGET $additionalopts -c
>>>> populate_sysroot
>>>> +
>>>> +    if [ $needtar = "1" ]; then
>>>> +       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS"
>>>> conf/local.conf
>>>> |awk -F':' '{print $1}'`
>>>> +    test -n "$NUM_THREAD_LINE" &&
>>>> +    sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g'
>>>> conf/local.conf
>>>> +
>>>> +    bitbake $TARTARGET -c populate_sysroot
>>>> +
>>>> +    test -n "$NUM_THREAD_LINE" &&
>>>> +    sed -i
>>>> ''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g'
>>>> conf/local.conf
>>>> +    fi
>>>
>>> We are *NOT* running sed over local.conf. What is the user used a
>>> different configuration file for example?
>>
>> I use the sed to disable NUM_THREAD_LINE in conf/local.conf if
>> NUM_THREAD_LINE has been enabled.
>>
>> After compile $TARTARGET, use the sed to enable NUM_THREAD_LINE
>> in conf/local.conf
>>
>>
>>
>>> If you're just building tar-native, is there a parallel race possible?
>>> I'm not sure that there is?
>>
>> Even if I just build tar-native, I still have several packages which
>> are needed to building.
>>
>>
>> $bitbake tar-replacement-native -g
>>
>> $ cat pn-buildlist
>> autoconf-native
>> libtool-native
>> m4-native
>> gettext-minimal-native
>> tar-replacement-native
>> quilt-native
>> gnu-config-native
>> automake-native
>>
>
> To avoid to use the incomplete tar binary, I think we could rename the
> binary name to "tar-native" before do_populate_sysroot, and then rename
> back to "tar" after do_populate_sysroot().
>
> Thanks.
>
> - Pascal
>

I think your suggestion is valuable to fix the similar problem,
we can adopt this method.

-Roy


>>
>> Thanks for your reply.
>> -Roy
>>>
>>> For the record I hate this script and what it has become, its horrible.
>>> It started as a workaround for pseudo-native, its becoming a dumping
>>> ground for a whole set of nasty workarounds :(.
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>
>>>
>>
>
>

Patch

diff --git a/scripts/bitbake b/scripts/bitbake
index 3772d82..eb6b144 100755
--- a/scripts/bitbake
+++ b/scripts/bitbake
@@ -134,7 +134,19 @@  if [ $buildpseudo -gt 0 ]; then
              fi
          done
      done
-    bitbake pseudo-native $TARTARGET $additionalopts -c populate_sysroot
+
+    if [ $needtar = "1" ]; then
+       NUM_THREAD_LINE=`grep -n "^\s*BB_NUMBER_THREADS" conf/local.conf 
|awk -F':' '{print $1}'`
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i ''"$NUM_THREAD_LINE"'s/^\s*BB_NUMBER_THREADS/#\0/g' conf/local.conf
+
+	bitbake $TARTARGET -c populate_sysroot
+
+	test -n "$NUM_THREAD_LINE" &&
+	sed -i 
''"$NUM_THREAD_LINE"'s/^#\s*BB_NUMBER_THREADS/BB_NUMBER_THREADS/g' 
conf/local.conf
+    fi
+
+    bitbake pseudo-native $additionalopts -c populate_sysroot