diff mbox series

[3/3] package: Fix handling of minidebuginfo with newer binutils

Message ID 20221103075606.208982-3-nathan@nathanrossi.com
State New
Headers show
Series [1/3] oeqa/selftest/minidebuginfo: Create selftest for minidebuginfo | expand

Commit Message

Nathan Rossi Nov. 3, 2022, 7:56 a.m. UTC 
From: Nathan Rossi <nathan.rossi@digi.com>

Newer versions of binutils (2.38+) have changed how the
"--only-keep-debug" of objcopy behaves when stripping non-debug sections
from an ELF.

  https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=68f543154e92ab0f5d6c569e0fa143f5e8bd2d80

This change causes associated sections to be correctly marked as NOBITS
with the section contents removed from the output. The side effect is
that this causes issues with objcopy's ability to perform symbol and
relocation stripping (-S/--strip-all) on the debug split ELF, such that
with some object files (e.g. kernel modules) objcopy fails to strip
symbols/relocations with an error like the following:

  .../.debug/nls_cp950.ko[.rodata]: file truncated

Because of this it is now problematic to generate minidebuginfo for
these types of ELF objects. However it is not typically useful to inject
minidebuginfo into these types of ELFs, and other distributions (e.g.
Fedora, referring to find-debuginfo.sh of debugedit) only insert
minidebuginfo into executables and shared libraries.

This change causes the minidebuginfo injection to only apply to EXEC/DYN
type ELFs, which limits the injection to executables and shared
libraires.

Additionally this change fixes the parsing of the sections from the
"readelf -W -S" output which was not accounting for the section index
column having leading spaces for single digit index values e.g. "[ 1]".

Signed-off-by: Nathan Rossi <nathan.rossi@digi.com>
---
 meta/classes-global/package.bbclass | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

---
2.37.2

Comments

Alexandre Belloni Nov. 6, 2022, 3:54 p.m. UTC | #1 
Hello,

This series causes the following failure on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4354/steps/14/logs/stdio

2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_bindir: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_machine: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_alternatives: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_basic: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_binary: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_extlayer: ERROR
2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_func: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_glob: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_todir_glob: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_invalid: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_orig: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_patch: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_postinstall: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_renamed: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_script: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_subdir: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_todir: ERROR
2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_wildcard: ERROR

On 03/11/2022 07:56:06+0000, Nathan Rossi wrote:
> From: Nathan Rossi <nathan.rossi@digi.com>
> 
> Newer versions of binutils (2.38+) have changed how the
> "--only-keep-debug" of objcopy behaves when stripping non-debug sections
> from an ELF.
> 
>   https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=68f543154e92ab0f5d6c569e0fa143f5e8bd2d80
> 
> This change causes associated sections to be correctly marked as NOBITS
> with the section contents removed from the output. The side effect is
> that this causes issues with objcopy's ability to perform symbol and
> relocation stripping (-S/--strip-all) on the debug split ELF, such that
> with some object files (e.g. kernel modules) objcopy fails to strip
> symbols/relocations with an error like the following:
> 
>   .../.debug/nls_cp950.ko[.rodata]: file truncated
> 
> Because of this it is now problematic to generate minidebuginfo for
> these types of ELF objects. However it is not typically useful to inject
> minidebuginfo into these types of ELFs, and other distributions (e.g.
> Fedora, referring to find-debuginfo.sh of debugedit) only insert
> minidebuginfo into executables and shared libraries.
> 
> This change causes the minidebuginfo injection to only apply to EXEC/DYN
> type ELFs, which limits the injection to executables and shared
> libraires.
> 
> Additionally this change fixes the parsing of the sections from the
> "readelf -W -S" output which was not accounting for the section index
> column having leading spaces for single digit index values e.g. "[ 1]".
> 
> Signed-off-by: Nathan Rossi <nathan.rossi@digi.com>
> ---
>  meta/classes-global/package.bbclass | 21 ++++++++++++++++++---
>  1 file changed, 18 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass
> index 2d985d8aff..7a0a428b30 100644
> --- a/meta/classes-global/package.bbclass
> +++ b/meta/classes-global/package.bbclass
> @@ -490,16 +490,31 @@ def inject_minidebuginfo(file, dvar, dv, d):
>          bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file))
>          return
>  
> +    # minidebuginfo does not make sense to apply to ELF objects other than
> +    # executables and shared libraries, skip applying the minidebuginfo
> +    # generation for objects like kernel modules.
> +    for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines():
> +        if not line.strip().startswith("Type:"):
> +            continue
> +        elftype = line.split(":")[1].strip()
> +        if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]):
> +            bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file))
> +            return
> +        break
> +
>      # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo.
>      # We will exclude all of these from minidebuginfo to save space.
>      remove_section_names = []
>      for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines():
> -        fields = line.split()
> -        if len(fields) < 8:
> +        # strip the leading "  [ 1]" section index to allow splitting on space
> +        if ']' not in line:
> +            continue
> +        fields = line[line.index(']') + 1:].split()
> +        if len(fields) < 7:
>              continue
>          name = fields[0]
>          type = fields[1]
> -        flags = fields[7]
> +        flags = fields[6]
>          # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them
>          if name.startswith('.debug_'):
>              continue
> ---
> 2.37.2

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#172617): https://lists.openembedded.org/g/openembedded-core/message/172617
> Mute This Topic: https://lists.openembedded.org/mt/94752153/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
Nathan Rossi Nov. 8, 2022, 3:33 a.m. UTC | #2 
On Mon, 7 Nov 2022 at 01:55, Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
>
> Hello,
>
> This series causes the following failure on the autobuilders:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4354/steps/14/logs/stdio

I looked into these failures, the SRC_URI error appears to be an issue
with the lic_checksum test itself, and it presents without this
series. I have posted a patch to resolve it
https://lists.openembedded.org/g/openembedded-core/message/172945.

The other errors around "test.bb" bash hash changes do not appear to
be errors from the recipetools bitbake execution (note the "2 ERROR
messages"). I suspect they are just captured during the fetch selftest
cases in a common stderr buffer? I was unable to reproduce them
locally when running fetch/recipetool.

Regards,
Nathan



>
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add: ERROR
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_bindir: ERROR
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_add_machine: ERROR
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_alternatives: ERROR
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_basic: ERROR
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_binary: ERROR
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_extlayer: ERROR
> 2022-11-06 00:37:54,099 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_func: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_glob: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_inst_todir_glob: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_invalid: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_orig: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_patch: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_postinstall: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_renamed: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_script: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_subdir: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_todir: ERROR
> 2022-11-06 00:37:54,100 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendTests.test_recipetool_appendfile_wildcard: ERROR
>
> On 03/11/2022 07:56:06+0000, Nathan Rossi wrote:
> > From: Nathan Rossi <nathan.rossi@digi.com>
> >
> > Newer versions of binutils (2.38+) have changed how the
> > "--only-keep-debug" of objcopy behaves when stripping non-debug sections
> > from an ELF.
> >
> >   https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=68f543154e92ab0f5d6c569e0fa143f5e8bd2d80
> >
> > This change causes associated sections to be correctly marked as NOBITS
> > with the section contents removed from the output. The side effect is
> > that this causes issues with objcopy's ability to perform symbol and
> > relocation stripping (-S/--strip-all) on the debug split ELF, such that
> > with some object files (e.g. kernel modules) objcopy fails to strip
> > symbols/relocations with an error like the following:
> >
> >   .../.debug/nls_cp950.ko[.rodata]: file truncated
> >
> > Because of this it is now problematic to generate minidebuginfo for
> > these types of ELF objects. However it is not typically useful to inject
> > minidebuginfo into these types of ELFs, and other distributions (e.g.
> > Fedora, referring to find-debuginfo.sh of debugedit) only insert
> > minidebuginfo into executables and shared libraries.
> >
> > This change causes the minidebuginfo injection to only apply to EXEC/DYN
> > type ELFs, which limits the injection to executables and shared
> > libraires.
> >
> > Additionally this change fixes the parsing of the sections from the
> > "readelf -W -S" output which was not accounting for the section index
> > column having leading spaces for single digit index values e.g. "[ 1]".
> >
> > Signed-off-by: Nathan Rossi <nathan.rossi@digi.com>
> > ---
> >  meta/classes-global/package.bbclass | 21 ++++++++++++++++++---
> >  1 file changed, 18 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass
> > index 2d985d8aff..7a0a428b30 100644
> > --- a/meta/classes-global/package.bbclass
> > +++ b/meta/classes-global/package.bbclass
> > @@ -490,16 +490,31 @@ def inject_minidebuginfo(file, dvar, dv, d):
> >          bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file))
> >          return
> >
> > +    # minidebuginfo does not make sense to apply to ELF objects other than
> > +    # executables and shared libraries, skip applying the minidebuginfo
> > +    # generation for objects like kernel modules.
> > +    for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines():
> > +        if not line.strip().startswith("Type:"):
> > +            continue
> > +        elftype = line.split(":")[1].strip()
> > +        if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]):
> > +            bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file))
> > +            return
> > +        break
> > +
> >      # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo.
> >      # We will exclude all of these from minidebuginfo to save space.
> >      remove_section_names = []
> >      for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines():
> > -        fields = line.split()
> > -        if len(fields) < 8:
> > +        # strip the leading "  [ 1]" section index to allow splitting on space
> > +        if ']' not in line:
> > +            continue
> > +        fields = line[line.index(']') + 1:].split()
> > +        if len(fields) < 7:
> >              continue
> >          name = fields[0]
> >          type = fields[1]
> > -        flags = fields[7]
> > +        flags = fields[6]
> >          # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them
> >          if name.startswith('.debug_'):
> >              continue
> > ---
> > 2.37.2
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#172617): https://lists.openembedded.org/g/openembedded-core/message/172617
> > Mute This Topic: https://lists.openembedded.org/mt/94752153/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
diff mbox series

Patch

diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass
index 2d985d8aff..7a0a428b30 100644
--- a/meta/classes-global/package.bbclass
+++ b/meta/classes-global/package.bbclass
@@ -490,16 +490,31 @@  def inject_minidebuginfo(file, dvar, dv, d):
         bb.debug(1, 'ELF file {} has no debuginfo, skipping minidebuginfo injection'.format(file))
         return
 
+    # minidebuginfo does not make sense to apply to ELF objects other than
+    # executables and shared libraries, skip applying the minidebuginfo
+    # generation for objects like kernel modules.
+    for line in subprocess.check_output([readelf, '-h', debugfile], universal_newlines=True).splitlines():
+        if not line.strip().startswith("Type:"):
+            continue
+        elftype = line.split(":")[1].strip()
+        if not any(elftype.startswith(i) for i in ["EXEC", "DYN"]):
+            bb.debug(1, 'ELF file {} is not executable/shared, skipping minidebuginfo injection'.format(file))
+            return
+        break
+
     # Find non-allocated PROGBITS, NOTE, and NOBITS sections in the debuginfo.
     # We will exclude all of these from minidebuginfo to save space.
     remove_section_names = []
     for line in subprocess.check_output([readelf, '-W', '-S', debugfile], universal_newlines=True).splitlines():
-        fields = line.split()
-        if len(fields) < 8:
+        # strip the leading "  [ 1]" section index to allow splitting on space
+        if ']' not in line:
+            continue
+        fields = line[line.index(']') + 1:].split()
+        if len(fields) < 7:
             continue
         name = fields[0]
         type = fields[1]
-        flags = fields[7]
+        flags = fields[6]
         # .debug_ sections will be removed by objcopy -S so no need to explicitly remove them
         if name.startswith('.debug_'):
             continue