Patchwork [6/6,v2] package_rpm: Allow translation of requirement to package name

login
register
mail settings
Submitter Mark Hatle
Date Nov. 13, 2013, 1:59 a.m.
Message ID <1384307965-10290-2-git-send-email-mark.hatle@windriver.com>
Download mbox | patch
Permalink /patch/61567/
State New
Headers show

Comments

Mark Hatle - Nov. 13, 2013, 1:59 a.m.
In the translate oe to smart function, we only translated package names.
However, it's allowed that people can put in a dependency name in the
IMAGE_INSTALL.  So on a failure to translate a package name, we fall back
and attempt to resolve based on a package's provide.

Note: it may be possible to generate an unsolvable install solution.  If the
dependency is provided by one or more things that conflict with something else
set to be installed.  We can't determine this until smart is run.

If this occurs, file a bug and we'll have to identify a way to deal with the
RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES with
actual package names.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/package_rpm.bbclass | 79 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 75 insertions(+), 4 deletions(-)
Paul Eggleton - Nov. 13, 2013, 10:33 a.m.
Hi Mark,

On Tuesday 12 November 2013 19:59:20 Mark Hatle wrote:
> In the translate oe to smart function, we only translated package names.
> However, it's allowed that people can put in a dependency name in the
> IMAGE_INSTALL.  So on a failure to translate a package name, we fall back
> and attempt to resolve based on a package's provide.
> 
> Note: it may be possible to generate an unsolvable install solution.  If the
> dependency is provided by one or more things that conflict with something
> else set to be installed.  We can't determine this until smart is run.
> 
> If this occurs, file a bug and we'll have to identify a way to deal with the
> RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES
> with actual package names.
> 
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
>  meta/classes/package_rpm.bbclass | 79
> ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 75 insertions(+),
> 4 deletions(-)
> 
> diff --git a/meta/classes/package_rpm.bbclass
> b/meta/classes/package_rpm.bbclass index 31265d9..9fe0e6c 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -166,6 +166,7 @@ translate_oe_to_smart() {
>  	[ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart
> --data-dir=${target_rootfs}/var/lib/smart query --output
> ${target_rootfs}/install/tmp/fullpkglist.query
> 
>  	pkgs_to_install=""
> +	not_found=""
>  	for pkg in "$@" ; do
>  		new_pkg="$pkg"
>  		if [ -z "$sdk_mode" ]; then
> @@ -184,7 +185,6 @@ translate_oe_to_smart() {
>  				fi
>  				subst=${pkg#${mlib}-}
>  				if [ "$subst" != "$pkg" ]; then
> -					feeds=$@
>  					while [ -n "$1" ]; do
>  						arch="$1"
>  						arch=`echo "$arch" | tr - _`
> @@ -197,6 +197,76 @@ translate_oe_to_smart() {
>  					done
>  					if [ "$pkg" = "$new_pkg" ]; then
>  						# Failed to translate, package not found!
> +						not_found="$not_found $pkg"
> +						continue
> +					fi
> +				fi
> +			done
> +		fi
> +		# Apparently not a multilib package...
> +		if [ "$pkg" = "$new_pkg" ]; then
> +			default_archs_fixed=`echo "$default_archs" | tr - _`
> +			for arch in $default_archs_fixed ; do
> +				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$'
> ${target_rootfs}/install/tmp/fullpkglist.query ; then
> +					new_pkg="$pkg@$arch"
> +					# First found is best match
> +					break
> +				fi
> +			done
> +			if [ "$pkg" = "$new_pkg" ]; then
> +				# Failed to translate, package not found!
> +				not_found="$not_found $pkg"
> +				continue
> +			fi
> +		fi
> +		#echo "$pkg -> $new_pkg" >&2
> +		pkgs_to_install="${pkgs_to_install} ${new_pkg}"
> +	done
> +
> +	# Parse the not_found items and see if they were dependencies (RPROVIDES)
> +	# Follow the parsing example above...
> +	for pkg in $not_found ; do
> +		new_pkg="$pkg"
> +		smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg
> --output ${target_rootfs}/install/tmp/provide.query +		grep '^[^@
> ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e
> 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' >
> ${target_rootfs}/install/tmp/provide.query.list || : +		prov=`echo
> $pkgs_to_install | xargs -n 1 echo | grep -f
> ${target_rootfs}/install/tmp/provide.query.list || :` +		if [ -n "$prov" 
];
> then
> +			# Nothing to do, already in the list
> +			#echo "Skipping $pkg -> $prov, already in install set" >&2
> +			continue
> +		fi
> +		if [ -z "$sdk_mode" ]; then
> +			for i in ${MULTILIB_PREFIX_LIST} ; do
> +				old_IFS="$IFS"
> +				IFS=":"
> +				set $i
> +				IFS="$old_IFS"
> +				mlib="$1"
> +				shift
> +				if [ "$mlib" = "default" ]; then
> +					if [ -z "$default_archs" ]; then
> +						default_archs=$@
> +					fi
> +					continue
> +				fi
> +				subst=${pkg#${mlib}-}
> +				if [ "$subst" != "$pkg" ]; then
> +					feeds=$@
> +					smart --data-dir=${target_rootfs}/var/lib/smart query
> --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
> +					grep '^[^@ ]*@[^@]*$' 
${target_rootfs}/install/tmp/provide.query |
> sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' >
> ${target_rootfs}/install/tmp/provide.query.list || : +					while [ 
-n "$1"
> ]; do
> +						arch="$1"
> +						arch=`echo "$arch" | tr - _`
> +						shift
> +						# Select first found, we don't know if one is better then 
another...
> +						prov=`grep '^[^@ ]*@'$arch'$'
> ${target_rootfs}/install/tmp/provide.query.list | head -n 1` +						
if [ -n
> "$prov" ]; then
> +							new_pkg=$prov
> +							break
> +						fi
> +					done
> +					if [ "$pkg" = "$new_pkg" ]; then
> +						# Failed to translate, package not found!
>  						echo "$attemptonly: $pkg not found in the $mlib feeds 
($feeds)." >&2
>  						if [ "$attemptonly" = "Error" ]; then
>  							exit 1
> @@ -210,9 +280,10 @@ translate_oe_to_smart() {
>  		if [ "$pkg" = "$new_pkg" ]; then
>  			default_archs_fixed=`echo "$default_archs" | tr - _`
>  			for arch in $default_archs_fixed ; do
> -				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$'
> ${target_rootfs}/install/tmp/fullpkglist.query ; then
> -					new_pkg="$pkg@$arch"
> -					# First found is best match
> +				# Select first found, we don't know if one is better then 
another...
> +				prov=`grep '^[^@ ]*@'$arch'$'
> ${target_rootfs}/install/tmp/provide.query.list | head -n 1` +				if 
[ -n
> "$prov" ]; then
> +					new_pkg=$prov
>  					break
>  				fi
>  			done

Sigh... this code is getting ridiculously complicated. Shouldn't Smart be 
doing more of the heavy lifting here?

Cheers,
Paul
Otavio Salvador - Nov. 13, 2013, 12:31 p.m.
On Wed, Nov 13, 2013 at 8:33 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Tuesday 12 November 2013 19:59:20 Mark Hatle wrote:
>> In the translate oe to smart function, we only translated package names.
>> However, it's allowed that people can put in a dependency name in the
>> IMAGE_INSTALL.  So on a failure to translate a package name, we fall back
>> and attempt to resolve based on a package's provide.
>>
>> Note: it may be possible to generate an unsolvable install solution.  If the
>> dependency is provided by one or more things that conflict with something
>> else set to be installed.  We can't determine this until smart is run.
>>
>> If this occurs, file a bug and we'll have to identify a way to deal with the
>> RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES
>> with actual package names.
>>
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>> ---
>>  meta/classes/package_rpm.bbclass | 79
>> ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 75 insertions(+),
>> 4 deletions(-)
>>
>> diff --git a/meta/classes/package_rpm.bbclass
>> b/meta/classes/package_rpm.bbclass index 31265d9..9fe0e6c 100644
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -166,6 +166,7 @@ translate_oe_to_smart() {
>>       [ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart
>> --data-dir=${target_rootfs}/var/lib/smart query --output
>> ${target_rootfs}/install/tmp/fullpkglist.query
>>
>>       pkgs_to_install=""
>> +     not_found=""
>>       for pkg in "$@" ; do
>>               new_pkg="$pkg"
>>               if [ -z "$sdk_mode" ]; then
>> @@ -184,7 +185,6 @@ translate_oe_to_smart() {
>>                               fi
>>                               subst=${pkg#${mlib}-}
>>                               if [ "$subst" != "$pkg" ]; then
>> -                                     feeds=$@
>>                                       while [ -n "$1" ]; do
>>                                               arch="$1"
>>                                               arch=`echo "$arch" | tr - _`
>> @@ -197,6 +197,76 @@ translate_oe_to_smart() {
>>                                       done
>>                                       if [ "$pkg" = "$new_pkg" ]; then
>>                                               # Failed to translate, package not found!
>> +                                             not_found="$not_found $pkg"
>> +                                             continue
>> +                                     fi
>> +                             fi
>> +                     done
>> +             fi
>> +             # Apparently not a multilib package...
>> +             if [ "$pkg" = "$new_pkg" ]; then
>> +                     default_archs_fixed=`echo "$default_archs" | tr - _`
>> +                     for arch in $default_archs_fixed ; do
>> +                             if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$'
>> ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> +                                     new_pkg="$pkg@$arch"
>> +                                     # First found is best match
>> +                                     break
>> +                             fi
>> +                     done
>> +                     if [ "$pkg" = "$new_pkg" ]; then
>> +                             # Failed to translate, package not found!
>> +                             not_found="$not_found $pkg"
>> +                             continue
>> +                     fi
>> +             fi
>> +             #echo "$pkg -> $new_pkg" >&2
>> +             pkgs_to_install="${pkgs_to_install} ${new_pkg}"
>> +     done
>> +
>> +     # Parse the not_found items and see if they were dependencies (RPROVIDES)
>> +     # Follow the parsing example above...
>> +     for pkg in $not_found ; do
>> +             new_pkg="$pkg"
>> +             smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg
>> --output ${target_rootfs}/install/tmp/provide.query +         grep '^[^@
>> ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e
>> 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' >
>> ${target_rootfs}/install/tmp/provide.query.list || : +                prov=`echo
>> $pkgs_to_install | xargs -n 1 echo | grep -f
>> ${target_rootfs}/install/tmp/provide.query.list || :` +               if [ -n "$prov"
> ];
>> then
>> +                     # Nothing to do, already in the list
>> +                     #echo "Skipping $pkg -> $prov, already in install set" >&2
>> +                     continue
>> +             fi
>> +             if [ -z "$sdk_mode" ]; then
>> +                     for i in ${MULTILIB_PREFIX_LIST} ; do
>> +                             old_IFS="$IFS"
>> +                             IFS=":"
>> +                             set $i
>> +                             IFS="$old_IFS"
>> +                             mlib="$1"
>> +                             shift
>> +                             if [ "$mlib" = "default" ]; then
>> +                                     if [ -z "$default_archs" ]; then
>> +                                             default_archs=$@
>> +                                     fi
>> +                                     continue
>> +                             fi
>> +                             subst=${pkg#${mlib}-}
>> +                             if [ "$subst" != "$pkg" ]; then
>> +                                     feeds=$@
>> +                                     smart --data-dir=${target_rootfs}/var/lib/smart query
>> --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
>> +                                     grep '^[^@ ]*@[^@]*$'
> ${target_rootfs}/install/tmp/provide.query |
>> sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' >
>> ${target_rootfs}/install/tmp/provide.query.list || : +                                        while [
> -n "$1"
>> ]; do
>> +                                             arch="$1"
>> +                                             arch=`echo "$arch" | tr - _`
>> +                                             shift
>> +                                             # Select first found, we don't know if one is better then
> another...
>> +                                             prov=`grep '^[^@ ]*@'$arch'$'
>> ${target_rootfs}/install/tmp/provide.query.list | head -n 1` +
> if [ -n
>> "$prov" ]; then
>> +                                                     new_pkg=$prov
>> +                                                     break
>> +                                             fi
>> +                                     done
>> +                                     if [ "$pkg" = "$new_pkg" ]; then
>> +                                             # Failed to translate, package not found!
>>                                               echo "$attemptonly: $pkg not found in the $mlib feeds
> ($feeds)." >&2
>>                                               if [ "$attemptonly" = "Error" ]; then
>>                                                       exit 1
>> @@ -210,9 +280,10 @@ translate_oe_to_smart() {
>>               if [ "$pkg" = "$new_pkg" ]; then
>>                       default_archs_fixed=`echo "$default_archs" | tr - _`
>>                       for arch in $default_archs_fixed ; do
>> -                             if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$'
>> ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> -                                     new_pkg="$pkg@$arch"
>> -                                     # First found is best match
>> +                             # Select first found, we don't know if one is better then
> another...
>> +                             prov=`grep '^[^@ ]*@'$arch'$'
>> ${target_rootfs}/install/tmp/provide.query.list | head -n 1` +                                if
> [ -n
>> "$prov" ]; then
>> +                                     new_pkg=$prov
>>                                       break
>>                               fi
>>                       done
>
> Sigh... this code is getting ridiculously complicated. Shouldn't Smart be
> doing more of the heavy lifting here?

Agreed; this seems OE is doing too much.

Another think which bothers me is it being RPM specific ... :-(
Mark Hatle - Nov. 13, 2013, 1:12 p.m.
On 11/13/13, 4:33 AM, Paul Eggleton wrote:
> Hi Mark,
>
> On Tuesday 12 November 2013 19:59:20 Mark Hatle wrote:
>> In the translate oe to smart function, we only translated package names.
>> However, it's allowed that people can put in a dependency name in the
>> IMAGE_INSTALL.  So on a failure to translate a package name, we fall back
>> and attempt to resolve based on a package's provide.
>>
>> Note: it may be possible to generate an unsolvable install solution.  If the
>> dependency is provided by one or more things that conflict with something
>> else set to be installed.  We can't determine this until smart is run.
>>
>> If this occurs, file a bug and we'll have to identify a way to deal with the
>> RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES
>> with actual package names.
>>
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>> ---
>>   meta/classes/package_rpm.bbclass | 79
>> ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 75 insertions(+),
>> 4 deletions(-)
>>
>> diff --git a/meta/classes/package_rpm.bbclass
>> b/meta/classes/package_rpm.bbclass index 31265d9..9fe0e6c 100644
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -166,6 +166,7 @@ translate_oe_to_smart() {
>>   	[ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart
>> --data-dir=${target_rootfs}/var/lib/smart query --output
>> ${target_rootfs}/install/tmp/fullpkglist.query
>>
>>   	pkgs_to_install=""
>> +	not_found=""
>>   	for pkg in "$@" ; do
>>   		new_pkg="$pkg"
>>   		if [ -z "$sdk_mode" ]; then
>> @@ -184,7 +185,6 @@ translate_oe_to_smart() {
>>   				fi
>>   				subst=${pkg#${mlib}-}
>>   				if [ "$subst" != "$pkg" ]; then
>> -					feeds=$@
>>   					while [ -n "$1" ]; do
>>   						arch="$1"
>>   						arch=`echo "$arch" | tr - _`
>> @@ -197,6 +197,76 @@ translate_oe_to_smart() {
>>   					done
>>   					if [ "$pkg" = "$new_pkg" ]; then
>>   						# Failed to translate, package not found!

Just to be clear the following diff is rather confusing.. there are two one line 
changes (the error to the no_found line below) in the existing code, and then 
the new chunk of code after both to process the not_found entries, which are 
presumed to be dependencies.)

If you apply this, or look at it in context it's easier to see..

>> +						not_found="$not_found $pkg"
>> +						continue
>> +					fi
>> +				fi
>> +			done
>> +		fi
>> +		# Apparently not a multilib package...
>> +		if [ "$pkg" = "$new_pkg" ]; then
>> +			default_archs_fixed=`echo "$default_archs" | tr - _`
>> +			for arch in $default_archs_fixed ; do
>> +				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$'
>> ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> +					new_pkg="$pkg@$arch"
>> +					# First found is best match
>> +					break
>> +				fi
>> +			done
>> +			if [ "$pkg" = "$new_pkg" ]; then
>> +				# Failed to translate, package not found!
>> +				not_found="$not_found $pkg"
>> +				continue
>> +			fi
>> +		fi
>> +		#echo "$pkg -> $new_pkg" >&2
>> +		pkgs_to_install="${pkgs_to_install} ${new_pkg}"
>> +	done
>> +
>> +	# Parse the not_found items and see if they were dependencies (RPROVIDES)
>> +	# Follow the parsing example above...
>> +	for pkg in $not_found ; do
>> +		new_pkg="$pkg"
>> +		smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg
>> --output ${target_rootfs}/install/tmp/provide.query +		grep '^[^@
>> ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e
>> 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' >
>> ${target_rootfs}/install/tmp/provide.query.list || : +		prov=`echo
>> $pkgs_to_install | xargs -n 1 echo | grep -f
>> ${target_rootfs}/install/tmp/provide.query.list || :` +		if [ -n "$prov"
> ];
>> then
>> +			# Nothing to do, already in the list
>> +			#echo "Skipping $pkg -> $prov, already in install set" >&2
>> +			continue
>> +		fi
>> +		if [ -z "$sdk_mode" ]; then
>> +			for i in ${MULTILIB_PREFIX_LIST} ; do
>> +				old_IFS="$IFS"
>> +				IFS=":"
>> +				set $i
>> +				IFS="$old_IFS"
>> +				mlib="$1"
>> +				shift
>> +				if [ "$mlib" = "default" ]; then
>> +					if [ -z "$default_archs" ]; then
>> +						default_archs=$@
>> +					fi
>> +					continue
>> +				fi
>> +				subst=${pkg#${mlib}-}
>> +				if [ "$subst" != "$pkg" ]; then
>> +					feeds=$@
>> +					smart --data-dir=${target_rootfs}/var/lib/smart query
>> --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
>> +					grep '^[^@ ]*@[^@]*$'
> ${target_rootfs}/install/tmp/provide.query |
>> sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' >
>> ${target_rootfs}/install/tmp/provide.query.list || : +					while [
> -n "$1"
>> ]; do
>> +						arch="$1"
>> +						arch=`echo "$arch" | tr - _`
>> +						shift
>> +						# Select first found, we don't know if one is better then
> another...
>> +						prov=`grep '^[^@ ]*@'$arch'$'
>> ${target_rootfs}/install/tmp/provide.query.list | head -n 1` +						
> if [ -n
>> "$prov" ]; then
>> +							new_pkg=$prov
>> +							break
>> +						fi
>> +					done
>> +					if [ "$pkg" = "$new_pkg" ]; then
>> +						# Failed to translate, package not found!
>>   						echo "$attemptonly: $pkg not found in the $mlib feeds
> ($feeds)." >&2
>>   						if [ "$attemptonly" = "Error" ]; then
>>   							exit 1
>> @@ -210,9 +280,10 @@ translate_oe_to_smart() {
>>   		if [ "$pkg" = "$new_pkg" ]; then
>>   			default_archs_fixed=`echo "$default_archs" | tr - _`
>>   			for arch in $default_archs_fixed ; do
>> -				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$'
>> ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> -					new_pkg="$pkg@$arch"
>> -					# First found is best match
>> +				# Select first found, we don't know if one is better then
> another...
>> +				prov=`grep '^[^@ ]*@'$arch'$'
>> ${target_rootfs}/install/tmp/provide.query.list | head -n 1` +				if
> [ -n
>> "$prov" ]; then
>> +					new_pkg=$prov
>>   					break
>>   				fi
>>   			done
>
> Sigh... this code is getting ridiculously complicated. Shouldn't Smart be
> doing more of the heavy lifting here?

This function translate_oe_to_smart (and the corresponding 
translate_smart_to_oe) are used to switch from one name space to another. 
Remember in the RPM world, we don't use prefixes like "lib32-" on our package 
names, instead we use specific arch's that refer to specific ABI and processor 
tunings.

This is what enables the seamless multilib install on RPM.

What this code does is allows for dependencies to be added to the mix.  If we 
didn't have multilibs then none of this code would be needed, but since we do -- 
the name space of both the package and dependencies need to be translate, not 
just packages as we'd done before.

Generally what this function does:

for each multilib prefx:
    if PN starts w/ a known multilib prefix:
       we translate it to: BPN@arch
       check if the package exists
          if it does add to pkg_list
          else add to "not_found"
anything remaining is translated to the default arch
    check if the package exists
        if it does add to pkg_list
        else add to "not_found"

if there are any "not_found", we assume they must be provides:
    for each multilib prefix:
       if the DEPENDENCY starts w/ a known multilib prefix:
          we translate it without the prefix
          ask smart what provides that dependency
          check if something already in the pkg_list is there, if so skip it
          else use priority ordering to select the best provider -- add to pkg_list
    for anything w/o a multilib prefix, follow the same steps

The reverse smart_to_oe function doesn't ever need to pay attention to 
dependencies so that is not an issue, and the translation is significantly more 
straight forward.

If we had unique namespaces in oe, and didn't use the lib* markers on 
multilibs.. then none of this would be necessary.. (but then of course dpkg and 
opkg wouldn't work properly.)  So the translation is a necessary evil.

Could it be done differently, most likely.  Perhaps identifying a way to do it 
in python would be quicker.. but it's actually pretty fast sine the majority of 
translations happen via a static list and a single grep.  It's only the 
(hopefully) small number of dependencies that happen via individual calls to 
smart which impose a time penalty.

--Mark

> Cheers,
> Paul
>
Richard Purdie - Nov. 15, 2013, 2:02 p.m.
On Tue, 2013-11-12 at 19:59 -0600, Mark Hatle wrote:
> In the translate oe to smart function, we only translated package names.
> However, it's allowed that people can put in a dependency name in the
> IMAGE_INSTALL.

For what its worth, I find this description hard to understand, I think
as you leave the keyword "provide" right until the end. A mention of
RPROVIDES earlier here would help a lot.

>   So on a failure to translate a package name, we fall back
> and attempt to resolve based on a package's provide.
> 
> Note: it may be possible to generate an unsolvable install solution.  If the
> dependency is provided by one or more things that conflict with something else
> set to be installed.  We can't determine this until smart is run.
> 
> If this occurs, file a bug and we'll have to identify a way to deal with the
> RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES with
> actual package names.
> 
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
>  meta/classes/package_rpm.bbclass | 79 ++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 75 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 31265d9..9fe0e6c 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -166,6 +166,7 @@ translate_oe_to_smart() {
>  	[ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart --data-dir=${target_rootfs}/var/lib/smart query --output ${target_rootfs}/install/tmp/fullpkglist.query
>  
>  	pkgs_to_install=""
> +	not_found=""
>  	for pkg in "$@" ; do
>  		new_pkg="$pkg"
>  		if [ -z "$sdk_mode" ]; then
> @@ -184,7 +185,6 @@ translate_oe_to_smart() {
>  				fi
>  				subst=${pkg#${mlib}-}
>  				if [ "$subst" != "$pkg" ]; then
> -					feeds=$@
>  					while [ -n "$1" ]; do
>  						arch="$1"
>  						arch=`echo "$arch" | tr - _`
> @@ -197,6 +197,76 @@ translate_oe_to_smart() {
>  					done
>  					if [ "$pkg" = "$new_pkg" ]; then
>  						# Failed to translate, package not found!
> +						not_found="$not_found $pkg"
> +						continue
> +					fi
> +				fi
> +			done
> +		fi
> +		# Apparently not a multilib package...
> +		if [ "$pkg" = "$new_pkg" ]; then
> +			default_archs_fixed=`echo "$default_archs" | tr - _`
> +			for arch in $default_archs_fixed ; do
> +				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
> +					new_pkg="$pkg@$arch"
> +					# First found is best match
> +					break
> +				fi
> +			done
> +			if [ "$pkg" = "$new_pkg" ]; then
> +				# Failed to translate, package not found!
> +				not_found="$not_found $pkg"
> +				continue
> +			fi
> +		fi
> +		#echo "$pkg -> $new_pkg" >&2
> +		pkgs_to_install="${pkgs_to_install} ${new_pkg}"
> +	done
> +
> +	# Parse the not_found items and see if they were dependencies (RPROVIDES)
> +	# Follow the parsing example above...
> +	for pkg in $not_found ; do
> +		new_pkg="$pkg"
> +		smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg --output ${target_rootfs}/install/tmp/provide.query
> +		grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
> +		prov=`echo $pkgs_to_install | xargs -n 1 echo | grep -f ${target_rootfs}/install/tmp/provide.query.list || :`
> +		if [ -n "$prov" ]; then
> +			# Nothing to do, already in the list
> +			#echo "Skipping $pkg -> $prov, already in install set" >&2
> +			continue
> +		fi
> +		if [ -z "$sdk_mode" ]; then
> +			for i in ${MULTILIB_PREFIX_LIST} ; do
> +				old_IFS="$IFS"
> +				IFS=":"
> +				set $i
> +				IFS="$old_IFS"
> +				mlib="$1"
> +				shift
> +				if [ "$mlib" = "default" ]; then
> +					if [ -z "$default_archs" ]; then
> +						default_archs=$@
> +					fi
> +					continue
> +				fi
> +				subst=${pkg#${mlib}-}
> +				if [ "$subst" != "$pkg" ]; then
> +					feeds=$@
> +					smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
> +					grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
> +					while [ -n "$1" ]; do
> +						arch="$1"
> +						arch=`echo "$arch" | tr - _`
> +						shift
> +						# Select first found, we don't know if one is better then another...
> +						prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
> +						if [ -n "$prov" ]; then
> +							new_pkg=$prov
> +							break
> +						fi
> +					done
> +					if [ "$pkg" = "$new_pkg" ]; then
> +						# Failed to translate, package not found!
>  						echo "$attemptonly: $pkg not found in the $mlib feeds ($feeds)." >&2
>  						if [ "$attemptonly" = "Error" ]; then
>  							exit 1
> @@ -210,9 +280,10 @@ translate_oe_to_smart() {
>  		if [ "$pkg" = "$new_pkg" ]; then
>  			default_archs_fixed=`echo "$default_archs" | tr - _`
>  			for arch in $default_archs_fixed ; do
> -				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
> -					new_pkg="$pkg@$arch"
> -					# First found is best match
> +				# Select first found, we don't know if one is better then another...
> +				prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
> +				if [ -n "$prov" ]; then
> +					new_pkg=$prov
>  					break
>  				fi
>  			done

This 'shell calling rpm commands' code with all the mangling is near
impossible to read/understand. This patch just makes the mess more
complicated and harder to follow.

What is the plan for turning this into something better?

My instinct says more functions and a language better at data
manipulation (python instead of shell) might start to resolve this.

One of the reasons for using smart was that we could probably enhance it
to better suit our needs. Can we consider that too?

I really don't want to see more patches like this one piling on top of
already crumbling foundations.

Cheers,

Richard
Mark Hatle - Nov. 15, 2013, 5:15 p.m.
On 11/15/13, 8:02 AM, Richard Purdie wrote:
> On Tue, 2013-11-12 at 19:59 -0600, Mark Hatle wrote:
>> In the translate oe to smart function, we only translated package names.
>> However, it's allowed that people can put in a dependency name in the
>> IMAGE_INSTALL.
>
> For what its worth, I find this description hard to understand, I think
> as you leave the keyword "provide" right until the end. A mention of
> RPROVIDES earlier here would help a lot.

Ok.

>>    So on a failure to translate a package name, we fall back
>> and attempt to resolve based on a package's provide.
>>
>> Note: it may be possible to generate an unsolvable install solution.  If the
>> dependency is provided by one or more things that conflict with something else
>> set to be installed.  We can't determine this until smart is run.
>>
>> If this occurs, file a bug and we'll have to identify a way to deal with the
>> RCONFLICTS and RREPLACES.  As a workaround replace the conflict REQUIRES with
>> actual package names.
>>
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>> ---
>>   meta/classes/package_rpm.bbclass | 79 ++++++++++++++++++++++++++++++++++++++--
>>   1 file changed, 75 insertions(+), 4 deletions(-)
>>
>> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
>> index 31265d9..9fe0e6c 100644
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -166,6 +166,7 @@ translate_oe_to_smart() {
>>   	[ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart --data-dir=${target_rootfs}/var/lib/smart query --output ${target_rootfs}/install/tmp/fullpkglist.query
>>
>>   	pkgs_to_install=""
>> +	not_found=""
>>   	for pkg in "$@" ; do
>>   		new_pkg="$pkg"
>>   		if [ -z "$sdk_mode" ]; then
>> @@ -184,7 +185,6 @@ translate_oe_to_smart() {
>>   				fi
>>   				subst=${pkg#${mlib}-}
>>   				if [ "$subst" != "$pkg" ]; then
>> -					feeds=$@
>>   					while [ -n "$1" ]; do
>>   						arch="$1"
>>   						arch=`echo "$arch" | tr - _`
>> @@ -197,6 +197,76 @@ translate_oe_to_smart() {
>>   					done
>>   					if [ "$pkg" = "$new_pkg" ]; then
>>   						# Failed to translate, package not found!
>> +						not_found="$not_found $pkg"
>> +						continue
>> +					fi
>> +				fi
>> +			done
>> +		fi
>> +		# Apparently not a multilib package...
>> +		if [ "$pkg" = "$new_pkg" ]; then
>> +			default_archs_fixed=`echo "$default_archs" | tr - _`
>> +			for arch in $default_archs_fixed ; do
>> +				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> +					new_pkg="$pkg@$arch"
>> +					# First found is best match
>> +					break
>> +				fi
>> +			done
>> +			if [ "$pkg" = "$new_pkg" ]; then
>> +				# Failed to translate, package not found!
>> +				not_found="$not_found $pkg"
>> +				continue
>> +			fi
>> +		fi
>> +		#echo "$pkg -> $new_pkg" >&2
>> +		pkgs_to_install="${pkgs_to_install} ${new_pkg}"
>> +	done
>> +
>> +	# Parse the not_found items and see if they were dependencies (RPROVIDES)
>> +	# Follow the parsing example above...
>> +	for pkg in $not_found ; do
>> +		new_pkg="$pkg"
>> +		smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg --output ${target_rootfs}/install/tmp/provide.query
>> +		grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
>> +		prov=`echo $pkgs_to_install | xargs -n 1 echo | grep -f ${target_rootfs}/install/tmp/provide.query.list || :`
>> +		if [ -n "$prov" ]; then
>> +			# Nothing to do, already in the list
>> +			#echo "Skipping $pkg -> $prov, already in install set" >&2
>> +			continue
>> +		fi
>> +		if [ -z "$sdk_mode" ]; then
>> +			for i in ${MULTILIB_PREFIX_LIST} ; do
>> +				old_IFS="$IFS"
>> +				IFS=":"
>> +				set $i
>> +				IFS="$old_IFS"
>> +				mlib="$1"
>> +				shift
>> +				if [ "$mlib" = "default" ]; then
>> +					if [ -z "$default_archs" ]; then
>> +						default_archs=$@
>> +					fi
>> +					continue
>> +				fi
>> +				subst=${pkg#${mlib}-}
>> +				if [ "$subst" != "$pkg" ]; then
>> +					feeds=$@
>> +					smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
>> +					grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
>> +					while [ -n "$1" ]; do
>> +						arch="$1"
>> +						arch=`echo "$arch" | tr - _`
>> +						shift
>> +						# Select first found, we don't know if one is better then another...
>> +						prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
>> +						if [ -n "$prov" ]; then
>> +							new_pkg=$prov
>> +							break
>> +						fi
>> +					done
>> +					if [ "$pkg" = "$new_pkg" ]; then
>> +						# Failed to translate, package not found!
>>   						echo "$attemptonly: $pkg not found in the $mlib feeds ($feeds)." >&2
>>   						if [ "$attemptonly" = "Error" ]; then
>>   							exit 1
>> @@ -210,9 +280,10 @@ translate_oe_to_smart() {
>>   		if [ "$pkg" = "$new_pkg" ]; then
>>   			default_archs_fixed=`echo "$default_archs" | tr - _`
>>   			for arch in $default_archs_fixed ; do
>> -				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
>> -					new_pkg="$pkg@$arch"
>> -					# First found is best match
>> +				# Select first found, we don't know if one is better then another...
>> +				prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
>> +				if [ -n "$prov" ]; then
>> +					new_pkg=$prov
>>   					break
>>   				fi
>>   			done
>
> This 'shell calling rpm commands' code with all the mangling is near
> impossible to read/understand. This patch just makes the mess more
> complicated and harder to follow.

The root of this is the namespace issue.  In RPM, the architecture (and arch 
compatibility), as well as file contents all provide the namespace and conflict 
resolution.  While in opkg/deb the name of the package itself is where the 
namespace is defined.  (i.e. lib32-...)

The translation is necessary due to this multilib support, otherwise we wouldn't 
need to do any of it.  Somehow we have to translate one namespace to another -- 
and since the translation can't occur until the packages are built, it'd be 
difficult to do it 'earlier'.

> What is the plan for turning this into something better?
>
> My instinct says more functions and a language better at data
> manipulation (python instead of shell) might start to resolve this.
>
> One of the reasons for using smart was that we could probably enhance it
> to better suit our needs. Can we consider that too?

The only thing we can do is build a python (or similar) app that takes the 
inputs of the translation 'namespace' IDs and the corresponding architectures, 
and RPM feed.  Then translate from one to the other.  My biggest concern with 
this translation is to cover all of the corner cases we could start getting more 
and more complex.

> I really don't want to see more patches like this one piling on top of
> already crumbling foundations.

Easiest way to avoid this is to make dependencies in the IMAGE/PACKAGE_INSTALL 
forbidden and do the translation at a higher level generically.  But frankly I 
don't see that happening -- so the solution becomes RPM specific and we end up 
with code like this.  I'm not sure if we can move this code into a python 
function -- or if we need to simply create an external translation program...

As far as plans, frankly I don't have any plans to focus on this work for the 
time being.  There are too many other things I need to get working properly 
first.  If someone else wants to step in and work on a solution, I'm more then 
happy to help them out.

--Mark

> Cheers,
>
> Richard
>

Patch

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 31265d9..9fe0e6c 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -166,6 +166,7 @@  translate_oe_to_smart() {
 	[ ! -e ${target_rootfs}/install/tmp/fullpkglist.query ] && smart --data-dir=${target_rootfs}/var/lib/smart query --output ${target_rootfs}/install/tmp/fullpkglist.query
 
 	pkgs_to_install=""
+	not_found=""
 	for pkg in "$@" ; do
 		new_pkg="$pkg"
 		if [ -z "$sdk_mode" ]; then
@@ -184,7 +185,6 @@  translate_oe_to_smart() {
 				fi
 				subst=${pkg#${mlib}-}
 				if [ "$subst" != "$pkg" ]; then
-					feeds=$@
 					while [ -n "$1" ]; do
 						arch="$1"
 						arch=`echo "$arch" | tr - _`
@@ -197,6 +197,76 @@  translate_oe_to_smart() {
 					done
 					if [ "$pkg" = "$new_pkg" ]; then
 						# Failed to translate, package not found!
+						not_found="$not_found $pkg"
+						continue
+					fi
+				fi
+			done
+		fi
+		# Apparently not a multilib package...
+		if [ "$pkg" = "$new_pkg" ]; then
+			default_archs_fixed=`echo "$default_archs" | tr - _`
+			for arch in $default_archs_fixed ; do
+				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
+					new_pkg="$pkg@$arch"
+					# First found is best match
+					break
+				fi
+			done
+			if [ "$pkg" = "$new_pkg" ]; then
+				# Failed to translate, package not found!
+				not_found="$not_found $pkg"
+				continue
+			fi
+		fi
+		#echo "$pkg -> $new_pkg" >&2
+		pkgs_to_install="${pkgs_to_install} ${new_pkg}"
+	done
+
+	# Parse the not_found items and see if they were dependencies (RPROVIDES)
+	# Follow the parsing example above...
+	for pkg in $not_found ; do
+		new_pkg="$pkg"
+		smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$pkg --output ${target_rootfs}/install/tmp/provide.query
+		grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
+		prov=`echo $pkgs_to_install | xargs -n 1 echo | grep -f ${target_rootfs}/install/tmp/provide.query.list || :`
+		if [ -n "$prov" ]; then
+			# Nothing to do, already in the list
+			#echo "Skipping $pkg -> $prov, already in install set" >&2
+			continue
+		fi
+		if [ -z "$sdk_mode" ]; then
+			for i in ${MULTILIB_PREFIX_LIST} ; do
+				old_IFS="$IFS"
+				IFS=":"
+				set $i
+				IFS="$old_IFS"
+				mlib="$1"
+				shift
+				if [ "$mlib" = "default" ]; then
+					if [ -z "$default_archs" ]; then
+						default_archs=$@
+					fi
+					continue
+				fi
+				subst=${pkg#${mlib}-}
+				if [ "$subst" != "$pkg" ]; then
+					feeds=$@
+					smart --data-dir=${target_rootfs}/var/lib/smart query --provides=$subst --output ${target_rootfs}/install/tmp/provide.query
+					grep '^[^@ ]*@[^@]*$' ${target_rootfs}/install/tmp/provide.query | sed -e 's,\(.*\)-[^-]*-[^-]*\(@[^@]*\)$,\1\2,' > ${target_rootfs}/install/tmp/provide.query.list || :
+					while [ -n "$1" ]; do
+						arch="$1"
+						arch=`echo "$arch" | tr - _`
+						shift
+						# Select first found, we don't know if one is better then another...
+						prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
+						if [ -n "$prov" ]; then
+							new_pkg=$prov
+							break
+						fi
+					done
+					if [ "$pkg" = "$new_pkg" ]; then
+						# Failed to translate, package not found!
 						echo "$attemptonly: $pkg not found in the $mlib feeds ($feeds)." >&2
 						if [ "$attemptonly" = "Error" ]; then
 							exit 1
@@ -210,9 +280,10 @@  translate_oe_to_smart() {
 		if [ "$pkg" = "$new_pkg" ]; then
 			default_archs_fixed=`echo "$default_archs" | tr - _`
 			for arch in $default_archs_fixed ; do
-				if grep -q '^'$pkg'-[^-]*-[^-]*@'$arch'$' ${target_rootfs}/install/tmp/fullpkglist.query ; then
-					new_pkg="$pkg@$arch"
-					# First found is best match
+				# Select first found, we don't know if one is better then another...
+				prov=`grep '^[^@ ]*@'$arch'$' ${target_rootfs}/install/tmp/provide.query.list | head -n 1`
+				if [ -n "$prov" ]; then
+					new_pkg=$prov
 					break
 				fi
 			done