Patchwork python-smartpm: Add an attempt install mode

login
register
mail settings
Submitter Mark Hatle
Date Sept. 11, 2013, 1:59 a.m.
Message ID <1378864775-7094-1-git-send-email-mark.hatle@windriver.com>
Download mbox | patch
Permalink /patch/57793/
State Accepted
Commit bdf07b1698d228dc7ff555199a269b1ff8ceca19
Headers show

Comments

Mark Hatle - Sept. 11, 2013, 1:59 a.m.
[ YOCTO #3723 ]

Add a mode to smart that will allow an installation to continue, instead of
failure in the case that one or more items is uninstallable.

Uninstallable packages are simply ignored, and no error is generated.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/package_rpm.bbclass                   |  7 +--
 .../python/python-smartpm/smart-attempt.patch      | 59 ++++++++++++++++++++++
 .../python/python-smartpm_1.4.1.bb                 |  1 +
 3 files changed, 62 insertions(+), 5 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
Mark Hatle - Sept. 11, 2013, 2:04 a.m.
Time measurements with:

MACHINE = "qemux86"
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs"
BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j 8"

image: core-image-sato

Quad Core i7 workstation...

With all of the recipes built, and just executing the do_rootfs:

Before this change:
  real    26m2.541s
  user    21m8.458s
  sys     7m9.683s

After this change:
  real    14m43.436s
  user    15m57.525s
  sys     5m16.412s

This is a significant performance boost when using the 'complementary' installs 
and attemptonly installs.  The performance is by doing the complementary 
installs as a batch instead of one at a time, ignoring failures.

--Mark

On 9/10/13 8:59 PM, Mark Hatle wrote:
> [ YOCTO #3723 ]
>
> Add a mode to smart that will allow an installation to continue, instead of
> failure in the case that one or more items is uninstallable.
>
> Uninstallable packages are simply ignored, and no error is generated.
>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
>   meta/classes/package_rpm.bbclass                   |  7 +--
>   .../python/python-smartpm/smart-attempt.patch      | 59 ++++++++++++++++++++++
>   .../python/python-smartpm_1.4.1.bb                 |  1 +
>   3 files changed, 62 insertions(+), 5 deletions(-)
>   create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
>
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index 8ab42c4..699d0f2 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -472,11 +472,8 @@ EOF
>   		echo "Attempting $pkgs_to_install"
>   		echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
>   		translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly
> -		for each_pkg in $pkgs_to_install ;  do
> -			# We need to try each package individually as a single dependency failure
> -			# will break the whole set otherwise.
> -			smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true
> -		done
> +		echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
> +		smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1
>   	fi
>   }
>
> diff --git a/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
> new file mode 100644
> index 0000000..0d603d3
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
> @@ -0,0 +1,59 @@
> +Add a mechanism to attempt the install operation, w/o failing.
> +
> +For complementary and 'attemptonly' packages, we need a way to instruct smart to
> +try to install, but ignore any failures.
> +
> +This option only works for the install operation.
> +
> +Upstream-Status: Pending
> +
> +Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> +
> +Index: smart-1.4.1/smart/commands/install.py
> +===================================================================
> +--- smart-1.4.1.orig/smart/commands/install.py
> ++++ smart-1.4.1/smart/commands/install.py
> +@@ -50,6 +50,8 @@ def option_parser():
> +     parser = OptionParser(usage=USAGE,
> +                           description=DESCRIPTION,
> +                           examples=EXAMPLES)
> ++    parser.add_option("--attempt", action="store_true",
> ++                      help=_("attempt to install packages, ignore failures"))
> +     parser.add_option("--stepped", action="store_true",
> +                       help=_("split operation in steps"))
> +     parser.add_option("--urls", action="store_true",
> +@@ -80,6 +82,9 @@ def main(ctrl, opts):
> +     if not opts.args:
> +         raise Error, _("no package(s) given")
> +
> ++    if opts.attempt:
> ++	sysconf.set("attempt-install", True, soft=True)
> ++
> +     if opts.explain:
> +         sysconf.set("explain-changesets", True, soft=True)
> +
> +Index: smart-1.4.1/smart/transaction.py
> +===================================================================
> +--- smart-1.4.1.orig/smart/transaction.py
> ++++ smart-1.4.1/smart/transaction.py
> +@@ -1216,9 +1216,17 @@ class Transaction(object):
> +                     else:
> +                         op = REMOVE
> +                 if op is INSTALL or op is REINSTALL:
> +-                    self._install(pkg, changeset, locked, pending)
> +-                    if pkg in changeset:
> +-                        changeset.setRequested(pkg, True)
> ++                    try:
> ++                        self._install(pkg, changeset, locked, pending)
> ++                        if pkg in changeset:
> ++                            changeset.setRequested(pkg, True)
> ++                    except Failed, e:
> ++                        if sysconf.has("attempt-install", soft=True):
> ++                            if pkg in changeset:
> ++                                del changeset[pkg]
> ++                            continue
> ++                        else:
> ++                            raise Failed, e
> +                 elif op is REMOVE:
> +                     self._remove(pkg, changeset, locked, pending)
> +                 elif op is UPGRADE:
> diff --git a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> index 5d673b9..f406d69 100644
> --- a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> +++ b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
> @@ -30,6 +30,7 @@ SRC_URI = "\
>             file://smart-flag-ignore-recommends.patch \
>             file://smart-flag-exclude-packages.patch \
>             file://smart-config-ignore-all-recommends.patch \
> +          file://smart-attempt.patch \
>             "
>
>   SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
>
Paul Eggleton - Sept. 11, 2013, 10:04 a.m.
Hi Mark,

On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
> Time measurements with:
> 
> MACHINE = "qemux86"
> PACKAGE_CLASSES = "package_rpm"
> EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs
> ptest-pkgs" BB_NUMBER_THREADS ?= "8"
> PARALLEL_MAKE ?= "-j 8"
> 
> image: core-image-sato
> 
> Quad Core i7 workstation...
> 
> With all of the recipes built, and just executing the do_rootfs:
> 
> Before this change:
>   real    26m2.541s
>   user    21m8.458s
>   sys     7m9.683s
> 
> After this change:
>   real    14m43.436s
>   user    15m57.525s
>   sys     5m16.412s
> 
> This is a significant performance boost when using the 'complementary'
> installs and attemptonly installs.  The performance is by doing the
> complementary installs as a batch instead of one at a time, ignoring
> failures.

This is a significant improvement, well done! My concern about not catching 
failures stands though, which is why I hadn't implemented this yet. How do we 
ensure that genuine failures are caught with this mode?

Cheers,
Paul
Mark Hatle - Sept. 11, 2013, 1:47 p.m.
On 9/11/13 5:04 AM, Paul Eggleton wrote:
> Hi Mark,
>
> On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
>> Time measurements with:
>>
>> MACHINE = "qemux86"
>> PACKAGE_CLASSES = "package_rpm"
>> EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs
>> ptest-pkgs" BB_NUMBER_THREADS ?= "8"
>> PARALLEL_MAKE ?= "-j 8"
>>
>> image: core-image-sato
>>
>> Quad Core i7 workstation...
>>
>> With all of the recipes built, and just executing the do_rootfs:
>>
>> Before this change:
>>    real    26m2.541s
>>    user    21m8.458s
>>    sys     7m9.683s
>>
>> After this change:
>>    real    14m43.436s
>>    user    15m57.525s
>>    sys     5m16.412s
>>
>> This is a significant performance boost when using the 'complementary'
>> installs and attemptonly installs.  The performance is by doing the
>> complementary installs as a batch instead of one at a time, ignoring
>> failures.
>
> This is a significant improvement, well done! My concern about not catching
> failures stands though, which is why I hadn't implemented this yet. How do we
> ensure that genuine failures are caught with this mode?

The code only ignores failures in the resolution of the dependencies.  As far as 
I'm aware, this is the correct behavior in the attempt only mode(s).  If the 
dependencies can't be resolved for some reason, we simply throw away that item 
and move on.

Any errors that occur during package installation -will- now cause a failure to 
exist, and they will be logged to the attemptonly log-file.

In the previous implementation, any error when installing an attemptonly package 
would be ignored, even if it turned out to be an installation problem and not a 
dependency issue.  So I think this is an improvement as well.

(For some of my testing I intentionally created dependency errors by adding 
things to the PACKAGE_EXCLUDE, and the system did what it was supposed to. 
Ignored the items that couldn't be installed.)

--Mark

> Cheers,
> Paul
>
Paul Eggleton - Sept. 11, 2013, 2:29 p.m.
On Wednesday 11 September 2013 08:47:33 Mark Hatle wrote:
> On 9/11/13 5:04 AM, Paul Eggleton wrote:
> > On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
> >> Time measurements with:
> >> 
> >> MACHINE = "qemux86"
> >> PACKAGE_CLASSES = "package_rpm"
> >> EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs
> >> ptest-pkgs" BB_NUMBER_THREADS ?= "8"
> >> PARALLEL_MAKE ?= "-j 8"
> >> 
> >> image: core-image-sato
> >> 
> >> Quad Core i7 workstation...
> >> 
> >> With all of the recipes built, and just executing the do_rootfs:
> >> 
> >> Before this change:
> >>    real    26m2.541s
> >>    user    21m8.458s
> >>    sys     7m9.683s
> >> 
> >> After this change:
> >>    real    14m43.436s
> >>    user    15m57.525s
> >>    sys     5m16.412s
> >> 
> >> This is a significant performance boost when using the 'complementary'
> >> installs and attemptonly installs.  The performance is by doing the
> >> complementary installs as a batch instead of one at a time, ignoring
> >> failures.
> > 
> > This is a significant improvement, well done! My concern about not
> > catching
> > failures stands though, which is why I hadn't implemented this yet. How do
> > we ensure that genuine failures are caught with this mode?
> 
> The code only ignores failures in the resolution of the dependencies.  As
> far as I'm aware, this is the correct behavior in the attempt only mode(s).
>  If the dependencies can't be resolved for some reason, we simply throw
> away that item and move on.
> 
> Any errors that occur during package installation -will- now cause a failure
> to exist, and they will be logged to the attemptonly log-file.
> 
> In the previous implementation, any error when installing an attemptonly
> package would be ignored, even if it turned out to be an installation
> problem and not a dependency issue.  So I think this is an improvement as
> well.
> 
> (For some of my testing I intentionally created dependency errors by adding
> things to the PACKAGE_EXCLUDE, and the system did what it was supposed to.
> Ignored the items that couldn't be installed.)

Sounds like a great improvement. Thanks!

Cheers,
Paul

Patch

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 8ab42c4..699d0f2 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -472,11 +472,8 @@  EOF
 		echo "Attempting $pkgs_to_install"
 		echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
 		translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly
-		for each_pkg in $pkgs_to_install ;  do
-			# We need to try each package individually as a single dependency failure
-			# will break the whole set otherwise.
-			smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true
-		done
+		echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
+		smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1
 	fi
 }
 
diff --git a/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
new file mode 100644
index 0000000..0d603d3
--- /dev/null
+++ b/meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
@@ -0,0 +1,59 @@ 
+Add a mechanism to attempt the install operation, w/o failing.
+
+For complementary and 'attemptonly' packages, we need a way to instruct smart to
+try to install, but ignore any failures.
+
+This option only works for the install operation.
+
+Upstream-Status: Pending
+
+Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
+
+Index: smart-1.4.1/smart/commands/install.py
+===================================================================
+--- smart-1.4.1.orig/smart/commands/install.py
++++ smart-1.4.1/smart/commands/install.py
+@@ -50,6 +50,8 @@ def option_parser():
+     parser = OptionParser(usage=USAGE,
+                           description=DESCRIPTION,
+                           examples=EXAMPLES)
++    parser.add_option("--attempt", action="store_true",
++                      help=_("attempt to install packages, ignore failures"))
+     parser.add_option("--stepped", action="store_true",
+                       help=_("split operation in steps"))
+     parser.add_option("--urls", action="store_true",
+@@ -80,6 +82,9 @@ def main(ctrl, opts):
+     if not opts.args:
+         raise Error, _("no package(s) given")
+ 
++    if opts.attempt:
++	sysconf.set("attempt-install", True, soft=True)
++
+     if opts.explain:
+         sysconf.set("explain-changesets", True, soft=True)
+ 
+Index: smart-1.4.1/smart/transaction.py
+===================================================================
+--- smart-1.4.1.orig/smart/transaction.py
++++ smart-1.4.1/smart/transaction.py
+@@ -1216,9 +1216,17 @@ class Transaction(object):
+                     else:
+                         op = REMOVE
+                 if op is INSTALL or op is REINSTALL:
+-                    self._install(pkg, changeset, locked, pending)
+-                    if pkg in changeset:
+-                        changeset.setRequested(pkg, True)
++                    try:
++                        self._install(pkg, changeset, locked, pending)
++                        if pkg in changeset:
++                            changeset.setRequested(pkg, True)
++                    except Failed, e:
++                        if sysconf.has("attempt-install", soft=True):
++                            if pkg in changeset:
++                                del changeset[pkg]
++                            continue
++                        else:
++                            raise Failed, e
+                 elif op is REMOVE:
+                     self._remove(pkg, changeset, locked, pending)
+                 elif op is UPGRADE:
diff --git a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
index 5d673b9..f406d69 100644
--- a/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
+++ b/meta/recipes-devtools/python/python-smartpm_1.4.1.bb
@@ -30,6 +30,7 @@  SRC_URI = "\
           file://smart-flag-ignore-recommends.patch \
           file://smart-flag-exclude-packages.patch \
           file://smart-config-ignore-all-recommends.patch \
+          file://smart-attempt.patch \
           "
 
 SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"