ptest-runner: return 1 instead of zero if any test fails

Submitted by Laszlo Papp on Aug. 22, 2014, 2:05 p.m.

Details

Message ID 1408716303-32264-1-git-send-email-root@pol-lx-007.polatis.com
State Accepted
Commit d6b1c9e7e29aaa8d71ccb485b7a88863a117a8b1
Headers show

Commit Message

Laszlo Papp Aug. 22, 2014, 2:05 p.m.
From: Laszlo Papp <lpapp@kde.org>

It is better to know this information from the exit code rather than parsing
test specific outputs.

Signed-off-by: Laszlo Papp <lpapp@kde.org>
---
 meta/recipes-support/ptest-runner/files/ptest-runner | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

Patch hide | download patch | download mbox

diff --git a/meta/recipes-support/ptest-runner/files/ptest-runner b/meta/recipes-support/ptest-runner/files/ptest-runner
index 3342413..c618f11 100644
--- a/meta/recipes-support/ptest-runner/files/ptest-runner
+++ b/meta/recipes-support/ptest-runner/files/ptest-runner
@@ -1,5 +1,5 @@ 
 #!/bin/sh
-
+ANYFAILED=no
 echo "START: $0"
 
 for libdir in /usr/lib*
@@ -15,9 +15,13 @@  do
         date "+%Y-%m-%dT%H:%M"
         echo "BEGIN: $x"
         cd "$x"
-        ./run-ptest
+        ./run-ptest || ANYFAILED=yes
         echo "END: $x"
         date "+%Y-%m-%dT%H:%M"
     done
 done
 echo "STOP: $0"
+if [ "$ANYFAILED" = "yes"  ]; then
+    exit 1
+fi
+exit 0

Comments

Corneliu Stoicescu Aug. 25, 2014, 6:42 a.m.
Hello,

From my experience with ptest, there will always be a few tests failing, so we will always get exit code 1. Should this better return exit code 0 if the individual tests performed correctly(weather they passed or failed) and 1 if there were any errors during the running of the tests?

Regards,
Corneliu
Yocto QA

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> Of root@myhostname
> Sent: Friday, August 22, 2014 5:05 PM
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH] ptest-runner: return 1 instead of zero if any test
> fails
> 
> From: Laszlo Papp <lpapp@kde.org>
> 
> It is better to know this information from the exit code rather than parsing
> test specific outputs.
> 
> Signed-off-by: Laszlo Papp <lpapp@kde.org>
> ---
>  meta/recipes-support/ptest-runner/files/ptest-runner | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/recipes-support/ptest-runner/files/ptest-runner
> b/meta/recipes-support/ptest-runner/files/ptest-runner
> index 3342413..c618f11 100644
> --- a/meta/recipes-support/ptest-runner/files/ptest-runner
> +++ b/meta/recipes-support/ptest-runner/files/ptest-runner
> @@ -1,5 +1,5 @@
>  #!/bin/sh
> -
> +ANYFAILED=no
>  echo "START: $0"
> 
>  for libdir in /usr/lib*
> @@ -15,9 +15,13 @@ do
>          date "+%Y-%m-%dT%H:%M"
>          echo "BEGIN: $x"
>          cd "$x"
> -        ./run-ptest
> +        ./run-ptest || ANYFAILED=yes
>          echo "END: $x"
>          date "+%Y-%m-%dT%H:%M"
>      done
>  done
>  echo "STOP: $0"
> +if [ "$ANYFAILED" = "yes"  ]; then
> +    exit 1
> +fi
> +exit 0
> --
> 2.0.4
> 
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
Laszlo Papp Aug. 29, 2014, 10:54 a.m.
On Mon, Aug 25, 2014 at 7:42 AM, Stoicescu, CorneliuX
<corneliux.stoicescu@intel.com> wrote:
> Hello,
>
> From my experience with ptest, there will always be a few tests failing, so we will always get exit code 1.

Ptest is not only about meta-core, you need to remember that. My use
case is actually quite different: running ptest only on my binaries,
and I will trust the rest of the system. For me, it is critical to see
if there is any issues in my system. This patch resolves that use
case, but even for Yocto, you really oughta aim for fixing your tests,
either with "known failure" or correct means.

> Should this better return exit code 0 if the individual tests performed correctly(weather they passed or failed) and 1 if there were any errors during the running of the tests?

I do not understand this question. This is what the current change tries to do.
Laszlo Papp Nov. 27, 2014, 5:53 p.m.
Ping. I have not received a reply for about three months now? What is
the problem?

On Fri, Aug 29, 2014 at 11:54 AM, Laszlo Papp <lpapp@kde.org> wrote:
> On Mon, Aug 25, 2014 at 7:42 AM, Stoicescu, CorneliuX
> <corneliux.stoicescu@intel.com> wrote:
>> Hello,
>>
>> From my experience with ptest, there will always be a few tests failing, so we will always get exit code 1.
>
> Ptest is not only about meta-core, you need to remember that. My use
> case is actually quite different: running ptest only on my binaries,
> and I will trust the rest of the system. For me, it is critical to see
> if there is any issues in my system. This patch resolves that use
> case, but even for Yocto, you really oughta aim for fixing your tests,
> either with "known failure" or correct means.
>
>> Should this better return exit code 0 if the individual tests performed correctly(weather they passed or failed) and 1 if there were any errors during the running of the tests?
>
> I do not understand this question. This is what the current change tries to do.
Ross Burton Nov. 27, 2014, 6:02 p.m.
On 27 November 2014 at 17:53, Laszlo Papp <lpapp@kde.org> wrote:

> Ping. I have not received a reply for about three months now? What is
> the problem?
>

http://git.openembedded.org/openembedded-core/commit/?id=d6b1c9e7e29aaa8d71ccb485b7a88863a117a8b1

Ross
Laszlo Papp Nov. 27, 2014, 6:05 p.m.
On Thu, Nov 27, 2014 at 6:02 PM, Burton, Ross <ross.burton@intel.com> wrote:
>
> On 27 November 2014 at 17:53, Laszlo Papp <lpapp@kde.org> wrote:
>>
>> Ping. I have not received a reply for about three months now? What is
>> the problem?
>
>
> http://git.openembedded.org/openembedded-core/commit/?id=d6b1c9e7e29aaa8d71ccb485b7a88863a117a8b1

Thanks.

> Ross