[ptest-runner] ptest-runner: Add support timeout failure type to XML result file

Submitted by Aníbal Limón on Jan. 3, 2019, 6:02 p.m. | Patch ID: 157546

Details

Message ID 20190103180218.6968-1-anibal.limon@linaro.org
State New
Headers show

Commit Message

Aníbal Limón Jan. 3, 2019, 6:02 p.m.
The ptest-runner support logging results to stdout and to a XML file
in stdout the ptest is mark as:

...
ERROR: Exit status is 1
TIMEOUT: ptest-directory
...

Add the same support in XML file for example,

...
<testcase classname='ptest-directory' name='run-ptest'>
  <failure type='exit_code' message='run-ptest exited with code: 1'></failure>
  <failure type='timeout'/>
</testcase>
...

[YOCTO #13088]

Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
---
 tests/data/reference.xml |  1 +
 tests/utils.c            |  4 ++--
 utils.c                  | 28 ++++++++++++++++------------
 utils.h                  |  2 +-
 4 files changed, 20 insertions(+), 15 deletions(-)

Patch hide | download patch | download mbox

diff --git a/tests/data/reference.xml b/tests/data/reference.xml
index 91522c7..17f91c2 100644
--- a/tests/data/reference.xml
+++ b/tests/data/reference.xml
@@ -4,5 +4,6 @@ 
 	</testcase>
 	<testcase classname='test2' name='run-ptest'>
 		<failure type='exit_code' message='run-ptest exited with code: 1'></failure>
+		<failure type='timeout'/>
 	</testcase>
 </testsuite>
diff --git a/tests/utils.c b/tests/utils.c
index cf09379..662abe8 100644
--- a/tests/utils.c
+++ b/tests/utils.c
@@ -257,8 +257,8 @@  START_TEST(test_xml_pass)
 	FILE *xp;
 	xp = xml_create(2, "./test.xml");
 	ck_assert(xp != NULL);
-	xml_add_case(xp, 0,"test1");
-	xml_add_case(xp, 1,"test2");
+	xml_add_case(xp, 0,"test1", 0);
+	xml_add_case(xp, 1,"test2", 1);
 	xml_finish(xp);
 
 	FILE *fp, *fr;
diff --git a/utils.c b/utils.c
index ed2eff7..4a38ea1 100644
--- a/utils.c
+++ b/utils.c
@@ -260,14 +260,13 @@  run_child(char *run_ptest, int fd_stdout, int fd_stderr)
 
 static inline int
 wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
-		int timeout, int *fds, FILE **fps)
+		int timeout, int *fds, FILE **fps, int *timeouted)
 {
 	struct pollfd pfds[2];
 	struct timespec sentinel;
 	clockid_t clock = CLOCK_MONOTONIC;
 	int r;
 
-	int timeouted = 0;
 	int status;
 	int waitflags;
 
@@ -281,6 +280,8 @@  wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
 		clock_gettime(clock, &sentinel);
 	}
 
+	*timeouted = 0;
+
 	while (1) {
 		waitflags = WNOHANG;
 
@@ -305,7 +306,7 @@  wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
 
 			clock_gettime(clock, &time);
 			if ((time.tv_sec - sentinel.tv_sec) > timeout) {
-				timeouted = 1;
+				*timeouted = 1;
 				kill(pid, SIGKILL);
 				waitflags = 0;
 			}
@@ -315,11 +316,6 @@  wait_child(const char *ptest_dir, const char *run_ptest, pid_t pid,
 			break;
 	}
 
-	if (status) {
-		fprintf(fps[0], "\nERROR: Exit status is %d\n", status);
-		if (timeouted)
-			fprintf(fps[0], "TIMEOUT: %s\n", ptest_dir);
-	}
 
 	return status;
 }
@@ -337,6 +333,7 @@  run_ptests(struct ptest_list *head, const struct ptest_options opts,
 	pid_t child;
 	int pipefd_stdout[2];
 	int pipefd_stderr[2];
+	int timeouted;
 
 	if (opts.xml_filename) {
 		xh = xml_create(ptest_list_length(head), opts.xml_filename);
@@ -380,12 +377,17 @@  run_ptests(struct ptest_list *head, const struct ptest_options opts,
 				fprintf(fp, "BEGIN: %s\n", ptest_dir);
 
 				status = wait_child(ptest_dir, p->run_ptest, child,
-						opts.timeout, fds, fps);
-				if (status)
+						opts.timeout, fds, fps, &timeouted);
+
+				if (status) {
+					fprintf(fps[0], "\nERROR: Exit status is %d\n", status);
 					rc += 1;
+				}
+				if (timeouted)
+					fprintf(fps[0], "TIMEOUT: %s\n", ptest_dir);
 
 				if (opts.xml_filename)
-					xml_add_case(xh, status, ptest_dir);
+					xml_add_case(xh, status, ptest_dir, timeouted);
 
 				fprintf(fp, "END: %s\n", ptest_dir);
 				fprintf(fp, "%s\n", get_stime(stime, GET_STIME_BUF_SIZE));
@@ -424,7 +426,7 @@  xml_create(int test_count, char *xml_filename)
 }
 
 void
-xml_add_case(FILE *xh, int status, const char *ptest_dir)
+xml_add_case(FILE *xh, int status, const char *ptest_dir, int timeouted)
 {
 	fprintf(xh, "\t<testcase classname='%s' name='run-ptest'>\n", ptest_dir);
 
@@ -433,6 +435,8 @@  xml_add_case(FILE *xh, int status, const char *ptest_dir)
 		fprintf(xh, " message='run-ptest exited with code: %d'>", status);
 		fprintf(xh, "</failure>\n");
 	}
+	if (timeouted)
+		fprintf(xh, "\t\t<failure type='timeout'/>\n");
 
 	fprintf(xh, "\t</testcase>\n");
 }
diff --git a/utils.h b/utils.h
index ee85163..880105f 100644
--- a/utils.h
+++ b/utils.h
@@ -48,7 +48,7 @@  extern int run_ptests(struct ptest_list *, const struct ptest_options,
 		const char *, FILE *, FILE *);
 
 extern FILE *xml_create(int, char *);
-extern void xml_add_case(FILE *, int, const char *);
+extern void xml_add_case(FILE *, int, const char *, int);
 extern void xml_finish(FILE *);
 
 #endif

Comments

Richard Purdie Jan. 3, 2019, 9:45 p.m.
On Thu, 2019-01-03 at 12:02 -0600, Aníbal Limón wrote:
> The ptest-runner support logging results to stdout and to a XML file
> in stdout the ptest is mark as:
> 
> ...
> ERROR: Exit status is 1
> TIMEOUT: ptest-directory
> ...
> 
> Add the same support in XML file for example,
> 
> ...
> <testcase classname='ptest-directory' name='run-ptest'>
>   <failure type='exit_code' message='run-ptest exited with code:
> 1'></failure>
>   <failure type='timeout'/>
> </testcase>
> ...
> 
> [YOCTO #13088]
> 
> Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>

Sounds good, thanks!

I have a feeling this will show up a few bugs but we should find and
fix the ones which are timing out...

I'd still be interested in the time each directory takes too if we can
get that as it would help us decide which tests to run.

Cheers,

Richard
Aníbal Limón Jan. 5, 2019, 5:30 p.m.
On Thu, 3 Jan 2019 at 15:45, <richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2019-01-03 at 12:02 -0600, Aníbal Limón wrote:
> > The ptest-runner support logging results to stdout and to a XML file
> > in stdout the ptest is mark as:
> >
> > ...
> > ERROR: Exit status is 1
> > TIMEOUT: ptest-directory
> > ...
> >
> > Add the same support in XML file for example,
> >
> > ...
> > <testcase classname='ptest-directory' name='run-ptest'>
> >   <failure type='exit_code' message='run-ptest exited with code:
> > 1'></failure>
> >   <failure type='timeout'/>
> > </testcase>
> > ...
> >
> > [YOCTO #13088]
> >
> > Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
>
> Sounds good, thanks!
>
> I have a feeling this will show up a few bugs but we should find and
> fix the ones which are timing out...
>
> I'd still be interested in the time each directory takes too if we can
> get that as it would help us decide which tests to run.
>

The timeout is set to 5 minutes (300 secs) by default and currently there
is no way to set a timeout per ptest,
I will create other patch to add the timeout value to stdout and XML file.

Anibal

>
> Cheers,
>
> Richard
>
>
Richard Purdie Jan. 5, 2019, 6:52 p.m.
On Sat, 2019-01-05 at 11:30 -0600, Anibal Limon wrote:
> 
> 
> On Thu, 3 Jan 2019 at 15:45, <richard.purdie@linuxfoundation.org>
> wrote:
> > On Thu, 2019-01-03 at 12:02 -0600, Aníbal Limón wrote:
> > > The ptest-runner support logging results to stdout and to a XML
> > file
> > > in stdout the ptest is mark as:
> > > 
> > > ...
> > > ERROR: Exit status is 1
> > > TIMEOUT: ptest-directory
> > > ...
> > > 
> > > Add the same support in XML file for example,
> > > 
> > > ...
> > > <testcase classname='ptest-directory' name='run-ptest'>
> > >   <failure type='exit_code' message='run-ptest exited with code:
> > > 1'></failure>
> > >   <failure type='timeout'/>
> > > </testcase>
> > > ...
> > > 
> > > [YOCTO #13088]
> > > 
> > > Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
> > 
> > Sounds good, thanks!
> > 
> > I have a feeling this will show up a few bugs but we should find
> > and
> > fix the ones which are timing out...
> > 
> > I'd still be interested in the time each directory takes too if we
> > can
> > get that as it would help us decide which tests to run.
> 
> The timeout is set to 5 minutes (300 secs) by default and currently
> there is no way to set a timeout per ptest, 
> I will create other patch to add the timeout value to stdout and XML
> file.

That is handy but what I mean is the execution time each one takes. If
we have test which take 5s its easier to run them regularly than tests
which take 500s. Having an idea of their relative execution time would
therefore be helpful.

I assume the timeout takes effect after no output for that period?

Cheers,

Richard
Aníbal Limón Jan. 5, 2019, 9:45 p.m.
On Sat, 5 Jan 2019 at 12:52, <richard.purdie@linuxfoundation.org> wrote:

> On Sat, 2019-01-05 at 11:30 -0600, Anibal Limon wrote:
> >
> >
> > On Thu, 3 Jan 2019 at 15:45, <richard.purdie@linuxfoundation.org>
> > wrote:
> > > On Thu, 2019-01-03 at 12:02 -0600, Aníbal Limón wrote:
> > > > The ptest-runner support logging results to stdout and to a XML
> > > file
> > > > in stdout the ptest is mark as:
> > > >
> > > > ...
> > > > ERROR: Exit status is 1
> > > > TIMEOUT: ptest-directory
> > > > ...
> > > >
> > > > Add the same support in XML file for example,
> > > >
> > > > ...
> > > > <testcase classname='ptest-directory' name='run-ptest'>
> > > >   <failure type='exit_code' message='run-ptest exited with code:
> > > > 1'></failure>
> > > >   <failure type='timeout'/>
> > > > </testcase>
> > > > ...
> > > >
> > > > [YOCTO #13088]
> > > >
> > > > Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
> > >
> > > Sounds good, thanks!
> > >
> > > I have a feeling this will show up a few bugs but we should find
> > > and
> > > fix the ones which are timing out...
> > >
> > > I'd still be interested in the time each directory takes too if we
> > > can
> > > get that as it would help us decide which tests to run.
> >
> > The timeout is set to 5 minutes (300 secs) by default and currently
> > there is no way to set a timeout per ptest,
> > I will create other patch to add the timeout value to stdout and XML
> > file.
>
> That is handy but what I mean is the execution time each one takes. If
> we have test which take 5s its easier to run them regularly than tests
> which take 500s. Having an idea of their relative execution time would
> therefore be helpful.
>

Got it, I can add a duration for example:.

In stdout will be (after end),

...
BEGIN: TIME
END: ptest-directory
DURATION: Ns
END: TIME
...

In XML:

...
<testcase classname='%s' name='run-ptest' duration='Ns'>
</testcase>
...

or

...
<testcase classname='%s' name='run-ptest' >
<duration>Ns</duration>
</testcase>
...

What is the better? or any other?


> I assume the timeout takes effect after no output for that period?
>

Right, the timeout is based on stdout data.

Anibal

>
> Cheers,
>
> Richard
>
>