Patchwork [1/1] report-error.bbclass: new class to save build information when errors occur

login
register
mail settings
Submitter Andreea Proca
Date Feb. 13, 2014, 3:28 p.m.
Message ID <a4ffa9d1c2f040a1b78b5cbfa92ee7ad9e3402e0.1392303990.git.andreea.b.proca@intel.com>
Download mbox | patch
Permalink /patch/66717/
State New
Headers show

Comments

Andreea Proca - Feb. 13, 2014, 3:28 p.m.
Class is used to save data about errors after every task that failed.
Debug data is saved in json files in ERROR_REPORT_DIR (defaults to tmp/log/error-report).
To use this class one has to add INHERIT += "report-error" to local.conf.

scripts/send-error-report is a simple script that sends the json file
to a HTTP server that collects data (git://git.yoctoproject.org/error-report-web
is a Django web interface that can be used to receive and visualize
the error reports). The script will give you an URL where you can
find your report.

Signed-off-by: Andreea Proca <andreea.b.proca@intel.com>
---
 meta/classes/report-error.bbclass |   67 +++++++++++++++++++++++++++++++
 scripts/send-error-report         |   78 +++++++++++++++++++++++++++++++++++++
 2 files changed, 145 insertions(+)
 create mode 100644 meta/classes/report-error.bbclass
 create mode 100755 scripts/send-error-report
Richard Purdie - Feb. 13, 2014, 11:37 p.m.
On Thu, 2014-02-13 at 17:28 +0200, Andreea Proca wrote:
> Class is used to save data about errors after every task that failed.
> Debug data is saved in json files in ERROR_REPORT_DIR (defaults to tmp/log/error-report).
> To use this class one has to add INHERIT += "report-error" to local.conf.
> 
> scripts/send-error-report is a simple script that sends the json file
> to a HTTP server that collects data (git://git.yoctoproject.org/error-report-web
> is a Django web interface that can be used to receive and visualize
> the error reports). The script will give you an URL where you can
> find your report.
> 
> Signed-off-by: Andreea Proca <andreea.b.proca@intel.com>
> ---
>  meta/classes/report-error.bbclass |   67 +++++++++++++++++++++++++++++++
>  scripts/send-error-report         |   78 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 145 insertions(+)
>  create mode 100644 meta/classes/report-error.bbclass
>  create mode 100755 scripts/send-error-report
> 
> diff --git a/meta/classes/report-error.bbclass b/meta/classes/report-error.bbclass
> new file mode 100644
> index 0000000..149b36c
> --- /dev/null
> +++ b/meta/classes/report-error.bbclass
> @@ -0,0 +1,67 @@
> +#
> +# Collects debug information in order to create error report files.
> +#
> +# Copyright (C) 2013 Intel Corporation
> +# Author: Andreea Brandusa Proca <andreea.b.proca@intel.com>
> +#
> +# Licensed under the MIT license, see COPYING.MIT for details
> +
> +ERR_REPORT_DIR ?= "${LOG_DIR}/error-report"
> +
> +def getdata(e):
> +    logpath = e.data.getVar('ERR_REPORT_DIR', True)
> +    datafile = os.path.join(logpath, "error-report.txt")
> +    f = open(datafile, "r")
> +    data = f.read()
> +    f.close()
> +    return data
> +
> +def savedata(e, newdata, file):
> +    import json
> +    logpath = e.data.getVar('ERR_REPORT_DIR', True)
> +    bb.utils.mkdirhier(logpath)
> +    datafile = os.path.join(logpath, file)
> +    f = open(datafile, "w")
> +    json.dump(newdata, f, indent=4, sort_keys=True)
> +    f.close()
> +    return datafile

Since these functions will end up in the main namespace, could we prefix
them so something like errorreport_getdata and errorreport_savedata?
This makes it clearer when you look at the names out of context of this
class.


> +python report () {
> +        import os, re, sys, json
> +
> +        if isinstance(e, bb.event.BuildStarted):
> +            data = {}
> +            machine = e.data.getVar("MACHINE")
> +            data['machine'] = machine
> +            data['build_sys'] = e.data.getVar("BUILD_SYS", True)
> +            data['nativelsb'] = e.data.getVar("NATIVELSBSTRING")
> +            data['distro'] = e.data.getVar("DISTRO")
> +            data['target_sys'] = e.data.getVar("TARGET_SYS", True)
> +            data['failures'] = []
> +            data['component'] = e.getPkgs()[0]
> +            data['branch_commit'] = base_detect_branch(e.data) + ": " + base_detect_revision(e.data)
> +            savedata(e, data, "error-report.txt")
> +
> +        elif isinstance(e, bb.build.TaskFailed):
> +            task = e.task
> +            taskdata={}
> +            log = e.data.getVar('BB_LOGFILE', True)
> +            logFile = open(log, 'r')
> +            taskdata['package'] = e.data.expand("${PF}")
> +            taskdata['task'] = task
> +            taskdata['log'] = logFile.read()
> +            logFile.close()
> +            jsondata = json.loads(getdata(e))
> +            jsondata['failures'].append(taskdata)
> +            savedata(e, jsondata, "error-report.txt")
> +
> +        elif isinstance(e, bb.event.BuildCompleted):
> +            jsondata = json.loads(getdata(e))
> +            failures = jsondata['failures']
> +            if(len(failures) > 0):
> +                filename = "error_report_" + e.data.getVar("BUILDNAME")+".txt"
> +                datafile = savedata(e, jsondata, filename)
> +                bb.note("The errors of this build are stored in: %s. You can send the errors to an upstream server by running: send-error-report %s [server]" % (datafile, datafile))
> +}
> +
> +addhandler report

Could you name this something like errorreport_handler for the same
reason as the functions and add an eventmask:

errorreport_handler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted bb.build.TaskFailed"

which means that handler only gets the events it is interested in. I
suspect this syntax was added after you started work on this code but it
would be nice to add.

I'm really looking forward to seeing this in action!

Cheers,

Richard

> diff --git a/scripts/send-error-report b/scripts/send-error-report
> new file mode 100755
> index 0000000..0d85776
> --- /dev/null
> +++ b/scripts/send-error-report
> @@ -0,0 +1,78 @@
> +#!/usr/bin/env python
> +
> +# Sends an error report (if the report-error class was enabled) to a remote server.
> +#
> +# Copyright (C) 2013 Intel Corporation
> +# Author: Andreea Proca <andreea.b.proca@intel.com>
> +
> +
> +
> +import httplib, urllib, os, sys, json
> +
> +
> +def sendData(json_file, server):
> +
> +    if os.path.isfile(json_file):
> +
> +        home = os.path.expanduser("~")
> +        userfile = os.path.join(home, ".oe-send-error")
> +        if os.path.isfile(userfile):
> +            with open(userfile) as g:
> +                username = g.readline()
> +                email = g.readline()
> +        else:
> +            print("Please enter your name and your email (optionally), they'll be saved in the file you send.")
> +            username = raw_input("Name: ")
> +            email = raw_input("E-mail (not required): ")
> +            if len(username) > 0 and len(username) < 50:
> +                with open(userfile, "w") as g:
> +                    g.write(username + "\n")
> +                    g.write(email + "\n")
> +            else:
> +                print("Invalid inputs, try again.")
> +                return
> +
> +        with open(json_file) as f:
> +            data = f.read()
> +
> +        try:
> +            jsondata = json.loads(data)
> +            jsondata['username'] = username.strip()
> +            jsondata['email'] = email.strip()
> +            data = json.dumps(jsondata, indent=4, sort_keys=True)
> +        except:
> +            print("Invalid json data")
> +            return
> +
> +        try:
> +            params = urllib.urlencode({'data': data})
> +            headers = {"Content-type": "application/json"}
> +            conn = httplib.HTTPConnection(server)
> +            conn.request("POST", "/ClientPost/", params, headers)
> +            response = conn.getresponse()
> +            print response.status, response.reason
> +            res = response.read()
> +            if response.status == 200:
> +                print res
> +            else:
> +                print("There was a problem submiting your data")
> +            conn.close()
> +        except:
> +                print("Server connection failed")
> +
> +    else:
> +        print("No data file found.")
> +
> +
> +if __name__ == '__main__':
> +    print ("\nSends an error report (if the report-error class was enabled) to a remote server.")
> +    if len(sys.argv) < 2:
> +        print("\nThis scripts sends the contents of a file to an upstream server.")
> +        print("\nUsage: send-error-report <error_fileName> [server]")
> +        print("\nIf this is the first when sending a report you'll be asked for your name and optionally your email address.")
> +        print("They will be associated with your report.\n")
> +
> +    elif len(sys.argv) == 3:
> +        sendData(sys.argv[1], sys.argv[2])
> +    else:
> +        sendData(sys.argv[1], "localhost:8000")

Patch

diff --git a/meta/classes/report-error.bbclass b/meta/classes/report-error.bbclass
new file mode 100644
index 0000000..149b36c
--- /dev/null
+++ b/meta/classes/report-error.bbclass
@@ -0,0 +1,67 @@ 
+#
+# Collects debug information in order to create error report files.
+#
+# Copyright (C) 2013 Intel Corporation
+# Author: Andreea Brandusa Proca <andreea.b.proca@intel.com>
+#
+# Licensed under the MIT license, see COPYING.MIT for details
+
+ERR_REPORT_DIR ?= "${LOG_DIR}/error-report"
+
+def getdata(e):
+    logpath = e.data.getVar('ERR_REPORT_DIR', True)
+    datafile = os.path.join(logpath, "error-report.txt")
+    f = open(datafile, "r")
+    data = f.read()
+    f.close()
+    return data
+
+def savedata(e, newdata, file):
+    import json
+    logpath = e.data.getVar('ERR_REPORT_DIR', True)
+    bb.utils.mkdirhier(logpath)
+    datafile = os.path.join(logpath, file)
+    f = open(datafile, "w")
+    json.dump(newdata, f, indent=4, sort_keys=True)
+    f.close()
+    return datafile
+
+python report () {
+        import os, re, sys, json
+
+        if isinstance(e, bb.event.BuildStarted):
+            data = {}
+            machine = e.data.getVar("MACHINE")
+            data['machine'] = machine
+            data['build_sys'] = e.data.getVar("BUILD_SYS", True)
+            data['nativelsb'] = e.data.getVar("NATIVELSBSTRING")
+            data['distro'] = e.data.getVar("DISTRO")
+            data['target_sys'] = e.data.getVar("TARGET_SYS", True)
+            data['failures'] = []
+            data['component'] = e.getPkgs()[0]
+            data['branch_commit'] = base_detect_branch(e.data) + ": " + base_detect_revision(e.data)
+            savedata(e, data, "error-report.txt")
+
+        elif isinstance(e, bb.build.TaskFailed):
+            task = e.task
+            taskdata={}
+            log = e.data.getVar('BB_LOGFILE', True)
+            logFile = open(log, 'r')
+            taskdata['package'] = e.data.expand("${PF}")
+            taskdata['task'] = task
+            taskdata['log'] = logFile.read()
+            logFile.close()
+            jsondata = json.loads(getdata(e))
+            jsondata['failures'].append(taskdata)
+            savedata(e, jsondata, "error-report.txt")
+
+        elif isinstance(e, bb.event.BuildCompleted):
+            jsondata = json.loads(getdata(e))
+            failures = jsondata['failures']
+            if(len(failures) > 0):
+                filename = "error_report_" + e.data.getVar("BUILDNAME")+".txt"
+                datafile = savedata(e, jsondata, filename)
+                bb.note("The errors of this build are stored in: %s. You can send the errors to an upstream server by running: send-error-report %s [server]" % (datafile, datafile))
+}
+
+addhandler report
diff --git a/scripts/send-error-report b/scripts/send-error-report
new file mode 100755
index 0000000..0d85776
--- /dev/null
+++ b/scripts/send-error-report
@@ -0,0 +1,78 @@ 
+#!/usr/bin/env python
+
+# Sends an error report (if the report-error class was enabled) to a remote server.
+#
+# Copyright (C) 2013 Intel Corporation
+# Author: Andreea Proca <andreea.b.proca@intel.com>
+
+
+
+import httplib, urllib, os, sys, json
+
+
+def sendData(json_file, server):
+
+    if os.path.isfile(json_file):
+
+        home = os.path.expanduser("~")
+        userfile = os.path.join(home, ".oe-send-error")
+        if os.path.isfile(userfile):
+            with open(userfile) as g:
+                username = g.readline()
+                email = g.readline()
+        else:
+            print("Please enter your name and your email (optionally), they'll be saved in the file you send.")
+            username = raw_input("Name: ")
+            email = raw_input("E-mail (not required): ")
+            if len(username) > 0 and len(username) < 50:
+                with open(userfile, "w") as g:
+                    g.write(username + "\n")
+                    g.write(email + "\n")
+            else:
+                print("Invalid inputs, try again.")
+                return
+
+        with open(json_file) as f:
+            data = f.read()
+
+        try:
+            jsondata = json.loads(data)
+            jsondata['username'] = username.strip()
+            jsondata['email'] = email.strip()
+            data = json.dumps(jsondata, indent=4, sort_keys=True)
+        except:
+            print("Invalid json data")
+            return
+
+        try:
+            params = urllib.urlencode({'data': data})
+            headers = {"Content-type": "application/json"}
+            conn = httplib.HTTPConnection(server)
+            conn.request("POST", "/ClientPost/", params, headers)
+            response = conn.getresponse()
+            print response.status, response.reason
+            res = response.read()
+            if response.status == 200:
+                print res
+            else:
+                print("There was a problem submiting your data")
+            conn.close()
+        except:
+                print("Server connection failed")
+
+    else:
+        print("No data file found.")
+
+
+if __name__ == '__main__':
+    print ("\nSends an error report (if the report-error class was enabled) to a remote server.")
+    if len(sys.argv) < 2:
+        print("\nThis scripts sends the contents of a file to an upstream server.")
+        print("\nUsage: send-error-report <error_fileName> [server]")
+        print("\nIf this is the first when sending a report you'll be asked for your name and optionally your email address.")
+        print("They will be associated with your report.\n")
+
+    elif len(sys.argv) == 3:
+        sendData(sys.argv[1], sys.argv[2])
+    else:
+        sendData(sys.argv[1], "localhost:8000")