[bitbake-devel,1/2] bb/utils.py: add iterate_chunks and hash_file helpers

Submitted by Rasmus Villemoes on July 10, 2018, 2:06 p.m. | Patch ID: 152496


Message ID 20180710140612.26245-1-rasmus.villemoes@prevas.dk
State New
Headers show

Commit Message

Rasmus Villemoes July 10, 2018, 2:06 p.m.
The current (md5,sha1,sha256)_file functions are somewhat inefficient in
that they loop over the lines of the given file. In a pathological case,
a huge binary file might have no \n characters at all, causing a lot of
realloc'ing on the way to providing the caller with the file as one big

For random binary data (and compressed tarballs are effectively
that) there's a \n roughly every 256 bytes, and for text files even more
often, but splitting the input buffer into lines is a waste of time, and
all hash methods work beter with nice aligned chunks.

So introduce two helpers: One that will iterate over a given file in
chunks, another that takes a hasher object and calls m.update() on the

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
 lib/bb/utils.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

Patch hide | download patch | download mbox

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 378e699e..cb7a6fa2 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -520,6 +520,28 @@  def unlockfile(lf):
     fcntl.flock(lf.fileno(), fcntl.LOCK_UN)
+def iterate_chunks(filename, chunk_size = 32768):
+    """
+    Return an iterator that yields the contents of filename in chunks
+    of size (up to) chunk_size.
+    """
+    with open(filename, "rb") as f:
+        while True:
+            chunk = f.read(chunk_size)
+            if chunk:
+                yield chunk
+            else:
+                return
+def hash_file(hasher, filename):
+    """
+    Update the hasher object with the contents of filename, returning
+    the hasher object.
+    """
+    for chunk in iterate_chunks(filename):
+        hasher.update(chunk)
+    return hasher
 def md5_file(filename):
     Return the hex string representation of the MD5 checksum of filename.