From patchwork Fri Feb 25 14:58:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 4305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC1DAC433EF for ; Fri, 25 Feb 2022 14:58:32 +0000 (UTC) Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by mx.groups.io with SMTP id smtpd.web08.7164.1645801112228385868 for ; Fri, 25 Feb 2022 06:58:32 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=H3y7qq0o; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.50, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f50.google.com with SMTP id j17so5010191wrc.0 for ; Fri, 25 Feb 2022 06:58:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=rYO4w8DIZLPA1cpSdr/USX1+OGONV5UqcTWWkD87fao=; b=H3y7qq0osXdeogYymC0nu5QkYYufxNAB4v6BNchOaErjeBS0zmbSdA3IrrecyLRrw8 1m5Bh+cDhiN+w/mflYi4BclY9eEsAM6h0Q8uuOq0M4KkKc6BCfdJ3eyYWKct25ItK7qA tE291mJoRcuX739w47hNql9hDW6/x27T0BgTA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=rYO4w8DIZLPA1cpSdr/USX1+OGONV5UqcTWWkD87fao=; b=KPN15WLLbe9AEJB01tup1/l2VYhRFAaXhDcjjsJkl3iK7KAgtdvh8KLkvp/MYdi4PJ 3YdryyXdpvzrOghfAk5dQgdTSpz71lkwmBstP65h+uVXnbBOoRKHERexL6RU6PuWgDeI LEatkYFE2CwQ/Y8Qf2Xg884e+EtXFosixgnshHMxS7LiL90zJ2WGpuZFPfLPtigoc/P5 L9pGkHm/nSMTGwifZgkhxfQRoHuq9z7QPERCNYhi7CMuBPwqBHoILg4m9HlQwDzhos8o 1sE0ONJoXYHT9ccqB1BpmpW9M1Ckuf+mHxgg5hNtJ7E9UYqe+/poTpfap9wDYBmdJK8C 2YSA== X-Gm-Message-State: AOAM530b3MzuSbLfV5RgXJTBa/WflChZlPUvArQSGm7ja5dtW5cB4HXC 6mVyPqrGKPVK/hKbsqHraabkyn9l0a3nEg5w X-Google-Smtp-Source: ABdhPJzKufZ1zLxU5DAEcLYuAIEXfD9wEE3GZ4igzW5ORRLdEmFIVf4EMNb8fmeqc//LzjSq9/t5rA== X-Received: by 2002:adf:8063:0:b0:1ef:78e3:330 with SMTP id 90-20020adf8063000000b001ef78e30330mr1087521wrk.424.1645801110017; Fri, 25 Feb 2022 06:58:30 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:8c05:a731:3c1:5d08]) by smtp.gmail.com with ESMTPSA id r2-20020a05600c35c200b00352cdcdd7b2sm17121733wmq.0.2022.02.25.06.58.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Feb 2022 06:58:29 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 1/3] python-pip: Improve reproducibility Date: Fri, 25 Feb 2022 14:58:26 +0000 Message-Id: <20220225145828.88238-1-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 25 Feb 2022 14:58:32 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/162399 Pip installed wheels are not reproducible currently. The direct_url files encode an installation path and the installed wheels compile the python files at their location, not their final install location which is incorrect. To fix this, simply disable the direct_urls and pass the "root" to the python compile function to strip that path out of the compiled files. Signed-off-by: Richard Purdie --- .../python/python3-pip/reproducible.patch | 74 +++++++++++++++++++ .../python/python3-pip_22.0.3.bb | 1 + 2 files changed, 75 insertions(+) create mode 100644 meta/recipes-devtools/python/python3-pip/reproducible.patch diff --git a/meta/recipes-devtools/python/python3-pip/reproducible.patch b/meta/recipes-devtools/python/python3-pip/reproducible.patch new file mode 100644 index 00000000000..538bb94f7ad --- /dev/null +++ b/meta/recipes-devtools/python/python3-pip/reproducible.patch @@ -0,0 +1,74 @@ +Pip installed wheels are not reproducible currently. The direct_url +files encode an installation path and the installed wheels compile +the python files at their location, not their final install location +which is incorrect. + +To fix this, simply disable the direct_urls and pass the "root" to +the python compile function to strip that path out of the compiled +files. + +A version of this patch, perhaps stripping root from the direct_urls +may be something that could be considered by upstream. + +Signed-off-by: Richard Purdie + +Upstream-Status: Pending + +Index: pip-22.0.3/src/pip/_internal/req/req_install.py +=================================================================== +--- pip-22.0.3.orig/src/pip/_internal/req/req_install.py ++++ pip-22.0.3/src/pip/_internal/req/req_install.py +@@ -758,7 +758,9 @@ class InstallRequirement: + if self.is_wheel: + assert self.local_file_path + direct_url = None +- if self.editable: ++ if '_PYTHON_SYSCONFIGDATA_NAME' in os.environ: ++ direct_url = None ++ elif self.editable: + direct_url = direct_url_for_editable(self.unpacked_source_directory) + elif self.original_link: + direct_url = direct_url_from_link( +@@ -775,6 +777,7 @@ class InstallRequirement: + warn_script_location=warn_script_location, + direct_url=direct_url, + requested=self.user_supplied, ++ root=root, + ) + self.install_succeeded = True + return +Index: pip-22.0.3/src/pip/_internal/operations/install/wheel.py +=================================================================== +--- pip-22.0.3.orig/src/pip/_internal/operations/install/wheel.py ++++ pip-22.0.3/src/pip/_internal/operations/install/wheel.py +@@ -436,6 +436,7 @@ def _install_wheel( + warn_script_location: bool = True, + direct_url: Optional[DirectUrl] = None, + requested: bool = False, ++ root: str = None, + ) -> None: + """Install a wheel. + +@@ -612,7 +613,7 @@ def _install_wheel( + with warnings.catch_warnings(): + warnings.filterwarnings("ignore") + for path in pyc_source_file_paths(): +- success = compileall.compile_file(path, force=True, quiet=True) ++ success = compileall.compile_file(path, force=True, quiet=True, stripdir=root) + if success: + pyc_path = pyc_output_path(path) + assert os.path.exists(pyc_path) +@@ -723,6 +724,7 @@ def install_wheel( + warn_script_location: bool = True, + direct_url: Optional[DirectUrl] = None, + requested: bool = False, ++ root: str = None, + ) -> None: + with ZipFile(wheel_path, allowZip64=True) as z: + with req_error_context(req_description): +@@ -735,4 +737,5 @@ def install_wheel( + warn_script_location=warn_script_location, + direct_url=direct_url, + requested=requested, ++ root=root, + ) diff --git a/meta/recipes-devtools/python/python3-pip_22.0.3.bb b/meta/recipes-devtools/python/python3-pip_22.0.3.bb index e02ea5bd3d2..7eb9fb69ba4 100644 --- a/meta/recipes-devtools/python/python3-pip_22.0.3.bb +++ b/meta/recipes-devtools/python/python3-pip_22.0.3.bb @@ -14,6 +14,7 @@ DEPENDS:append:class-native = " unzip-native" SRC_URI += "file://0001-change-shebang-to-python3.patch" SRC_URI += "file://no_shebang_mangling.patch" +SRC_URI += "file://reproducible.patch" SRC_URI[sha256sum] = "f29d589df8c8ab99c060e68ad294c4a9ed896624f6368c5349d70aa581b333d0" From patchwork Fri Feb 25 14:58:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 4306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D071CC433FE for ; Fri, 25 Feb 2022 14:58:33 +0000 (UTC) Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by mx.groups.io with SMTP id smtpd.web12.7272.1645801112870070517 for ; Fri, 25 Feb 2022 06:58:33 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=acPOnUSY; spf=pass (domain: linuxfoundation.org, ip: 209.85.128.47, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wm1-f47.google.com with SMTP id n13-20020a05600c3b8d00b0037bff8a24ebso1842414wms.4 for ; Fri, 25 Feb 2022 06:58:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=9EE5uXwyZFQqEVSExCWf03Yto8pZOcYKn35SnXQAago=; b=acPOnUSYjnpUO4kr4D0N6JNLFCKdA1cSH2IUcphp3RkRz7Ys/Ew0w+yHJj0l3UHWxQ 83gZEJzimihY8aCviY8Wu5o9lTl0htM99YxKCPbtYZB1+mhg572uHbggl+dS856Qf+Re wGhNQTO7qYPhmxyrpD2HUUd0mu0oDTWTuq+ZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9EE5uXwyZFQqEVSExCWf03Yto8pZOcYKn35SnXQAago=; b=T/+E8ixnNMDcs+0fmvxuOfaSlnykrQRc2XEs/vaa4HESi776ylBZihoRSak5UEKwdv N4v8DLrFiANUTqQEzFkU8irpJAMz4YmbliS9TwYms6CclQd0nN5fWxB3RxQ+G4ot3RB3 A13mWt2bIRYxZ74LM/9SYZtbRUm+27sQVu3NkeWS02TzZuGNJNA0W966Ze0lqGwcRfBl zAzy39vwYpKvD+sj6RyDT9x1TI6AVVhPODnNxGrYmM/o1nXJTTJO+fF2Um3SE0AQChHi ml5pVJeZextC4Ub94xqheRZKoaP40e4BShBBu/LcioWhK6f4VGu68QRLKba6WYw4V+sr z3sA== X-Gm-Message-State: AOAM532YW53kfxrO68oGwNPJ4VCEjk2WMnjOefN2swm3RX19cvpU0ozj +9f4HBabp07LJAQv/U2O/DQFVQsXdOiiwANu X-Google-Smtp-Source: ABdhPJx9a+wbBbtUlenWp9EUlOD1ie88cGckTgm5EGr6DxDnvJn/17/xSv4UGy5Dwv8DnLEVsteEgA== X-Received: by 2002:a1c:f018:0:b0:37b:c13c:3128 with SMTP id a24-20020a1cf018000000b0037bc13c3128mr3049655wmb.157.1645801110757; Fri, 25 Feb 2022 06:58:30 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:8c05:a731:3c1:5d08]) by smtp.gmail.com with ESMTPSA id r2-20020a05600c35c200b00352cdcdd7b2sm17121733wmq.0.2022.02.25.06.58.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Feb 2022 06:58:30 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 2/3] pip_install_wheel: Recompile modified files Date: Fri, 25 Feb 2022 14:58:27 +0000 Message-Id: <20220225145828.88238-2-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220225145828.88238-1-richard.purdie@linuxfoundation.org> References: <20220225145828.88238-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 25 Feb 2022 14:58:33 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/162400 If we modify the file, we need to recompile the pyc files since the file hash has changed. This would otherwise result in reproducibility failrues. Signed-off-by: Richard Purdie --- meta/classes/pip_install_wheel.bbclass | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/meta/classes/pip_install_wheel.bbclass b/meta/classes/pip_install_wheel.bbclass index f0312e0b1eb..8a848c0ebab 100644 --- a/meta/classes/pip_install_wheel.bbclass +++ b/meta/classes/pip_install_wheel.bbclass @@ -32,6 +32,11 @@ pip_install_wheel_do_install () { sed -i -e "1s,#!.*nativepython3,#!${USRBINPATH}/env ${PIP_INSTALL_PYTHON}," $i sed -i -e "s:${PYTHON}:${USRBINPATH}/env\ ${PIP_INSTALL_PYTHON}:g" $i sed -i -e "s:${STAGING_BINDIR_NATIVE}:${bindir}:g" $i + # Recompile after modifying it + cd ${D} + file=`echo $i | sed 's:^${D}/::'` + ${STAGING_BINDIR_NATIVE}/python3-native/python3 -c "from py_compile import compile; compile('$file')" + cd - fi done } From patchwork Fri Feb 25 14:58:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Purdie X-Patchwork-Id: 4307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA1B8C4332F for ; Fri, 25 Feb 2022 14:58:34 +0000 (UTC) Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by mx.groups.io with SMTP id smtpd.web10.7449.1645801113745080204 for ; Fri, 25 Feb 2022 06:58:34 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@linuxfoundation.org header.s=google header.b=PrxjAdCz; spf=pass (domain: linuxfoundation.org, ip: 209.85.221.54, mailfrom: richard.purdie@linuxfoundation.org) Received: by mail-wr1-f54.google.com with SMTP id j22so4877446wrb.13 for ; Fri, 25 Feb 2022 06:58:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=9/kz/6mcCtHBtAGnnUJASyhp3aXvcVITxQEsX4yGDaw=; b=PrxjAdCzLOw+Zzk97/SnzK+c2RRkLZ46d9XbEUh/bTCCaqK31nHA1T7yBihlal8g3B Gk0Fkz7DlBkhL4ik06GzkG94xp7CM1iC6CbrYkXQ1fDZGlNfxxnku9sf3tk1F/iDv0LF e24kHu8UeYZf0GA4eO0lWWGdjyNBP2VdTdnSQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9/kz/6mcCtHBtAGnnUJASyhp3aXvcVITxQEsX4yGDaw=; b=1+6IHKSaOC33BIvJz562G1voShiK/jOin4u3WjI/Todlo6a/rLADM0lBmt0oX3Qw4l tmN4zvQ90mzi72yW8VYZ8WpxxxTGOO7k9Wrkp3o2RF3pb4UHoRR48YVxdhFR+HFokZ+v x1rSFKyogC66clWSyoQdmY8D8k+5+1PkJWmlFwl3//0ltcgJaAuPg83acl8sgB0vggcT ByFTjAWwe/PJ0h7Ljm+KPXwfDsGYSU8tPHnSgBC1eBSnuFU0m/jn39ZaBezYdBjXixWz Me7/BBBOqZ8gc5qemJJdYfLyDiSToIp3+rp+Kv3tT5nDcpAR939XE2+LHuz2WJG+nOgD gqAQ== X-Gm-Message-State: AOAM5302iVvoOVXnjBAeJ/GMdJJDZ5r7/wO6zOmohg1tF9F3KZpappTQ XDCQyXK52f+uoWPMV4a8gDzXo0onypCKKL75 X-Google-Smtp-Source: ABdhPJwOWGW58twbe+43l3pue/BpqVdKf/h6pKOPYsAdZ6RnmeYf5qFYj3UJx9g0IcAkWk11JtNihg== X-Received: by 2002:a5d:6148:0:b0:1ed:b74e:508a with SMTP id y8-20020a5d6148000000b001edb74e508amr6215658wrt.90.1645801111785; Fri, 25 Feb 2022 06:58:31 -0800 (PST) Received: from hex.int.rpsys.net ([2001:8b0:aba:5f3c:8c05:a731:3c1:5d08]) by smtp.gmail.com with ESMTPSA id r2-20020a05600c35c200b00352cdcdd7b2sm17121733wmq.0.2022.02.25.06.58.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Feb 2022 06:58:31 -0800 (PST) From: Richard Purdie To: openembedded-core@lists.openembedded.org Subject: [PATCH 3/3] python3-numpy: Fix pyc determinism issue Date: Fri, 25 Feb 2022 14:58:28 +0000 Message-Id: <20220225145828.88238-3-richard.purdie@linuxfoundation.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220225145828.88238-1-richard.purdie@linuxfoundation.org> References: <20220225145828.88238-1-richard.purdie@linuxfoundation.org> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 25 Feb 2022 14:58:34 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/162401 Using frozenset causes problems for pyc file determinism. For now remove the problematic pyc file as we do in the main python3 recipe. Signed-off-by: Richard Purdie --- meta/recipes-devtools/python/python3-numpy_1.22.2.bb | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/meta/recipes-devtools/python/python3-numpy_1.22.2.bb b/meta/recipes-devtools/python/python3-numpy_1.22.2.bb index d2b2f3e38d5..43e7427eab5 100644 --- a/meta/recipes-devtools/python/python3-numpy_1.22.2.bb +++ b/meta/recipes-devtools/python/python3-numpy_1.22.2.bb @@ -30,6 +30,15 @@ do_compile:prepend() { export NPY_DISABLE_SVML=1 } +# Unfortunately the following pyc files are non-deterministc due to 'frozenset' +# being written without strict ordering, even with PYTHONHASHSEED = 0 +# Upstream is discussing ways to solve the issue properly, until then let's +# just not install the problematic files. +# More info: http://benno.id.au/blog/2013/01/15/python-determinism +do_install:append() { + rm ${D}${PYTHON_SITEPACKAGES_DIR}/numpy/typing/tests/data/pass/__pycache__/literal.cpython* +} + FILES:${PN}-staticdev += "${PYTHON_SITEPACKAGES_DIR}/numpy/core/lib/*.a ${PYTHON_SITEPACKAGES_DIR}/numpy/random/lib/*.a" # install what is needed for numpy.test()