From patchwork Wed Oct 18 12:01:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55F86CDB483 for ; Wed, 18 Oct 2023 12:01:26 +0000 (UTC) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) by mx.groups.io with SMTP id smtpd.web11.279502.1697630479380958008 for ; Wed, 18 Oct 2023 05:01:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=kVA6CqE1; spf=pass (domain: baylibre.com, ip: 209.85.167.42, mailfrom: jstephan@baylibre.com) Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-507a62d4788so6054730e87.0 for ; Wed, 18 Oct 2023 05:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1697630477; x=1698235277; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=mEaB40U+u3nn/bEeyzm7t6UtVYmcyOEd+1ZuyF2+9Bg=; b=kVA6CqE1w5cyfyYyG5CE46CRTwKq9LGfD7fqu5dx5TnPrV+Z3zysWw6HCLXoQ+49qg 6aq2sPPj+UROU6Xg8jqf8UF1mnjVKL5ARma1kfrF3sD2S4M1uSLe/ure0r2sNWWqcK6K av5EiUiAsJMybaPRl8qwxVBOLI8/afTBBGiPCk0L2CfmiHzsWlXQyqIt7GTSp46Q7dfv R/I8ROEYQjueuEr5QAVDMH/GOiMQjVJOcRf9pvVcnNqPN5fVLQ1gh3iDWbg/BTm5vOXY l5vePt7IobX/4aDQb1HjbErdh+8kLozHVQSyDAKIVJTe1yA3gUdWgV3Rk9qdO+/hOnnn IpLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697630477; x=1698235277; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mEaB40U+u3nn/bEeyzm7t6UtVYmcyOEd+1ZuyF2+9Bg=; b=ggkLyfAKPlw+1URgxE2t6rXOpNC05fWFOXrcG/w/oZ2MjWGCgvNoxzpyRHywmQWbo9 wphBsCgCYa5wOfSoNTKh8kAb5pUXqC532ZNgWwp3DIWPDq6mqsdUyu5zlpqHcIrmUduO Q3wQ3m8n/KgOFf4YzTX1Zhcjt0Ph39i2YEionVeE04tN55b8wUi29wP9j6iovx2UY7wO O9GCWGun/IefirXKv5R2gLo+CveM1PzKgJESc1n3B8I9tKG3q6rDuxXZi/lmCVgIABAl KzJC8L0ytlRQ14wQxsf3TFIwrG0JpizQnXM6KFk2N8NObmfsTNdI5XmQAp5RsxK4Bh5X 9tng== X-Gm-Message-State: AOJu0YyIvE5Mje7UBgqkmm6Ze4uzoIEdmO73Kw0zB3EP4Cqghb/6gx/x dSYwj2dCQLxTjsc9Q7wVrGdLHxQm8jA6cg4/ilo6mg== X-Google-Smtp-Source: AGHT+IHj4J89FGh/sDXJRMP4XhNjqfngVIofZZbznZNpsOhqaM10c+7SC6hdf9UHHReGliEtWJE2lA== X-Received: by 2002:a05:6512:4018:b0:507:9a0b:3fd7 with SMTP id br24-20020a056512401800b005079a0b3fd7mr5062555lfb.13.1697630476508; Wed, 18 Oct 2023 05:01:16 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id j4-20020adfe504000000b0032da7454ebesm1960312wrm.79.2023.10.18.05.01.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 05:01:16 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 1/4] scripts:recipetool:create_buildsys_python: fix license note Date: Wed, 18 Oct 2023 14:01:11 +0200 Message-ID: <20231018120114.1222475-1-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 18 Oct 2023 12:01:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189385 License field of setup is not always standardized, so we usually use the classifier to determine the correct license format to use in the recipe. A warning note is added above the LICENSE field of the create recipe in case a license is provided in setup. But when the plugin is called, "LICENSE =" is not yet present so we can never display this note. Replace the "LICENSE =" condition with "##LICENSE_PLACEHOLDER##" to actually be able to display the note message Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create_buildsys_python.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 92468b22546..d2c7e251bf7 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -254,7 +254,7 @@ class PythonRecipeHandler(RecipeHandler): if license_str: for i, line in enumerate(lines_before): - if line.startswith('LICENSE = '): + if ine.startswith('##LICENSE_PLACEHOLDER##'): lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str) break From patchwork Wed Oct 18 12:01:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BF4DCDB47E for ; Wed, 18 Oct 2023 12:01:26 +0000 (UTC) Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) by mx.groups.io with SMTP id smtpd.web11.279501.1697630479296882548 for ; Wed, 18 Oct 2023 05:01:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=bepXXArt; spf=pass (domain: baylibre.com, ip: 209.85.167.53, mailfrom: jstephan@baylibre.com) Received: by mail-lf1-f53.google.com with SMTP id 2adb3069b0e04-507a3b8b113so5462698e87.0 for ; Wed, 18 Oct 2023 05:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1697630477; x=1698235277; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HFG+NgKDD+a3pHQn1XSBG0pv5Z6pNQtH02miX+DoJT4=; b=bepXXArt4YpwNmDFzkd6u6LAd99JQL7w8lYn8oTTAql6TbCVm5unBsEZRnU8+izTY0 Wof7NjGqXLRs+13o9nG8MjlDZadu/wb4yatUutNh0+iBtTJJNROdmBn5OiL++0b5qyGn NV+t8orG4M9EPcstD2sJ2Cs4GKXaaPhwF/h+YF91zOw9rBbK8SbwSst9iyjR5Osj0Tvm qRvO8YAqsAf0msHG/mesb8SpsANJn+sLEZzn513pjNZIeymfxzC1zh311Bj+6HUmBu/1 FiiqEijmZAUQVsyA9kJH8qC/3J8cqISnLoaDmLIN+njp08DpNt2Tb3kVnQRJhDaTNmHW t9oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697630477; x=1698235277; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HFG+NgKDD+a3pHQn1XSBG0pv5Z6pNQtH02miX+DoJT4=; b=HmD83lHIpFQxMdvfTaQK+eRTwk9CbnuOAHZTWp4ZtCGhd0B87wFGZNDDyymyofDrB8 uFXjK4KO5oyxsKe3kUIVr2b71jlJGknFyboMKU6btbKiwCNCdiO89hmKWA6oTeu2Uwn/ 9kWY/+CshsPG694Ll11hnPW30XFd4c9YZ9mpNYMv9aFCq1W8CK8skRWqzYBAK5TD1IKm dyt8DTNtF0ORtBx3Ty721RZfxwjnTvYhVx8whbO85Ohbu3kx4FXI5TuptghZbSYiTzco rQZWAGYBcv48lqps3/TjgewcgAtPpeP3KUW7MAeUFT7aZZP+yMJl9MU93VeKXJ1F7Zmd KJdQ== X-Gm-Message-State: AOJu0YzcIxc78WxLckMC94gwxlE5pbLsFXJ1Ccmal6wwp75ow7PEgS/m M83FSwvbZYIA4NOFzZH9hTKlDPsKW+0pVTVmOFV/sw== X-Google-Smtp-Source: AGHT+IFj97iYtM/S1mqHsMogn7AaIeyPRM3ki8GQk/MAXWuHRxSBUiMfT02k37sV/w0bP5W+1wTn3Q== X-Received: by 2002:a05:6512:401e:b0:504:7ff8:3430 with SMTP id br30-20020a056512401e00b005047ff83430mr4346094lfb.10.1697630477003; Wed, 18 Oct 2023 05:01:17 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id j4-20020adfe504000000b0032da7454ebesm1960312wrm.79.2023.10.18.05.01.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 05:01:16 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 2/4] scripts:recipetool:create_buildsys_python: prefix created recipes with python3- Date: Wed, 18 Oct 2023 14:01:12 +0200 Message-ID: <20231018120114.1222475-2-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231018120114.1222475-1-jstephan@baylibre.com> References: <20231018120114.1222475-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 18 Oct 2023 12:01:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189384 By convention, all python recipes start with "python3-" so update create_buildsys_python to do this This rule doesn't apply for packages already starting with "python" Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create_buildsys_python.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index d2c7e251bf7..7726ba7f161 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -297,6 +297,11 @@ class PythonRecipeHandler(RecipeHandler): value = ' '.join(str(v) for v in values if v) bbvar = self.bbvar_map[field] + if bbvar == "PN": + # by convention python recipes start with "python3-" + if not value.startswith('python'): + value = 'python3-' + value + if bbvar not in extravalues and value: extravalues[bbvar] = value From patchwork Wed Oct 18 12:01:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6649CC41513 for ; Wed, 18 Oct 2023 12:01:26 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web11.279504.1697630480026698970 for ; Wed, 18 Oct 2023 05:01:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=EdOZvSZ/; spf=pass (domain: baylibre.com, ip: 209.85.221.52, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-313e742a787so4157330f8f.1 for ; Wed, 18 Oct 2023 05:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1697630478; x=1698235278; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UxRR5IH8LAEWxprYRzbg7W5u8EqLU6jExnU4wR8TKlI=; b=EdOZvSZ/iSiL66wn381jPVtSXzOE5p9r6hP6NWAzEWx9EZ2ppipFAiKgk7DMdqr2fI ty16VvS1vC5N53ANoBSLJqeErieV2/ss1zQN1gzPpEaITiH8uqWwFUkGBKaP7qmFYyAS wIZgHxDxqln0tLRohzOhIAMJCpJEYBDb4Yk4eSmD4bMz073jTrXT/SqUtJ3/suE9RWcm BJjfBNW3g7ndX8gDTOnb7jPsDgSfimRKWkHy2fEF2KvJ9h7JLZ0l4Pu0qe2sYa3GPWTv 63NLND99dghLLjk24oPpffy1D9xDBWQYBSW8pA9co0fID7/bKS7zhwK1Mvv2BiC1CAZB bDFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697630478; x=1698235278; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UxRR5IH8LAEWxprYRzbg7W5u8EqLU6jExnU4wR8TKlI=; b=wjD9Ev7b3EnfpDbWC31VqRb9/0C5GqIfI/WCaceFvDgCtVW3+rZFH2lVCwdedV/IAV P+G2wEOOxE+xnhV/z0vSD8ncnvsFB7uOKUkA3Fq2MT0sILo7a3y6aCk69Xp+wqRTVzkP ZP3QNKLoOyTk2YGN2VYNcWYBk59Wz7i68kRSvyyJafbY1iN5bwscloe2PdF6NFgOAGsI F3AZBIPAsFnJiCFdznjwtj45gEwbUl+hbyu0KyrvmB9KL8vN4Pl4n5+950WaJWJ+dTbp 6iHG27y2QVBxel0lSHdgxh/+FxfGsrCiHY7FuKjoSgOEH2UiRg5efbpSYXKpprisME9I FtBA== X-Gm-Message-State: AOJu0Yz//xEVa75RJlYb/i13zOv4BqM7UIaro7oVxFWqn1fimVzOCb5Z hwLzMjHv0dpg2PDAlCculNuUByVubLpT9snsZ404Rw== X-Google-Smtp-Source: AGHT+IEXCKvse3fNY1Q/qMrG/i2yAYME+q+//KXQSspONsM8305qSoyET5iFqZXQ9jTyu8jj+PRyIA== X-Received: by 2002:adf:a49c:0:b0:32d:a3f7:f0dd with SMTP id g28-20020adfa49c000000b0032da3f7f0ddmr4487250wrb.25.1697630477544; Wed, 18 Oct 2023 05:01:17 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id j4-20020adfe504000000b0032da7454ebesm1960312wrm.79.2023.10.18.05.01.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 05:01:17 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 3/4] scripts:recipetool:create_buildsys_python: refactor code for futur PEP517 addition Date: Wed, 18 Oct 2023 14:01:13 +0200 Message-ID: <20231018120114.1222475-3-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231018120114.1222475-1-jstephan@baylibre.com> References: <20231018120114.1222475-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 18 Oct 2023 12:01:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189386 In order to prepare the support for pyproject.toml (PEP517 [1]) enabled projects, refactor the code and move setup.py specific code into a specific class in order to allow sharing the PythonRecipeHandler class No functionnal changes expected [1]: https://peps.python.org/pep-0517/#source-tree Signed-off-by: Julien Stephan --- .../lib/recipetool/create_buildsys_python.py | 750 +++++++++--------- 1 file changed, 386 insertions(+), 364 deletions(-) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 7726ba7f161..69f6f5ca511 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -37,63 +37,8 @@ class PythonRecipeHandler(RecipeHandler): assume_provided = ['builtins', 'os.path'] # Assumes that the host python3 builtin_module_names is sane for target too assume_provided = assume_provided + list(sys.builtin_module_names) + excluded_fields = [] - bbvar_map = { - 'Name': 'PN', - 'Version': 'PV', - 'Home-page': 'HOMEPAGE', - 'Summary': 'SUMMARY', - 'Description': 'DESCRIPTION', - 'License': 'LICENSE', - 'Requires': 'RDEPENDS:${PN}', - 'Provides': 'RPROVIDES:${PN}', - 'Obsoletes': 'RREPLACES:${PN}', - } - # PN/PV are already set by recipetool core & desc can be extremely long - excluded_fields = [ - 'Description', - ] - setup_parse_map = { - 'Url': 'Home-page', - 'Classifiers': 'Classifier', - 'Description': 'Summary', - } - setuparg_map = { - 'Home-page': 'url', - 'Classifier': 'classifiers', - 'Summary': 'description', - 'Description': 'long-description', - } - # Values which are lists, used by the setup.py argument based metadata - # extraction method, to determine how to process the setup.py output. - setuparg_list_fields = [ - 'Classifier', - 'Requires', - 'Provides', - 'Obsoletes', - 'Platform', - 'Supported-Platform', - ] - setuparg_multi_line_values = ['Description'] - replacements = [ - ('License', r' +$', ''), - ('License', r'^ +', ''), - ('License', r' ', '-'), - ('License', r'^GNU-', ''), - ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''), - ('License', r'^UNKNOWN$', ''), - - # Remove currently unhandled version numbers from these variables - ('Requires', r' *\([^)]*\)', ''), - ('Provides', r' *\([^)]*\)', ''), - ('Obsoletes', r' *\([^)]*\)', ''), - ('Install-requires', r'^([^><= ]+).*', r'\1'), - ('Extras-require', r'^([^><= ]+).*', r'\1'), - ('Tests-require', r'^([^><= ]+).*', r'\1'), - - # Remove unhandled dependency on particular features (e.g. foo[PDF]) - ('Install-requires', r'\[[^\]]+\]$', ''), - ] classifier_license_map = { 'License :: OSI Approved :: Academic Free License (AFL)': 'AFL', @@ -166,122 +111,34 @@ class PythonRecipeHandler(RecipeHandler): def __init__(self): pass - def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): - if 'buildsystem' in handled: - return False - - # Check for non-zero size setup.py files - setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py']) - for fn in setupfiles: - if os.path.getsize(fn): - break - else: - return False - - # setup.py is always parsed to get at certain required information, such as - # distutils vs setuptools - # - # If egg info is available, we use it for both its PKG-INFO metadata - # and for its requires.txt for install_requires. - # If PKG-INFO is available but no egg info is, we use that for metadata in preference to - # the parsed setup.py, but use the install_requires info from the - # parsed setup.py. - - setupscript = os.path.join(srctree, 'setup.py') - try: - setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript) - except Exception: - logger.exception("Failed to parse setup.py") - setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], [] - - egginfo = glob.glob(os.path.join(srctree, '*.egg-info')) - if egginfo: - info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO')) - requires_txt = os.path.join(egginfo[0], 'requires.txt') - if os.path.exists(requires_txt): - with codecs.open(requires_txt) as f: - inst_req = [] - extras_req = collections.defaultdict(list) - current_feature = None - for line in f.readlines(): - line = line.rstrip() - if not line: - continue - - if line.startswith('['): - # PACKAGECONFIG must not contain expressions or whitespace - line = line.replace(" ", "") - line = line.replace(':', "") - line = line.replace('.', "-dot-") - line = line.replace('"', "") - line = line.replace('<', "-smaller-") - line = line.replace('>', "-bigger-") - line = line.replace('_', "-") - line = line.replace('(', "") - line = line.replace(')', "") - line = line.replace('!', "-not-") - line = line.replace('=', "-equals-") - current_feature = line[1:-1] - elif current_feature: - extras_req[current_feature].append(line) - else: - inst_req.append(line) - info['Install-requires'] = inst_req - info['Extras-require'] = extras_req - elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']): - info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO')) - - if setup_info: - if 'Install-requires' in setup_info: - info['Install-requires'] = setup_info['Install-requires'] - if 'Extras-require' in setup_info: - info['Extras-require'] = setup_info['Extras-require'] - else: - if setup_info: - info = setup_info - else: - info = self.get_setup_args_info(setupscript) - - # Grab the license value before applying replacements - license_str = info.get('License', '').strip() - - self.apply_info_replacements(info) - - if uses_setuptools: - classes.append('setuptools3') - else: - classes.append('distutils3') - - if license_str: - for i, line in enumerate(lines_before): - if ine.startswith('##LICENSE_PLACEHOLDER##'): - lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str) - break - - if 'Classifier' in info: - existing_licenses = info.get('License', '') - licenses = [] - for classifier in info['Classifier']: - if classifier in self.classifier_license_map: - license = self.classifier_license_map[classifier] - if license == 'Apache' and 'Apache-2.0' in existing_licenses: - license = 'Apache-2.0' - elif license == 'GPL': - if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses: - license = 'GPL-2.0' - elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses: - license = 'GPL-3.0' - elif license == 'LGPL': - if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses: - license = 'LGPL-2.1' - elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses: - license = 'LGPL-2.0' - elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses: - license = 'LGPL-3.0' - licenses.append(license) - - if licenses: - info['License'] = ' & '.join(licenses) + def handle_classifier_license(self, classifiers, existing_licenses=""): + + licenses = [] + for classifier in classifiers: + if classifier in self.classifier_license_map: + license = self.classifier_license_map[classifier] + if license == 'Apache' and 'Apache-2.0' in existing_licenses: + license = 'Apache-2.0' + elif license == 'GPL': + if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses: + license = 'GPL-2.0' + elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses: + license = 'GPL-3.0' + elif license == 'LGPL': + if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses: + license = 'LGPL-2.1' + elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses: + license = 'LGPL-2.0' + elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses: + license = 'LGPL-3.0' + licenses.append(license) + + if licenses: + return ' & '.join(licenses) + + return None + + def map_info_to_bbvar(self, info, extravalues): # Map PKG-INFO & setup.py fields to bitbake variables for field, values in info.items(): @@ -305,86 +162,221 @@ class PythonRecipeHandler(RecipeHandler): if bbvar not in extravalues and value: extravalues[bbvar] = value - mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals) - - extras_req = set() - if 'Extras-require' in info: - extras_req = info['Extras-require'] - if extras_req: - lines_after.append('# The following configs & dependencies are from setuptools extras_require.') - lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.') - lines_after.append('# The upstream names may not correspond exactly to bitbake package names.') - lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.') - lines_after.append('#') - lines_after.append('# Uncomment this line to enable all the optional features.') - lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req))) - for feature, feature_reqs in extras_req.items(): - unmapped_deps.difference_update(feature_reqs) - - feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs)) - lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps))) - - inst_reqs = set() - if 'Install-requires' in info: - if extras_req: - lines_after.append('') - inst_reqs = info['Install-requires'] - if inst_reqs: - unmapped_deps.difference_update(inst_reqs) - - inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs)) - lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These') - lines_after.append('# upstream names may not correspond exactly to bitbake package names.') - lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps))) + def apply_info_replacements(self, info): + if not self.replacements: + return - if mapped_deps: - name = info.get('Name') - if name and name[0] in mapped_deps: - # Attempt to avoid self-reference - mapped_deps.remove(name[0]) - mapped_deps -= set(self.excluded_pkgdeps) - if inst_reqs or extras_req: - lines_after.append('') - lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the') - lines_after.append('# python sources, and might not be 100% accurate.') - lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps)))) + for variable, search, replace in self.replacements: + if variable not in info: + continue - unmapped_deps -= set(extensions) - unmapped_deps -= set(self.assume_provided) - if unmapped_deps: - if mapped_deps: - lines_after.append('') - lines_after.append('# WARNING: We were unable to map the following python package/module') - lines_after.append('# dependencies to the bitbake packages which include them:') - lines_after.extend('# {}'.format(d) for d in sorted(unmapped_deps)) + def replace_value(search, replace, value): + if replace is None: + if re.search(search, value): + return None + else: + new_value = re.sub(search, replace, value) + if value != new_value: + return new_value + return value - handled.append('buildsystem') + value = info[variable] + if isinstance(value, str): + new_value = replace_value(search, replace, value) + if new_value is None: + del info[variable] + elif new_value != value: + info[variable] = new_value + elif hasattr(value, 'items'): + for dkey, dvalue in list(value.items()): + new_list = [] + for pos, a_value in enumerate(dvalue): + new_value = replace_value(search, replace, a_value) + if new_value is not None and new_value != value: + new_list.append(new_value) - def get_pkginfo(self, pkginfo_fn): - msg = email.message_from_file(open(pkginfo_fn, 'r')) - msginfo = {} - for field in msg.keys(): - values = msg.get_all(field) - if len(values) == 1: - msginfo[field] = values[0] + if value != new_list: + value[dkey] = new_list else: - msginfo[field] = values - return msginfo + new_list = [] + for pos, a_value in enumerate(value): + new_value = replace_value(search, replace, a_value) + if new_value is not None and new_value != value: + new_list.append(new_value) - def parse_setup_py(self, setupscript='./setup.py'): - with codecs.open(setupscript) as f: - info, imported_modules, non_literals, extensions = gather_setup_info(f) + if value != new_list: + info[variable] = new_list - def _map(key): - key = key.replace('_', '-') - key = key[0].upper() + key[1:] - if key in self.setup_parse_map: - key = self.setup_parse_map[key] - return key - # Naive mapping of setup() arguments to PKG-INFO field names - for d in [info, non_literals]: - for key, value in list(d.items()): + def scan_python_dependencies(self, paths): + deps = set() + try: + dep_output = self.run_command(['pythondeps', '-d'] + paths) + except (OSError, subprocess.CalledProcessError): + pass + else: + for line in dep_output.splitlines(): + line = line.rstrip() + dep, filename = line.split('\t', 1) + if filename.endswith('/setup.py'): + continue + deps.add(dep) + + try: + provides_output = self.run_command(['pythondeps', '-p'] + paths) + except (OSError, subprocess.CalledProcessError): + pass + else: + provides_lines = (l.rstrip() for l in provides_output.splitlines()) + provides = set(l for l in provides_lines if l and l != 'setup') + deps -= provides + + return deps + + def parse_pkgdata_for_python_packages(self): + pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR') + + ldata = tinfoil.config_data.createCopy() + bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True) + python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR') + + dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload') + python_dirs = [python_sitedir + os.sep, + os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep, + os.path.dirname(python_sitedir) + os.sep] + packages = {} + for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)): + files_info = None + with open(pkgdatafile, 'r') as f: + for line in f.readlines(): + field, value = line.split(': ', 1) + if field.startswith('FILES_INFO'): + files_info = ast.literal_eval(value) + break + else: + continue + + for fn in files_info: + for suffix in importlib.machinery.all_suffixes(): + if fn.endswith(suffix): + break + else: + continue + + if fn.startswith(dynload_dir + os.sep): + if '/.debug/' in fn: + continue + base = os.path.basename(fn) + provided = base.split('.', 1)[0] + packages[provided] = os.path.basename(pkgdatafile) + continue + + for python_dir in python_dirs: + if fn.startswith(python_dir): + relpath = fn[len(python_dir):] + relstart, _, relremaining = relpath.partition(os.sep) + if relstart.endswith('.egg'): + relpath = relremaining + base, _ = os.path.splitext(relpath) + + if '/.debug/' in base: + continue + if os.path.basename(base) == '__init__': + base = os.path.dirname(base) + base = base.replace(os.sep + os.sep, os.sep) + provided = base.replace(os.sep, '.') + packages[provided] = os.path.basename(pkgdatafile) + return packages + + @classmethod + def run_command(cls, cmd, **popenargs): + if 'stderr' not in popenargs: + popenargs['stderr'] = subprocess.STDOUT + try: + return subprocess.check_output(cmd, **popenargs).decode('utf-8') + except OSError as exc: + logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc) + raise + except subprocess.CalledProcessError as exc: + logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output) + raise + +class PythonSetupPyRecipeHandler(PythonRecipeHandler): + bbvar_map = { + 'Name': 'PN', + 'Version': 'PV', + 'Home-page': 'HOMEPAGE', + 'Summary': 'SUMMARY', + 'Description': 'DESCRIPTION', + 'License': 'LICENSE', + 'Requires': 'RDEPENDS:${PN}', + 'Provides': 'RPROVIDES:${PN}', + 'Obsoletes': 'RREPLACES:${PN}', + } + # PN/PV are already set by recipetool core & desc can be extremely long + excluded_fields = [ + 'Description', + ] + setup_parse_map = { + 'Url': 'Home-page', + 'Classifiers': 'Classifier', + 'Description': 'Summary', + } + setuparg_map = { + 'Home-page': 'url', + 'Classifier': 'classifiers', + 'Summary': 'description', + 'Description': 'long-description', + } + # Values which are lists, used by the setup.py argument based metadata + # extraction method, to determine how to process the setup.py output. + setuparg_list_fields = [ + 'Classifier', + 'Requires', + 'Provides', + 'Obsoletes', + 'Platform', + 'Supported-Platform', + ] + setuparg_multi_line_values = ['Description'] + + replacements = [ + ('License', r' +$', ''), + ('License', r'^ +', ''), + ('License', r' ', '-'), + ('License', r'^GNU-', ''), + ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''), + ('License', r'^UNKNOWN$', ''), + + # Remove currently unhandled version numbers from these variables + ('Requires', r' *\([^)]*\)', ''), + ('Provides', r' *\([^)]*\)', ''), + ('Obsoletes', r' *\([^)]*\)', ''), + ('Install-requires', r'^([^><= ]+).*', r'\1'), + ('Extras-require', r'^([^><= ]+).*', r'\1'), + ('Tests-require', r'^([^><= ]+).*', r'\1'), + + # Remove unhandled dependency on particular features (e.g. foo[PDF]) + ('Install-requires', r'\[[^\]]+\]$', ''), + ] + + def __init__(self): + pass + + def parse_setup_py(self, setupscript='./setup.py'): + with codecs.open(setupscript) as f: + info, imported_modules, non_literals, extensions = gather_setup_info(f) + + def _map(key): + key = key.replace('_', '-') + key = key[0].upper() + key[1:] + if key in self.setup_parse_map: + key = self.setup_parse_map[key] + return key + + # Naive mapping of setup() arguments to PKG-INFO field names + for d in [info, non_literals]: + for key, value in list(d.items()): if key is None: continue new_key = _map(key) @@ -445,47 +437,16 @@ class PythonRecipeHandler(RecipeHandler): info[fields[lineno]] = line return info - def apply_info_replacements(self, info): - for variable, search, replace in self.replacements: - if variable not in info: - continue - - def replace_value(search, replace, value): - if replace is None: - if re.search(search, value): - return None - else: - new_value = re.sub(search, replace, value) - if value != new_value: - return new_value - return value - - value = info[variable] - if isinstance(value, str): - new_value = replace_value(search, replace, value) - if new_value is None: - del info[variable] - elif new_value != value: - info[variable] = new_value - elif hasattr(value, 'items'): - for dkey, dvalue in list(value.items()): - new_list = [] - for pos, a_value in enumerate(dvalue): - new_value = replace_value(search, replace, a_value) - if new_value is not None and new_value != value: - new_list.append(new_value) - - if value != new_list: - value[dkey] = new_list + def get_pkginfo(self, pkginfo_fn): + msg = email.message_from_file(open(pkginfo_fn, 'r')) + msginfo = {} + for field in msg.keys(): + values = msg.get_all(field) + if len(values) == 1: + msginfo[field] = values[0] else: - new_list = [] - for pos, a_value in enumerate(value): - new_value = replace_value(search, replace, a_value) - if new_value is not None and new_value != value: - new_list.append(new_value) - - if value != new_list: - info[variable] = new_list + msginfo[field] = values + return msginfo def scan_setup_python_deps(self, srctree, setup_info, setup_non_literals): if 'Package-dir' in setup_info: @@ -540,99 +501,160 @@ class PythonRecipeHandler(RecipeHandler): unmapped_deps.add(dep) return mapped_deps, unmapped_deps - def scan_python_dependencies(self, paths): - deps = set() - try: - dep_output = self.run_command(['pythondeps', '-d'] + paths) - except (OSError, subprocess.CalledProcessError): - pass + def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): + + if 'buildsystem' in handled: + return False + + # Check for non-zero size setup.py files + setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py']) + for fn in setupfiles: + if os.path.getsize(fn): + break else: - for line in dep_output.splitlines(): - line = line.rstrip() - dep, filename = line.split('\t', 1) - if filename.endswith('/setup.py'): - continue - deps.add(dep) + return False + + # setup.py is always parsed to get at certain required information, such as + # distutils vs setuptools + # + # If egg info is available, we use it for both its PKG-INFO metadata + # and for its requires.txt for install_requires. + # If PKG-INFO is available but no egg info is, we use that for metadata in preference to + # the parsed setup.py, but use the install_requires info from the + # parsed setup.py. + setupscript = os.path.join(srctree, 'setup.py') try: - provides_output = self.run_command(['pythondeps', '-p'] + paths) - except (OSError, subprocess.CalledProcessError): - pass + setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript) + except Exception: + logger.exception("Failed to parse setup.py") + setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], [] + + egginfo = glob.glob(os.path.join(srctree, '*.egg-info')) + if egginfo: + info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO')) + requires_txt = os.path.join(egginfo[0], 'requires.txt') + if os.path.exists(requires_txt): + with codecs.open(requires_txt) as f: + inst_req = [] + extras_req = collections.defaultdict(list) + current_feature = None + for line in f.readlines(): + line = line.rstrip() + if not line: + continue + + if line.startswith('['): + # PACKAGECONFIG must not contain expressions or whitespace + line = line.replace(" ", "") + line = line.replace(':', "") + line = line.replace('.', "-dot-") + line = line.replace('"', "") + line = line.replace('<', "-smaller-") + line = line.replace('>', "-bigger-") + line = line.replace('_', "-") + line = line.replace('(', "") + line = line.replace(')', "") + line = line.replace('!', "-not-") + line = line.replace('=', "-equals-") + current_feature = line[1:-1] + elif current_feature: + extras_req[current_feature].append(line) + else: + inst_req.append(line) + info['Install-requires'] = inst_req + info['Extras-require'] = extras_req + elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']): + info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO')) + + if setup_info: + if 'Install-requires' in setup_info: + info['Install-requires'] = setup_info['Install-requires'] + if 'Extras-require' in setup_info: + info['Extras-require'] = setup_info['Extras-require'] else: - provides_lines = (l.rstrip() for l in provides_output.splitlines()) - provides = set(l for l in provides_lines if l and l != 'setup') - deps -= provides + if setup_info: + info = setup_info + else: + info = self.get_setup_args_info(setupscript) - return deps + # Grab the license value before applying replacements + license_str = info.get('License', '').strip() - def parse_pkgdata_for_python_packages(self): - pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR') + self.apply_info_replacements(info) - ldata = tinfoil.config_data.createCopy() - bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True) - python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR') + if uses_setuptools: + classes.append('setuptools3') + else: + classes.append('distutils3') - dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload') - python_dirs = [python_sitedir + os.sep, - os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep, - os.path.dirname(python_sitedir) + os.sep] - packages = {} - for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)): - files_info = None - with open(pkgdatafile, 'r') as f: - for line in f.readlines(): - field, value = line.split(': ', 1) - if field.startswith('FILES_INFO'): - files_info = ast.literal_eval(value) - break - else: - continue + if license_str: + for i, line in enumerate(lines_before): + if line.startswith('##LICENSE_PLACEHOLDER##'): + lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str) + break - for fn in files_info: - for suffix in importlib.machinery.all_suffixes(): - if fn.endswith(suffix): - break - else: - continue + if 'Classifier' in info: + license = self.handle_classifier_license(info['Classifier'], info.get('License', '')) + if license: + info['License'] = license - if fn.startswith(dynload_dir + os.sep): - if '/.debug/' in fn: - continue - base = os.path.basename(fn) - provided = base.split('.', 1)[0] - packages[provided] = os.path.basename(pkgdatafile) - continue + self.map_info_to_bbvar(info, extravalues) - for python_dir in python_dirs: - if fn.startswith(python_dir): - relpath = fn[len(python_dir):] - relstart, _, relremaining = relpath.partition(os.sep) - if relstart.endswith('.egg'): - relpath = relremaining - base, _ = os.path.splitext(relpath) + mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals) - if '/.debug/' in base: - continue - if os.path.basename(base) == '__init__': - base = os.path.dirname(base) - base = base.replace(os.sep + os.sep, os.sep) - provided = base.replace(os.sep, '.') - packages[provided] = os.path.basename(pkgdatafile) - return packages + extras_req = set() + if 'Extras-require' in info: + extras_req = info['Extras-require'] + if extras_req: + lines_after.append('# The following configs & dependencies are from setuptools extras_require.') + lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.') + lines_after.append('# The upstream names may not correspond exactly to bitbake package names.') + lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.') + lines_after.append('#') + lines_after.append('# Uncomment this line to enable all the optional features.') + lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req))) + for feature, feature_reqs in extras_req.items(): + unmapped_deps.difference_update(feature_reqs) - @classmethod - def run_command(cls, cmd, **popenargs): - if 'stderr' not in popenargs: - popenargs['stderr'] = subprocess.STDOUT - try: - return subprocess.check_output(cmd, **popenargs).decode('utf-8') - except OSError as exc: - logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc) - raise - except subprocess.CalledProcessError as exc: - logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output) - raise + feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs)) + lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps))) + inst_reqs = set() + if 'Install-requires' in info: + if extras_req: + lines_after.append('') + inst_reqs = info['Install-requires'] + if inst_reqs: + unmapped_deps.difference_update(inst_reqs) + + inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs)) + lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These') + lines_after.append('# upstream names may not correspond exactly to bitbake package names.') + lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps))) + + if mapped_deps: + name = info.get('Name') + if name and name[0] in mapped_deps: + # Attempt to avoid self-reference + mapped_deps.remove(name[0]) + mapped_deps -= set(self.excluded_pkgdeps) + if inst_reqs or extras_req: + lines_after.append('') + lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the') + lines_after.append('# python sources, and might not be 100% accurate.') + lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps)))) + + unmapped_deps -= set(extensions) + unmapped_deps -= set(self.assume_provided) + if unmapped_deps: + if mapped_deps: + lines_after.append('') + lines_after.append('# WARNING: We were unable to map the following python package/module') + lines_after.append('# dependencies to the bitbake packages which include them:') + lines_after.extend('# {}'.format(d) for d in sorted(unmapped_deps)) + + handled.append('buildsystem') def gather_setup_info(fileobj): parsed = ast.parse(fileobj.read(), fileobj.name) @@ -748,4 +770,4 @@ def has_non_literals(value): def register_recipe_handlers(handlers): # We need to make sure this is ahead of the makefile fallback handler - handlers.append((PythonRecipeHandler(), 70)) + handlers.append((PythonSetupPyRecipeHandler(), 70)) From patchwork Wed Oct 18 12:01:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62FA6CDB482 for ; Wed, 18 Oct 2023 12:01:26 +0000 (UTC) Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) by mx.groups.io with SMTP id smtpd.web11.279505.1697630480327394058 for ; Wed, 18 Oct 2023 05:01:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=EEnVjJzQ; spf=pass (domain: baylibre.com, ip: 209.85.167.52, mailfrom: jstephan@baylibre.com) Received: by mail-lf1-f52.google.com with SMTP id 2adb3069b0e04-507962561adso7993107e87.0 for ; Wed, 18 Oct 2023 05:01:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1697630478; x=1698235278; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Zf3+mF+ADEXhiKuFNw47M820pdeUykGEqBItJp/P2ZM=; b=EEnVjJzQV2RJzxNGm5hYLkvMY+9s3N2E8gLxAjBf1uAf89N33Wt6jdH5rS/Qv7+aUU /HKtJ+kiwfBLARQT4b8sCkdDBNNh9VNqthuvFGqLgaGjXLyfZdb3B5DJuZKtRAhRHKxs D9PGxXENkZuKPgTdGqM3KoZTHQSqGtt5iQnJeunioOwfJpMTyBWx92i2hB96s03l44EX L5D5rgbamh5B15TXmQ3Ukrzjf1zGM0eat+bse0rOXLhiXLuJ897cTl/3jntA4YRbjdun ih9/D/A268sXFbn1j+8gvixEZXkNxo3armWKbrleIB1CMTd6CmiPwAuDd3aBhnKdPnSc M34Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697630478; x=1698235278; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Zf3+mF+ADEXhiKuFNw47M820pdeUykGEqBItJp/P2ZM=; b=nk62iL8+eJ8LXOTMb4zFzjOspGG9nKkm3JFR2juHFfki1HtlHXgBumdvU8MnAiikmS nSuGZVTFG+mO4L7JnCOkMOTSCnGMIMcezdJSgO8M62IsuuLZTNFhSoUzxpQh5teaSeky AzKssKSSbsTU8DmMnYtlKAuydAEmaMObcaqRIoPDTTo/mMv0/u9dmhqX1vL2FdEymtS6 b69VVeNIEU0HCZx4+5RctmvoNI3ikXu0x6CpaqASpsSbBYD9EBFdjsIzxN+gPWLioixj 02ZlAbahhDuPRoKgl+blHe6jTvLdUkoHWSuPULn9rdOngEt21BatynhbGRory4mA3dnE cJVw== X-Gm-Message-State: AOJu0YzHhDgkRkYXZC1LYyyYwbSTl3k57fpZK8Pb/qZwC/aGiymXJEPN LLZkQ64UdpiA1MMXjy1e0W6i23DiROu/E9ML4R9xzA== X-Google-Smtp-Source: AGHT+IFQGK8ptK9W61I8b9aVjtyv2EYVqrfqusRxcs9bkxubcooU7j1bpjjCoeJr/jWQ5s5SkKCY4A== X-Received: by 2002:ac2:4424:0:b0:4fe:db6:cb41 with SMTP id w4-20020ac24424000000b004fe0db6cb41mr3822980lfl.39.1697630478189; Wed, 18 Oct 2023 05:01:18 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id j4-20020adfe504000000b0032da7454ebesm1960312wrm.79.2023.10.18.05.01.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 05:01:17 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support Date: Wed, 18 Oct 2023 14:01:14 +0200 Message-ID: <20231018120114.1222475-4-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231018120114.1222475-1-jstephan@baylibre.com> References: <20231018120114.1222475-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 18 Oct 2023 12:01:26 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189387 add support for PEP517 [1] if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method. [YOCTO #14737] [1]: https://peps.python.org/pep-0517/ Signed-off-by: Julien Stephan --- .../lib/recipetool/create_buildsys_python.py | 234 +++++++++++++++++- 1 file changed, 233 insertions(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 69f6f5ca511..0b601d50a4b 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -18,6 +18,7 @@ import os import re import sys import subprocess +import toml from recipetool.create import RecipeHandler logger = logging.getLogger('recipetool') @@ -656,6 +657,235 @@ class PythonSetupPyRecipeHandler(PythonRecipeHandler): handled.append('buildsystem') +class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler): + """Base class to support PEP517 and PEP518 + + PEP517 https://peps.python.org/pep-0517/#source-trees + PEP518 https://peps.python.org/pep-0518/#build-system-table + """ + + # PEP621: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/ + # add only the ones that map to a BB var + # potentially missing: optional-dependencies + bbvar_map = { + "name": "PN", + "version": "PV", + "Homepage": "HOMEPAGE", + "description": "SUMMARY", + "license": "LICENSE", + "dependencies": "RDEPENDS:${PN}", + "requires": "DEPENDS", + } + + replacements = [ + ("license", r" +$", ""), + ("license", r"^ +", ""), + ("license", r" ", "-"), + ("license", r"^GNU-", ""), + ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""), + ("license", r"^UNKNOWN$", ""), + # Remove currently unhandled version numbers from these variables + ("requires", r"\[[^\]]+\]$", ""), + ("requires", r"^([^><= ]+).*", r"\1"), + ("dependencies", r"\[[^\]]+\]$", ""), + ("dependencies", r"^([^><= ]+).*", r"\1"), + ] + + build_backend_map = { + "setuptools.build_meta": "python_setuptools_build_meta", + "poetry.core.masonry.api": "python_poetry_core", + "flit_core.buildapi": "python_flit_core", + } + + excluded_native_pkgdeps = [ + # already provided by python_setuptools_build_meta.bbclass + "python3-setuptools-native", + "python3-wheel-native", + # already provided by python_poetry_core.bbclass + "python3-poetry-core-native", + # already provided by python_flit_core.bbclass + "python3-flit-core-native", + ] + + # add here a list of known and often used packages and the corresponding bitbake package + known_deps_map = { + "setuptools": "python3-setuptools", + "wheel": "python3-wheel", + "poetry-core": "python3-poetry-core", + "flit_core": "python3-flit-core", + "setuptools-scm": "python3-setuptools-scm", + } + + def __init__(self): + pass + + def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): + info = {} + + if 'buildsystem' in handled: + return False + + # Check for non-zero size setup.py files + setupfiles = RecipeHandler.checkfiles(srctree, ["pyproject.toml"]) + for fn in setupfiles: + if os.path.getsize(fn): + break + else: + return False + + setupscript = os.path.join(srctree, "pyproject.toml") + + try: + config = self.parse_pyproject_toml(setupscript) + build_backend = config["build-system"]["build-backend"] + if build_backend in self.build_backend_map: + classes.append(self.build_backend_map[build_backend]) + else: + logger.error( + "Unsupported build-backend: %s, cannot use pyproject.toml. Will try to use legacy setup.py" + % build_backend + ) + return False + + licfile = "" + if "project" in config: + for field, values in config["project"].items(): + if field == "license": + value = values.get("text", "") + if not value: + licfile = values.get("file", "") + elif isinstance(values, dict): + for k, v in values.items(): + info[k] = v + continue + else: + value = values + + info[field] = value + + # Grab the license value before applying replacements + license_str = info.get("license", "").strip() + + if license_str: + for i, line in enumerate(lines_before): + if line.startswith("##LICENSE_PLACEHOLDER##"): + lines_before.insert( + i, "# NOTE: License in pyproject.toml is: %s" % license_str + ) + break + + info["requires"] = config["build-system"]["requires"] + + self.apply_info_replacements(info) + + if "classifiers" in info: + license = self.handle_classifier_license( + info["classifiers"], info.get("license", "") + ) + if license: + if licfile: + lines = [] + md5value = bb.utils.md5_file(os.path.join(srctree, licfile)) + lines.append('LICENSE = "%s"' % license) + lines.append( + 'LIC_FILES_CHKSUM = "file://%s;md5=%s"' + % (licfile, md5value) + ) + lines.append("") + + # Replace the placeholder so we get the values in the right place in the recipe file + try: + pos = lines_before.index("##LICENSE_PLACEHOLDER##") + except ValueError: + pos = -1 + if pos == -1: + lines_before.extend(lines) + else: + lines_before[pos : pos + 1] = lines + + handled.append(("license", [license, licfile, md5value])) + else: + info["license"] = license + + provided_packages = self.parse_pkgdata_for_python_packages() + provided_packages.update(self.known_deps_map) + native_mapped_deps, native_unmapped_deps = set(), set() + mapped_deps, unmapped_deps = set(), set() + + if "requires" in info: + for require in info["requires"]: + mapped = provided_packages.get(require) + + if mapped: + logger.error("Mapped %s to %s" % (require, mapped)) + native_mapped_deps.add(mapped) + else: + logger.error("Could not map %s" % require) + native_unmapped_deps.add(require) + + info.pop("requires") + + if native_mapped_deps != set(): + native_mapped_deps = { + item + "-native" for item in native_mapped_deps + } + native_mapped_deps -= set(self.excluded_native_pkgdeps) + if native_mapped_deps != set(): + info["requires"] = " ".join(sorted(native_mapped_deps)) + + if native_unmapped_deps: + lines_after.append("") + lines_after.append( + "# WARNING: We were unable to map the following python package/module" + ) + lines_after.append( + "# dependencies to the bitbake packages which include them:" + ) + lines_after.extend( + "# {}".format(d) for d in sorted(native_unmapped_deps) + ) + + if "dependencies" in info: + for dependency in info["dependencies"]: + mapped = provided_packages.get(dependency) + if mapped: + logger.error("Mapped %s to %s" % (dependency, mapped)) + mapped_deps.add(mapped) + else: + logger.error("Could not map %s" % dependency) + unmapped_deps.add(dependency) + + info.pop("dependencies") + + if mapped_deps != set(): + if mapped_deps != set(): + info["dependencies"] = " ".join(sorted(mapped_deps)) + + if unmapped_deps: + lines_after.append("") + lines_after.append( + "# WARNING: We were unable to map the following python package/module" + ) + lines_after.append( + "# runtime dependencies to the bitbake packages which include them:" + ) + lines_after.extend( + "# {}".format(d) for d in sorted(unmapped_deps) + ) + + self.map_info_to_bbvar(info, extravalues) + + handled.append("buildsystem") + except Exception: + logger.exception("Failed to parse pyproject.toml") + return False + + def parse_pyproject_toml(self, setupscript): + with open(setupscript, "r") as f: + config = toml.load(f) + return config + + def gather_setup_info(fileobj): parsed = ast.parse(fileobj.read(), fileobj.name) visitor = SetupScriptVisitor() @@ -769,5 +999,7 @@ def has_non_literals(value): def register_recipe_handlers(handlers): - # We need to make sure this is ahead of the makefile fallback handler + # We need to make sure these are ahead of the makefile fallback handler + # and the pyproject.toml handler ahead of the setup.py handler + handlers.append((PythonPyprojectTomlRecipeHandler(), 75)) handlers.append((PythonSetupPyRecipeHandler(), 70))