From patchwork Wed Oct 25 15:46:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33AE5C07545 for ; Wed, 25 Oct 2023 15:47:09 +0000 (UTC) Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by mx.groups.io with SMTP id smtpd.web10.176910.1698248827231984276 for ; Wed, 25 Oct 2023 08:47:07 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=1hLEvEGR; spf=pass (domain: baylibre.com, ip: 209.85.167.51, mailfrom: jstephan@baylibre.com) Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-507c78d258fso8486138e87.2 for ; Wed, 25 Oct 2023 08:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1698248825; x=1698853625; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kZKSAkB8AXb+krDxx6iPZFIE4+JVtHFgnD7HuH9JG14=; b=1hLEvEGRIbQNJVphCb7HWMnAkA2aMFAtZ9zrzvh8BIxAxqz7zq+P3NbDc6xekFDZa+ vP5XAuv3XriJkHaff40VxVu8+YW4S2kZNm/b21n9+NWGurAZ/ppL79iJbd1+irjQ/dD0 6Wzg8iHcGTRWtnzTTsTLH0Hu7KukwgoOK4AAf+Zf6w7WWQ7czEP9BX2vKNSYp3y2ffaN FO11XgezRq57dmZIiu4daTwEyNnR6sHIoFrvOtDi1/2IugC0nibDrclPZ7Z8HvQRoVtc k00LH1R4bOIMNTk6whB2OgzVj0JxqvLRHcojoyCTerLdgpdYqW/NrIG4eXLWjFNSzzsO J5Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698248825; x=1698853625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kZKSAkB8AXb+krDxx6iPZFIE4+JVtHFgnD7HuH9JG14=; b=CeF0zqVmN1fB+naBfVog3dbFL0Bvl7ZiXtOC3NyVCIIrB3vW8H6wQ5NtNp7WXVidx6 jkn4gQv2ca8n35oObDd2cfqVBzHpKFRHt7Nq/Kh8/dTRWzxvafBdqc6WouNvqk4DXejA dxRqAMzUEUVgHsZ93IEG2iXzVvrPu1hI3KyNL5u3wPppUQq1JexP9punKs1xNNdGbEEq yHrhx/kCcpd40T8gKChGyoCOTsFFBKBXyHn8EyUPUkDBXAY5lGQsTJuX9NoUA8TVfU6k uJb3eC5v25GqXmo2QQlgz5kg2uYS4W9eHbA1AWSM7kFBOXRmBT0mCV5kO9ezqxlBdOY1 iAlw== X-Gm-Message-State: AOJu0YyEDMr/U6eVIUjmrtGr7xxOyB+gtG9UheXEo7ikILpUPFuwS1GM 5tk4o/Acjl0MaRBAa2SxG20KnxRNxsiUbQcrEjWKsQ== X-Google-Smtp-Source: AGHT+IGJLoKowMg0RZva3TH+S9j6gg8KA0uDArGe9RXoGmsM7DWKv9I4NzaCpE2i4SANzGPCwM/j9A== X-Received: by 2002:ac2:560a:0:b0:507:c9d5:39a9 with SMTP id v10-20020ac2560a000000b00507c9d539a9mr11230334lfd.52.1698248825001; Wed, 25 Oct 2023 08:47:05 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id a10-20020adfe5ca000000b0032415213a6fsm12301964wrn.87.2023.10.25.08.47.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 08:47:04 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 1/5] scripts:recipetool:create_buildsys_python: fix license note Date: Wed, 25 Oct 2023 17:46:55 +0200 Message-ID: <20231025154659.2442373-2-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231025154659.2442373-1-jstephan@baylibre.com> References: <20231025154659.2442373-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 25 Oct 2023 15:47:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189682 License field of setup is not always standardized, so we usually use the classifier to determine the correct license format to use in the recipe. A warning note is added above the LICENSE field of the create recipe in case a license is provided in setup. But when the plugin is called, "LICENSE =" is not yet present so we can never display this note. Replace the "LICENSE =" condition with "##LICENSE_PLACEHOLDER##" to actually be able to display the note message Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create_buildsys_python.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 92468b22546..321d0ba257d 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -254,7 +254,7 @@ class PythonRecipeHandler(RecipeHandler): if license_str: for i, line in enumerate(lines_before): - if line.startswith('LICENSE = '): + if line.startswith('##LICENSE_PLACEHOLDER##'): lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str) break From patchwork Wed Oct 25 15:46:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B344C0032E for ; Wed, 25 Oct 2023 15:47:09 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web10.176911.1698248827540170585 for ; Wed, 25 Oct 2023 08:47:07 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=zg8FOXwr; spf=pass (domain: baylibre.com, ip: 209.85.128.43, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-4084b0223ccso45967725e9.2 for ; Wed, 25 Oct 2023 08:47:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1698248825; x=1698853625; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Qy5zuMpicCozS0T/p7DwIkyVEA5KFMZFCqYTyDt2reY=; b=zg8FOXwraThFdcfy+VV9Iy2YmOPQYWVBZMiQQzcYvzxxIubQc1HF738fdcWmPb35Go HoMuQQK0w/0Ra7sC5xcftwau7owhP76vdj/4YU+3CNFCxKxi/AK/zewJzqHvm8nGAglr ggrYvwMEMeH0YWmKwTbNLR3tHKApdYSyGc4mVXAtrpAsmkrRQBQoprdY7HFQW3kJZxXl P8qzG5jSgJyoNlznafeTlKFn0eWz54LJpWYLtL6pJvFw+CDgNVmkFJDTFilnZOFqCk1S JI5zV1ojv9rbjlDmY6+DT2iicJou27AVaNYC6vAN1xGykafQ8K0HysCAKfCsf/cxA/IV q4Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698248825; x=1698853625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Qy5zuMpicCozS0T/p7DwIkyVEA5KFMZFCqYTyDt2reY=; b=KAIrYR8bb2sGTHvG3UU99q8DLRqCCr3PMLCFM4DOB8W+UuKqaiiZsQ2MUUbrUzOIy/ rRgt+MaCjOXsb2xuEbK02tD9l2FGgP7G/5jDy3iRTxOWQWrSHiPdtqn3JtoOR0rhdjvk FgFy7UbwUrnyxqO9qp8Kh3v+nSJNWBEWhulE1baZ1MvBUzXy4mnAOIZcR9pAugJNMUNi 2Ald/6d2kMnMiSS/1fQQ83zq+O2c/5mKgscnLioMhAvHtgsrlYavUQUOgsJd7XMDOvmF IR/fy04MBXbOCqKS6jEri+pUQmZIRQIcZFX5tH0lp6OYKHJtwuib27ixxLew38DsJJIA bTtw== X-Gm-Message-State: AOJu0YyjQ2kwjp65PU1I573YQu0a9BHIspS9c/5mf61jbUg8c12deMNo urMYDGCho50DjNBx9MKxB7Oof3g/+txuTErGUBYRAA== X-Google-Smtp-Source: AGHT+IEj18AKMPn57CcgdVO1hoPxmrNpQuuoZOOQv4hGw3GPFuuoqLM/yalu/PvI/owz9S5UgVNbBQ== X-Received: by 2002:a5d:67c5:0:b0:32d:b9c5:82fb with SMTP id n5-20020a5d67c5000000b0032db9c582fbmr10987535wrw.30.1698248825505; Wed, 25 Oct 2023 08:47:05 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id a10-20020adfe5ca000000b0032415213a6fsm12301964wrn.87.2023.10.25.08.47.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 08:47:05 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 2/5] scripts:recipetool:create_buildsys_python: prefix created recipes with python3- Date: Wed, 25 Oct 2023 17:46:56 +0200 Message-ID: <20231025154659.2442373-3-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231025154659.2442373-1-jstephan@baylibre.com> References: <20231025154659.2442373-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 25 Oct 2023 15:47:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189683 By convention, all python recipes start with "python3-" so update create_buildsys_python to do this This rule doesn't apply for packages already starting with "python" Update recipetool's selftest accordingly Signed-off-by: Julien Stephan --- meta/lib/oeqa/selftest/cases/recipetool.py | 4 ++-- scripts/lib/recipetool/create_buildsys_python.py | 5 +++++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 48661bee6f2..d3aea74228f 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -445,7 +445,7 @@ class RecipetoolCreateTests(RecipetoolBase): # Basic test to see if github URL mangling works temprecipe = os.path.join(self.tempdir, 'recipe') os.makedirs(temprecipe) - recipefile = os.path.join(temprecipe, 'meson_git.bb') + recipefile = os.path.join(temprecipe, 'python3-meson_git.bb') srcuri = 'https://github.com/mesonbuild/meson;rev=0.32.0' result = runCmd(['recipetool', 'create', '-o', temprecipe, srcuri]) self.assertTrue(os.path.isfile(recipefile)) @@ -479,7 +479,7 @@ class RecipetoolCreateTests(RecipetoolBase): temprecipe = os.path.join(self.tempdir, 'recipe') os.makedirs(temprecipe) pv = '0.32.0' - recipefile = os.path.join(temprecipe, 'meson_%s.bb' % pv) + recipefile = os.path.join(temprecipe, 'python3-meson_%s.bb' % pv) srcuri = 'https://github.com/mesonbuild/meson/releases/download/%s/meson-%s.tar.gz' % (pv, pv) result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) self.assertTrue(os.path.isfile(recipefile)) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 321d0ba257d..502e1dfbc3d 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -297,6 +297,11 @@ class PythonRecipeHandler(RecipeHandler): value = ' '.join(str(v) for v in values if v) bbvar = self.bbvar_map[field] + if bbvar == "PN": + # by convention python recipes start with "python3-" + if not value.startswith('python'): + value = 'python3-' + value + if bbvar not in extravalues and value: extravalues[bbvar] = value From patchwork Wed Oct 25 15:46:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C38FC25B6D for ; Wed, 25 Oct 2023 15:47:09 +0000 (UTC) Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by mx.groups.io with SMTP id smtpd.web11.47146.1698248828292836013 for ; Wed, 25 Oct 2023 08:47:08 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=yDMBuSC1; spf=pass (domain: baylibre.com, ip: 209.85.221.48, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-32dc918d454so3715878f8f.2 for ; Wed, 25 Oct 2023 08:47:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1698248826; x=1698853626; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=USMCgTKeXzLAaKbN4c5Kpv5mmnijD+t4SluhNvlO2U4=; b=yDMBuSC1R8fZM7HtMQ00OJWXNWxJdJT0eST/h7NL0ItC0eyy4ha4xVt17E9yyywXeT Phl606UPyyjZ9keQyFEQWAK1zbGL7CQ859Fh2PFPaHRM5/m/Cd1epCyVMftxIjurykca IQpiu5cgwEd6NXLa7M4R3Ot6wAs11/GcRAl1+jBkAaQwf1uZOD4m6PKmpERTk3SuAIpu BIQ4/Y9H+GeEBKDcfKF1gfFH7P0ZLoW6pdA1t4HCrVsQsrsO71H4jKXR46jE1MfloiPN pVC089xdQV9ohFSy0Xunpat/PQZ4IrupjqmaA49UW8NxpVvD5m5/OvEk2NQIIhNPcpUR owxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698248826; x=1698853626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=USMCgTKeXzLAaKbN4c5Kpv5mmnijD+t4SluhNvlO2U4=; b=LxbfSJreaeOf8PxGNuQ5ozJZ775CddU72HsCvTwC8DGDDuvLzLE6uJ0lZokOWY+uPG D+BWZCBOAji+N+HGKmJtI/9rybL8v36Gh/lCgsWKgpUaEUTBPjOOMLEbeqoneN8pZLuj G203R4YJRN5/iGglg/t3hrnCjqQQR+I9I7XXRtU5Ubefsr1C6nd1WGSN58eCHQ3n7jRO uFnp+LAPkaMs+sJHjQTjxSQNES4W3Y5/TJ32FuiNG3VWMmFgdQTTn2uEJNUp1bAwxp30 +KcSiRtEzc60BYuqnCJe8kgvU8MA6sc36Rf4KqgyDsbZSpoQzm7ujGGTPWyKdnhBreOQ v/JA== X-Gm-Message-State: AOJu0YzCIyl8qu6AS+7nUReiULFuWXB0tdxcilwrSFUaseZ7fi5zj+cP 1UT3pXQ1CJzUkpqhQB5uZ3WfXoyOCqQQsjWvK3Rneg== X-Google-Smtp-Source: AGHT+IEZ6Ue1v9wCjR23efucXatCYzJqJQEv2pMVD9GoU4LLwtP+0xE1d7Fm2S7pMW94JnsGLS8Guw== X-Received: by 2002:adf:f7c6:0:b0:32d:b8f8:2b18 with SMTP id a6-20020adff7c6000000b0032db8f82b18mr11368072wrq.32.1698248826064; Wed, 25 Oct 2023 08:47:06 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id a10-20020adfe5ca000000b0032415213a6fsm12301964wrn.87.2023.10.25.08.47.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 08:47:05 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 3/5] scripts:recipetool:create_buildsys_python: refactor code for futur PEP517 addition Date: Wed, 25 Oct 2023 17:46:57 +0200 Message-ID: <20231025154659.2442373-4-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231025154659.2442373-1-jstephan@baylibre.com> References: <20231025154659.2442373-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 25 Oct 2023 15:47:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189686 In order to prepare the support for pyproject.toml (PEP517 [1]) enabled projects, refactor the code and move setup.py specific code into a specific class in order to allow sharing the PythonRecipeHandler class No functionnal changes expected [1]: https://peps.python.org/pep-0517/#source-tree Signed-off-by: Julien Stephan --- .../lib/recipetool/create_buildsys_python.py | 748 +++++++++--------- 1 file changed, 385 insertions(+), 363 deletions(-) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 502e1dfbc3d..69f6f5ca511 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -37,63 +37,8 @@ class PythonRecipeHandler(RecipeHandler): assume_provided = ['builtins', 'os.path'] # Assumes that the host python3 builtin_module_names is sane for target too assume_provided = assume_provided + list(sys.builtin_module_names) + excluded_fields = [] - bbvar_map = { - 'Name': 'PN', - 'Version': 'PV', - 'Home-page': 'HOMEPAGE', - 'Summary': 'SUMMARY', - 'Description': 'DESCRIPTION', - 'License': 'LICENSE', - 'Requires': 'RDEPENDS:${PN}', - 'Provides': 'RPROVIDES:${PN}', - 'Obsoletes': 'RREPLACES:${PN}', - } - # PN/PV are already set by recipetool core & desc can be extremely long - excluded_fields = [ - 'Description', - ] - setup_parse_map = { - 'Url': 'Home-page', - 'Classifiers': 'Classifier', - 'Description': 'Summary', - } - setuparg_map = { - 'Home-page': 'url', - 'Classifier': 'classifiers', - 'Summary': 'description', - 'Description': 'long-description', - } - # Values which are lists, used by the setup.py argument based metadata - # extraction method, to determine how to process the setup.py output. - setuparg_list_fields = [ - 'Classifier', - 'Requires', - 'Provides', - 'Obsoletes', - 'Platform', - 'Supported-Platform', - ] - setuparg_multi_line_values = ['Description'] - replacements = [ - ('License', r' +$', ''), - ('License', r'^ +', ''), - ('License', r' ', '-'), - ('License', r'^GNU-', ''), - ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''), - ('License', r'^UNKNOWN$', ''), - - # Remove currently unhandled version numbers from these variables - ('Requires', r' *\([^)]*\)', ''), - ('Provides', r' *\([^)]*\)', ''), - ('Obsoletes', r' *\([^)]*\)', ''), - ('Install-requires', r'^([^><= ]+).*', r'\1'), - ('Extras-require', r'^([^><= ]+).*', r'\1'), - ('Tests-require', r'^([^><= ]+).*', r'\1'), - - # Remove unhandled dependency on particular features (e.g. foo[PDF]) - ('Install-requires', r'\[[^\]]+\]$', ''), - ] classifier_license_map = { 'License :: OSI Approved :: Academic Free License (AFL)': 'AFL', @@ -166,122 +111,34 @@ class PythonRecipeHandler(RecipeHandler): def __init__(self): pass - def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): - if 'buildsystem' in handled: - return False - - # Check for non-zero size setup.py files - setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py']) - for fn in setupfiles: - if os.path.getsize(fn): - break - else: - return False - - # setup.py is always parsed to get at certain required information, such as - # distutils vs setuptools - # - # If egg info is available, we use it for both its PKG-INFO metadata - # and for its requires.txt for install_requires. - # If PKG-INFO is available but no egg info is, we use that for metadata in preference to - # the parsed setup.py, but use the install_requires info from the - # parsed setup.py. - - setupscript = os.path.join(srctree, 'setup.py') - try: - setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript) - except Exception: - logger.exception("Failed to parse setup.py") - setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], [] - - egginfo = glob.glob(os.path.join(srctree, '*.egg-info')) - if egginfo: - info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO')) - requires_txt = os.path.join(egginfo[0], 'requires.txt') - if os.path.exists(requires_txt): - with codecs.open(requires_txt) as f: - inst_req = [] - extras_req = collections.defaultdict(list) - current_feature = None - for line in f.readlines(): - line = line.rstrip() - if not line: - continue - - if line.startswith('['): - # PACKAGECONFIG must not contain expressions or whitespace - line = line.replace(" ", "") - line = line.replace(':', "") - line = line.replace('.', "-dot-") - line = line.replace('"', "") - line = line.replace('<', "-smaller-") - line = line.replace('>', "-bigger-") - line = line.replace('_', "-") - line = line.replace('(', "") - line = line.replace(')', "") - line = line.replace('!', "-not-") - line = line.replace('=', "-equals-") - current_feature = line[1:-1] - elif current_feature: - extras_req[current_feature].append(line) - else: - inst_req.append(line) - info['Install-requires'] = inst_req - info['Extras-require'] = extras_req - elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']): - info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO')) - - if setup_info: - if 'Install-requires' in setup_info: - info['Install-requires'] = setup_info['Install-requires'] - if 'Extras-require' in setup_info: - info['Extras-require'] = setup_info['Extras-require'] - else: - if setup_info: - info = setup_info - else: - info = self.get_setup_args_info(setupscript) - - # Grab the license value before applying replacements - license_str = info.get('License', '').strip() - - self.apply_info_replacements(info) - - if uses_setuptools: - classes.append('setuptools3') - else: - classes.append('distutils3') - - if license_str: - for i, line in enumerate(lines_before): - if line.startswith('##LICENSE_PLACEHOLDER##'): - lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str) - break - - if 'Classifier' in info: - existing_licenses = info.get('License', '') - licenses = [] - for classifier in info['Classifier']: - if classifier in self.classifier_license_map: - license = self.classifier_license_map[classifier] - if license == 'Apache' and 'Apache-2.0' in existing_licenses: - license = 'Apache-2.0' - elif license == 'GPL': - if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses: - license = 'GPL-2.0' - elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses: - license = 'GPL-3.0' - elif license == 'LGPL': - if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses: - license = 'LGPL-2.1' - elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses: - license = 'LGPL-2.0' - elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses: - license = 'LGPL-3.0' - licenses.append(license) - - if licenses: - info['License'] = ' & '.join(licenses) + def handle_classifier_license(self, classifiers, existing_licenses=""): + + licenses = [] + for classifier in classifiers: + if classifier in self.classifier_license_map: + license = self.classifier_license_map[classifier] + if license == 'Apache' and 'Apache-2.0' in existing_licenses: + license = 'Apache-2.0' + elif license == 'GPL': + if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses: + license = 'GPL-2.0' + elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses: + license = 'GPL-3.0' + elif license == 'LGPL': + if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses: + license = 'LGPL-2.1' + elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses: + license = 'LGPL-2.0' + elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses: + license = 'LGPL-3.0' + licenses.append(license) + + if licenses: + return ' & '.join(licenses) + + return None + + def map_info_to_bbvar(self, info, extravalues): # Map PKG-INFO & setup.py fields to bitbake variables for field, values in info.items(): @@ -305,85 +162,220 @@ class PythonRecipeHandler(RecipeHandler): if bbvar not in extravalues and value: extravalues[bbvar] = value - mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals) - - extras_req = set() - if 'Extras-require' in info: - extras_req = info['Extras-require'] - if extras_req: - lines_after.append('# The following configs & dependencies are from setuptools extras_require.') - lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.') - lines_after.append('# The upstream names may not correspond exactly to bitbake package names.') - lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.') - lines_after.append('#') - lines_after.append('# Uncomment this line to enable all the optional features.') - lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req))) - for feature, feature_reqs in extras_req.items(): - unmapped_deps.difference_update(feature_reqs) - - feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs)) - lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps))) - - inst_reqs = set() - if 'Install-requires' in info: - if extras_req: - lines_after.append('') - inst_reqs = info['Install-requires'] - if inst_reqs: - unmapped_deps.difference_update(inst_reqs) - - inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs)) - lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These') - lines_after.append('# upstream names may not correspond exactly to bitbake package names.') - lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps))) + def apply_info_replacements(self, info): + if not self.replacements: + return - if mapped_deps: - name = info.get('Name') - if name and name[0] in mapped_deps: - # Attempt to avoid self-reference - mapped_deps.remove(name[0]) - mapped_deps -= set(self.excluded_pkgdeps) - if inst_reqs or extras_req: - lines_after.append('') - lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the') - lines_after.append('# python sources, and might not be 100% accurate.') - lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps)))) + for variable, search, replace in self.replacements: + if variable not in info: + continue - unmapped_deps -= set(extensions) - unmapped_deps -= set(self.assume_provided) - if unmapped_deps: - if mapped_deps: - lines_after.append('') - lines_after.append('# WARNING: We were unable to map the following python package/module') - lines_after.append('# dependencies to the bitbake packages which include them:') - lines_after.extend('# {}'.format(d) for d in sorted(unmapped_deps)) + def replace_value(search, replace, value): + if replace is None: + if re.search(search, value): + return None + else: + new_value = re.sub(search, replace, value) + if value != new_value: + return new_value + return value - handled.append('buildsystem') + value = info[variable] + if isinstance(value, str): + new_value = replace_value(search, replace, value) + if new_value is None: + del info[variable] + elif new_value != value: + info[variable] = new_value + elif hasattr(value, 'items'): + for dkey, dvalue in list(value.items()): + new_list = [] + for pos, a_value in enumerate(dvalue): + new_value = replace_value(search, replace, a_value) + if new_value is not None and new_value != value: + new_list.append(new_value) - def get_pkginfo(self, pkginfo_fn): - msg = email.message_from_file(open(pkginfo_fn, 'r')) - msginfo = {} - for field in msg.keys(): - values = msg.get_all(field) - if len(values) == 1: - msginfo[field] = values[0] + if value != new_list: + value[dkey] = new_list else: - msginfo[field] = values - return msginfo + new_list = [] + for pos, a_value in enumerate(value): + new_value = replace_value(search, replace, a_value) + if new_value is not None and new_value != value: + new_list.append(new_value) - def parse_setup_py(self, setupscript='./setup.py'): - with codecs.open(setupscript) as f: - info, imported_modules, non_literals, extensions = gather_setup_info(f) + if value != new_list: + info[variable] = new_list - def _map(key): - key = key.replace('_', '-') - key = key[0].upper() + key[1:] - if key in self.setup_parse_map: - key = self.setup_parse_map[key] - return key - # Naive mapping of setup() arguments to PKG-INFO field names - for d in [info, non_literals]: + def scan_python_dependencies(self, paths): + deps = set() + try: + dep_output = self.run_command(['pythondeps', '-d'] + paths) + except (OSError, subprocess.CalledProcessError): + pass + else: + for line in dep_output.splitlines(): + line = line.rstrip() + dep, filename = line.split('\t', 1) + if filename.endswith('/setup.py'): + continue + deps.add(dep) + + try: + provides_output = self.run_command(['pythondeps', '-p'] + paths) + except (OSError, subprocess.CalledProcessError): + pass + else: + provides_lines = (l.rstrip() for l in provides_output.splitlines()) + provides = set(l for l in provides_lines if l and l != 'setup') + deps -= provides + + return deps + + def parse_pkgdata_for_python_packages(self): + pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR') + + ldata = tinfoil.config_data.createCopy() + bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True) + python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR') + + dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload') + python_dirs = [python_sitedir + os.sep, + os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep, + os.path.dirname(python_sitedir) + os.sep] + packages = {} + for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)): + files_info = None + with open(pkgdatafile, 'r') as f: + for line in f.readlines(): + field, value = line.split(': ', 1) + if field.startswith('FILES_INFO'): + files_info = ast.literal_eval(value) + break + else: + continue + + for fn in files_info: + for suffix in importlib.machinery.all_suffixes(): + if fn.endswith(suffix): + break + else: + continue + + if fn.startswith(dynload_dir + os.sep): + if '/.debug/' in fn: + continue + base = os.path.basename(fn) + provided = base.split('.', 1)[0] + packages[provided] = os.path.basename(pkgdatafile) + continue + + for python_dir in python_dirs: + if fn.startswith(python_dir): + relpath = fn[len(python_dir):] + relstart, _, relremaining = relpath.partition(os.sep) + if relstart.endswith('.egg'): + relpath = relremaining + base, _ = os.path.splitext(relpath) + + if '/.debug/' in base: + continue + if os.path.basename(base) == '__init__': + base = os.path.dirname(base) + base = base.replace(os.sep + os.sep, os.sep) + provided = base.replace(os.sep, '.') + packages[provided] = os.path.basename(pkgdatafile) + return packages + + @classmethod + def run_command(cls, cmd, **popenargs): + if 'stderr' not in popenargs: + popenargs['stderr'] = subprocess.STDOUT + try: + return subprocess.check_output(cmd, **popenargs).decode('utf-8') + except OSError as exc: + logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc) + raise + except subprocess.CalledProcessError as exc: + logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output) + raise + +class PythonSetupPyRecipeHandler(PythonRecipeHandler): + bbvar_map = { + 'Name': 'PN', + 'Version': 'PV', + 'Home-page': 'HOMEPAGE', + 'Summary': 'SUMMARY', + 'Description': 'DESCRIPTION', + 'License': 'LICENSE', + 'Requires': 'RDEPENDS:${PN}', + 'Provides': 'RPROVIDES:${PN}', + 'Obsoletes': 'RREPLACES:${PN}', + } + # PN/PV are already set by recipetool core & desc can be extremely long + excluded_fields = [ + 'Description', + ] + setup_parse_map = { + 'Url': 'Home-page', + 'Classifiers': 'Classifier', + 'Description': 'Summary', + } + setuparg_map = { + 'Home-page': 'url', + 'Classifier': 'classifiers', + 'Summary': 'description', + 'Description': 'long-description', + } + # Values which are lists, used by the setup.py argument based metadata + # extraction method, to determine how to process the setup.py output. + setuparg_list_fields = [ + 'Classifier', + 'Requires', + 'Provides', + 'Obsoletes', + 'Platform', + 'Supported-Platform', + ] + setuparg_multi_line_values = ['Description'] + + replacements = [ + ('License', r' +$', ''), + ('License', r'^ +', ''), + ('License', r' ', '-'), + ('License', r'^GNU-', ''), + ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''), + ('License', r'^UNKNOWN$', ''), + + # Remove currently unhandled version numbers from these variables + ('Requires', r' *\([^)]*\)', ''), + ('Provides', r' *\([^)]*\)', ''), + ('Obsoletes', r' *\([^)]*\)', ''), + ('Install-requires', r'^([^><= ]+).*', r'\1'), + ('Extras-require', r'^([^><= ]+).*', r'\1'), + ('Tests-require', r'^([^><= ]+).*', r'\1'), + + # Remove unhandled dependency on particular features (e.g. foo[PDF]) + ('Install-requires', r'\[[^\]]+\]$', ''), + ] + + def __init__(self): + pass + + def parse_setup_py(self, setupscript='./setup.py'): + with codecs.open(setupscript) as f: + info, imported_modules, non_literals, extensions = gather_setup_info(f) + + def _map(key): + key = key.replace('_', '-') + key = key[0].upper() + key[1:] + if key in self.setup_parse_map: + key = self.setup_parse_map[key] + return key + + # Naive mapping of setup() arguments to PKG-INFO field names + for d in [info, non_literals]: for key, value in list(d.items()): if key is None: continue @@ -445,47 +437,16 @@ class PythonRecipeHandler(RecipeHandler): info[fields[lineno]] = line return info - def apply_info_replacements(self, info): - for variable, search, replace in self.replacements: - if variable not in info: - continue - - def replace_value(search, replace, value): - if replace is None: - if re.search(search, value): - return None - else: - new_value = re.sub(search, replace, value) - if value != new_value: - return new_value - return value - - value = info[variable] - if isinstance(value, str): - new_value = replace_value(search, replace, value) - if new_value is None: - del info[variable] - elif new_value != value: - info[variable] = new_value - elif hasattr(value, 'items'): - for dkey, dvalue in list(value.items()): - new_list = [] - for pos, a_value in enumerate(dvalue): - new_value = replace_value(search, replace, a_value) - if new_value is not None and new_value != value: - new_list.append(new_value) - - if value != new_list: - value[dkey] = new_list + def get_pkginfo(self, pkginfo_fn): + msg = email.message_from_file(open(pkginfo_fn, 'r')) + msginfo = {} + for field in msg.keys(): + values = msg.get_all(field) + if len(values) == 1: + msginfo[field] = values[0] else: - new_list = [] - for pos, a_value in enumerate(value): - new_value = replace_value(search, replace, a_value) - if new_value is not None and new_value != value: - new_list.append(new_value) - - if value != new_list: - info[variable] = new_list + msginfo[field] = values + return msginfo def scan_setup_python_deps(self, srctree, setup_info, setup_non_literals): if 'Package-dir' in setup_info: @@ -540,99 +501,160 @@ class PythonRecipeHandler(RecipeHandler): unmapped_deps.add(dep) return mapped_deps, unmapped_deps - def scan_python_dependencies(self, paths): - deps = set() - try: - dep_output = self.run_command(['pythondeps', '-d'] + paths) - except (OSError, subprocess.CalledProcessError): - pass + def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): + + if 'buildsystem' in handled: + return False + + # Check for non-zero size setup.py files + setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py']) + for fn in setupfiles: + if os.path.getsize(fn): + break else: - for line in dep_output.splitlines(): - line = line.rstrip() - dep, filename = line.split('\t', 1) - if filename.endswith('/setup.py'): - continue - deps.add(dep) + return False + + # setup.py is always parsed to get at certain required information, such as + # distutils vs setuptools + # + # If egg info is available, we use it for both its PKG-INFO metadata + # and for its requires.txt for install_requires. + # If PKG-INFO is available but no egg info is, we use that for metadata in preference to + # the parsed setup.py, but use the install_requires info from the + # parsed setup.py. + setupscript = os.path.join(srctree, 'setup.py') try: - provides_output = self.run_command(['pythondeps', '-p'] + paths) - except (OSError, subprocess.CalledProcessError): - pass + setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript) + except Exception: + logger.exception("Failed to parse setup.py") + setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], [] + + egginfo = glob.glob(os.path.join(srctree, '*.egg-info')) + if egginfo: + info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO')) + requires_txt = os.path.join(egginfo[0], 'requires.txt') + if os.path.exists(requires_txt): + with codecs.open(requires_txt) as f: + inst_req = [] + extras_req = collections.defaultdict(list) + current_feature = None + for line in f.readlines(): + line = line.rstrip() + if not line: + continue + + if line.startswith('['): + # PACKAGECONFIG must not contain expressions or whitespace + line = line.replace(" ", "") + line = line.replace(':', "") + line = line.replace('.', "-dot-") + line = line.replace('"', "") + line = line.replace('<', "-smaller-") + line = line.replace('>', "-bigger-") + line = line.replace('_', "-") + line = line.replace('(', "") + line = line.replace(')', "") + line = line.replace('!', "-not-") + line = line.replace('=', "-equals-") + current_feature = line[1:-1] + elif current_feature: + extras_req[current_feature].append(line) + else: + inst_req.append(line) + info['Install-requires'] = inst_req + info['Extras-require'] = extras_req + elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']): + info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO')) + + if setup_info: + if 'Install-requires' in setup_info: + info['Install-requires'] = setup_info['Install-requires'] + if 'Extras-require' in setup_info: + info['Extras-require'] = setup_info['Extras-require'] else: - provides_lines = (l.rstrip() for l in provides_output.splitlines()) - provides = set(l for l in provides_lines if l and l != 'setup') - deps -= provides + if setup_info: + info = setup_info + else: + info = self.get_setup_args_info(setupscript) - return deps + # Grab the license value before applying replacements + license_str = info.get('License', '').strip() - def parse_pkgdata_for_python_packages(self): - pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR') + self.apply_info_replacements(info) - ldata = tinfoil.config_data.createCopy() - bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True) - python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR') + if uses_setuptools: + classes.append('setuptools3') + else: + classes.append('distutils3') - dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload') - python_dirs = [python_sitedir + os.sep, - os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep, - os.path.dirname(python_sitedir) + os.sep] - packages = {} - for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)): - files_info = None - with open(pkgdatafile, 'r') as f: - for line in f.readlines(): - field, value = line.split(': ', 1) - if field.startswith('FILES_INFO'): - files_info = ast.literal_eval(value) - break - else: - continue + if license_str: + for i, line in enumerate(lines_before): + if line.startswith('##LICENSE_PLACEHOLDER##'): + lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str) + break - for fn in files_info: - for suffix in importlib.machinery.all_suffixes(): - if fn.endswith(suffix): - break - else: - continue + if 'Classifier' in info: + license = self.handle_classifier_license(info['Classifier'], info.get('License', '')) + if license: + info['License'] = license - if fn.startswith(dynload_dir + os.sep): - if '/.debug/' in fn: - continue - base = os.path.basename(fn) - provided = base.split('.', 1)[0] - packages[provided] = os.path.basename(pkgdatafile) - continue + self.map_info_to_bbvar(info, extravalues) - for python_dir in python_dirs: - if fn.startswith(python_dir): - relpath = fn[len(python_dir):] - relstart, _, relremaining = relpath.partition(os.sep) - if relstart.endswith('.egg'): - relpath = relremaining - base, _ = os.path.splitext(relpath) + mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals) - if '/.debug/' in base: - continue - if os.path.basename(base) == '__init__': - base = os.path.dirname(base) - base = base.replace(os.sep + os.sep, os.sep) - provided = base.replace(os.sep, '.') - packages[provided] = os.path.basename(pkgdatafile) - return packages + extras_req = set() + if 'Extras-require' in info: + extras_req = info['Extras-require'] + if extras_req: + lines_after.append('# The following configs & dependencies are from setuptools extras_require.') + lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.') + lines_after.append('# The upstream names may not correspond exactly to bitbake package names.') + lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.') + lines_after.append('#') + lines_after.append('# Uncomment this line to enable all the optional features.') + lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req))) + for feature, feature_reqs in extras_req.items(): + unmapped_deps.difference_update(feature_reqs) - @classmethod - def run_command(cls, cmd, **popenargs): - if 'stderr' not in popenargs: - popenargs['stderr'] = subprocess.STDOUT - try: - return subprocess.check_output(cmd, **popenargs).decode('utf-8') - except OSError as exc: - logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc) - raise - except subprocess.CalledProcessError as exc: - logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output) - raise + feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs)) + lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps))) + inst_reqs = set() + if 'Install-requires' in info: + if extras_req: + lines_after.append('') + inst_reqs = info['Install-requires'] + if inst_reqs: + unmapped_deps.difference_update(inst_reqs) + + inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs)) + lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These') + lines_after.append('# upstream names may not correspond exactly to bitbake package names.') + lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps))) + + if mapped_deps: + name = info.get('Name') + if name and name[0] in mapped_deps: + # Attempt to avoid self-reference + mapped_deps.remove(name[0]) + mapped_deps -= set(self.excluded_pkgdeps) + if inst_reqs or extras_req: + lines_after.append('') + lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the') + lines_after.append('# python sources, and might not be 100% accurate.') + lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps)))) + + unmapped_deps -= set(extensions) + unmapped_deps -= set(self.assume_provided) + if unmapped_deps: + if mapped_deps: + lines_after.append('') + lines_after.append('# WARNING: We were unable to map the following python package/module') + lines_after.append('# dependencies to the bitbake packages which include them:') + lines_after.extend('# {}'.format(d) for d in sorted(unmapped_deps)) + + handled.append('buildsystem') def gather_setup_info(fileobj): parsed = ast.parse(fileobj.read(), fileobj.name) @@ -748,4 +770,4 @@ def has_non_literals(value): def register_recipe_handlers(handlers): # We need to make sure this is ahead of the makefile fallback handler - handlers.append((PythonRecipeHandler(), 70)) + handlers.append((PythonSetupPyRecipeHandler(), 70)) From patchwork Wed Oct 25 15:46:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69FBCC25B6F for ; Wed, 25 Oct 2023 15:47:09 +0000 (UTC) Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mx.groups.io with SMTP id smtpd.web11.47148.1698248828406549875 for ; Wed, 25 Oct 2023 08:47:08 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=iHn1IuGp; spf=pass (domain: baylibre.com, ip: 209.85.221.53, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-307d58b3efbso3926933f8f.0 for ; Wed, 25 Oct 2023 08:47:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1698248826; x=1698853626; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ItcQgBfUjVT6z9AgtopLXVF802NEDrxnfCNLnj+QcBQ=; b=iHn1IuGp2KFxo2WtA49U/Tyne6outrLE26yihBVnq10bgvGKgxw+pe1ETQL39ZKe8f BjwBRMB6PCkd20c0t6zZiFMqLOgO/0uIAgpBYXO+KPYY4RafN3FBWef8n6RCuj0h4/U2 TY1rKkVm8aJSWybkH02ELvyGMUNGb3lS3wOpax0dP8hucJEgSFO+DjZxIpOJe2O3/k8C kR6uJF4MMPmP38O3WyDKcj3P/cTNlA4aaOloVQQr7kJFzRsFN+l33XhaogTorKNwNTGz DRCsJDJ7SPVWdb+5oZFWSNcyyMCH37ep79p5QgN6skcRDsEZf/cQIoeHQLnWBACTST61 mtIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698248826; x=1698853626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ItcQgBfUjVT6z9AgtopLXVF802NEDrxnfCNLnj+QcBQ=; b=P1ngL+2Icid57wwnIMP8Ae9olbNLCdp9eHgD1G9YUvVEl3A2JmLjKFBib1RtTC0KJN AU2z5wBlO9elTVn+k4/+KA9YmtE6vaZPsaJ2TWsRy6wO5zqyhAGNLMBRSJgy8+/vFy/h Hgg+sLTxyfMyUSl1uZhFBtvz86uK43aT0pxOvD6gENtC4x6Ke4qWtGrILCIyo3N+qZeb aF22wWSQ1M2KM1Va0tBA+Qqle00dbvb+zI96W/7eBqB4M3jWep/L1V1luXVby1pnnbOG u/BhP7GryE9M+QrD/geSfcW8AHcX649TLZPKGTnDUsvxi44PoxXzMsd4K1EXXHLDjL4K 7QtQ== X-Gm-Message-State: AOJu0Yz0qKSe7FN6oECzVVc4nAU9OObvZZIvTM5SR+tjgH7/bj9qLHMa b2FEzVS21FIu6/wTfiBvkkWF3Wrj74RqGdGafG/yrg== X-Google-Smtp-Source: AGHT+IG9RZhq94rfCeoOanzhxMUXspaDAhR73HX4ARdLt1ljcJZ6Ivt+G+HknPvXtdCX96dpwUeEzg== X-Received: by 2002:a5d:453a:0:b0:32d:96e0:8048 with SMTP id j26-20020a5d453a000000b0032d96e08048mr11968557wra.9.1698248826609; Wed, 25 Oct 2023 08:47:06 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id a10-20020adfe5ca000000b0032415213a6fsm12301964wrn.87.2023.10.25.08.47.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 08:47:06 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 4/5] scripts:recipetool:create_buildsys_python: add PEP517 support Date: Wed, 25 Oct 2023 17:46:58 +0200 Message-ID: <20231025154659.2442373-5-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231025154659.2442373-1-jstephan@baylibre.com> References: <20231025154659.2442373-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 25 Oct 2023 15:47:09 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189685 Add basic support for PEP517 [1] for the 3 following backends that are supported by bitbake: - setuptools.build_meta - poetry.core.masonry.api - flit_core.buildapi If a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method. Some projects can declare a minimal pyproject.toml file, and put all the metadata in setup.py/setup.cfg/requirements.txt .. theses cases are not handled. If a pyproject.toml file is found, assumes it has all necessary metadata. As for the old setup.py method, version numbers for dependencies are not handled. Some features may be missing, such as the extra dependencies. [YOCTO #14737] [1]: https://peps.python.org/pep-0517/ Signed-off-by: Julien Stephan --- .../lib/recipetool/create_buildsys_python.py | 268 +++++++++++++++++- 1 file changed, 267 insertions(+), 1 deletion(-) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 69f6f5ca511..9e7f22c0db0 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -656,6 +656,270 @@ class PythonSetupPyRecipeHandler(PythonRecipeHandler): handled.append('buildsystem') +class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler): + """Base class to support PEP517 and PEP518 + + PEP517 https://peps.python.org/pep-0517/#source-trees + PEP518 https://peps.python.org/pep-0518/#build-system-table + """ + # bitbake currently support the 3 following backends + build_backend_map = { + "setuptools.build_meta": "python_setuptools_build_meta", + "poetry.core.masonry.api": "python_poetry_core", + "flit_core.buildapi": "python_flit_core", + } + + # setuptools.build_meta and flit declare project metadata into the "project" section of pyproject.toml + # according to PEP-621: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/#declaring-project-metadata + # while poetry uses the "tool.poetry" section according to its official documentation: https://python-poetry.org/docs/pyproject/ + # keys from "project" and "tool.poetry" sections are almost the same except for the HOMEPAGE which is "homepage" for tool.poetry + # and "Homepage" for "project" section. So keep both + bbvar_map = { + "name": "PN", + "version": "PV", + "Homepage": "HOMEPAGE", + "homepage": "HOMEPAGE", + "description": "SUMMARY", + "license": "LICENSE", + "dependencies": "RDEPENDS:${PN}", + "requires": "DEPENDS", + } + + replacements = [ + ("license", r" +$", ""), + ("license", r"^ +", ""), + ("license", r" ", "-"), + ("license", r"^GNU-", ""), + ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""), + ("license", r"^UNKNOWN$", ""), + # Remove currently unhandled version numbers from these variables + ("requires", r"\[[^\]]+\]$", ""), + ("requires", r"^([^><= ]+).*", r"\1"), + ("dependencies", r"\[[^\]]+\]$", ""), + ("dependencies", r"^([^><= ]+).*", r"\1"), + ] + + excluded_native_pkgdeps = [ + # already provided by python_setuptools_build_meta.bbclass + "python3-setuptools-native", + "python3-wheel-native", + # already provided by python_poetry_core.bbclass + "python3-poetry-core-native", + # already provided by python_flit_core.bbclass + "python3-flit-core-native", + ] + + # add here a list of known and often used packages and the corresponding bitbake package + known_deps_map = { + "setuptools": "python3-setuptools", + "wheel": "python3-wheel", + "poetry-core": "python3-poetry-core", + "flit_core": "python3-flit-core", + "setuptools-scm": "python3-setuptools-scm", + } + + def __init__(self): + pass + + def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): + info = {} + + if 'buildsystem' in handled: + return False + + # Check for non-zero size setup.py files + setupfiles = RecipeHandler.checkfiles(srctree, ["pyproject.toml"]) + for fn in setupfiles: + if os.path.getsize(fn): + break + else: + return False + + setupscript = os.path.join(srctree, "pyproject.toml") + + try: + try: + import tomllib + except ImportError: + try: + import tomli as tomllib + except ImportError: + logger.exception("Neither 'tomllib' nor 'tomli' could be imported. Please use python3.11 or above or install tomli module") + return False + except Exception: + logger.exception("Failed to parse pyproject.toml") + return False + + with open(setupscript, "rb") as f: + config = tomllib.load(f) + build_backend = config["build-system"]["build-backend"] + if build_backend in self.build_backend_map: + classes.append(self.build_backend_map[build_backend]) + else: + logger.error( + "Unsupported build-backend: %s, cannot use pyproject.toml. Will try to use legacy setup.py" + % build_backend + ) + return False + + licfile = "" + + if build_backend == "poetry.core.masonry.api": + if "tool" in config and "poetry" in config["tool"]: + metadata = config["tool"]["poetry"] + else: + if "project" in config: + metadata = config["project"] + + if metadata: + for field, values in metadata.items(): + if field == "license": + # For setuptools.build_meta and flit, licence is a table + # but for poetry licence is a string + if build_backend == "poetry.core.masonry.api": + value = values + else: + value = values.get("text", "") + if not value: + licfile = values.get("file", "") + continue + elif field == "dependencies" and build_backend == "poetry.core.masonry.api": + # For poetry backend, "dependencies" section looks like: + # [tool.poetry.dependencies] + # requests = "^2.13.0" + # requests = { version = "^2.13.0", source = "private" } + # See https://python-poetry.org/docs/master/pyproject/#dependencies-and-dependency-groups for more details + # This class doesn't handle versions anyway, so we just get the dependencies name here and construct a list + value = [] + for k in values.keys(): + value.append(k) + elif isinstance(values, dict): + for k, v in values.items(): + info[k] = v + continue + else: + value = values + + info[field] = value + + # Grab the license value before applying replacements + license_str = info.get("license", "").strip() + + if license_str: + for i, line in enumerate(lines_before): + if line.startswith("##LICENSE_PLACEHOLDER##"): + lines_before.insert( + i, "# NOTE: License in pyproject.toml is: %s" % license_str + ) + break + + info["requires"] = config["build-system"]["requires"] + + self.apply_info_replacements(info) + + if "classifiers" in info: + license = self.handle_classifier_license( + info["classifiers"], info.get("license", "") + ) + if license: + if licfile: + lines = [] + md5value = bb.utils.md5_file(os.path.join(srctree, licfile)) + lines.append('LICENSE = "%s"' % license) + lines.append( + 'LIC_FILES_CHKSUM = "file://%s;md5=%s"' + % (licfile, md5value) + ) + lines.append("") + + # Replace the placeholder so we get the values in the right place in the recipe file + try: + pos = lines_before.index("##LICENSE_PLACEHOLDER##") + except ValueError: + pos = -1 + if pos == -1: + lines_before.extend(lines) + else: + lines_before[pos : pos + 1] = lines + + handled.append(("license", [license, licfile, md5value])) + else: + info["license"] = license + + provided_packages = self.parse_pkgdata_for_python_packages() + provided_packages.update(self.known_deps_map) + native_mapped_deps, native_unmapped_deps = set(), set() + mapped_deps, unmapped_deps = set(), set() + + if "requires" in info: + for require in info["requires"]: + mapped = provided_packages.get(require) + + if mapped: + logger.debug("Mapped %s to %s" % (require, mapped)) + native_mapped_deps.add(mapped) + else: + logger.debug("Could not map %s" % require) + native_unmapped_deps.add(require) + + info.pop("requires") + + if native_mapped_deps != set(): + native_mapped_deps = { + item + "-native" for item in native_mapped_deps + } + native_mapped_deps -= set(self.excluded_native_pkgdeps) + if native_mapped_deps != set(): + info["requires"] = " ".join(sorted(native_mapped_deps)) + + if native_unmapped_deps: + lines_after.append("") + lines_after.append( + "# WARNING: We were unable to map the following python package/module" + ) + lines_after.append( + "# dependencies to the bitbake packages which include them:" + ) + lines_after.extend( + "# {}".format(d) for d in sorted(native_unmapped_deps) + ) + + if "dependencies" in info: + for dependency in info["dependencies"]: + mapped = provided_packages.get(dependency) + if mapped: + logger.debug("Mapped %s to %s" % (dependency, mapped)) + mapped_deps.add(mapped) + else: + logger.debug("Could not map %s" % dependency) + unmapped_deps.add(dependency) + + info.pop("dependencies") + + if mapped_deps != set(): + if mapped_deps != set(): + info["dependencies"] = " ".join(sorted(mapped_deps)) + + if unmapped_deps: + lines_after.append("") + lines_after.append( + "# WARNING: We were unable to map the following python package/module" + ) + lines_after.append( + "# runtime dependencies to the bitbake packages which include them:" + ) + lines_after.extend( + "# {}".format(d) for d in sorted(unmapped_deps) + ) + + self.map_info_to_bbvar(info, extravalues) + + handled.append("buildsystem") + except Exception: + logger.exception("Failed to correctly handle pyproject.toml, falling back to another method") + return False + + def gather_setup_info(fileobj): parsed = ast.parse(fileobj.read(), fileobj.name) visitor = SetupScriptVisitor() @@ -769,5 +1033,7 @@ def has_non_literals(value): def register_recipe_handlers(handlers): - # We need to make sure this is ahead of the makefile fallback handler + # We need to make sure these are ahead of the makefile fallback handler + # and the pyproject.toml handler ahead of the setup.py handler + handlers.append((PythonPyprojectTomlRecipeHandler(), 75)) handlers.append((PythonSetupPyRecipeHandler(), 70)) From patchwork Wed Oct 25 15:46:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 32919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 702A2C0032E for ; Wed, 25 Oct 2023 15:47:19 +0000 (UTC) Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by mx.groups.io with SMTP id smtpd.web11.47150.1698248829386200673 for ; Wed, 25 Oct 2023 08:47:09 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=sbowIteh; spf=pass (domain: baylibre.com, ip: 209.85.128.54, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-408425c7c10so47828915e9.0 for ; Wed, 25 Oct 2023 08:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1698248827; x=1698853627; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rHdX3Cq9hsw0p513sEHr6CUJF5Qu2IPBlZ8uYt17DKw=; b=sbowItehpLFDywR/bOwg3einft710e/uiynTqcajshwsBFuFR96m/a2qNCc2WTRSDM CtIM90+RbehN8abEsU6FgKSTpUXg1WlcBhxYSwodOy0gz52aIBuI/IAWgdbCUcZ1vr0U S+mCr0xhc2MNoQvXOBOR6kL4+LzLhSFDBx03QvC0QXqgpPlVeUhDrUUBLGTGrmLspckY BPwbrTSax40VIkPoUF0WZsX6P2inORyKVpS2TXM8JSHTaADBkdMM3ysVl6OnGSw7xYf9 z9yXu9FGvr67b3vhmBw4YaJYqLJzI02lS5QTidLQccltAeSC5i8X5CKsqktca2UfMtkx H0iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698248827; x=1698853627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rHdX3Cq9hsw0p513sEHr6CUJF5Qu2IPBlZ8uYt17DKw=; b=fVPW4RoiaFPbN7vl4MiREDs4fPC8SLYChEDC+w9TYv7PBooXGagmGyTthb4EdGk7Mo KeuAnWjUT9dz3V/3vzkmpS3gcyyUynuQ0sMiiGnNy/dh9kWtjvydwGQOdgFoLfg7NCNI TzMVKDcsgcn9ZlYpZDh1l0dfBfNQ5wzcseTYbHw+cW7q7q5Mhx7rlBUtWrGgWwPE3zGB rTJRHCFqAwjvWJYq8NudE8CSSAwzfRltVrRj3Ox16CwqOYnJbIp8R7UU1H/HrY5SXD+c W774p4gzqsjTKpxntiChyW8l3j+CigPJSENh3SbHqlw9WaBImfJx945x70+Fc7W++2yv W5EQ== X-Gm-Message-State: AOJu0YyAqE6aB1woUEQfWznoCYo40welApJexrG5OHHAl4XJVxG3b8Xq CPWLCRjjfpwP3o5Gq0XbqzOFj0CxEFLfJa6nEAMi3w== X-Google-Smtp-Source: AGHT+IHodaw+R6nalKullEGPgN7MSMlBCGkn1t3/c60dWYhJun5eaEGtG39z2pk1ut3vzRsO213qMg== X-Received: by 2002:adf:e852:0:b0:31f:fdd8:7d56 with SMTP id d18-20020adfe852000000b0031ffdd87d56mr10461791wrn.12.1698248827393; Wed, 25 Oct 2023 08:47:07 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id a10-20020adfe5ca000000b0032415213a6fsm12301964wrn.87.2023.10.25.08.47.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 08:47:06 -0700 (PDT) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH v4 5/5] oeqa/selftest/recipetool: add selftest for PEP-517 recipe creation Date: Wed, 25 Oct 2023 17:46:59 +0200 Message-ID: <20231025154659.2442373-6-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231025154659.2442373-1-jstephan@baylibre.com> References: <20231025154659.2442373-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Wed, 25 Oct 2023 15:47:19 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/189687 Add 3 tests to check the creation of PEP-517 project using the 3 backends supported by bitbake: - setuptools.build_meta - poetry.core.masonry.api - flit_core.buildapi Theses tests requires the tomllib python module, so skip theses tests if module is not present. tomllib module is part of python starting from 3.11 Signed-off-by: Julien Stephan --- meta/lib/oeqa/selftest/cases/recipetool.py | 102 +++++++++++++++++++++ 1 file changed, 102 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index d3aea74228f..8e0fc995f7e 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -474,6 +474,108 @@ class RecipetoolCreateTests(RecipetoolBase): inherits = ['setuptools3'] self._test_recipe_contents(recipefile, checkvars, inherits) + def test_recipetool_create_python3_pep517_setuptools_build_meta(self): + # This test require python 3.11 or above for the tomllib module + # or tomli module to be installed + try: + import tomllib + except ImportError: + try: + import tomli + except ImportError: + self.skipTest('Test requires python 3.11 or above for tomllib module or tomli module') + + # Test creating python3 package from tarball (using setuptools.build_meta class) + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + pn = 'webcolors' + pv = '1.13' + recipefile = os.path.join(temprecipe, 'python3-%s_%s.bb' % (pn, pv)) + srcuri = 'https://files.pythonhosted.org/packages/a1/fb/f95560c6a5d4469d9c49e24cf1b5d4d21ffab5608251c6020a965fb7791c/%s-%s.tar.gz' % (pn, pv) + result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['SUMMARY'] = 'A library for working with the color formats defined by HTML and CSS.' + checkvars['LICENSE'] = set(['BSD-3-Clause']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=702b1ef12cf66832a88f24c8f2ee9c19' + checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/a1/fb/f95560c6a5d4469d9c49e24cf1b5d4d21ffab5608251c6020a965fb7791c/webcolors-${PV}.tar.gz' + checkvars['SRC_URI[md5sum]'] = 'c9be30c5b0cf1cad32e4cbacbb2229e9' + checkvars['SRC_URI[sha1sum]'] = 'c90b84fb65eed9b4c9dea7f08c657bfac0e820a5' + checkvars['SRC_URI[sha256sum]'] = 'c225b674c83fa923be93d235330ce0300373d02885cef23238813b0d5668304a' + checkvars['SRC_URI[sha384sum]'] = '45652af349660f19f68d01361dd5bda287789e5ea63608f52a8cea526ac04465614db2ea236103fb8456b1fcaea96ed7' + checkvars['SRC_URI[sha512sum]'] = '074aaf135ac6b0025b88b731d1d6dfa4c539b4fff7195658cc58a4326bb9f0449a231685d312b4a1ec48ca535a838bfa5c680787fe0e61473a2a092c448937d0' + inherits = ['python_setuptools_build_meta'] + + self._test_recipe_contents(recipefile, checkvars, inherits) + + def test_recipetool_create_python3_pep517_poetry_core_masonry_api(self): + # This test require python 3.11 or above for the tomllib module + # or tomli module to be installed + try: + import tomllib + except ImportError: + try: + import tomli + except ImportError: + self.skipTest('Test requires python 3.11 or above for tomllib module or tomli module') + + # Test creating python3 package from tarball (using poetry.core.masonry.api class) + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + pn = 'iso8601' + pv = '2.1.0' + recipefile = os.path.join(temprecipe, 'python3-%s_%s.bb' % (pn, pv)) + srcuri = 'https://files.pythonhosted.org/packages/b9/f3/ef59cee614d5e0accf6fd0cbba025b93b272e626ca89fb70a3e9187c5d15/%s-%s.tar.gz' % (pn, pv) + result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['SUMMARY'] = 'Simple module to parse ISO 8601 dates' + checkvars['LICENSE'] = set(['MIT']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=aab31f2ef7ba214a5a341eaa47a7f367' + checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/b9/f3/ef59cee614d5e0accf6fd0cbba025b93b272e626ca89fb70a3e9187c5d15/iso8601-${PV}.tar.gz' + checkvars['SRC_URI[md5sum]'] = '6e33910eba87066b3be7fcf3d59d16b5' + checkvars['SRC_URI[sha1sum]'] = 'efd225b2c9fa7d9e4a1ec6ad94f3295cee982e61' + checkvars['SRC_URI[sha256sum]'] = '6b1d3829ee8921c4301998c909f7829fa9ed3cbdac0d3b16af2d743aed1ba8df' + checkvars['SRC_URI[sha384sum]'] = '255002433fe65c19adfd6b91494271b613cb25ef6a35ac77436de1e03d60cc07bf89fd716451b917f1435e4384860ef6' + checkvars['SRC_URI[sha512sum]'] = 'db57ab2a25ef91e3bc479c8539d27e853cf1fbf60986820b8999ae15d7e566425a1e0cfba47d0f3b23aa703db0576db368e6c110ba2a2f46c9a34e8ee3611fb7' + inherits = ['python_poetry_core'] + + self._test_recipe_contents(recipefile, checkvars, inherits) + + def test_recipetool_create_python3_pep517_flit_core_buildapi(self): + # This test require python 3.11 or above for the tomllib module + # or tomli module to be installed + try: + import tomllib + except ImportError: + try: + import tomli + except ImportError: + self.skipTest('Test requires python 3.11 or above for tomllib module or tomli module') + + # Test creating python3 package from tarball (using flit_core.buildapi class) + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + pn = 'typing-extensions' + pv = '4.8.0' + recipefile = os.path.join(temprecipe, 'python3-%s_%s.bb' % (pn, pv)) + srcuri = 'https://files.pythonhosted.org/packages/1f/7a/8b94bb016069caa12fc9f587b28080ac33b4fbb8ca369b98bc0a4828543e/typing_extensions-%s.tar.gz' % pv + result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['SUMMARY'] = 'Backported and Experimental Type Hints for Python 3.8+' + checkvars['LICENSE'] = set(['PSF-2.0']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=fcf6b249c2641540219a727f35d8d2c2' + checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/1f/7a/8b94bb016069caa12fc9f587b28080ac33b4fbb8ca369b98bc0a4828543e/typing_extensions-${PV}.tar.gz' + checkvars['SRC_URI[md5sum]'] = '74bafe841fbd1c27324afdeb099babdf' + checkvars['SRC_URI[sha1sum]'] = 'f8bed69cbad4a57a1a67bf8a31b62b657b47f7a3' + checkvars['SRC_URI[sha256sum]'] = 'df8e4339e9cb77357558cbdbceca33c303714cf861d1eef15e1070055ae8b7ef' + checkvars['SRC_URI[sha384sum]'] = '0bd0112234134d965c6884f3c1f95d27b6ae49cfb08108101158e31dff33c2dce729331628b69818850f1acb68f6c8d0' + checkvars['SRC_URI[sha512sum]'] = '5fbff10e085fbf3ac2e35d08d913608d8c8bca66903435ede91cdc7776d775689a53d64f5f0615fe687c6c228ac854c8651d99eb1cb96ec61c56b7ca01fdd440' + inherits = ['python_flit_core'] + + self._test_recipe_contents(recipefile, checkvars, inherits) + def test_recipetool_create_github_tarball(self): # Basic test to ensure github URL mangling doesn't apply to release tarballs temprecipe = os.path.join(self.tempdir, 'recipe')