From patchwork Fri Nov 17 11:12:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 34812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCEC6C5AE4B for ; Fri, 17 Nov 2023 11:12:43 +0000 (UTC) Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by mx.groups.io with SMTP id smtpd.web10.9054.1700219559532038603 for ; Fri, 17 Nov 2023 03:12:39 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=nUsfhsyk; spf=pass (domain: baylibre.com, ip: 209.85.128.44, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-40906fc54fdso15768295e9.0 for ; Fri, 17 Nov 2023 03:12:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1700219557; x=1700824357; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ETVTJ5File0JK0UXg/b/p+iziwLzQW1xU1GCRisKjB4=; b=nUsfhsykuTHh+bspXgmGN/pqhUqzEX1z0Gx2NBjdCuCt1ecn5Dqk/cyYkJz0gCtODd 0bk/WhOfukUGz1BlSI+IJYK5+ST+6UlebnhsAJraKweZR1Yqh6FmJeQ21QgVMivA6e6A bMyEqeZX4E9ETrKiQcqMN7+we+cFHOfaW0dTK1Fzf3WvQxroT+M4sM/lO/Lcu0WN/tkM I2bNI72DBNSx5y1xPCXisHDdFkwgJBsQkP1AAATuj4UsdiaZDwT1+aEBSmgtgwUnG7oG 4+jflcZY973d2pT45dQZfFGtz9V/bXQxOgJtFGP9zO0JD6tYBqT5D2APhsnqBMFOeHBf lwrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700219557; x=1700824357; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ETVTJ5File0JK0UXg/b/p+iziwLzQW1xU1GCRisKjB4=; b=xUcPaenxXggPFW+dXb9BXLLSsEEjlr75dtrtIubWkLRxdRRP2toTEn6QSrUbGaKAAW 44R2AVWwkTMMlChHQWELxtK87dQYYlCFdg/lx9aW2LkF4EbaAHuwJIjqFBv7dgi2Qs6b 3CDZhd6Y7wQcXKGQ1KDc0MQxvbEgJ4NspqgsZ4E4qHWOYJ/8++bkkC/mJZeKnH+N6Tp4 3ENmQZfqVHMyecYaqAEPNFfzqssojpgjGs5nCATdV9x1bpx7T+YzGz0S9uDWvBtHB85v T6VoTEiSC/myWW03K8JCjqR36VjY9vJuNcMldDxTBAW9G2PKAKz0WMv5890gjIAH5SW7 c9Wg== X-Gm-Message-State: AOJu0YwkMg3IyidFCZzVdP9tX9SvaU+a1I0vPRBmMCtDH4oG8ZZIcHg+ PFxNmxeympAc28p254RWIUbdeDMx9Gpnh8HektpzDw== X-Google-Smtp-Source: AGHT+IEbrm+vAypHT8HLW6sZ3x2baeFAgY1bwwOa2Nk9aMwngkTScP9TYcgrvASThAJNqEuLdzlQ9Q== X-Received: by 2002:a5d:6d06:0:b0:32d:857c:d51c with SMTP id e6-20020a5d6d06000000b0032d857cd51cmr15869399wrq.43.1700219556795; Fri, 17 Nov 2023 03:12:36 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id z5-20020adff745000000b0032d829e10c0sm1982440wrp.28.2023.11.17.03.12.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 03:12:36 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 1/7] bitbake: utils: remove spaces on empty lines Date: Fri, 17 Nov 2023 12:12:12 +0100 Message-ID: <20231117111218.3344066-2-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231117111218.3344066-1-jstephan@baylibre.com> References: <20231117111218.3344066-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 17 Nov 2023 11:12:43 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190837 Signed-off-by: Julien Stephan --- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index b401fa5ec7a..61ffad92ce3 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx @@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2] @@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next @@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix): @@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return default From patchwork Fri Nov 17 11:12:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 34813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F0E9C5ACB3 for ; Fri, 17 Nov 2023 11:12:43 +0000 (UTC) Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) by mx.groups.io with SMTP id smtpd.web10.9055.1700219560326233052 for ; Fri, 17 Nov 2023 03:12:40 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=ReU1Rg1V; spf=pass (domain: baylibre.com, ip: 209.85.221.47, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-32dc9ff4a8fso1169672f8f.1 for ; Fri, 17 Nov 2023 03:12:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1700219558; x=1700824358; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H0y3nqiWo+iM/VgSe9noW2bvm/Ew1cYjCiWg4MOuKPs=; b=ReU1Rg1VEAFN+No2IStetkBMut7Vx7XZAuJF7UVf+v4CjFv0j4PxitByzbYBDMeo6h QPsGh9s65XLrNEDlR3a8UAKUlOzvDfKjmiT/qw6l+29Cs+9oWiQOydMFVCFRCCQLdQus Htzki+ADAsbJDg7iayy9QW1ZJ5PNL4RAWFr+VDy5h144XpZZbzH2xQcpG/VFXx0Uh73Q GJpBSD7fz5ViAM7WLwD5e7R3WiTefn51IlbGLoZg30IVeRtqQwxfmPaGrWEGhQNySQVa IrjgUvEDgHXoAsm4YBgFlMhuTWF6OoHGE8Yhao9Hau8ZSWZqqCw+kjwGycIPYkuCsAxq N9rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700219558; x=1700824358; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H0y3nqiWo+iM/VgSe9noW2bvm/Ew1cYjCiWg4MOuKPs=; b=fDSuFcxP2KdItzRyVs2FL/oGxxzIcAwKHHyMK+gygO2QG4r4mOsOls3/cOt21eWW6S vFV2gS2zbiWBtBmmtf92aV+Z+Q5Ho09ih5dq2S5tBPgAEo02E4XRpqIly+XMhZwy+HD2 dKKAG6k08Iqhboydbgjn9y2yHR4qgW2rtcYKKRuwDALKfxTTjSsbxdX4dml1g0gO0daW 4ntNC+cfBUXz5jISSLCNjtOauswuVsQck0BdM5KVKTZariBPHYWRo6omJ14Q9Ao04B98 0HPTB6mn3IwwKNjRbUUZ6WbLyqKNk50ZB0/Ya4kk2l+gnriaQWtizhIVxJfFLXIWxFD4 IOAQ== X-Gm-Message-State: AOJu0YyV2WZtLLbUnPAh/PPpDtTS2mkQ3khdpkblFygOvMGMYNguAtOc 27VBApZNxPXEgqvExpdNhAIpI29ZbXkMPdnVtpBOqw== X-Google-Smtp-Source: AGHT+IEqXbFgmvkjAdBfJ0aWencxc0o77r4LvFwISKD7ILPHTnh928BY6MSTCSe+bCUghbJXtkaoBg== X-Received: by 2002:a5d:64c7:0:b0:32c:eeee:d438 with SMTP id f7-20020a5d64c7000000b0032ceeeed438mr14469317wri.54.1700219558355; Fri, 17 Nov 2023 03:12:38 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id z5-20020adff745000000b0032d829e10c0sm1982440wrp.28.2023.11.17.03.12.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 03:12:36 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 2/7] recipetool: create_buildsys_python.py: initialize metadata Date: Fri, 17 Nov 2023 12:12:13 +0100 Message-ID: <20231117111218.3344066-3-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231117111218.3344066-1-jstephan@baylibre.com> References: <20231117111218.3344066-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 17 Nov 2023 11:12:43 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190838 In the case pyproject.toml doesn't contains metadatas, the metadata variable is not initialized and the plugin throws an error and falls back to another plugin, which is not the desired behaviour. So just ignore metadata if we don't have them Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create_buildsys_python.py | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 9312e4abf13..b620e3271b1 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -726,6 +726,7 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler): def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): info = {} + metadata = {} if 'buildsystem' in handled: return False From patchwork Fri Nov 17 11:12:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 34811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D253C197A0 for ; Fri, 17 Nov 2023 11:12:43 +0000 (UTC) Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by mx.groups.io with SMTP id smtpd.web10.9057.1700219561845513103 for ; Fri, 17 Nov 2023 03:12:42 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=pPEm/fE+; spf=pass (domain: baylibre.com, ip: 209.85.167.41, mailfrom: jstephan@baylibre.com) Received: by mail-lf1-f41.google.com with SMTP id 2adb3069b0e04-507cee17b00so2405063e87.2 for ; Fri, 17 Nov 2023 03:12:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1700219559; x=1700824359; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yxheyAbeqtzfBn8BlEtRKNU/86m2MEWVHj4M/cqyaDk=; b=pPEm/fE+QcqfuguVbwM3Z+SC9iGo/F/8TghbN32ScuZ8CswAqvo4k5TlR+Vwv6Zter pQUC8NV/40r49RShidHNUGs61SN/tcVWgBde5vevslrIiFkq3NCoPxiWdbhKe5yjc9p2 BVcYCRESb4oilzdsbaSx/xsaqIn/KUURKMpTrjcqghc7HQ4xqCFrx1Mal+vAuAom1nMS Xtp2rToAXxqZtJ3ej8IyruIUqDKm+aJZap2O8ZE8AQ87udVB9AlbbtC7Ci/TFc+/+lk2 lB2TtWjYWetVoQFFlE8iBG6wO4C3c9VPv8kwDPaNvOslvzU/F0L95dwqboUEyBZx2Ug8 G/CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700219559; x=1700824359; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yxheyAbeqtzfBn8BlEtRKNU/86m2MEWVHj4M/cqyaDk=; b=n5KHqOipAeOFW1nDmTo/mWwH57Bd/Nm3TbCDXIvVBeapyLbiq4xWB4wgHY1H3WggT7 2LEpVPAKFvygcvSM/FGXSSDvXEugfTitzz1t6o9ALTUwTzzV47aBdr3gK2zwePy2gPlt 6AOXyXArnQnK96+vMUnsZ4paRoUV6AMlniRg3TfyZqlLAjys6Ln79CWUmt6U7nKSLrsu hq4dQtpN3JoVVr63jgs74w81NipcCGRzwAFSBrNIwlq8ARoKINryBWGLPQ71gJvZ88Fm gaNHKRLDENTnNNOjyo7wZwi3lOoIfNNoMW6xQobkwGFSE0VEJpdosJ/5h40K4XnOk1wX Uq1Q== X-Gm-Message-State: AOJu0YwB2r+2Ba5MopdoNNI2/ZuJgZRpUsqwm6WcT4a93WywUIOfuyy/ swJWGNXACydt5Q87jUk1E4MDVfDH0xhrRp4HwwK+nQ== X-Google-Smtp-Source: AGHT+IFjmj1dVYKRzOG8r4FAtJeW9QbGgysM7sAp8/IspvLWY06qCfRiKTEKgC4VMGpcmbVc1u8pdg== X-Received: by 2002:a05:6512:39c9:b0:50a:77c3:10c with SMTP id k9-20020a05651239c900b0050a77c3010cmr16085864lfu.27.1700219559081; Fri, 17 Nov 2023 03:12:39 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id z5-20020adff745000000b0032d829e10c0sm1982440wrp.28.2023.11.17.03.12.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 03:12:38 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 3/7] recipetool: create: add trailing newlines Date: Fri, 17 Nov 2023 12:12:14 +0100 Message-ID: <20231117111218.3344066-4-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231117111218.3344066-1-jstephan@baylibre.com> References: <20231117111218.3344066-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 17 Nov 2023 11:12:43 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190839 create_recipe() function relies on oe.recipeutils.patch_recipe_lines() which relies on bb.utils.edit_metadata(). edit_metada expect lines to have trailing newlines, so add it to each line before calling patch_recipe_lines, otherwise edit_metadata will not be able to squash blank line if there are two consecutive blanks after a removal Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 293198d1c88..f5d541eb6c1 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -873,8 +873,10 @@ def create_recipe(args): outlines.append('') outlines.extend(lines_after) + outlines = [ line.rstrip('\n') +"\n" for line in outlines] + if extravalues: - _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=False) + _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=True) if args.extract_to: scriptutils.git_convert_standalone_clone(srctree) @@ -890,7 +892,7 @@ def create_recipe(args): log_info_cond('Source extracted to %s' % args.extract_to, args.devtool) if outfile == '-': - sys.stdout.write('\n'.join(outlines) + '\n') + sys.stdout.write(''.join(outlines) + '\n') else: with open(outfile, 'w') as f: lastline = None @@ -898,7 +900,7 @@ def create_recipe(args): if not lastline and not line: # Skip extra blank lines continue - f.write('%s\n' % line) + f.write('%s' % line) lastline = line log_info_cond('Recipe %s has been created; further editing may be required to make it fully functional' % outfile, args.devtool) tinfoil.modified_files() From patchwork Fri Nov 17 11:12:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 34814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB721C5AE4A for ; Fri, 17 Nov 2023 11:12:43 +0000 (UTC) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx.groups.io with SMTP id smtpd.web10.9059.1700219561980575118 for ; Fri, 17 Nov 2023 03:12:42 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=iOaGnXq1; spf=pass (domain: baylibre.com, ip: 209.85.128.51, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4083cd3917eso15600795e9.3 for ; Fri, 17 Nov 2023 03:12:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1700219559; x=1700824359; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kJhNSOkg8w/cEYhk7pYpbcVolYBKJavvLGDPT7R3tyM=; b=iOaGnXq1+M14o0HUBuT/4aB5DjUlIPCLElVIQaqf4h7eegIL6Pcd2c4mhU5KK4RwOj Dfe3pZZabpSz27QlhhCqH87dMi1yGBQ6obIl7x5wr2lChTEpj/G+luYu4dC2e6fyu9Wo LtqZ5NSSCiWoZPuVzm6/ds1YWZyhGJ5VOmFVEy7IZ3Mmwd9F++3n1FqP3Cq6Fkukg7S5 74hVp963SsfpwB/Y42PP1iVsLhCn0c0eZ3sEaq42+ebWHJgs/DKKpUcN6YpSb+Bj1gGU sWOnjHUsW0DuiNbns1MUKDn0jwPdpFWjMMXujIMShEoK8mJvXV/uUpz0bV0KiyejGlkY 0lfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700219559; x=1700824359; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kJhNSOkg8w/cEYhk7pYpbcVolYBKJavvLGDPT7R3tyM=; b=iHHj+QiOiuDYFDhgOKA0OaN846qF5CKLjsGHQRc9xg0/p1QTdUW/dwNi8YgRSjYXcw 3Df2ZPNXZ41RQybOUeZHAvejLgTRBg2jZKP0mBsWD8DDDX81p6mT5H2bRF6hU01cnbhA hiQcGucZPfufNOLNCl57dtNB3bSlz7Miwemb5aVhJ+41tNNq6eMsGgGdSnMO5sRdoJWd 7cr6mLf/XhDHEhn9C2kvm/CZxErwGb9qFqmPnevoIlQ9k8RWyat7hFWtnaxJBQjYZwPz WWr/a3RvRd7VI6aWWt0XsyPkdTsGu6pCtfYel5gJyuSgaC9Qgaktzi47HWuznx0NqVj6 nNkg== X-Gm-Message-State: AOJu0YxRBSQwOCSs1WW/d15fVK5UCVHbZCwAUhyfao3o1okwUx+0BfIj GMpisvmsR5+Z7zQPoyeA/kkFnIxR6Hr8nk9cDgYYNw== X-Google-Smtp-Source: AGHT+IGGRT44QzMQkKDgxTTMoTsZl1/DuGe3dB1XRVSwjsTD/kdzfL8HlQQ3QTzT8LNxg1J4Lakm3g== X-Received: by 2002:a5d:47a2:0:b0:32d:a476:527a with SMTP id 2-20020a5d47a2000000b0032da476527amr17170557wrb.50.1700219559591; Fri, 17 Nov 2023 03:12:39 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id z5-20020adff745000000b0032d829e10c0sm1982440wrp.28.2023.11.17.03.12.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 03:12:39 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 4/7] recipetool: create: add new optional process_url callback for plugins Date: Fri, 17 Nov 2023 12:12:15 +0100 Message-ID: <20231117111218.3344066-5-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231117111218.3344066-1-jstephan@baylibre.com> References: <20231117111218.3344066-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 17 Nov 2023 11:12:43 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190840 Add a new process_url callback that plugins can optionally implement if they which to handle url. Plugins can implement this callback for example, to: * transform the url * add special variables using extravalues * add extra classes * ... If a plugin handles the url, it must append 'url' to the handled list and must return the fetchuri No functional changes expected for plugins non implementing this optional callback Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create.py | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index f5d541eb6c1..5c5ac7ae403 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -423,6 +423,36 @@ def create_recipe(args): storeTagName = '' pv_srcpv = False + handled = [] + classes = [] + + # Find all plugins that want to register handlers + logger.debug('Loading recipe handlers') + raw_handlers = [] + for plugin in plugins: + if hasattr(plugin, 'register_recipe_handlers'): + plugin.register_recipe_handlers(raw_handlers) + # Sort handlers by priority + handlers = [] + for i, handler in enumerate(raw_handlers): + if isinstance(handler, tuple): + handlers.append((handler[0], handler[1], i)) + else: + handlers.append((handler, 0, i)) + handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True) + for handler, priority, _ in handlers: + logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority)) + setattr(handler, '_devtool', args.devtool) + handlers = [item[0] for item in handlers] + + fetchuri = None + for handler in handlers: + if hasattr(handler, 'process_url'): + ret = handler.process_url(args, classes, handled, extravalues) + if 'url' in handled and ret: + fetchuri = ret + break + if os.path.isfile(source): source = 'file://%s' % os.path.abspath(source) @@ -431,7 +461,8 @@ def create_recipe(args): if re.match(r'https?://github.com/[^/]+/[^/]+/archive/.+(\.tar\..*|\.zip)$', source): logger.warning('github archive files are not guaranteed to be stable and may be re-generated over time. If the latter occurs, the checksums will likely change and the recipe will fail at do_fetch. It is recommended that you point to an actual commit or tag in the repository instead (using the repository URL in conjunction with the -S/--srcrev option).') # Fetch a URL - fetchuri = reformat_git_uri(urldefrag(source)[0]) + if not fetchuri: + fetchuri = reformat_git_uri(urldefrag(source)[0]) if args.binary: # Assume the archive contains the directory structure verbatim # so we need to extract to a subdirectory @@ -638,8 +669,6 @@ def create_recipe(args): # We'll come back and replace this later in handle_license_vars() lines_before.append('##LICENSE_PLACEHOLDER##') - handled = [] - classes = [] # FIXME This is kind of a hack, we probably ought to be using bitbake to do this pn = None @@ -718,25 +747,6 @@ def create_recipe(args): if args.npm_dev: extravalues['NPM_INSTALL_DEV'] = 1 - # Find all plugins that want to register handlers - logger.debug('Loading recipe handlers') - raw_handlers = [] - for plugin in plugins: - if hasattr(plugin, 'register_recipe_handlers'): - plugin.register_recipe_handlers(raw_handlers) - # Sort handlers by priority - handlers = [] - for i, handler in enumerate(raw_handlers): - if isinstance(handler, tuple): - handlers.append((handler[0], handler[1], i)) - else: - handlers.append((handler, 0, i)) - handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True) - for handler, priority, _ in handlers: - logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority)) - setattr(handler, '_devtool', args.devtool) - handlers = [item[0] for item in handlers] - # Apply the handlers if args.binary: classes.append('bin_package') From patchwork Fri Nov 17 11:12:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 34815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8C74C5AE5B for ; Fri, 17 Nov 2023 11:12:43 +0000 (UTC) Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by mx.groups.io with SMTP id smtpd.web10.9060.1700219563296514452 for ; Fri, 17 Nov 2023 03:12:43 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=YHCOqn2/; spf=pass (domain: baylibre.com, ip: 209.85.221.41, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-3316d3d11e1so136302f8f.0 for ; Fri, 17 Nov 2023 03:12:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1700219561; x=1700824361; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ra4gWIdDWe4zN8Heer+jb5dwJ+yzP2r+TOZ9rqTKa70=; b=YHCOqn2/Xokev1GOBSzUisldLg0cqQVXV2xSyNVuidaVxovgk6cG5viYMGN1SiUT4F b4Ri4cpPQ2Nc3g7gv8dKPaj51gwUZ55t467LEzmHmv7Nm0HcLrRFC8BJe8Dnq4ZhyA2b d+JC38QP20Q9SVvSMbjqGKn9V9qM5ipogEtOASsqxvlWPD8eNGB2FYCiooCX6W9EFCE2 fg+oQUid3t0EeNDNjBCoQypU9SQJMAM/PcNV/ERXslGue/gsC4BFqjGh22lARImWSBB0 89NwBAWUGCILSdXu0jQ9IoS07AYp6fjuiYMKd/ltnnfWXfQbTy5e2B5AykqTGj+ZyL6G CgPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700219561; x=1700824361; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ra4gWIdDWe4zN8Heer+jb5dwJ+yzP2r+TOZ9rqTKa70=; b=FOBN+IWTWE/J0noa1PJOWU1l5QYpLajSNVHT2T1xmlN4Hyuxhl2v5xjUzQuaknFVZt 31D6IkCOUyq/wqSoSJru8PgJLBdgVEXJQwKIVyZCA1EaqxKE6Qf06BazxLBWxWyc/WHB dBzhBTys2ZJmFs88QDcwr2Kp744HXwKIqj/AUAt+ToNHno9oneQSNsmbD9QtCDqKBfbk yvvTMKTerKh+LpP3QaZNqEmIqQYcbrQVF8A4p59UOS76xwBLdBT4cBaRlUCNEmwpYB0j aFb1NzcKLpWTUe32knselrEB9ZxRQmvVvKf4V1gieFqeYWkWy98K1qGBo+C19IQjDtIG 14Qg== X-Gm-Message-State: AOJu0Yxxp2Qgl7VtER8iF+bOyPbOUFFS6ywTLAz9G4aHOSIdLVP80dxb iZYYAvocUhajhrS+G1GJYyniLjnmC+OBfx1nJZKvvA== X-Google-Smtp-Source: AGHT+IHHmxD5Cc4d1n+BlpRV5HVzvdkDgYcL3/CFKln6siiEcPKgLmiYDtqhpaN9XILFj7Eyu8Usow== X-Received: by 2002:adf:d1e3:0:b0:32f:89ce:f66b with SMTP id g3-20020adfd1e3000000b0032f89cef66bmr4378532wrd.10.1700219561095; Fri, 17 Nov 2023 03:12:41 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id z5-20020adff745000000b0032d829e10c0sm1982440wrp.28.2023.11.17.03.12.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 03:12:39 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 5/7] recipetool: create_buildsys_python: add pypi support Date: Fri, 17 Nov 2023 12:12:16 +0100 Message-ID: <20231117111218.3344066-6-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231117111218.3344066-1-jstephan@baylibre.com> References: <20231117111218.3344066-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 17 Nov 2023 11:12:43 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190841 Today, we can use devtool/recipetool to create recipes for python projects using the github url or the direct realse tarball of the project, but the create_buildsys_python plugin doesn't support the pypi class, since we cannot know from the extracted source if the package is available on pypi or not. By implementing the new optional process_url callback, we can detect that the url is a pypi one (i.e 'https://pypi.org/project/') and retrieve the release tarball location. Also detect if the url points to a release tarball hosted on "files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...) In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx' variables from the created recipe as they will be handled by the pypi class and add the PYPI_PACKAGE variable This helps to produce cleaner recipes when package is hosted on pypi. If the url points to a github url or a release tarball not coming from "files.pythonhosted.org" website, the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to devtool/recipetool to NOT inherit from pypi class on matching url, to keep legacy behaviour. To create a recipe for a pypi package, one can now use one of the new following syntax (using recipetool create / devtool add): * recipetool create https://pypi.org/project/ * recipetool create https://pypi.org/project// * recipetool create https://pypi.org/project/ --version or the old syntax: * recipetool create https://files.pythonhosted.org/packages/<...> Signed-off-by: Julien Stephan --- scripts/lib/devtool/standard.py | 3 + scripts/lib/recipetool/create.py | 1 + .../lib/recipetool/create_buildsys_python.py | 72 +++++++++++++++++++ 3 files changed, 76 insertions(+) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index d53fb810071..f18ebaa70d3 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -147,6 +147,8 @@ def add(args, config, basepath, workspace): extracmdopts += ' -a' if args.npm_dev: extracmdopts += ' --npm-dev' + if args.no_pypi: + extracmdopts += ' --no-pypi' if args.mirrors: extracmdopts += ' --mirrors' if args.srcrev: @@ -2260,6 +2262,7 @@ def register_commands(subparsers, context): group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true") parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI') parser_add.add_argument('--npm-dev', help='For npm, also fetch devDependencies', action="store_true") + parser_add.add_argument('--no-pypi', help='Do not inherit pypi class', action="store_true") parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)') parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true") group = parser_add.add_mutually_exclusive_group() diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 5c5ac7ae403..963aa91421e 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -1413,6 +1413,7 @@ def register_commands(subparsers): parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)') parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)') parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies') + parser_create.add_argument('--no-pypi', action="store_true", help='Do not inherit pypi class') parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS) parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).') parser_create.set_defaults(func=create_recipe) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index b620e3271b1..5e07222ece1 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -18,7 +18,11 @@ import os import re import sys import subprocess +import json +import urllib.request from recipetool.create import RecipeHandler +from urllib.parse import urldefrag +from recipetool.create import determine_from_url logger = logging.getLogger('recipetool') @@ -111,6 +115,74 @@ class PythonRecipeHandler(RecipeHandler): def __init__(self): pass + def process_url(self, args, classes, handled, extravalues): + """ + Convert any pypi url https://pypi.org/project// into https://files.pythonhosted.org/packages/source/... + which corresponds to the archive location, and add pypi class + """ + + if 'url' in handled: + return None + + fetch_uri = None + source = args.source + required_version = args.version if args.version else None + match = re.match(r'https?://pypi.org/project/([^/]+)(?:/([^/]+))?/?$', urldefrag(source)[0]) + if match: + package = match.group(1) + version = match.group(2) if match.group(2) else required_version + + json_url = f"https://pypi.org/pypi/%s/json" % package + response = urllib.request.urlopen(json_url) + if response.status == 200: + data = json.loads(response.read()) + if not version: + # grab latest version + version = data["info"]["version"] + pypi_package = data["info"]["name"] + for release in reversed(data["releases"][version]): + if release["packagetype"] == "sdist": + fetch_uri = release["url"] + break + else: + logger.warning("Cannot handle pypi url %s: cannot fetch package information using %s", source, json_url) + return None + else: + match = re.match(r'^https?://files.pythonhosted.org/packages.*/(.*)-.*$', source) + if match: + fetch_uri = source + pypi_package = match.group(1) + _, version = determine_from_url(fetch_uri) + + if match and not args.no_pypi: + if required_version and version != required_version: + raise Exception("Version specified using --version/-V (%s) and version specified in the url (%s) do not match" % (required_version, version)) + # This is optionnal if BPN looks like "python-" or "python3-" (see pypi.bbclass) + # but at this point we cannot know because because user can specify the output name of the recipe on the command line + extravalues["PYPI_PACKAGE"] = pypi_package + # If the tarball extension is not 'tar.gz' (default value in pypi.bblcass) whe should set PYPI_PACKAGE_EXT in the recipe + pypi_package_ext = re.match(r'.*%s-%s\.(.*)$' % (pypi_package, version), fetch_uri) + if pypi_package_ext: + pypi_package_ext = pypi_package_ext.group(1) + if pypi_package_ext != "tar.gz": + extravalues["PYPI_PACKAGE_EXT"] = pypi_package_ext + + # Pypi class will handle S and SRC_URIxxx variables, so remove them + # TODO: allow oe.recipeutils.patch_recipe_lines() to accept regexp so we can simplify the following to: + # extravalues['SRC_URI(?:\[.*?\])?'] = None + extravalues['S'] = None + extravalues['SRC_URI'] = None + extravalues['SRC_URI[md5sum]'] = None + extravalues['SRC_URI[sha1sum]'] = None + extravalues['SRC_URI[sha256sum]'] = None + extravalues['SRC_URI[sha384sum]'] = None + extravalues['SRC_URI[sha512sum]'] = None + + classes.append('pypi') + + handled.append('url') + return fetch_uri + def handle_classifier_license(self, classifiers, existing_licenses=""): licenses = [] From patchwork Fri Nov 17 11:12:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 34816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB00EC072A2 for ; Fri, 17 Nov 2023 11:12:53 +0000 (UTC) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by mx.groups.io with SMTP id smtpd.web11.9415.1700219563959808441 for ; Fri, 17 Nov 2023 03:12:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=vf+7wHJj; spf=pass (domain: baylibre.com, ip: 209.85.221.52, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-32f9268bf8cso1323506f8f.0 for ; Fri, 17 Nov 2023 03:12:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1700219562; x=1700824362; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ALFBlN3RXfM8G++9dipMmPS4BJazwTbEs56PXQLqvUY=; b=vf+7wHJjB8w7tN7sRt6Zp1Qy5lGNXBwR4Ob099WL/lbdjaZfcrH0rI42KSvtF3trUy FUl7jXHruKzUos1kAcJ2YZhKBfdj4MztIbW84uj0Merp0BqceYueiWzql37bvTjChZW2 /GM6Vcwt9eYPtbaJJJvYgI6bUy1NkqwrYimFhj+s+bbpB0bOYOJbQUQtMY0XnL9ifdRt QiY2NhsQJI1qQCrGEBIstu5dFywbUqnm/RSXymcV/L/EizPtWpspwidCiHbtoNk6n6DX VEbarQjPrZOJr4rrsxF7J/wRRjIbYibA+/oCuoWDE8/CphJ9rVZuVHLt8rt+pOqr9onI o8Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700219562; x=1700824362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ALFBlN3RXfM8G++9dipMmPS4BJazwTbEs56PXQLqvUY=; b=IsWkDHMlRBvqRRlZUw5pfz1463dgqO2r+N7Wf2SpTzefbwra77YIDROlPoyjiMCGKU kJPekTAy8iFIH6TD9PO1qG6vqh02FGjOdk9tQLZUPhfEl0nL1GQDham9K0yfvWgUEXz8 5c8quXUsKIr3NG64hyJsgEvT8sW1zKIojeUJcA82j0dkMAw9sxRdLmeMKpAOGzDfhInp KEmrub4icxJ1tzDUNDFxYd28xEQIvu4uQqhKoQivvo2YaiQDF+nHl2pWzWGotFEcveAn 5QBHKpcXf1ohsuZYvYdsAYObN6YpC80NhhRIvEBA6GiIIq4Ziicbo3CCkQn4o/ru1I5z ZGfQ== X-Gm-Message-State: AOJu0YwzE/LgQasgKamdNrL21tRwYPKjC45h8ovlUKgy8UwHfhFRX6dm jZ96qQdu23sd5gCjthI+gkAYj0iZGV7/R8ovHqBDCw== X-Google-Smtp-Source: AGHT+IEpG79JYfs8FZpy7VFy+eHos6ksLfVEgn7tJxjwZAtIzInIX5F6h7sExWW8TtICEQLjsxmldQ== X-Received: by 2002:a5d:47ca:0:b0:32d:82d8:3442 with SMTP id o10-20020a5d47ca000000b0032d82d83442mr19285529wrc.53.1700219562306; Fri, 17 Nov 2023 03:12:42 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id z5-20020adff745000000b0032d829e10c0sm1982440wrp.28.2023.11.17.03.12.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 03:12:41 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 6/7] oeqa/selftest/recipetool: remove spaces on empty lines Date: Fri, 17 Nov 2023 12:12:17 +0100 Message-ID: <20231117111218.3344066-7-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231117111218.3344066-1-jstephan@baylibre.com> References: <20231117111218.3344066-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 17 Nov 2023 11:12:53 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190842 Signed-off-by: Julien Stephan --- meta/lib/oeqa/selftest/cases/recipetool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 55cbba9ca74..4bc28a4f2ee 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -853,7 +853,7 @@ class RecipetoolTests(RecipetoolBase): self._test_recipe_contents(deps_require_file, checkvars, []) - + def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths): dstdir = basedstdir self.assertTrue(os.path.exists(dstdir)) From patchwork Fri Nov 17 11:12:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 34817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7B08C5ACB3 for ; Fri, 17 Nov 2023 11:12:53 +0000 (UTC) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mx.groups.io with SMTP id smtpd.web11.9416.1700219565228652137 for ; Fri, 17 Nov 2023 03:12:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=grpYsNFj; spf=pass (domain: baylibre.com, ip: 209.85.221.44, mailfrom: jstephan@baylibre.com) Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-3316bd84749so206510f8f.2 for ; Fri, 17 Nov 2023 03:12:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1700219563; x=1700824363; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tvn93zXlOozJpVGbSP2G/Dvv7AsjgFtSstz+fmtTgHE=; b=grpYsNFj6cjCi5mHko86M/BvMwC5YYBGmKxUwNafTMAK5uR/PH9B2z7PSq1h9mHdM7 hwEdc1GTokwG2sklf4ziuDH9UZGdmSCxnhrxyLXlsTzWt385cUpfIQWFV7QUXrAgAykD cvn4zGiN+V7iNaF07nvOCo4AyQHy9WmvK523AC3G5WDPXQHX0VHJuBHhE+8j2v/GAmH6 za0p1/ib9BAryrMRRtuL7e/S0bXjw9KFqG9ARHpDdrL11pk7UXwL8GnJNltL2EKEVJ9J AJr7WXbvSztM4aY94OJ4yN3LFlkhzyfIQzS13il4oAgrnZBqe0KEbRL16xeIv95M9fIF ao1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700219563; x=1700824363; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tvn93zXlOozJpVGbSP2G/Dvv7AsjgFtSstz+fmtTgHE=; b=EACHm38qz6evvZ1q0j5EbG660kkgyxTxRcU29cbvSwsZPRFQMsjlB2hDsFO5zJzeb0 wuP7UgpfoXG/ZIfclpsdFbJkviKdhwYPCtMtCOv+C1WibGI0bZvlR4ssuFrz4A5YVpHH VxkaAGACjQV4M1lwBzB9g8NQYM9+UguuC847kAi/2eFWd6uWQ7o6SU/AF5SRuE+Yyvl6 a02VPAzxGZ2LT/u/oxJW79PQOtEdqfsoipatOW1wS6pZkJwwss9faqphU6yahAWNHdI9 e3eHDYlVMrEwhv9kAzZGmO6TWIEw4AK+4d6WSRGnQIn9oq/oLNP7j3YcLfBSOLPvAoen a9qA== X-Gm-Message-State: AOJu0YyY1TfXmjw7Cdgda7oE8F9YZ50rSndSOCM+4lqfoP9h1M+YnSfk brG/lUdumZO0u8EVrY8UPMi6kVILDW/aKFbyYNT8OQ== X-Google-Smtp-Source: AGHT+IElFDdX8tt/S5nP4FwXe02nvTzOwYoT1pujSh9gjEnq6zsWedWtaTyd2pGqhs1YbqywXOxUXA== X-Received: by 2002:a05:6000:178c:b0:32f:9ac4:6f29 with SMTP id e12-20020a056000178c00b0032f9ac46f29mr12268660wrg.70.1700219562986; Fri, 17 Nov 2023 03:12:42 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id z5-20020adff745000000b0032d829e10c0sm1982440wrp.28.2023.11.17.03.12.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Nov 2023 03:12:42 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [PATCH 7/7] oeqa/selftest/recipetool/devtool: add test for pypi class Date: Fri, 17 Nov 2023 12:12:18 +0100 Message-ID: <20231117111218.3344066-8-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231117111218.3344066-1-jstephan@baylibre.com> References: <20231117111218.3344066-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 17 Nov 2023 11:12:53 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/190843 recipetool now supports the pypi class and python recipes can by created using the new following syntax: * recipetool create https://pypi.org/project/ * recipetool create https://pypi.org/project// * recipetool create https://pypi.org/project/ --version or the old syntax: * recipetool create https://files.pythonhosted.org/packages/<...> So add tests for the new syntax and modify old tests Signed-off-by: Julien Stephan --- meta/lib/oeqa/selftest/cases/devtool.py | 4 +- meta/lib/oeqa/selftest/cases/recipetool.py | 112 +++++++++++++++------ 2 files changed, 85 insertions(+), 31 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index b5c488be8e8..3d9a4901ea5 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -543,7 +543,7 @@ class DevtoolAddTests(DevtoolBase): self.track_for_cleanup(self.workspacedir) self.add_command_to_tearDown('bitbake -c cleansstate %s' % testrecipe) self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') - result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url)) + result = runCmd('devtool add --no-pypi %s %s -f %s' % (testrecipe, srcdir, url)) self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created. %s' % result.output) self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory') self.assertTrue(os.path.isdir(os.path.join(srcdir, '.git')), 'git repository for external source tree was not created') @@ -562,7 +562,7 @@ class DevtoolAddTests(DevtoolBase): result = runCmd('devtool reset -n %s' % testrecipe) shutil.rmtree(srcdir) fakever = '1.9' - result = runCmd('devtool add %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever)) + result = runCmd('devtool add --no-pypi %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever)) self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory') # Test devtool status result = runCmd('devtool status') diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 4bc28a4f2ee..b59e53f5994 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -457,13 +457,15 @@ class RecipetoolCreateTests(RecipetoolBase): def test_recipetool_create_python3_setuptools(self): # Test creating python3 package from tarball (using setuptools3 class) + # Use the --no-pypi switch to avoid creating a pypi enabled recipe and + # and check the created recipe as if it was a more general tarball temprecipe = os.path.join(self.tempdir, 'recipe') os.makedirs(temprecipe) pn = 'python-magic' pv = '0.4.15' recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv)) srcuri = 'https://files.pythonhosted.org/packages/84/30/80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/python-magic-%s.tar.gz' % pv - result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + result = runCmd('recipetool create --no-pypi -o %s %s' % (temprecipe, srcuri)) self.assertTrue(os.path.isfile(recipefile)) checkvars = {} checkvars['LICENSE'] = set(['MIT']) @@ -474,6 +476,82 @@ class RecipetoolCreateTests(RecipetoolBase): inherits = ['setuptools3'] self._test_recipe_contents(recipefile, checkvars, inherits) + def test_recipetool_create_python3_setuptools_pypi_tarball(self): + # Test creating python3 package from tarball (using setuptools3 and pypi classes) + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + pn = 'python-magic' + pv = '0.4.15' + recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv)) + srcuri = 'https://files.pythonhosted.org/packages/84/30/80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/python-magic-%s.tar.gz' % pv + result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88' + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', 'pypi'] + self._test_recipe_contents(recipefile, checkvars, inherits) + + def test_recipetool_create_python3_setuptools_pypi(self): + # Test creating python3 package from pypi url (using setuptools3 and pypi classes) + # Intentionnaly using setuptools3 class here instead of any of the pep517 class + # to avoid the toml dependency and allows this test to run on host autobuilders + # with older version of python + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + pn = 'python-magic' + pv = '0.4.15' + recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv)) + # First specify the required version in the url + srcuri = 'https://pypi.org/project/%s/%s' % (pn, pv) + runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88' + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', "pypi"] + self._test_recipe_contents(recipefile, checkvars, inherits) + + # Now specify the version as a recipetool parameter + runCmd('rm -rf %s' % recipefile) + self.assertFalse(os.path.isfile(recipefile)) + srcuri = 'https://pypi.org/project/%s' % pn + runCmd('recipetool create -o %s %s --version %s' % (temprecipe, srcuri, pv)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88' + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', "pypi"] + self._test_recipe_contents(recipefile, checkvars, inherits) + + # Now, try to grab latest version of the package, so we cannot guess the name of the recipe, + # unless hardcoding the latest version but it means we will need to update the test for each release, + # so use a regexp + runCmd('rm -rf %s' % recipefile) + self.assertFalse(os.path.isfile(recipefile)) + recipefile_re = r'%s_(.*)\.bb' % pn + result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + dirlist = os.listdir(temprecipe) + if len(dirlist) > 1: + self.fail('recipetool created more than just one file; output:\n%s\ndirlist:\n%s' % (result.output, str(dirlist))) + if len(dirlist) < 1 or not os.path.isfile(os.path.join(temprecipe, dirlist[0])): + self.fail('recipetool did not create recipe file; output:\n%s\ndirlist:\n%s' % (result.output, str(dirlist))) + import re + match = re.match(recipefile_re, dirlist[0]) + self.assertTrue(match) + latest_pv = match.group(1) + self.assertTrue(latest_pv != pv) + recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, latest_pv)) + # Do not check LIC_FILES_CHKSUM here to avoid having updating the test on each release + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', "pypi"] + self._test_recipe_contents(recipefile, checkvars, inherits) + def test_recipetool_create_python3_pep517_setuptools_build_meta(self): # This test require python 3.11 or above for the tomllib module # or tomli module to be installed @@ -498,13 +576,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['SUMMARY'] = 'A library for working with the color formats defined by HTML and CSS.' checkvars['LICENSE'] = set(['BSD-3-Clause']) checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=702b1ef12cf66832a88f24c8f2ee9c19' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/a1/fb/f95560c6a5d4469d9c49e24cf1b5d4d21ffab5608251c6020a965fb7791c/webcolors-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = 'c9be30c5b0cf1cad32e4cbacbb2229e9' - checkvars['SRC_URI[sha1sum]'] = 'c90b84fb65eed9b4c9dea7f08c657bfac0e820a5' - checkvars['SRC_URI[sha256sum]'] = 'c225b674c83fa923be93d235330ce0300373d02885cef23238813b0d5668304a' - checkvars['SRC_URI[sha384sum]'] = '45652af349660f19f68d01361dd5bda287789e5ea63608f52a8cea526ac04465614db2ea236103fb8456b1fcaea96ed7' - checkvars['SRC_URI[sha512sum]'] = '074aaf135ac6b0025b88b731d1d6dfa4c539b4fff7195658cc58a4326bb9f0449a231685d312b4a1ec48ca535a838bfa5c680787fe0e61473a2a092c448937d0' - inherits = ['python_setuptools_build_meta'] + inherits = ['python_setuptools_build_meta', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits) @@ -532,13 +604,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['SUMMARY'] = 'Simple module to parse ISO 8601 dates' checkvars['LICENSE'] = set(['MIT']) checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=aab31f2ef7ba214a5a341eaa47a7f367' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/b9/f3/ef59cee614d5e0accf6fd0cbba025b93b272e626ca89fb70a3e9187c5d15/iso8601-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = '6e33910eba87066b3be7fcf3d59d16b5' - checkvars['SRC_URI[sha1sum]'] = 'efd225b2c9fa7d9e4a1ec6ad94f3295cee982e61' - checkvars['SRC_URI[sha256sum]'] = '6b1d3829ee8921c4301998c909f7829fa9ed3cbdac0d3b16af2d743aed1ba8df' - checkvars['SRC_URI[sha384sum]'] = '255002433fe65c19adfd6b91494271b613cb25ef6a35ac77436de1e03d60cc07bf89fd716451b917f1435e4384860ef6' - checkvars['SRC_URI[sha512sum]'] = 'db57ab2a25ef91e3bc479c8539d27e853cf1fbf60986820b8999ae15d7e566425a1e0cfba47d0f3b23aa703db0576db368e6c110ba2a2f46c9a34e8ee3611fb7' - inherits = ['python_poetry_core'] + inherits = ['python_poetry_core', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits) @@ -566,13 +632,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['SUMMARY'] = 'Backported and Experimental Type Hints for Python 3.8+' checkvars['LICENSE'] = set(['PSF-2.0']) checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=fcf6b249c2641540219a727f35d8d2c2' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/1f/7a/8b94bb016069caa12fc9f587b28080ac33b4fbb8ca369b98bc0a4828543e/typing_extensions-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = '74bafe841fbd1c27324afdeb099babdf' - checkvars['SRC_URI[sha1sum]'] = 'f8bed69cbad4a57a1a67bf8a31b62b657b47f7a3' - checkvars['SRC_URI[sha256sum]'] = 'df8e4339e9cb77357558cbdbceca33c303714cf861d1eef15e1070055ae8b7ef' - checkvars['SRC_URI[sha384sum]'] = '0bd0112234134d965c6884f3c1f95d27b6ae49cfb08108101158e31dff33c2dce729331628b69818850f1acb68f6c8d0' - checkvars['SRC_URI[sha512sum]'] = '5fbff10e085fbf3ac2e35d08d913608d8c8bca66903435ede91cdc7776d775689a53d64f5f0615fe687c6c228ac854c8651d99eb1cb96ec61c56b7ca01fdd440' - inherits = ['python_flit_core'] + inherits = ['python_flit_core', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits) @@ -601,13 +661,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['HOMEPAGE'] = 'https://github.com/python-jsonschema/jsonschema' checkvars['LICENSE'] = set(['MIT']) checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=7a60a81c146ec25599a3e1dabb8610a8 file://json/LICENSE;md5=9d4de43111d33570c8fe49b4cb0e01af' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/e4/43/087b24516db11722c8687e0caf0f66c7785c0b1c51b0ab951dfde924e3f5/jsonschema-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = '4d6667ce76f820c35082c2d60a4896ab' - checkvars['SRC_URI[sha1sum]'] = '9173714cb88964d07f3a3f4fcaaef638b8ceac0c' - checkvars['SRC_URI[sha256sum]'] = 'ec84cc37cfa703ef7cd4928db24f9cb31428a5d0fa77747b8b51a847458e0bbf' - checkvars['SRC_URI[sha384sum]'] = '7a53181f0e679aa3dc3eb4d05a420877b7b9bff2d02e81f5c289a37ed1127d6c0cca1f5a5f9e4e166f089ab36bcc2be9' - checkvars['SRC_URI[sha512sum]'] = '60fa769faf6e3fc2c14eb9acd189c86e9d366b157230a5681d36552af0c159cb1ad33fd920668a36afdab98bc97253f91501704c5c07b5009fdaf9d29b52060d' - inherits = ['python_hatchling'] + inherits = ['python_hatchling', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits)