From patchwork Mon Dec 4 15:59:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 35631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A781EC46CA3 for ; Mon, 4 Dec 2023 15:59:52 +0000 (UTC) Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by mx.groups.io with SMTP id smtpd.web10.73097.1701705583979718006 for ; Mon, 04 Dec 2023 07:59:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=fUUXEtSA; spf=pass (domain: baylibre.com, ip: 209.85.128.43, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-40bd5eaa66eso26736035e9.3 for ; Mon, 04 Dec 2023 07:59:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1701705582; x=1702310382; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ETVTJ5File0JK0UXg/b/p+iziwLzQW1xU1GCRisKjB4=; b=fUUXEtSAv3+uTrf624Qa9jA6qDL+7QomsInMQ3f9BMDnZs9rqo9lhyxxrtP/PVHDUs kyfjpPYEevdHvl6HEnMhqF59VaIR1mdXvs+Snz+N9SMANsrafIlrDcO+g2kWOSiuWITa MRz7X5ioqXnAMlfnu6gZWuDAYybgF0hkI0qgEQEjhC3+V+wCAuZ2ri1VDaQXhPqrEb0/ Q48XlkpjTBZcYPBTJknCl7ONEbybLJ7R6r8ciGGPHxjncNaYwfYhwj5GLVLIsJppsTIV 6aTo7n06Qxmtafnor7AGYlS9Nb4O6Nl6HRIENESpxpUInAHBBRF6AZH1upd+op7l2stm xUnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701705582; x=1702310382; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ETVTJ5File0JK0UXg/b/p+iziwLzQW1xU1GCRisKjB4=; b=g3f0gmKCX/dDhWxIzupO67FnSo8p66Ji3E96vovz6xIVZAEQIA9PLTc2PsnTwAjv7Z GRpJSAgVKciwnnccsmoxX6FdW7VIEW0Owr8RQKUZIokWUmU5gWf4cgOLiFWR54/EDPtq ndIlQ3yzW2sBRnRBPsb+irWrfCTg5r90Hwk1l5vjiS/WFJsBiFqiXm+MWADOT+VcBrhV ObTwdQSBOjSqUFRw5CuM/XkacuO3jlOSOEJV9bI28Qet/MJRY+BQdtrKiKPRHevYm9Ex B/RRwy397ApkvBOfVpIqlfEHSfgcytFYW2zOc/yhvrwg8NQzV+NxkG509ZEoQxjh4Xx4 Yr6g== X-Gm-Message-State: AOJu0YxVy0fVuULq76zchVO/kYfOn7qRQhSSc/jaXIkeONQoKHNHaTOi 0z+GYJPf7MLHHBLENEfcLNwDD0a0ACM6aZkQKE0GEA== X-Google-Smtp-Source: AGHT+IGd0D2pHsjgDPObbXjqvpdh0PwiyRxurclSVd4+/5qpGj8CDD+GGbPfKc+SeKnJLcqeumoiDA== X-Received: by 2002:a05:600c:3545:b0:40b:4520:45af with SMTP id i5-20020a05600c354500b0040b452045afmr2662929wmq.23.1701705581996; Mon, 04 Dec 2023 07:59:41 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id g16-20020a05600c4ed000b0040b47c53610sm15479595wmq.14.2023.12.04.07.59.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 07:59:41 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines Date: Mon, 4 Dec 2023 16:59:28 +0100 Message-ID: <20231204155934.836992-2-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231204155934.836992-1-jstephan@baylibre.com> References: <20231204155934.836992-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Dec 2023 15:59:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/191754 Signed-off-by: Julien Stephan --- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index b401fa5ec7a..61ffad92ce3 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx @@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2] @@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next @@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix): @@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return default From patchwork Mon Dec 4 15:59:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 35629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0C67C10DCE for ; Mon, 4 Dec 2023 15:59:52 +0000 (UTC) Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by mx.groups.io with SMTP id smtpd.web11.72358.1701705584468221458 for ; Mon, 04 Dec 2023 07:59:44 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=0dmpu8pC; spf=pass (domain: baylibre.com, ip: 209.85.128.48, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-40c09dfd82aso17796105e9.0 for ; Mon, 04 Dec 2023 07:59:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1701705583; x=1702310383; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H0y3nqiWo+iM/VgSe9noW2bvm/Ew1cYjCiWg4MOuKPs=; b=0dmpu8pCokz3lngVxQeV62Bqa55SZibwtnNU08pVIjnUdNg4m12C9IOA61M2QqEWBM GV+2sJ8dVtvtxWfLrgsakJyeOfZqgME5pdPL1k7jWUtU4Wti6OYoxa9RwDxJCbfDIO6t utVRuPD2showUuET1q6Xu9Er8VpZ+aT3Xu68kAhjBhRfIutAUqRHFJpr/y3+NMbGrI/T a0nQoP29anbcccys13TMZsII5AdBE5DmV0GH+OY1CcIUXXYzER/26VHIrfMP0NeuyHGq uQAPIdmLZDSC7XZwl2bqFhHKf4ctcQBVQ3Ti4zRuKZr5/qPRQuyDiMSTGFuu+kWZhNeq ivAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701705583; x=1702310383; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H0y3nqiWo+iM/VgSe9noW2bvm/Ew1cYjCiWg4MOuKPs=; b=Tx/9BNB9RO3mkWBwQrVCgJSAp+TVkg7HMgeZpaO1hV0VRv/9PO2UE+twkJpT3zJWy9 flhy/FwTh35Hci06cAbBvo09cXiYFgkDuk6nWnVrMcfDLQ+1Q5K/QDjyJz1BLUyFV+m4 Nn+ncL+N7P1rvEXu+VGANADsOVl/78skutQkOU0illdiSoj5H3ykeaqrYbBEw8+JXfJF KEdpLQbfNY1gIHFjB/spMJmHWyM6M0wFjnL24bcB5zVvN3yTd8XDNMN258c6XKC0Z+IV ZfPrsaNAGu+s6M5v3hftSYBeEiP/we3jcEkVdMGJP352xXUiEpjn4SovEXNO0++t0gJW T2Ig== X-Gm-Message-State: AOJu0Yyt0pGDI8+GmlCipb9bmtQvNiVd3YehP6liVq8zwH3MtGRHawCT puCriW7+GrrCmPoZ+87Q0hAsBlop30B47MUNEfH7Ew== X-Google-Smtp-Source: AGHT+IGZBEMg+cY0/O1qa/AfgHHdovOAbOUpnhtv6YRrB9/ccVpqX3x1ceDPkuLGsRuXv9vS8o0Q1A== X-Received: by 2002:a1c:7c0a:0:b0:40b:5e21:dd4c with SMTP id x10-20020a1c7c0a000000b0040b5e21dd4cmr3200346wmc.122.1701705582880; Mon, 04 Dec 2023 07:59:42 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id g16-20020a05600c4ed000b0040b47c53610sm15479595wmq.14.2023.12.04.07.59.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 07:59:42 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [RESEND][PATCH 2/7] recipetool: create_buildsys_python.py: initialize metadata Date: Mon, 4 Dec 2023 16:59:29 +0100 Message-ID: <20231204155934.836992-3-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231204155934.836992-1-jstephan@baylibre.com> References: <20231204155934.836992-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Dec 2023 15:59:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/191756 In the case pyproject.toml doesn't contains metadatas, the metadata variable is not initialized and the plugin throws an error and falls back to another plugin, which is not the desired behaviour. So just ignore metadata if we don't have them Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create_buildsys_python.py | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index 9312e4abf13..b620e3271b1 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -726,6 +726,7 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler): def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): info = {} + metadata = {} if 'buildsystem' in handled: return False From patchwork Mon Dec 4 15:59:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 35633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0DF5C10F07 for ; Mon, 4 Dec 2023 15:59:52 +0000 (UTC) Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by mx.groups.io with SMTP id smtpd.web10.73100.1701705585550210147 for ; Mon, 04 Dec 2023 07:59:45 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=qFxLOUTd; spf=pass (domain: baylibre.com, ip: 209.85.128.46, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-40838915cecso47118765e9.2 for ; Mon, 04 Dec 2023 07:59:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1701705583; x=1702310383; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/f9cm/lL6ltdCUfGCbC8zquWoEImJOxxnBFdz4FribU=; b=qFxLOUTdse4t+ThtTg1scap5d3LuiEW27QNhezzJRdxfmcwGQIcN6bSyLT4ijfSpMy mjWQktV+cY1Ee0HgwYgJa7M0MpqvzBf9MZ6GvrqQ4qKHc/6hGXH+RMr6eN9EBvbtBVWi Nq0ght4m4E8lz9JW1/uyncyO3ucqywWPd4x/DJKTca6V9f571IJElfuevNvCod1KCMO0 Wc84biK91epAN/ipwRG+91iz2KyEz815PSBrrlufk4vGRmKGVIcDZDu4TisVIRNvkMrj wmpvCnT+hQa75ri15E8ZaKul6wCvYqLmgH/lptBXH7h5ALO8+z/AneJ9xCUFNzAREnr9 Txlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701705583; x=1702310383; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/f9cm/lL6ltdCUfGCbC8zquWoEImJOxxnBFdz4FribU=; b=Wl6us9PX6jxOviioAJ0UClvt7SpLr4juwtpXDZA20W4O2nwPqG01h1F1ER2SWTJgz6 ZNmrK87uMmrhCO3TpyR/TlIIJMQwkGWonmtYu0SthZ5bv1bK9kDDw/I/lQ8Aalv5+Qn9 Fiz8kquCpPLTtp54954ROukCNMWph1Pj9/A2H/Qlz2oAvT2HNGZtfaO1tq6YkxuqEtPX s8obSgqzpaD0dsvtVNdR1tq6SJxpwOF7nnYG5EbfGqPGQopQMdA0dBkYbCz+AcYwxQqQ VUEnCfdPISMVPTAb8+oOwolvL2701aLLyLB4otXrjAdz0QZbC9eqTJf6FolWcOyqdBQ7 AKxw== X-Gm-Message-State: AOJu0YwyQJKk0VEQPh6LNhcM2o9EJcV8/BzutqehiV2/zAqdvEARpyCS nsojzEvgMIwzwJ6DKQpoOIa87HWdHV2nJA2wtIjqpg== X-Google-Smtp-Source: AGHT+IF4qI41TphjhYzqszwpkZd4gpH5hUa5k9KezyCfZjI8ITnQKVnI/IeTaM4vz5ObKsUR/RWGHQ== X-Received: by 2002:a7b:cd0e:0:b0:40c:ec3:fbd0 with SMTP id f14-20020a7bcd0e000000b0040c0ec3fbd0mr405050wmj.85.1701705583436; Mon, 04 Dec 2023 07:59:43 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id g16-20020a05600c4ed000b0040b47c53610sm15479595wmq.14.2023.12.04.07.59.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 07:59:43 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [RESEND][PATCH 3/7] recipetool: create: add trailing newlines Date: Mon, 4 Dec 2023 16:59:30 +0100 Message-ID: <20231204155934.836992-4-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231204155934.836992-1-jstephan@baylibre.com> References: <20231204155934.836992-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Dec 2023 15:59:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/191757 create_recipe() function relies on oe.recipeutils.patch_recipe_lines() which relies on bb.utils.edit_metadata(). edit_metada expect lines to have trailing newlines, so add it to each lines before calling patch_recipe_lines, otherwise edit_metadata will not be able to squash blank line if there are two consecutive blanks after a removal Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 293198d1c88..f5d541eb6c1 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -873,8 +873,10 @@ def create_recipe(args): outlines.append('') outlines.extend(lines_after) + outlines = [ line.rstrip('\n') +"\n" for line in outlines] + if extravalues: - _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=False) + _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=True) if args.extract_to: scriptutils.git_convert_standalone_clone(srctree) @@ -890,7 +892,7 @@ def create_recipe(args): log_info_cond('Source extracted to %s' % args.extract_to, args.devtool) if outfile == '-': - sys.stdout.write('\n'.join(outlines) + '\n') + sys.stdout.write(''.join(outlines) + '\n') else: with open(outfile, 'w') as f: lastline = None @@ -898,7 +900,7 @@ def create_recipe(args): if not lastline and not line: # Skip extra blank lines continue - f.write('%s\n' % line) + f.write('%s' % line) lastline = line log_info_cond('Recipe %s has been created; further editing may be required to make it fully functional' % outfile, args.devtool) tinfoil.modified_files() From patchwork Mon Dec 4 15:59:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 35635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C671AC46CA0 for ; Mon, 4 Dec 2023 15:59:52 +0000 (UTC) Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by mx.groups.io with SMTP id smtpd.web10.73101.1701705586623122380 for ; Mon, 04 Dec 2023 07:59:46 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=JYbrwg0W; spf=pass (domain: baylibre.com, ip: 209.85.128.46, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-40b538d5c4eso41746505e9.1 for ; Mon, 04 Dec 2023 07:59:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1701705584; x=1702310384; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kJhNSOkg8w/cEYhk7pYpbcVolYBKJavvLGDPT7R3tyM=; b=JYbrwg0WKVRh9ftGoanBwPqQor6X83vPhVINa29cFJ9DFlv/HBEcw/9U+zC22qra4N 17EMxFiuwG2sR7kpBSNA4thydGtyIShwbA+9IL8h9JHShx7xoPm/2s8U7WQQYmd4jK5T VYk6jxJrMzl8bL7TjUSZuSf/5raxcox+Sw5Bg5bjDyPaJ8U03HQtz8u8CfaWKwyVRsXy Odm/v0vgp/9vZTqg9Z3nBrcHXBeaKor5iyQWRHZUIL45WeJQEKXjMb6oIqd//nZnlX8/ HnjOWceU9jqCoFGj0ot3qQd4MFImFO6kyCJVpMY7Ia2dPNxzeG4sIo41XjBsaIJiL9/K q1XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701705584; x=1702310384; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kJhNSOkg8w/cEYhk7pYpbcVolYBKJavvLGDPT7R3tyM=; b=u3bVSHh43Tbi87u6FJUP1iPL25xV7f7ZYJMq+KzaGRM7BrBDpySEIpho7sLD3Ojr4O ci48+SFVhfd8Wjo8LsPw1rfZ0X5eBlY6TrVv67N3XY0gfdzM90OucyFgnTMGOeGujFu2 Q/5cRS6itmAMNHsbfNAOMYlroI3oky4ShW7701ugog/W8ii2GEmAyE+nFH4S4XprVt4G U4s3MVW5aIBXePytOqsAyi/4XtVQjwR3fXr9I6U9jCnoegWIk7aAI5sraCygmG2rBwB6 PT6lZuWjLxhXNqLzFeLCMkZp4DqTr/+peeOU5ny0GKUg0/NHImlE3/8+h8jsDy3HYaiP BNqw== X-Gm-Message-State: AOJu0Ywc8bRxHSMMHqQdRZLyr9X34lYQLUL6zaKs6aBCX9U4SbLS1qhA SngBMga5OzDfujodKh2kkjPk+C0nulZzW23AcdHlww== X-Google-Smtp-Source: AGHT+IGPjAjVj5Zq1sRUzbgoj8yi1S434tjVZXe2AOM1piq1aXaRgDdxlyerGC2yXBF+Q6N1RNYJ5Q== X-Received: by 2002:a05:600c:a2a2:b0:40b:5e59:b7d9 with SMTP id hu34-20020a05600ca2a200b0040b5e59b7d9mr2497911wmb.182.1701705584584; Mon, 04 Dec 2023 07:59:44 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id g16-20020a05600c4ed000b0040b47c53610sm15479595wmq.14.2023.12.04.07.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 07:59:43 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [RESEND][PATCH 4/7] recipetool: create: add new optional process_url callback for plugins Date: Mon, 4 Dec 2023 16:59:31 +0100 Message-ID: <20231204155934.836992-5-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231204155934.836992-1-jstephan@baylibre.com> References: <20231204155934.836992-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Dec 2023 15:59:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/191758 Add a new process_url callback that plugins can optionally implement if they which to handle url. Plugins can implement this callback for example, to: * transform the url * add special variables using extravalues * add extra classes * ... If a plugin handles the url, it must append 'url' to the handled list and must return the fetchuri No functional changes expected for plugins non implementing this optional callback Signed-off-by: Julien Stephan --- scripts/lib/recipetool/create.py | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index f5d541eb6c1..5c5ac7ae403 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -423,6 +423,36 @@ def create_recipe(args): storeTagName = '' pv_srcpv = False + handled = [] + classes = [] + + # Find all plugins that want to register handlers + logger.debug('Loading recipe handlers') + raw_handlers = [] + for plugin in plugins: + if hasattr(plugin, 'register_recipe_handlers'): + plugin.register_recipe_handlers(raw_handlers) + # Sort handlers by priority + handlers = [] + for i, handler in enumerate(raw_handlers): + if isinstance(handler, tuple): + handlers.append((handler[0], handler[1], i)) + else: + handlers.append((handler, 0, i)) + handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True) + for handler, priority, _ in handlers: + logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority)) + setattr(handler, '_devtool', args.devtool) + handlers = [item[0] for item in handlers] + + fetchuri = None + for handler in handlers: + if hasattr(handler, 'process_url'): + ret = handler.process_url(args, classes, handled, extravalues) + if 'url' in handled and ret: + fetchuri = ret + break + if os.path.isfile(source): source = 'file://%s' % os.path.abspath(source) @@ -431,7 +461,8 @@ def create_recipe(args): if re.match(r'https?://github.com/[^/]+/[^/]+/archive/.+(\.tar\..*|\.zip)$', source): logger.warning('github archive files are not guaranteed to be stable and may be re-generated over time. If the latter occurs, the checksums will likely change and the recipe will fail at do_fetch. It is recommended that you point to an actual commit or tag in the repository instead (using the repository URL in conjunction with the -S/--srcrev option).') # Fetch a URL - fetchuri = reformat_git_uri(urldefrag(source)[0]) + if not fetchuri: + fetchuri = reformat_git_uri(urldefrag(source)[0]) if args.binary: # Assume the archive contains the directory structure verbatim # so we need to extract to a subdirectory @@ -638,8 +669,6 @@ def create_recipe(args): # We'll come back and replace this later in handle_license_vars() lines_before.append('##LICENSE_PLACEHOLDER##') - handled = [] - classes = [] # FIXME This is kind of a hack, we probably ought to be using bitbake to do this pn = None @@ -718,25 +747,6 @@ def create_recipe(args): if args.npm_dev: extravalues['NPM_INSTALL_DEV'] = 1 - # Find all plugins that want to register handlers - logger.debug('Loading recipe handlers') - raw_handlers = [] - for plugin in plugins: - if hasattr(plugin, 'register_recipe_handlers'): - plugin.register_recipe_handlers(raw_handlers) - # Sort handlers by priority - handlers = [] - for i, handler in enumerate(raw_handlers): - if isinstance(handler, tuple): - handlers.append((handler[0], handler[1], i)) - else: - handlers.append((handler, 0, i)) - handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True) - for handler, priority, _ in handlers: - logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority)) - setattr(handler, '_devtool', args.devtool) - handlers = [item[0] for item in handlers] - # Apply the handlers if args.binary: classes.append('bin_package') From patchwork Mon Dec 4 15:59:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 35634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFB60C10F05 for ; Mon, 4 Dec 2023 15:59:52 +0000 (UTC) Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by mx.groups.io with SMTP id smtpd.web10.73102.1701705587783915818 for ; Mon, 04 Dec 2023 07:59:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=cF8zCIgE; spf=pass (domain: baylibre.com, ip: 209.85.128.42, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-40a4848c6e1so48107815e9.1 for ; Mon, 04 Dec 2023 07:59:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1701705585; x=1702310385; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FBbEqi6s9no++CKjmIJSgIW7ZpeYRQv6BzDLZ4a8D3c=; b=cF8zCIgEOXdfaZAXhYSsA5MydT860WBYtrV1nSawNgLCbH+pqD00MDks2HNpn9mbxK 9swuOD58xfXab7EewolmmEukjrOf9SPhUL3T/nCoSEDHNvAcghBfWOTMKAclBkrG4HeJ ecQywta1+Di7PsxLXuuh54MBmk0VdJmqyhM7uPbeNEaSAUHleKf6ll+Y8U0iI2tVbgX2 PFnMadp1QeelVDkOkJFw75IDmtOCPQ5z94txGrSAMTkXiEw4+SKk41oeBSmK4Z1l0TxV ChIvUGfjtOUGArnZJS5cti/YDEigBmeGuY3zook55BYX11/LjKobajJ+DCtnd2JRqYWh OPkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701705585; x=1702310385; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FBbEqi6s9no++CKjmIJSgIW7ZpeYRQv6BzDLZ4a8D3c=; b=ndb68BfyyfdkrNFo8mFvA4TNgylH8RpKw9vFg/XX0WP8pumkvirKG0RdTBwtU6AQ2r TCemVfEaFGodOGrvKAANOOQYK3U2Fw5KgDnQ0mvDEimTIl/LVsw7rtm4VLCbVtBDtJZc Pn7o3X9ignS+peBaWDb3PpIQvZV1t4RBaQcNgYSzGZFyqdB/rwIE2M3jKEWCNa4pNSsT 0w17JoRLVILkIuO+b4f51JajGswdviK6Fnjq8IhxcDGyXX6pSfCUkuboQVXM4M5Gyykl +RmFqxC/3tKJbM1Xq9P+hTFKPWHQYuAylPPkx8QMLU6HBJM35sJY0S0VEhTmbY48L3v4 2LBg== X-Gm-Message-State: AOJu0YxhmXDsaGf2Z5hQox94SOkfEWzSxYmbH9zdFON9yU+xrLFi8pfF 9JZE/mLFOjDrCkdm9oRo7rD+TtTwmDu4rfTGVHPyzQ== X-Google-Smtp-Source: AGHT+IHxd9IfDIS0zRJIq13/t09INSVU5c+lU+oGoPyogKuBstLpH+C9+BENV5Gf8dkgFuYqbsscTw== X-Received: by 2002:a7b:cb42:0:b0:40b:5e4a:40a2 with SMTP id v2-20020a7bcb42000000b0040b5e4a40a2mr2348191wmj.194.1701705585607; Mon, 04 Dec 2023 07:59:45 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id g16-20020a05600c4ed000b0040b47c53610sm15479595wmq.14.2023.12.04.07.59.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 07:59:44 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [RESEND][PATCH 5/7] recipetool: create_buildsys_python: add pypi support Date: Mon, 4 Dec 2023 16:59:32 +0100 Message-ID: <20231204155934.836992-6-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231204155934.836992-1-jstephan@baylibre.com> References: <20231204155934.836992-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Dec 2023 15:59:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/191759 Today, we can use devtool/recipetool to create recipes for python projects using the github url or the direct release tarball of the project, but the create_buildsys_python plugin doesn't support the pypi class, since we cannot know from the extracted source if the package is available on pypi or not. By implementing the new optional process_url callback, we can detect that the url is a pypi one (i.e 'https://pypi.org/project/') and retrieve the release tarball location. Also detect if the url points to a release tarball hosted on "files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...) In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx' variables from the created recipe as they will be handled by the pypi class and add the PYPI_PACKAGE variable This helps to produce cleaner recipes when package is hosted on pypi. If the url points to a github url or a release tarball not coming from "files.pythonhosted.org", the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to NOT inherit from pypi class on matching url, to keep legacy behaviour. To create a recipe for a pypi package, one can now use one of the new following syntax (using recipetool create / devtool add): * recipetool create https://pypi.org/project/ * recipetool create https://pypi.org/project// * recipetool create https://pypi.org/project/ --version or the old syntax: * recipetool create https://files.pythonhosted.org/packages/<...> Signed-off-by: Julien Stephan --- scripts/lib/devtool/standard.py | 3 + scripts/lib/recipetool/create.py | 1 + .../lib/recipetool/create_buildsys_python.py | 72 +++++++++++++++++++ 3 files changed, 76 insertions(+) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index d53fb810071..f18ebaa70d3 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -147,6 +147,8 @@ def add(args, config, basepath, workspace): extracmdopts += ' -a' if args.npm_dev: extracmdopts += ' --npm-dev' + if args.no_pypi: + extracmdopts += ' --no-pypi' if args.mirrors: extracmdopts += ' --mirrors' if args.srcrev: @@ -2260,6 +2262,7 @@ def register_commands(subparsers, context): group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true") parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI') parser_add.add_argument('--npm-dev', help='For npm, also fetch devDependencies', action="store_true") + parser_add.add_argument('--no-pypi', help='Do not inherit pypi class', action="store_true") parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)') parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true") group = parser_add.add_mutually_exclusive_group() diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py index 5c5ac7ae403..963aa91421e 100644 --- a/scripts/lib/recipetool/create.py +++ b/scripts/lib/recipetool/create.py @@ -1413,6 +1413,7 @@ def register_commands(subparsers): parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)') parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)') parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies') + parser_create.add_argument('--no-pypi', action="store_true", help='Do not inherit pypi class') parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS) parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).') parser_create.set_defaults(func=create_recipe) diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py index b620e3271b1..5e07222ece1 100644 --- a/scripts/lib/recipetool/create_buildsys_python.py +++ b/scripts/lib/recipetool/create_buildsys_python.py @@ -18,7 +18,11 @@ import os import re import sys import subprocess +import json +import urllib.request from recipetool.create import RecipeHandler +from urllib.parse import urldefrag +from recipetool.create import determine_from_url logger = logging.getLogger('recipetool') @@ -111,6 +115,74 @@ class PythonRecipeHandler(RecipeHandler): def __init__(self): pass + def process_url(self, args, classes, handled, extravalues): + """ + Convert any pypi url https://pypi.org/project// into https://files.pythonhosted.org/packages/source/... + which corresponds to the archive location, and add pypi class + """ + + if 'url' in handled: + return None + + fetch_uri = None + source = args.source + required_version = args.version if args.version else None + match = re.match(r'https?://pypi.org/project/([^/]+)(?:/([^/]+))?/?$', urldefrag(source)[0]) + if match: + package = match.group(1) + version = match.group(2) if match.group(2) else required_version + + json_url = f"https://pypi.org/pypi/%s/json" % package + response = urllib.request.urlopen(json_url) + if response.status == 200: + data = json.loads(response.read()) + if not version: + # grab latest version + version = data["info"]["version"] + pypi_package = data["info"]["name"] + for release in reversed(data["releases"][version]): + if release["packagetype"] == "sdist": + fetch_uri = release["url"] + break + else: + logger.warning("Cannot handle pypi url %s: cannot fetch package information using %s", source, json_url) + return None + else: + match = re.match(r'^https?://files.pythonhosted.org/packages.*/(.*)-.*$', source) + if match: + fetch_uri = source + pypi_package = match.group(1) + _, version = determine_from_url(fetch_uri) + + if match and not args.no_pypi: + if required_version and version != required_version: + raise Exception("Version specified using --version/-V (%s) and version specified in the url (%s) do not match" % (required_version, version)) + # This is optionnal if BPN looks like "python-" or "python3-" (see pypi.bbclass) + # but at this point we cannot know because because user can specify the output name of the recipe on the command line + extravalues["PYPI_PACKAGE"] = pypi_package + # If the tarball extension is not 'tar.gz' (default value in pypi.bblcass) whe should set PYPI_PACKAGE_EXT in the recipe + pypi_package_ext = re.match(r'.*%s-%s\.(.*)$' % (pypi_package, version), fetch_uri) + if pypi_package_ext: + pypi_package_ext = pypi_package_ext.group(1) + if pypi_package_ext != "tar.gz": + extravalues["PYPI_PACKAGE_EXT"] = pypi_package_ext + + # Pypi class will handle S and SRC_URIxxx variables, so remove them + # TODO: allow oe.recipeutils.patch_recipe_lines() to accept regexp so we can simplify the following to: + # extravalues['SRC_URI(?:\[.*?\])?'] = None + extravalues['S'] = None + extravalues['SRC_URI'] = None + extravalues['SRC_URI[md5sum]'] = None + extravalues['SRC_URI[sha1sum]'] = None + extravalues['SRC_URI[sha256sum]'] = None + extravalues['SRC_URI[sha384sum]'] = None + extravalues['SRC_URI[sha512sum]'] = None + + classes.append('pypi') + + handled.append('url') + return fetch_uri + def handle_classifier_license(self, classifiers, existing_licenses=""): licenses = [] From patchwork Mon Dec 4 15:59:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 35632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5504C10DC1 for ; Mon, 4 Dec 2023 15:59:52 +0000 (UTC) Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx.groups.io with SMTP id smtpd.web11.72362.1701705588637596096 for ; Mon, 04 Dec 2023 07:59:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=aWT5GQcv; spf=pass (domain: baylibre.com, ip: 209.85.128.51, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-40c0a03eb87so14578785e9.3 for ; Mon, 04 Dec 2023 07:59:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1701705586; x=1702310386; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ALFBlN3RXfM8G++9dipMmPS4BJazwTbEs56PXQLqvUY=; b=aWT5GQcvM7rZ0BWnvjh1EeM3Fe03sAEwoORkZHdNqUCl2Andw5oo3LBjsJuIP7FgwB xQk+SDfRbD4bAuK20qw/ru6sXaGRzX5k2mjpo0kW/Mc/uGEkXAL9IGbj8krPl4H/cK3l OxXYVOfxbYCUy1bUWbGu0XWyXi4YHzO6UvLrRPC6v9eojomHsgnQlPBbI5PDsaL1kxlG fBjOj4qhWi5PMMBoCeG/ynylBsCVhafEbmfIRbdBzWCl6K472LJwLe1eM/I5n06oSm2a wdGKZ+e1LMcov1qg3Uv1Pt2gOSR71S/leip6QObofSEiia7VDf9vL7KEs9lTbApg5ZxN Q0Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701705586; x=1702310386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ALFBlN3RXfM8G++9dipMmPS4BJazwTbEs56PXQLqvUY=; b=b2PkVOh9Y5YhgpfQpkxvPvplS+uLPsMDFK2TuvvC+l/4aeK1DEj2eLlt4L07LwaGrf Xj4O7Rug+nu4GkCkIaUl7+UIL7sxCHkadgT49pZ2NNHBD19FWbmgZUJ2P7GUijHW0YZV Gt0IH1obC4wv8HF5KiKmuOhSEo3P0mE7k9a9F/0hRVKLSBC3ySgoQc6T845+3UEdA7Gt PBBzNFFoSek5Jq66CnZM8mLWqV/Qxl7ekpg9NLwgYkDhLdEIsSI972vpCesL/9vc9qa7 ASoYt5JrnsWve0KfuVlEo0pNdyBzDi5m5HYOwfMq4wIdyyo63exvy9nhAY0iZqhULphf zKpg== X-Gm-Message-State: AOJu0YxwY6r9EVyg2e8qkfF5ZaegKb4BbaRX/fGzJZ4xcJ4h7p2NNx9V lbOsPSVvjof841+IRpBEQIzB8YKvvFcG5qe+Fm2h6A== X-Google-Smtp-Source: AGHT+IG6G7yIosMQzG2R0PY+KaNVw3gF9QYhXWsOc+JKzPvANlTR9S95PkXnRMSmchprOn1Y4H6pSw== X-Received: by 2002:a7b:ca50:0:b0:40c:aa3:c65b with SMTP id m16-20020a7bca50000000b0040c0aa3c65bmr529306wml.313.1701705586608; Mon, 04 Dec 2023 07:59:46 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id g16-20020a05600c4ed000b0040b47c53610sm15479595wmq.14.2023.12.04.07.59.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 07:59:45 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [RESEND][PATCH 6/7] oeqa/selftest/recipetool: remove spaces on empty lines Date: Mon, 4 Dec 2023 16:59:33 +0100 Message-ID: <20231204155934.836992-7-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231204155934.836992-1-jstephan@baylibre.com> References: <20231204155934.836992-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Dec 2023 15:59:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/191760 Signed-off-by: Julien Stephan --- meta/lib/oeqa/selftest/cases/recipetool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 55cbba9ca74..4bc28a4f2ee 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -853,7 +853,7 @@ class RecipetoolTests(RecipetoolBase): self._test_recipe_contents(deps_require_file, checkvars, []) - + def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths): dstdir = basedstdir self.assertTrue(os.path.exists(dstdir)) From patchwork Mon Dec 4 15:59:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Stephan X-Patchwork-Id: 35630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A28FCC4167B for ; Mon, 4 Dec 2023 15:59:52 +0000 (UTC) Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by mx.groups.io with SMTP id smtpd.web11.72364.1701705589691338267 for ; Mon, 04 Dec 2023 07:59:50 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@baylibre-com.20230601.gappssmtp.com header.s=20230601 header.b=DEw21Cd4; spf=pass (domain: baylibre.com, ip: 209.85.128.53, mailfrom: jstephan@baylibre.com) Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-40c0a03eb87so14578925e9.3 for ; Mon, 04 Dec 2023 07:59:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1701705587; x=1702310387; darn=lists.openembedded.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tvn93zXlOozJpVGbSP2G/Dvv7AsjgFtSstz+fmtTgHE=; b=DEw21Cd4uIQp70slznUHZ8Dmcgmh2OvYA2UBeFmnanVpVOczluIc1xedNj6uwm8+yL 5kw60JFxSmMA25ZJgP9ptm+6I2J9OuViHl4QeEMyRxmOCk+iuVzNCUOyfs94XnKjdNp9 fJ4xfXi3ODeWlhJvawvRlmFY0S1zIdKap3/kFzsiB+zYXOnlZ/Jtw10iSYl66GnsHMB0 cxo7bSk85OZwLH5rWh6jIJQy6wxdk0mfSbzpxol1Hi46z6AFiAhfpwkMOOZBLu17DytI 09Vhs0N7r7MoaxjDJlrc1UEubAyXgdEY6T53UNlDxGvDGDhkPxBKe7sKacMXdnY1s0pS GdLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701705587; x=1702310387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tvn93zXlOozJpVGbSP2G/Dvv7AsjgFtSstz+fmtTgHE=; b=siwDO5RIoxdW0BE3VX17PgVoqo5jFwkYsK2rXmklHZJdlqMvfVutIlS5t1BHqd3ANa F17wOBZDcilztAblONzMz3D2p6ljVwE11tKpT672Bh52pa9fNnPQerUjkl5DhNk5IEk3 HZzNXgUKEw8xl8nkap8bfpPNKqokCTgtqX7mWMdiolm9IUXyF0+KUudEx7pjwjXUbGAS p7nPBi3hrRYLUhok56UxXHbXka7ZgUVPMiu48S6PMFT/lk0+w46pSnDn7M/Kv4S8ErQ8 bkxCY+hqTCl/VpEXd5w8ypxQRj/0adssMCgHEjoRyugNLaYDzaG0K7oPueQ+BX/XIBsR Dj1A== X-Gm-Message-State: AOJu0YxCscoPcfA1S9z3yyryzD68776vsPFe3snOZpx/aH7AMpWOPv0f 9m99cTcEGwWJ3cSQRliu6/WrPt4HibzzM/KzOyMFbw== X-Google-Smtp-Source: AGHT+IG890eBmxZNEEVpg+nlJE/fgDq+xbb4iXrRxR8SeH6Izq0QTHN9GMFlQkethPljlyIpplRB8Q== X-Received: by 2002:a05:600c:1f92:b0:40b:5f03:b3d1 with SMTP id je18-20020a05600c1f9200b0040b5f03b3d1mr1413175wmb.243.1701705587630; Mon, 04 Dec 2023 07:59:47 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:55f:21e0:9e19:4376:dea6:dbfa]) by smtp.gmail.com with ESMTPSA id g16-20020a05600c4ed000b0040b47c53610sm15479595wmq.14.2023.12.04.07.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 07:59:46 -0800 (PST) From: Julien Stephan To: openembedded-core@lists.openembedded.org Cc: Julien Stephan Subject: [RESEND][PATCH 7/7] oeqa/selftest/recipetool/devtool: add test for pypi class Date: Mon, 4 Dec 2023 16:59:34 +0100 Message-ID: <20231204155934.836992-8-jstephan@baylibre.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231204155934.836992-1-jstephan@baylibre.com> References: <20231204155934.836992-1-jstephan@baylibre.com> MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 04 Dec 2023 15:59:52 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/191761 recipetool now supports the pypi class and python recipes can by created using the new following syntax: * recipetool create https://pypi.org/project/ * recipetool create https://pypi.org/project// * recipetool create https://pypi.org/project/ --version or the old syntax: * recipetool create https://files.pythonhosted.org/packages/<...> So add tests for the new syntax and modify old tests Signed-off-by: Julien Stephan --- meta/lib/oeqa/selftest/cases/devtool.py | 4 +- meta/lib/oeqa/selftest/cases/recipetool.py | 112 +++++++++++++++------ 2 files changed, 85 insertions(+), 31 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index b5c488be8e8..3d9a4901ea5 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -543,7 +543,7 @@ class DevtoolAddTests(DevtoolBase): self.track_for_cleanup(self.workspacedir) self.add_command_to_tearDown('bitbake -c cleansstate %s' % testrecipe) self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') - result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url)) + result = runCmd('devtool add --no-pypi %s %s -f %s' % (testrecipe, srcdir, url)) self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created. %s' % result.output) self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory') self.assertTrue(os.path.isdir(os.path.join(srcdir, '.git')), 'git repository for external source tree was not created') @@ -562,7 +562,7 @@ class DevtoolAddTests(DevtoolBase): result = runCmd('devtool reset -n %s' % testrecipe) shutil.rmtree(srcdir) fakever = '1.9' - result = runCmd('devtool add %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever)) + result = runCmd('devtool add --no-pypi %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever)) self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory') # Test devtool status result = runCmd('devtool status') diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py index 4bc28a4f2ee..b59e53f5994 100644 --- a/meta/lib/oeqa/selftest/cases/recipetool.py +++ b/meta/lib/oeqa/selftest/cases/recipetool.py @@ -457,13 +457,15 @@ class RecipetoolCreateTests(RecipetoolBase): def test_recipetool_create_python3_setuptools(self): # Test creating python3 package from tarball (using setuptools3 class) + # Use the --no-pypi switch to avoid creating a pypi enabled recipe and + # and check the created recipe as if it was a more general tarball temprecipe = os.path.join(self.tempdir, 'recipe') os.makedirs(temprecipe) pn = 'python-magic' pv = '0.4.15' recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv)) srcuri = 'https://files.pythonhosted.org/packages/84/30/80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/python-magic-%s.tar.gz' % pv - result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + result = runCmd('recipetool create --no-pypi -o %s %s' % (temprecipe, srcuri)) self.assertTrue(os.path.isfile(recipefile)) checkvars = {} checkvars['LICENSE'] = set(['MIT']) @@ -474,6 +476,82 @@ class RecipetoolCreateTests(RecipetoolBase): inherits = ['setuptools3'] self._test_recipe_contents(recipefile, checkvars, inherits) + def test_recipetool_create_python3_setuptools_pypi_tarball(self): + # Test creating python3 package from tarball (using setuptools3 and pypi classes) + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + pn = 'python-magic' + pv = '0.4.15' + recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv)) + srcuri = 'https://files.pythonhosted.org/packages/84/30/80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/python-magic-%s.tar.gz' % pv + result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88' + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', 'pypi'] + self._test_recipe_contents(recipefile, checkvars, inherits) + + def test_recipetool_create_python3_setuptools_pypi(self): + # Test creating python3 package from pypi url (using setuptools3 and pypi classes) + # Intentionnaly using setuptools3 class here instead of any of the pep517 class + # to avoid the toml dependency and allows this test to run on host autobuilders + # with older version of python + temprecipe = os.path.join(self.tempdir, 'recipe') + os.makedirs(temprecipe) + pn = 'python-magic' + pv = '0.4.15' + recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv)) + # First specify the required version in the url + srcuri = 'https://pypi.org/project/%s/%s' % (pn, pv) + runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88' + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', "pypi"] + self._test_recipe_contents(recipefile, checkvars, inherits) + + # Now specify the version as a recipetool parameter + runCmd('rm -rf %s' % recipefile) + self.assertFalse(os.path.isfile(recipefile)) + srcuri = 'https://pypi.org/project/%s' % pn + runCmd('recipetool create -o %s %s --version %s' % (temprecipe, srcuri, pv)) + self.assertTrue(os.path.isfile(recipefile)) + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88' + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', "pypi"] + self._test_recipe_contents(recipefile, checkvars, inherits) + + # Now, try to grab latest version of the package, so we cannot guess the name of the recipe, + # unless hardcoding the latest version but it means we will need to update the test for each release, + # so use a regexp + runCmd('rm -rf %s' % recipefile) + self.assertFalse(os.path.isfile(recipefile)) + recipefile_re = r'%s_(.*)\.bb' % pn + result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri)) + dirlist = os.listdir(temprecipe) + if len(dirlist) > 1: + self.fail('recipetool created more than just one file; output:\n%s\ndirlist:\n%s' % (result.output, str(dirlist))) + if len(dirlist) < 1 or not os.path.isfile(os.path.join(temprecipe, dirlist[0])): + self.fail('recipetool did not create recipe file; output:\n%s\ndirlist:\n%s' % (result.output, str(dirlist))) + import re + match = re.match(recipefile_re, dirlist[0]) + self.assertTrue(match) + latest_pv = match.group(1) + self.assertTrue(latest_pv != pv) + recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, latest_pv)) + # Do not check LIC_FILES_CHKSUM here to avoid having updating the test on each release + checkvars = {} + checkvars['LICENSE'] = set(['MIT']) + checkvars['PYPI_PACKAGE'] = pn + inherits = ['setuptools3', "pypi"] + self._test_recipe_contents(recipefile, checkvars, inherits) + def test_recipetool_create_python3_pep517_setuptools_build_meta(self): # This test require python 3.11 or above for the tomllib module # or tomli module to be installed @@ -498,13 +576,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['SUMMARY'] = 'A library for working with the color formats defined by HTML and CSS.' checkvars['LICENSE'] = set(['BSD-3-Clause']) checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=702b1ef12cf66832a88f24c8f2ee9c19' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/a1/fb/f95560c6a5d4469d9c49e24cf1b5d4d21ffab5608251c6020a965fb7791c/webcolors-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = 'c9be30c5b0cf1cad32e4cbacbb2229e9' - checkvars['SRC_URI[sha1sum]'] = 'c90b84fb65eed9b4c9dea7f08c657bfac0e820a5' - checkvars['SRC_URI[sha256sum]'] = 'c225b674c83fa923be93d235330ce0300373d02885cef23238813b0d5668304a' - checkvars['SRC_URI[sha384sum]'] = '45652af349660f19f68d01361dd5bda287789e5ea63608f52a8cea526ac04465614db2ea236103fb8456b1fcaea96ed7' - checkvars['SRC_URI[sha512sum]'] = '074aaf135ac6b0025b88b731d1d6dfa4c539b4fff7195658cc58a4326bb9f0449a231685d312b4a1ec48ca535a838bfa5c680787fe0e61473a2a092c448937d0' - inherits = ['python_setuptools_build_meta'] + inherits = ['python_setuptools_build_meta', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits) @@ -532,13 +604,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['SUMMARY'] = 'Simple module to parse ISO 8601 dates' checkvars['LICENSE'] = set(['MIT']) checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=aab31f2ef7ba214a5a341eaa47a7f367' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/b9/f3/ef59cee614d5e0accf6fd0cbba025b93b272e626ca89fb70a3e9187c5d15/iso8601-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = '6e33910eba87066b3be7fcf3d59d16b5' - checkvars['SRC_URI[sha1sum]'] = 'efd225b2c9fa7d9e4a1ec6ad94f3295cee982e61' - checkvars['SRC_URI[sha256sum]'] = '6b1d3829ee8921c4301998c909f7829fa9ed3cbdac0d3b16af2d743aed1ba8df' - checkvars['SRC_URI[sha384sum]'] = '255002433fe65c19adfd6b91494271b613cb25ef6a35ac77436de1e03d60cc07bf89fd716451b917f1435e4384860ef6' - checkvars['SRC_URI[sha512sum]'] = 'db57ab2a25ef91e3bc479c8539d27e853cf1fbf60986820b8999ae15d7e566425a1e0cfba47d0f3b23aa703db0576db368e6c110ba2a2f46c9a34e8ee3611fb7' - inherits = ['python_poetry_core'] + inherits = ['python_poetry_core', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits) @@ -566,13 +632,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['SUMMARY'] = 'Backported and Experimental Type Hints for Python 3.8+' checkvars['LICENSE'] = set(['PSF-2.0']) checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=fcf6b249c2641540219a727f35d8d2c2' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/1f/7a/8b94bb016069caa12fc9f587b28080ac33b4fbb8ca369b98bc0a4828543e/typing_extensions-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = '74bafe841fbd1c27324afdeb099babdf' - checkvars['SRC_URI[sha1sum]'] = 'f8bed69cbad4a57a1a67bf8a31b62b657b47f7a3' - checkvars['SRC_URI[sha256sum]'] = 'df8e4339e9cb77357558cbdbceca33c303714cf861d1eef15e1070055ae8b7ef' - checkvars['SRC_URI[sha384sum]'] = '0bd0112234134d965c6884f3c1f95d27b6ae49cfb08108101158e31dff33c2dce729331628b69818850f1acb68f6c8d0' - checkvars['SRC_URI[sha512sum]'] = '5fbff10e085fbf3ac2e35d08d913608d8c8bca66903435ede91cdc7776d775689a53d64f5f0615fe687c6c228ac854c8651d99eb1cb96ec61c56b7ca01fdd440' - inherits = ['python_flit_core'] + inherits = ['python_flit_core', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits) @@ -601,13 +661,7 @@ class RecipetoolCreateTests(RecipetoolBase): checkvars['HOMEPAGE'] = 'https://github.com/python-jsonschema/jsonschema' checkvars['LICENSE'] = set(['MIT']) checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=7a60a81c146ec25599a3e1dabb8610a8 file://json/LICENSE;md5=9d4de43111d33570c8fe49b4cb0e01af' - checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/e4/43/087b24516db11722c8687e0caf0f66c7785c0b1c51b0ab951dfde924e3f5/jsonschema-${PV}.tar.gz' - checkvars['SRC_URI[md5sum]'] = '4d6667ce76f820c35082c2d60a4896ab' - checkvars['SRC_URI[sha1sum]'] = '9173714cb88964d07f3a3f4fcaaef638b8ceac0c' - checkvars['SRC_URI[sha256sum]'] = 'ec84cc37cfa703ef7cd4928db24f9cb31428a5d0fa77747b8b51a847458e0bbf' - checkvars['SRC_URI[sha384sum]'] = '7a53181f0e679aa3dc3eb4d05a420877b7b9bff2d02e81f5c289a37ed1127d6c0cca1f5a5f9e4e166f089ab36bcc2be9' - checkvars['SRC_URI[sha512sum]'] = '60fa769faf6e3fc2c14eb9acd189c86e9d366b157230a5681d36552af0c159cb1ad33fd920668a36afdab98bc97253f91501704c5c07b5009fdaf9d29b52060d' - inherits = ['python_hatchling'] + inherits = ['python_hatchling', 'pypi'] self._test_recipe_contents(recipefile, checkvars, inherits)