From patchwork Thu Feb 1 03:30:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: ChenQi X-Patchwork-Id: 38531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB3A6C47258 for ; Thu, 1 Feb 2024 03:31:18 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web10.7745.1706758277155275150 for ; Wed, 31 Jan 2024 19:31:17 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriver.com header.s=PPS06212021 header.b=jaldy2sH; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=276184b32d=qi.chen@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 41125UGO025555 for ; Wed, 31 Jan 2024 19:31:17 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from:to:subject:date:message-id:content-transfer-encoding :content-type:mime-version; s=PPS06212021; bh=tthfk7lUuSy/In55rl uhxDNDi+Xkn6RtuH+gc04oxAc=; b=jaldy2sHwQhSA813PEIubw3BSEpbzZ69SL d8Y2c9jWZexd2u0Jgz71FJ8aNXfRossLMQLVA+vAAFMtIAPaTo0WKdXcXCZggJEi ezXOo6dES8MbsDVnBeiIab/xZeT/uQ25ESbhGsxPZkeMwjFEqd0pP67TZGSGOqVy mgg7+d8DxsYZ3olkkNDBXY0um/vBrlpMLS8XlybtnIMyMTpTPHwMdN20oqnCuVAj T8LAN+Rh0NDZJZzKs+wDG1NnzeFheBKxRr4697ZryQUHs1b/RUzLgUZ38CrooNK5 wzRkhgkCHR6LFvRHtrBRgJOKpB7Bf/4grYjoIaOkhTUs4Hx++4aQ== Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3w0276g1q4-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 31 Jan 2024 19:31:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VsKiCYikddBnmLd4SKONhKg5QG+ZX0+rHLUskYEqFRoeJPm2klUrUhtM6PoIkbuj3+b4eJR3vskqXZMFjalIh+w9iHtB16ibNMiGfdSZTwUpCPw7I+lHJ4rP9bGcT9z6S9wzd9rZj/zptqKMVi5AjqJQvyRSdldOvqEcqK2wMkBOlmbqdkbRewYFEjXQ9dqgD7bdPh1QBesbr4qGTyMMrR31se+6kF9waRNpGtFt94d2L60TcvGYZqBS2sB8wwoLmyNjBd+/sfjA6U+nvqttJnXPsnh9cnqzi9rYwn8jSLNRkj7d4hypB28cRlaHwctBrdENTT2D6PJwv2AzfGoY+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tthfk7lUuSy/In55rluhxDNDi+Xkn6RtuH+gc04oxAc=; b=BoGXxsWrLwMc4HRZZc6/k45f2bzXnXM5/64izbKbK8KdC1Dt34f+xVEOHs8joCAvXz9QeXjsRsTvzMzht8HMuxSV3Hi+0yRpwOYU2HHJGgMnRd6GmyV6Xj6rO1Ns6Suq+U/rjinHb+Jjh97asXN6wH+HDADyNe6mmytMu8siDFIVeVmAoFiOt8v++ckUoDRNLWGjaNuRX08LoiIQ+cjAyA0qzG+zhPN3uurhoCgWS4RvqFDJpaaUTJLR3Z+eNfdz7H5dxulQ+48UE95E5YhXkhTZDy51pXnDP3qJuRz/674G9z1A+LjplYFiaL1Vwb1ptl8AgQdtTu7d5Q5a68TC4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) by PH7PR11MB8598.namprd11.prod.outlook.com (2603:10b6:510:2fd::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.22; Thu, 1 Feb 2024 03:31:13 +0000 Received: from CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::5d84:2b01:5029:9315]) by CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::5d84:2b01:5029:9315%7]) with mapi id 15.20.7249.025; Thu, 1 Feb 2024 03:31:13 +0000 From: Qi.Chen@windriver.com To: bitbake-devel@lists.openembedded.org Subject: [bitbake-devel][PATCH 1/3] fetch2/git.py: fix a corner case in try_premirror Date: Wed, 31 Jan 2024 19:30:58 -0800 Message-ID: <20240201033100.3617421-1-Qi.Chen@windriver.com> X-Mailer: git-send-email 2.42.0 X-ClientProxiedBy: SJ0P220CA0013.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:41b::20) To CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR11MB5602:EE_|PH7PR11MB8598:EE_ X-MS-Office365-Filtering-Correlation-Id: 45575933-3e7e-4ec9-b583-08dc22d63da6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CvdwHDgoKrVD0YLNUaZzHZiHzZUsbEdVCRodKxQc4/uylp/UPGb5EckyKoPu4h6orKbeyqxVMF6kGTgtR+q1w5NGNZEKGipcRg6HrRdFuN0/++PNDH5/nj62LzXhAvFxICWYQ3+UZxOspNLlGymUUuRx+FSodCsfYS8bFuu9z/m0R28xKBZGm1+bMWUIGkgzIdHQvRmhZ1DDtrOWnHKq7QERsDN/GVe54sf1jsZq+TNhvnXgb2w8jLFC6sgAb+i2Pv0hi5PD1cl3eAr3gmonUSrxrfeJp7KeidvQBJ+mJn1M4jP0k8+T+MmsCW4vYX1xpnpqHLwvJYyGR7/JgbdIhAsO8i94TXe65aSsewa8YUnoY1/TG1aa12QJ8iXtCKz06fkAWo99LRR/Q/MuaRG6y85aFer0r/I7IPnIdY0iOZbr6dlkdMR4MddyZ1Nbrr4UHBJ3qJIgOazwNXn6jIS7GDWxmus+iz9WFqx1LYJwJG5HthVpu9LcXzBcaEpvCEMk2BY/pjq8Ar+wo5UFSusXNSkLd3U8/Oh9Uk0EHuD2HW5u0IUqD7RUPKxAn1VAwwzVYLfGYATelHrej+l9Fg3iXpAhy6JlHHA+9pTkBmpwWKOozDYCrLeZdPFwtWWDVyvg X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR11MB5602.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(39850400004)(396003)(136003)(366004)(346002)(230922051799003)(64100799003)(451199024)(186009)(1800799012)(38350700005)(26005)(6486002)(66946007)(478600001)(5660300002)(2906002)(83380400001)(9686003)(6666004)(66476007)(6506007)(8676002)(52116002)(6916009)(8936002)(316002)(2616005)(6512007)(1076003)(66556008)(38100700002)(86362001)(36756003)(41300700001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: RrvUZ8Nh7lcixs4jntXlPxMyz2n1qYv4HfkbjyS9JhAEmxn5IJXhpURCbSECh/Uj1AFf1j8Ko9SkbApq7/T9DL6RrSFFd3YujhV5MGgnG9Rk39qeTSjt5nT82rYELAprUoJi7oH9WvwzDF+soWk41FRFtPlt5iVsqT86j6Kb0ClIBw6jQdC4o2hzKqt6hNbI1aNDx92/DnHroXhbWr46CBneegurheLESfAXgl2SvBI5UUz+N5vffpWSa9e1aOemoeZuioDsAPVc8EVSc+oQ/d8tWdTNz913sKZc4qzAK5Ga+/T751IUnV6oKQCe+F6VMjbf94QRgR0vus5lgPsbZGdyOHxFo5uY1BmkykhWGM3ekDl9plRP9FSgIde9fYqUq1z7BN5o3eNT+cvT6WgiYHxsJ+g5HIWxD53cXVZYzO2rl8p/KV0fAWNyAu2cas0qtrWjyxVeqqW0sGYWWHltPFe1U6qSnN2+0voN7b/Sf+LXFLvuG2AzjDe8BFjFY9i0DCHJD49UM76ktsu08khFYOE7aMQHp233Y/O+LUIvVN2eyRQw3jn26DmBuxzxBvSddmdCrXBZ+RTikKIB0XT6+AVY0AuYcaIR7GcOAWzOTJBtk55Uta2TObwk/oLT8DmrZGKGyxqjWcgQAqNvBIXARKVhfY2KTjHMb+CmBZCRKF7fb9QGBaks+qUR8DcGjhblfA3eKpu7ReiPD/Tq6eCinwH39mngGvsXlGydPLI/M+CsnpP8VBL/jUBXPstxIl34q1yvbiALQ7DIayjpJVuYrWDLstcFDq8jZdQ2pURegoj3EwwfcvevO7wy/jJ74JkPs06XkMxKHK9dAMzLJ1aRtSVMrqHEKvVMLMNN+uSOkRwYEr9CcqiCMdWfHZV0XBY9pgpoU1b9y7HsNBuctT7tShx9zIuTsPZ1ZCgsRKTyGMx7wMckouqEMtmY23p3/t0UhBsd2HpSFIbJ+JvY0L6C+PvEfD5ulI8pYKfLQWnOarSeibOZMShyhmb47S4tOFncuFg92Jr9o7Vfr7yI72CtB1xm6AeY9Jgiz3p7abNb+ApV4+/HpGSGaonyks0K+MNfiGQcgsV9VCcp4ovcZhDAH0v2743gHbreAMou83Nk/EPheH3BfJbh35dXCWr2BqslGIwuW3M83B9Eo8OyO8dn10HOmue57NvNPH0QJtURY8GReArrC1efiNsoH5riF/wfYU2tAy1oTVIiMC//LuHscX16gRn1QhJldxqPIOMLEda6ANEjwTTsqfegBm/B9YdUe+O4gyfNvueThRTWrJdAw0Fab7rnBjHzhNIw+A4QDB8eTppMP0WxDYIw62rkqIG+e61nZs+p6lfuifRW5mHMh//FLGT8bgMZhCsIQunK3nl5GA/sryB87YI1N/3F/HRD54yKpZCLnb7XMEHhSG7UBbtF0I22oqeS0dt1dR6U+RCjJfGPY+smRZyfudZnsmNAj9mNzLSv2YPqa7gYhUuGcnhwcohobEDAdAVval/aC3fFwxMVmIvSaFGStd7+87d1PobY7bZvxHXxeBpJoAGEDtgHTZCzxcpt+vriuwCWOSVYxCtde+7N1pqQhuuRErQJ X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45575933-3e7e-4ec9-b583-08dc22d63da6 X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5602.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2024 03:31:13.3395 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 93yxo0SZm5TzFDeWeAWfAUKSNr9UKoPUE5gKHjPJ6ywp1bPv6Twir8fk1DRJTfcrGInpm7q8c7hgVh5kdl4OkA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8598 X-Proofpoint-GUID: TheyGmzc5P0EV41boyBhH1mgdZixHnoE X-Proofpoint-ORIG-GUID: TheyGmzc5P0EV41boyBhH1mgdZixHnoE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-31_10,2024-01-31_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 adultscore=0 mlxlogscore=999 lowpriorityscore=0 bulkscore=0 malwarescore=0 spamscore=0 phishscore=0 impostorscore=0 clxscore=1015 priorityscore=1501 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2401190000 definitions=main-2402010026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 01 Feb 2024 03:31:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15817 From: Chen Qi For gitsm recipes, it's possible that some URL is used more than once. e.g., A -> B:rev1 (B is a submodule of A) A -> C (C is a submodule of A) C -> B:rev2 (B is a submodule of C) A anc C are both using B as submodules, but on different revs. Now if we have: B:rev1 -> D B:rev2 -> E Then, the mirror will not be fully used. Say we have all repo mirrors for A, B, C, D, E, then in theory it's not necessary to reach out to any network for downloading. But it's not the case. After downloading B(rev1) and its submodule D from mirrors, the fetch process continues to download C, thus B(rev2) and E. Now it finds that B needs an update because its submodule E needs an update. Of course this is true because E is not downloaded yet. Now the problem comes to whether to use mirror or not. The git.py defines try_premirror to return 'False' when the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and try_mirror returns False, resulting in not using mirror and going to upstream directly. We can see that the mirrors are not fully used. This is usually not problem, as the cost is only some network download. But in case the following two settings are there, we get errors. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.allowed.domain" In such case, the gitsm recipe A will fail to fetch. Note that all contents that A needs are in mirrors and now it's failing to fetch. This is unexpected. Note that the different revs of the same repo in gitsm recipe is not the only way to reveal this problem. For example, there might be a recipe call B that uses B:rev3. Check the protobuf and grpc recipes as an example. For now, we can use the following steps to reproduce this issue. To be clear, the grpc recipe in meta-oe is now 1.60.0. 1. Add in local.conf: DL_DIR = "${TOPDIR}/downloads-premirror" bitbake grpc -c fetch 2. Comment out the DL_DIR setting in local.conf and add the following lines: PREMIRRORS:append = " \ git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \ " 3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'. This command succeeds and this shows that the premirror holds everything we need. 4. Add the following lines and then 'bitbake grpc -c fetch'. BB_NO_NETWORK = "0" BB_ALLOWED_NETWORKS = "*.some.domain" After step 4, the error message is as below: ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used This patch fixes this problem by handling this corner case, that is, if the URL is not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should try premirrors because trying to reach upstream is destined to fail. Signed-off-by: Chen Qi --- bitbake/lib/bb/fetch2/git.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 0deeb5cee1..322a9366da 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -87,6 +87,7 @@ from contextlib import contextmanager from bb.fetch2 import FetchMethod from bb.fetch2 import runfetchcmd from bb.fetch2 import logger +from bb.fetch2 import trusted_network sha1_re = re.compile(r'^[0-9a-f]{40}$') @@ -355,6 +356,11 @@ class Git(FetchMethod): # is not possible if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")): return True + # If the url is not in trusted network, that is, BB_NO_NETWORK is set to 0 + # and BB_ALLOWED_NETWORKS does not contain the host that ud.url uses, then + # we need to try premirrors first as using upstream is destined to fail. + if not trusted_network(d, ud.url): + return True if os.path.exists(ud.clonedir): return False return True From patchwork Thu Feb 1 03:30:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: ChenQi X-Patchwork-Id: 38530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3B5AC4828C for ; Thu, 1 Feb 2024 03:31:18 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web10.7744.1706758276556223050 for ; Wed, 31 Jan 2024 19:31:16 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriver.com header.s=PPS06212021 header.b=BUTxkb20; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=276184b32d=qi.chen@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 41125UGN025555 for ; Wed, 31 Jan 2024 19:31:16 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from:to:subject:date:message-id:in-reply-to:references :content-transfer-encoding:content-type:mime-version; s= PPS06212021; bh=69MjcHK5rPQ2K+WqgCjwY5X7q5Az4qcRhj5To9dOoxA=; b= BUTxkb20bsIDUXDhH4pJbPC2VmIOrwtzGvvLuAqD1YD8WRhORXJ4+zMfgqGrAGCG hAwjmSRtXz8FH0VGFLyU4E/JhhRcWe+3TR3h5ziZujuBajSqWRiLwObpQ2RnNPZQ qfVASd4PDv27bp84xstLMuCVhOao3lWtZzOtutqIJ2pGpMh2G785OvdF79q6MS5M kO4StOw7NuRd+ocNnxWng6WEauAUqk7l2u3GEZwWch42gjkgPeEa+ObJog2axd3c eVYgSVgEDnlH75a2HkO/eHBNSv2wSp66dDryPY6cC6acChgReAAmsu9HJjjEs+dF sKd8iAPU0c8r60jTlZv+Yg== Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3w0276g1q4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 31 Jan 2024 19:31:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gQYz1+UkF3zjIC7yNV1wl4dq7I5255uhloh8ADpRVnKHi67dc21cqTH2bKTId9NtVgjDB1HgeDh7AjTtM9mzNr1c6XwvSVamD9QcnXYkMpJ6HoXxvJFYavAryOsKjFPcnblpH9+fsm90hjerEbvpAxSIQhLP+0rhZZvU4MvXvFKQOgjj7FhnrVRlG4TdzuqR3Uflapbz7GQ1j8s4pHnyHaFk56GtqbqvsA+skB8FtIT/V6mqYoKoXgbI3Xe3CWGWue9sXLHeUtCZIKK4r7Ty9T5mQk6umjeoTiZ7ueBbyMuVljJZqpHiazX7VO13oFMrdWXB7kzhID6AFs5TFEZ/JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=69MjcHK5rPQ2K+WqgCjwY5X7q5Az4qcRhj5To9dOoxA=; b=h+XCOe2QjWjSMh/xweof5UwzxVzTUxgq5PplyDHvO3sSHaZSYVveC5WwL/o/jj5lefdaKgbU7NwmiAlDWUv+ECvOBIvNT8IKczgOj3wWLSIgwwNdsZRMnSdHz2Ns+ib4fUego4+GLmKAOHtkf7I9h5zwUBtGdyMxrPufYVqR8T+UuKrkvponyKNuu2sEp7f1R44RO5F3HLU5CIquM6wTJfL8SsnGikZ20dr5WKoR9WujOA4fbErDSC7Q/Xz3B58oBdCG07msMPKk8xX/y1BHiwdal970paCTAsYvnJBbHK+CkkvpAxLNqFPU/OJ68m1zhYXX7Ld4kmy4QkBC4mjlNA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) by PH7PR11MB8598.namprd11.prod.outlook.com (2603:10b6:510:2fd::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.22; Thu, 1 Feb 2024 03:31:13 +0000 Received: from CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::5d84:2b01:5029:9315]) by CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::5d84:2b01:5029:9315%7]) with mapi id 15.20.7249.025; Thu, 1 Feb 2024 03:31:13 +0000 From: Qi.Chen@windriver.com To: bitbake-devel@lists.openembedded.org Subject: [bitbake-devel][PATCH 2/3] tests/fetch.py: add test case for using premirror in restricted network Date: Wed, 31 Jan 2024 19:30:59 -0800 Message-ID: <20240201033100.3617421-2-Qi.Chen@windriver.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240201033100.3617421-1-Qi.Chen@windriver.com> References: <20240201033100.3617421-1-Qi.Chen@windriver.com> X-ClientProxiedBy: SJ0P220CA0013.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:41b::20) To CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR11MB5602:EE_|PH7PR11MB8598:EE_ X-MS-Office365-Filtering-Correlation-Id: 9cf955fd-2214-488c-ed72-08dc22d63dd1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: m8jPlPzmz9kh9varcQjH95AcDbvGN+eXcJx/rSOYLE7VoOo9XCb9uCX5WGq7AAUIsOPXPXRGD8AzSbpsvFzQwz6QzZ68M7Kx1RduB4XcJ/ZIecj+vkx4CXLaPBAHllrCItLg64myg9n/WDQ5mePth7NuezQDFYUQHS0QRVOC9dg/0I6MLzkexxDBuIzLOrVrH5NvLSjG458RLdk0YdXqm5JLxIWjstPz7VIPM6A0/qaYuqvnpQNDTjqeJODJtjqT0s2kBEr2lZTcFdeEZBZ5xjlYGccZeMCX7iV5srYMOnwGOIN/5TvPL3iTfixwjdYBIbojcEFsfQz8LjTG7HKPjb+siG8f7y0/IRJjzJT1eclXIbzYoa317PKSjr09vKRFNTWM5872D/+Yi4bp6OoMRrIwkfF8RRjCVUa5kqo2LZwDIkw5qUj+lOgQxeswA3xaKZXW26kpsH+trhltMmcEdGLA+orjEH0jTGHCZHu2JNF9T2r4mIdQ28u05WBl5gl0J/aEeTMNzI9/9i7a5pxEaxpsqjDpkWLTzA08OK6jPc8wNbDAtTIObm06D5UlCgx2UozCldodUEFMK++4c73L8qkbO9bLd+agOUPujfWlvB2f6GcVoiLPSeGJZa/EvmvYn9Qrl0qZ3c4+qlvoy3rsNZQXIbUl4GRAZk6WpbxjDNA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR11MB5602.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(39850400004)(396003)(136003)(366004)(346002)(230273577357003)(230173577357003)(230922051799003)(64100799003)(451199024)(186009)(1800799012)(38350700005)(26005)(6486002)(66946007)(478600001)(5660300002)(2906002)(83380400001)(9686003)(6666004)(66476007)(6506007)(8676002)(52116002)(6916009)(8936002)(316002)(2616005)(6512007)(1076003)(66556008)(38100700002)(86362001)(36756003)(41300700001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: M6cS24Et11dkDrTH8MfMA8xLpWepELw6jgJodNWfA+MT0DfM/i3r7JNTqwxzXSnidiw1EXRamk4fHUECunGv5+lasW0asESJVN1xiRP2F7sB1aGbM0EA+NTrgVOTKtqTxMDRCwJ5FPlGb5TKQbyZ+lBLy96zxWSfhEOATuj8wo7EZr56hu+TBdWsCvAUr3lQc3iNwGCJpVqQddZ0mWHlg16LgB9Nr438upDen1TiXW/zFD5Y9N6Pw0k1KdkUoXDuh7Pyxh4gsZyAVh7rkrD3zmItd1O+yH17bYXdFYkv+M1OpSPSTYnoXSqS5r7kV4LCRZHUaWGlvzgsQabxZDawRkHUC+irwhyKuLTnRiDRNA5wh4ETL11eefk/ezefDfDhUFUASPRTsn0Wn6Ek4V7CiJ+a/Wjiiw+TrmJfyUpIycMg3F1+XvRxb92L/yQBpdGK4li5XC9duV5samMDVG7Yriy/t9dkPJqcR1Ce+VpvChHDyETw4u1fUEEO5vkx9McbboDGtOKRjRbVNwJSQWqKV/U3oX63sdHg6OVBiiZVH8h29RKUF68NaF0BBBRyzH5D5g9LCpnc6iGYilxYd78z2p41kQaiEAX7ALczou3RNa3I+GLLIN8b2UjW4FwpAADRAFcIVvlQvz7LlKx4MO2I5TwU95HjF3hpxuzM0Zhq6EKKq/XISKIEYfxHKNCN6z6H7Lofy0OHo9N+Ma0ovTlBymL7xvvn0Dh7LNhHeR/4HV+jLJONpyOIn7nFYLE5i2YvQ2jEMwC3/RcpNI4SmSfLutO1cMsiHs0OMXX5mB/sBylMwfCein5LnUf5HurO2dcwRM0MsLZjWyXZknZnRoHvAzqfhP6yMeDKic/2Pbn01fFMQH30C4Z0DACAzN2FbBl5o8047w3vywgKVbYmx250Cj7VK5CJ9RlaPPS3G6Js+cIxgIucDx44pJ5vOA/Ob9Nn/gFFNgmCceJc1c7zxi+RGhBUNzr1uYZtBqKj+majddyqtXKoTujIr0cOMs3N1sOb17K4RckohHf0OvuAAIzKdmwRasDFgpkpesuC+MemUKMU1yjc6YKOdPCAqLdWDrd4wdJMX1AhGj8s8A2Lg3yiKrX9FFacnsMVBWWFMa4aOM2XNKXhIqsikWJKhC1IKnILQvKYo8ayH8VN75QuDedq4TifQdD7mE7BxciWJmmZFUw+HuNhwhm7XEyRSR9AJX2tez1SyvwSopEOy/sYMpJIHkFK/M+uYV35eBD9Z4aAG1voiDTn2q9B5J4q8SADG2dB/ELaaWlj1d9ez3Yb1QEiSMt/Bo+QszZfQW2UwnWWQ0NrZEDiDPsQCM58D/Iy8nqs2jn2NeVjScfNDnHAiseWuxsTCOThrIY9t2KKr96d7K+6JEVXRa5tqaY/Nvzu2Ckb4dQxlwU9RnQC3UvleKqCyyRK1TajqFp6/nMwMhe4Wfi0qoEQkwLhYcx4C9Sbo7eUuqXhfUF+0suPvkEK88GAcCVPzK69RcVXTjj3NzsMuys2NBVmxCTZy5I1wIE9q9KF6d9jKHCzXXG96R7UExqMLNWpY0efV1lf1jaE92IF4gSibTRoTasOggUd71aImww2 X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9cf955fd-2214-488c-ed72-08dc22d63dd1 X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5602.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2024 03:31:13.6356 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: K3G3/ts8S+gZT6rXP9w/li0tCymFHZ34+yOdCAsXTDWrjzx0/+LcmWC1VMFvW4mvbZcsHDs277CWn0aUG3zWhw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8598 X-Proofpoint-GUID: iYVOddFlgTxeHQqRs_1gx7AdZUTYOw2C X-Proofpoint-ORIG-GUID: iYVOddFlgTxeHQqRs_1gx7AdZUTYOw2C X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-31_10,2024-01-31_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 adultscore=0 mlxlogscore=486 lowpriorityscore=0 bulkscore=0 malwarescore=0 spamscore=0 phishscore=0 impostorscore=0 clxscore=1015 priorityscore=1501 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2401190000 definitions=main-2402010026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 01 Feb 2024 03:31:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15816 From: Chen Qi We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch' failed even with all contents available in PREMIRRORS. Add a test case to ensure no regression in the future. Signed-off-by: Chen Qi --- bitbake/lib/bb/tests/fetch.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index c7a23407c1..179511232d 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -1107,6 +1107,25 @@ class FetcherNetworkTest(FetcherTest): if os.path.exists(os.path.join(repo_path, 'bitbake-gitsm-test1')): self.assertTrue(os.path.exists(os.path.join(repo_path, 'bitbake-gitsm-test1', 'bitbake')), msg='submodule of submodule missing') + @skipIfNoNetwork() + def test_git_submodule_restricted_network_premirrors(self): + # this test is to ensure that premirrors will be tried in restricted network + # that is, BB_ALLOWED_NETWORKS does not contain the domain the url uses + url = "gitsm://github.com/grpc/grpc.git;protocol=https;name=grpc;branch=v1.60.x;rev=0ef13a7555dbaadd4633399242524129eef5e231" + # create a download directory to be used as premirror later + tempdir = tempfile.mkdtemp(prefix="bitbake-fetch-") + dl_premirror = os.path.join(tempdir, "download-premirror") + os.mkdir(dl_premirror) + self.d.setVar("DL_DIR", dl_premirror) + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + # now use the premirror in restricted network + self.d.setVar("DL_DIR", self.dldir) + self.d.setVar("PREMIRRORS", "gitsm://.*/.* gitsm://%s/git2/MIRRORNAME;protocol=file" % dl_premirror) + self.d.setVar("BB_ALLOWED_NETWORKS", "*.some.domain") + fetcher = bb.fetch.Fetch([url], self.d) + fetcher.download() + @skipIfNoNetwork() def test_git_submodule_dbus_broker(self): # The following external repositories have show failures in fetch and unpack operations From patchwork Thu Feb 1 03:31:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: ChenQi X-Patchwork-Id: 38529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0046EC4828E for ; Thu, 1 Feb 2024 03:31:18 +0000 (UTC) Received: from mx0a-0064b401.pphosted.com (mx0a-0064b401.pphosted.com [205.220.166.238]) by mx.groups.io with SMTP id smtpd.web10.7746.1706758277709518154 for ; Wed, 31 Jan 2024 19:31:17 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@windriver.com header.s=PPS06212021 header.b=oxYtEy2u; spf=permerror, err=parse error for token &{10 18 %{ir}.%{v}.%{d}.spf.has.pphosted.com}: invalid domain name (domain: windriver.com, ip: 205.220.166.238, mailfrom: prvs=276184b32d=qi.chen@windriver.com) Received: from pps.filterd (m0250810.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 41125UGP025555 for ; Wed, 31 Jan 2024 19:31:17 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from:to:subject:date:message-id:in-reply-to:references :content-transfer-encoding:content-type:mime-version; s= PPS06212021; bh=WWZ8mWvxqW5lNVrsXArQ1Sw5BAHyFB9uMC4xanVyCsg=; b= oxYtEy2uEUENdsQB2JZSbhL68VbYZG5FBGH+yzyz6UkE2NxvCOblnOmMQtuX6tBa 3dihpDijcqG2q0oyAGW191VjyxqO3hYH3wmmkiNKmTYe3lV2Fx1znClTvT5VVn3B omtY63PXeMUQ+x99xwA1H8d2QvvTQJEr50e3IespM4Bf3EZll2P5kauttORPHm0q OAFe9JhUr5m2Z0AvQArGZu2R4sEcZ1+T11YfvGEc6EVMjRzkvsitMqmmMVht/309 jJlZ8BkTE3f7yQbq2ctqpKZhNzdKo8wSZnEOPZH2xAyYrfY7/HSARDQl+9n0tfFP i9HRe2liDWi7pdF1cWFpyA== Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168]) by mx0a-0064b401.pphosted.com (PPS) with ESMTPS id 3w0276g1q4-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 31 Jan 2024 19:31:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h4GLuGTHwk6qLkZQAiMg1sHorM1A2/n2o3gxsWTiU6iPE9Xi8G7w8aeVGs0gcYAiH/qJKIrh0+xgRbug5qhLLFQjFm0CZNIbM7fhUaT4raXpDCW31dVvus5ijtV7jldRabBib2bOYHjdELDUuulENRy+O0tAVev+VVdWpOTatEi0nMM6fSLHz3rSNNHQ/yrVnv6PUDzh7LVHmaMBNyZ7C1lKrLvXOf79wTkZEsixHvZOo3modCNRDBsilnHsMOouL1qOk2dt+2SS9/w8e8Put1Ryo4FeGaTviY+24EVlSmrhx7CpUQpvFZrnjSxqcsHN5ICPHQ5gbb8F5AKHYSf0mQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WWZ8mWvxqW5lNVrsXArQ1Sw5BAHyFB9uMC4xanVyCsg=; b=oCaj8TtgWmMVJGNUvStfP7VnHNtRfJHGckb8v8TGAvsP3Hbs7vuSvPiysiuwVfQ3W+tD5uhCzjRLos4+y2vKdDfzCrEu8VJ3rbx0UQ75g6E0aj+yK1+uLGzz/c/nu4UQOZp6pQ8a/DvJJDP7odZz7DlFJM7LbItqtmsF8HhtaYTvVnyMtvn3mz5N0IWKZxcl4WTvTqkHBsVN9SQW4p+EXHStmbG5fMsLnRBo9oGfyfshmiHupSwAA0UTa19ObGfVqmncE08kEcBsErwES/yTGMGfb9sjh/oqH5XeHqyzkwQebqbfSU/Ig5oqza9JgYF3Qj1ff/qRGIc6vcoe6/0VMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Received: from CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) by PH7PR11MB8598.namprd11.prod.outlook.com (2603:10b6:510:2fd::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.22; Thu, 1 Feb 2024 03:31:14 +0000 Received: from CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::5d84:2b01:5029:9315]) by CO6PR11MB5602.namprd11.prod.outlook.com ([fe80::5d84:2b01:5029:9315%7]) with mapi id 15.20.7249.025; Thu, 1 Feb 2024 03:31:14 +0000 From: Qi.Chen@windriver.com To: bitbake-devel@lists.openembedded.org Subject: [bitbake-devel][PATCH 3/3] fetch2/git.py: add comment in try_premirrors Date: Wed, 31 Jan 2024 19:31:00 -0800 Message-ID: <20240201033100.3617421-3-Qi.Chen@windriver.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240201033100.3617421-1-Qi.Chen@windriver.com> References: <20240201033100.3617421-1-Qi.Chen@windriver.com> X-ClientProxiedBy: SJ0P220CA0013.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:41b::20) To CO6PR11MB5602.namprd11.prod.outlook.com (2603:10b6:303:13a::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR11MB5602:EE_|PH7PR11MB8598:EE_ X-MS-Office365-Filtering-Correlation-Id: 6119cccc-725c-406d-d4cd-08dc22d63dfd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: viFTyWvu9xG7ry95iCY3uIDNUZsGqX+5WXN9BhWBZlTR54jFbBAJMFrUU7uypjzIObVqLNuCUkXnAEwWMOJ0LDPh4azIzRdPqdcuFBaa4OpxKzCkCd/DGQNWZL2S+QJ6vZ7tzIwT6MLdBGHFpseBgPY2Cr5Ot7SQkfF25cV3bOP2MCODBcR5ILZf9z6Ppr/bsTW5ZYaxhZcK40O22UFe0tmEr6zVuXCHlDRR7OOVZCX0BadANo/DzGjW/E5QtcFT2a0PpF5EQxg7BH6zIWaACzwB3cqoShC2nX7hunhn9lMr6WU5NhQ3YXRgo78YcB43LRAk1ROJa0s2DiHnplM9td6k8+eJjvf/sT6mqRa9mumvVsWPfDUdUwHZDkBgL5LesI3Gykn3czIECuL2jMWkBxVPfQjfOuaISIS4JIabXyKDbuemO3VebmbEMnaD/04cTuQzvj9yXZSLGvJBgjkGr3shGZwP+OVqY8/NxsYv3belXzUz2jpA7mnXBi6OGLkrOsG/c1i4vwz7sBuWseljdBrQ2iCu8JUueGIC7rTBrCrNNP2LTiNlw0lmnRuigdZNktHlfpNMlVFsO7utnsBSJVbgb4r+lMIxatOPhR/O7OXBqzp3Tyz+dVQHJU54lmly X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR11MB5602.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376002)(39850400004)(396003)(136003)(366004)(346002)(230922051799003)(64100799003)(451199024)(186009)(1800799012)(38350700005)(26005)(6486002)(66946007)(478600001)(5660300002)(2906002)(83380400001)(9686003)(6666004)(66476007)(6506007)(8676002)(52116002)(6916009)(8936002)(316002)(2616005)(6512007)(1076003)(66556008)(38100700002)(86362001)(36756003)(41300700001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +EODGdVlBGAmTVfpJu/ug1fdfPKJtbfS23fy9SnzpkKPXYHeqXYzoaIHjTjjGnfiV/qwz38jtjR1Rdz7RogEeqcd8aNGkehRCH7ZmJA2KMG+QVu0hgW2jAGQS+2gKizMJKQygUZK+D+VAE0jX7KURwtuagu+z7YcCnlqt24h6B+oeiiZQU0R7zc9mC4ohLCGkB8IH7ooIxYc7u5IYWrT8xmsYN7xFQT1saRtsJJEM+4Oq7rgYaF4AXDkkxIfI2k343ENYz4ggbWPb5JT/pww1i9QdUl0fPTmAB/4eTOjT03xqtbdkv48IzfsuHsIpX/j1+Pv3rYwOW1M4pAwIVDe1SbW52qefOH4pe+KNP7m63bRY9klIp1/TlXe/zwD8qOmYDWnnXp6pSYHtP1tpx8CXGFjvYzzc2IqTo6ycBMeX06BNxGqEWaOnyUJS93bYbxI1xTI33dVcmuXyKia8CBwXIm8NP4Fv6zyCGl/Oxkh9719gTul0gx3PcLxSdnpzOeHQVdMAdcs8Ezimr0h7qG7kN/pLR3AWVzObjo0ym4Tj33JwQP0L40h6owg6mVH21JOX85Vsdg+QyQmdd5qWEuw5ywWtfHoUyjJPPSojimcjKo2BMKipGsNkwenVU01Cvbm99NRGGBKyviABeMJQlY1YExwYE/DI5P+eJUxQ9OR8L+1tDVxlKF+gbn74ogW4onJ4rOwGKFmxbuuceMJ78o3nrLoJnEE1Wz13sCmYijjZ5rGSjMcGFUUZdIldAETwCmGkkNV6+THvBknd1VZUu4rsF09A8Ycj8rhVSV069ZWzi7FB56PGkcLaVDAfFwYCDRDy2JSVRZkHas8B6AvA/+AldCNZHag/4zf/I6dOgIwd/SmlHdE6Ik4UUkFfko9VJ9dOx1kTHGbsYIADQO7di2SQUiF7moQwVMN9z7A9GQdIcrjjAn4xrXIyjwQxpzJRJGplEfW/9rNhC8iF5ei9tQ1aDSHaQmG61s8Y7jJZ7jg/kowVt/dDllqf7vTDGnhtdfvgmqJx8gA7+pey1DxwmJ9eUG+vlZJvSw/7XgdNn7NAgeooKmhJxUZVhdSuC3jpF4IleuIsm3N3BDOiGNs+Q6aGI2+yKuSF0C3Rq+Z5ojPrlEaP0W+8jLEdOXfq5fKk1nALYf20nYj/t9L/F5s0pti/ssMXhvuEVLgij2f2ln+jvMD3T3jjmglOU9DY9H6rnR++gobZMNS2LJLSmqjlNXeAcDmAR1DusqzjemF5f1XqAz6GLgnzrQd2yYnknAGOBrn+qQr2fZh+OWk7yYMfw673eMmNdsDc4tv2mwvv7CsbPa/nGgklweNoGIJ8y/vmKS+G/ioCnTGEqhi054wMnKGm7yiTWHEE4qFWJrzASt2W0pMVl8utkLwJUngYVBKjgZdtGUWgowCWUZLNbwVhCh/fGvzBmDgMBAppVnhIezBwkSa87uKziDymrrRkkvMyX0ccJm/C+OoKxylajdZBpL1RlC5BWy0raW50Yq74AzZTrrYuD7w3Aqg808f9vlGurTfQD5lQw0WfkqawhZCNaC/BmzntGBYdVx5TCkFR/Ouwora9pljmRlQXNw/yq+svdv6 X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6119cccc-725c-406d-d4cd-08dc22d63dfd X-MS-Exchange-CrossTenant-AuthSource: CO6PR11MB5602.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2024 03:31:13.9729 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: itwSQPDIFtIyqTugfal5BeJjdGACrTG1DKICTemgHLvyUjTCQjOic/jgeTxP569isgiTifvyhD5qzkeer7HS2Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8598 X-Proofpoint-GUID: Dt7MZb0qM1IW3St4oomVvGo4TBzpIpEQ X-Proofpoint-ORIG-GUID: Dt7MZb0qM1IW3St4oomVvGo4TBzpIpEQ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-01-31_10,2024-01-31_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 adultscore=0 mlxlogscore=736 lowpriorityscore=0 bulkscore=0 malwarescore=0 spamscore=0 phishscore=0 impostorscore=0 clxscore=1015 priorityscore=1501 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2401190000 definitions=main-2402010026 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 01 Feb 2024 03:31:18 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15818 From: Chen Qi The purpose of ensuring 'incremental fetch' is not easy to see from the codes. So add comments to explain this. Signed-off-by: Chen Qi --- bitbake/lib/bb/fetch2/git.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 322a9366da..fa8c597ea2 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -361,6 +361,11 @@ class Git(FetchMethod): # we need to try premirrors first as using upstream is destined to fail. if not trusted_network(d, ud.url): return True + # the following check is to ensure incremental fetch in downloads, this is + # because the premirror might be old and does not contain the new rev required, + # and this will cause a total removal and new clone. So if we can reach to + # network, we prefer upstream over premirror, though the premirror might contain + # the new rev. if os.path.exists(ud.clonedir): return False return True