[v2,0/3] oeqa/utils/gitarchive: fix tag name computation

Hello,
this series brings a fix to a sporadic tag push issue observed in
autobuilder. The bug is documented in bugzilla #15140 ([1]). Basically,
whenever the autobuilder creates a new tag on test results, it is only
aware of "local" tags, which is kind of faulty since used repository is a
shallow clone.
A first attempt has been merged a few days ago, but quickly reverted
because of a new issue linked to the fact that some use cases involve git
repositories without any remote configured, which makes the new git-ls
remote strategy fail (detected by Richard, see [2])

V2:
- mutualize ls-remote usage in a "get_tags" helper
- add a fallback case in the helper if no remote is configured (use push
  url if provided

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15140
[2] https://lore.kernel.org/openembedded-core/16b82ce9979d4a09ad3dc42ebef2cbb85c1729c3.camel@linuxfoundation.org/

Alexis Lothoré (3):
  oeqa/selftest: introduce gitarchive tests
  oeqa/utils/gitarchive: fix tag computation when creating archive
  oeqa/selftest/gitarchive: add tests about tags lisiting when no remote
    is configured

 .../oeqa/selftest/cases/gitarchivetests.py    | 127 ++++++++++++++++++
 meta/lib/oeqa/utils/gitarchive.py             |  36 ++++-
 2 files changed, 159 insertions(+), 4 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/gitarchivetests.py

Hello Richard,

On 8/18/23 16:17, Alexis Lothoré via lists.openembedded.org wrote:
> From: Alexis Lothoré <alexis.lothore@bootlin.com>
> 
> Add specific tests on gitarchive for when tag listing is required but no
> remote is configured in target directory: it should either succeed if valid
> url is provided, or fail is url is not provided or wrong
> 
> Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
> ---
>  .../oeqa/selftest/cases/gitarchivetests.py    | 35 +++++++++++++++++--
>  1 file changed, 33 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/cases/gitarchivetests.py b/meta/lib/oeqa/selftest/cases/gitarchivetests.py
> index 4f7acd3311d5..11b88daab860 100644
> --- a/meta/lib/oeqa/selftest/cases/gitarchivetests.py
> +++ b/meta/lib/oeqa/selftest/cases/gitarchivetests.py
> @@ -10,6 +10,7 @@ basepath = os.path.abspath(os.path.dirname(__file__) + '/../../../../../')
>  lib_path = basepath + '/scripts/lib'
>  sys.path = sys.path + [lib_path]
>  import oeqa.utils.gitarchive as ga
> +from oeqa.utils.git import GitError
>  import tempfile
>  import shutil
>  import scriptutils
> @@ -17,7 +18,7 @@ from oeqa.selftest.case import OESelftestTestCase
>  
>  logger = scriptutils.logger_create('resulttool')
>  
> -def create_fake_repository(commit, tag_list=[]):
> +def create_fake_repository(commit, tag_list=[], add_remote=True):
>      """ Create a testing git directory
>  
>      Initialize a simple git repository with one initial commit, and as many
> @@ -31,7 +32,8 @@ def create_fake_repository(commit, tag_list=[]):
>      fake_data_file = "fake_data.txt"
>      tempdir = tempfile.mkdtemp(prefix='fake_results.')
>      repo = ga.init_git_repo(tempdir, False, False, logger)
> -    repo.run_cmd(["remote", "add", "origin", "."])
> +    if add_remote:
> +        repo.run_cmd(["remote", "add", "origin", "."])
>      with open(os.path.join(tempdir, fake_data_file), "w") as fake_data:
>          fake_data.write("Fake data")
>      if commit:
> @@ -94,3 +96,32 @@ class GitArchiveTests(OESelftestTestCase):
>          self.assertEqual(len(revs[0].tags), 2)
>          self.assertEqual(revs[0].tags, ['main/10-g0f7d5df/0', 'main/10-g0f7d5df/1'])
>          delete_fake_repository(path)
> +
> +    def test_get_tags_without_valid_remote(self):
> +        url = 'git://git.yoctoproject.org/poky'
> +        path, git_obj = create_fake_repository(False, None, False)
> +
> +        tags = ga.get_tags(git_obj, pattern="yocto-*", url=url)
> +        """Test for some well established tags (released tags)"""
> +        self.assertIn("yocto-4.0", tags)
> +        self.assertIn("yocto-4.1", tags)
> +        self.assertIn("yocto-4.2", tags)
> +        delete_fake_repository(path)
> +
> +    def test_get_tags_without_valid_remote_neither_url(self):
> +        url = 'git://git.yoctoproject.org/poky'
> +        path, git_obj = create_fake_repository(False, None, False)
> +
> +        """Test for some well established tags (released tags)"""
> +        with self.assertRaises(GitError):
> +            tags = ga.get_tags(git_obj, pattern="yocto-*")
> +        delete_fake_repository(path)
> +
> +    def test_get_tags_without_valid_remote_and_wrong_url(self):
> +        url = 'git://git.foo.org/bar'
> +        path, git_obj = create_fake_repository(False, None, False)
> +
> +        """Test for some well established tags (released tags)"""
> +        with self.assertRaises(GitError):
> +            tags = ga.get_tags(git_obj, pattern="yocto-*", url=url)
> +        delete_fake_repository(path)

Now that this v2 has been merged, I observe that the last raised failure is now
passed, but a new one occurs a bit later, because I missed some use cases where
gitarchive APIs that can be called without a gitarchive object being created before:

https://autobuilder.yoctoproject.org/typhoon/#/builders/133/builds/2125/steps/12/logs/stdio

Sorry for the disturbance, I am working on it right now.

> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186369): https://lists.openembedded.org/g/openembedded-core/message/186369
> Mute This Topic: https://lists.openembedded.org/mt/100821194/7394887
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexis.lothore@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>