[bitbake-devel,4/5] fetch2/gitsm: Add clean function

Submitted by Robert Yang on March 14, 2019, 9:28 a.m. | Patch ID: 159570

Details

Message ID 8074fc829447468b0f42672f17f31b83d3f3888b.1552555629.git.liezhi.yang@windriver.com
State New
Headers show

Commit Message

Robert Yang March 14, 2019, 9:28 a.m.
The git's clean can only remove parent git repo from DL_DIR, but doesn't remove
submodules, this patch fixes the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

Patch hide | download patch | download mbox

diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index b21fed2..bb61093 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -76,8 +76,8 @@  class GitSM(Git):
         for name in ud.names:
             try:
                 gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
-            except:
-                # No submodules to update
+            except Exception as esc:
+                logger.info("No submodules found: %s" % workdir)
                 continue
 
             for m, md in parse_gitmodules(gitmodules).items():
@@ -164,6 +164,17 @@  class GitSM(Git):
         Git.download(self, ud, d)
         self.process_submodules(ud, ud.clonedir, download_submodule, d)
 
+    def clean(self, ud, d):
+        def clean_submodule(ud, url, module, modpath, d):
+            try:
+                newfetch = Fetch([url], d, cache=False)
+                newfetch.clean()
+            except Exception as e:
+                logger.warn('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e)))
+
+        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
+        Git.clean(self, ud, d)
+
     def unpack(self, ud, destdir, d):
         def unpack_submodules(ud, url, module, modpath, d):
             url += ";bareclone=1;nobranch=1"

Comments

Lisicki, Raphael March 14, 2019, 12:34 p.m.
I have not fully understand the logic but I am wondering what happens here is a submodule is located in two different repositories? Wouldn't a clean of one module delete submodules of the other one as well?

Best regards
Raphael

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
> bounces@lists.openembedded.org> On Behalf Of Robert Yang
> Sent: Thursday, March 14, 2019 10:28 AM
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH 4/5] fetch2/gitsm: Add clean function
> 
> The git's clean can only remove parent git repo from DL_DIR, but doesn't
> remove
> submodules, this patch fixes the problem.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
> index b21fed2..bb61093 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -76,8 +76,8 @@ class GitSM(Git):
>          for name in ud.names:
>              try:
>                  gitmodules = runfetchcmd("%s show %s:.gitmodules" %
> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
> -            except:
> -                # No submodules to update
> +            except Exception as esc:
> +                logger.info("No submodules found: %s" % workdir)
>                  continue
> 
>              for m, md in parse_gitmodules(gitmodules).items():
> @@ -164,6 +164,17 @@ class GitSM(Git):
>          Git.download(self, ud, d)
>          self.process_submodules(ud, ud.clonedir, download_submodule, d)
> 
> +    def clean(self, ud, d):
> +        def clean_submodule(ud, url, module, modpath, d):
> +            try:
> +                newfetch = Fetch([url], d, cache=False)
> +                newfetch.clean()
> +            except Exception as e:
> +                logger.warn('gitsm: submodule clean failed: %s %s' %
> (type(e).__name__, str(e)))
> +
> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
> +        Git.clean(self, ud, d)
> +
>      def unpack(self, ud, destdir, d):
>          def unpack_submodules(ud, url, module, modpath, d):
>              url += ";bareclone=1;nobranch=1"
> --
> 2.7.4
> 
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Mark Hatle March 14, 2019, 2:41 p.m.
On 3/14/19 4:28 AM, Robert Yang wrote:
> The git's clean can only remove parent git repo from DL_DIR, but doesn't remove
> submodules, this patch fixes the problem.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
> index b21fed2..bb61093 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -76,8 +76,8 @@ class GitSM(Git):
>          for name in ud.names:
>              try:
>                  gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
> -            except:
> -                # No submodules to update
> +            except Exception as esc:
> +                logger.info("No submodules found: %s" % workdir)

We don't want the log message.  git submodules are processed recursively, and
the end of the recursion will -always- have 'gitsm' items that have no
submodules.  This is also why it was a generic except, because we really don't
care WHAT the exception is, only that if one happens -- we know the command
failed and to simply ignore it and finish the iteration.

                  continue
>  
>              for m, md in parse_gitmodules(gitmodules).items():
> @@ -164,6 +164,17 @@ class GitSM(Git):
>          Git.download(self, ud, d)
>          self.process_submodules(ud, ud.clonedir, download_submodule, d)
>  
> +    def clean(self, ud, d):
> +        def clean_submodule(ud, url, module, modpath, d):
> +            try:
> +                newfetch = Fetch([url], d, cache=False)
> +                newfetch.clean()
> +            except Exception as e:
> +                logger.warn('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e)))
> +
> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
> +        Git.clean(self, ud, d)
> +

I don't see 'clean_submodule' implemented in this patch.  The process_submodules
will simply call that function, if it's not defined it will error or not do
anything if it finds a submodule.

>      def unpack(self, ud, destdir, d):
>          def unpack_submodules(ud, url, module, modpath, d):
>              url += ";bareclone=1;nobranch=1"
>
Robert Yang March 15, 2019, 2:50 a.m.
Hi,

On 3/14/19 8:34 PM, Lisicki, Raphael wrote:
> I have not fully understand the logic but I am wondering what happens here is a submodule is located in two different repositories? Wouldn't a clean of one module delete submodules of the other one as well?

Here are more explanations about the logic, for example, repo_A has two
submodules repo_B and repo_C, then after do_fetch,
they will be"
DL_DIR/repo_A
DL_DIR/repo_B
DL_DIR/repo_C

Without this patch, do_cleanall only removes DL_DIR/repo_A, but leaves
repo_B and repo_C there, this patch makes it remove repo_B and C.


For your question, if repo_B is submodule of both repo_A and repo_D, it
is like foo and foo-native use the same git source, when you run
"bitbake foo -cccleanall", both foo and foo-native's source will be
removed since they use the same source, the similar to gitsm, so it is not
a problem, the source will be re-downloaded automatically when needed.

// Robert

> 
> Best regards
> Raphael
> 
>> -----Original Message-----
>> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>> bounces@lists.openembedded.org> On Behalf Of Robert Yang
>> Sent: Thursday, March 14, 2019 10:28 AM
>> To: bitbake-devel@lists.openembedded.org
>> Subject: [bitbake-devel] [PATCH 4/5] fetch2/gitsm: Add clean function
>>
>> The git's clean can only remove parent git repo from DL_DIR, but doesn't
>> remove
>> submodules, this patch fixes the problem.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>>   1 file changed, 13 insertions(+), 2 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>> index b21fed2..bb61093 100644
>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> @@ -76,8 +76,8 @@ class GitSM(Git):
>>           for name in ud.names:
>>               try:
>>                   gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>> -            except:
>> -                # No submodules to update
>> +            except Exception as esc:
>> +                logger.info("No submodules found: %s" % workdir)
>>                   continue
>>
>>               for m, md in parse_gitmodules(gitmodules).items():
>> @@ -164,6 +164,17 @@ class GitSM(Git):
>>           Git.download(self, ud, d)
>>           self.process_submodules(ud, ud.clonedir, download_submodule, d)
>>
>> +    def clean(self, ud, d):
>> +        def clean_submodule(ud, url, module, modpath, d):
>> +            try:
>> +                newfetch = Fetch([url], d, cache=False)
>> +                newfetch.clean()
>> +            except Exception as e:
>> +                logger.warn('gitsm: submodule clean failed: %s %s' %
>> (type(e).__name__, str(e)))
>> +
>> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
>> +        Git.clean(self, ud, d)
>> +
>>       def unpack(self, ud, destdir, d):
>>           def unpack_submodules(ud, url, module, modpath, d):
>>               url += ";bareclone=1;nobranch=1"
>> --
>> 2.7.4
>>
>> --
>> _______________________________________________
>> bitbake-devel mailing list
>> bitbake-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Robert Yang March 15, 2019, 2:52 a.m.
Hi Mark,

On 3/14/19 10:41 PM, Mark Hatle wrote:
> On 3/14/19 4:28 AM, Robert Yang wrote:
>> The git's clean can only remove parent git repo from DL_DIR, but doesn't remove
>> submodules, this patch fixes the problem.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>>   1 file changed, 13 insertions(+), 2 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>> index b21fed2..bb61093 100644
>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> @@ -76,8 +76,8 @@ class GitSM(Git):
>>           for name in ud.names:
>>               try:
>>                   gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>> -            except:
>> -                # No submodules to update
>> +            except Exception as esc:
>> +                logger.info("No submodules found: %s" % workdir)
> 
> We don't want the log message.  git submodules are processed recursively, and
> the end of the recursion will -always- have 'gitsm' items that have no
> submodules.  This is also why it was a generic except, because we really don't
> care WHAT the exception is, only that if one happens -- we know the command
> failed and to simply ignore it and finish the iteration.
> 
>                    continue
>>   
>>               for m, md in parse_gitmodules(gitmodules).items():
>> @@ -164,6 +164,17 @@ class GitSM(Git):
>>           Git.download(self, ud, d)
>>           self.process_submodules(ud, ud.clonedir, download_submodule, d)
>>   
>> +    def clean(self, ud, d):
>> +        def clean_submodule(ud, url, module, modpath, d):
>> +            try:
>> +                newfetch = Fetch([url], d, cache=False)
>> +                newfetch.clean()
>> +            except Exception as e:
>> +                logger.warn('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e)))
>> +
>> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
>> +        Git.clean(self, ud, d)
>> +
> 
> I don't see 'clean_submodule' implemented in this patch.  The process_submodules
> will simply call that function, if it's not defined it will error or not do
> anything if it finds a submodule.


The clean_submodule is inside of 'def clean', right under 'def clean':

+    def clean(self, ud, d):
+        def clean_submodule(ud, url, module, modpath, d):

// Robert

> 
>>       def unpack(self, ud, destdir, d):
>>           def unpack_submodules(ud, url, module, modpath, d):
>>               url += ";bareclone=1;nobranch=1"
>>
> 
>
Robert Yang March 15, 2019, 3:48 a.m.
Hi Mark,

On 3/14/19 10:41 PM, Mark Hatle wrote:
> On 3/14/19 4:28 AM, Robert Yang wrote:
>> The git's clean can only remove parent git repo from DL_DIR, but doesn't remove
>> submodules, this patch fixes the problem.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>>   1 file changed, 13 insertions(+), 2 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>> index b21fed2..bb61093 100644
>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> @@ -76,8 +76,8 @@ class GitSM(Git):
>>           for name in ud.names:
>>               try:
>>                   gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>> -            except:
>> -                # No submodules to update
>> +            except Exception as esc:
>> +                logger.info("No submodules found: %s" % workdir)
> 
> We don't want the log message.  git submodules are processed recursively, and
> the end of the recursion will -always- have 'gitsm' items that have no
> submodules.  This is also why it was a generic except, because we really don't
> care WHAT the exception is, only that if one happens -- we know the command
> failed and to simply ignore it and finish the iteration.

Sorry, I forgot to reply on this just now. Submodule is a little different and
complicated then other SCMs, for exmaple, when I worked on gitsm' clean()
function, it always didn't work which puzzled me a lot, at last I found that
I had called Git.clean() before self.process_submodules:

Git.clean(self, ud, d)
self.process_submodules(ud, ud.clonedir, clean_submodule, d)

So that the parent repo had gone before submodules, but there was no message
told me this, it just failed quietly, so I had to debug it line by line.

The logger.info() goes into log.do_fetch which won't affect users by default,
but it would help to understand what happend when they need debug.

// Robert

> 
>                    continue
>>   
>>               for m, md in parse_gitmodules(gitmodules).items():
>> @@ -164,6 +164,17 @@ class GitSM(Git):
>>           Git.download(self, ud, d)
>>           self.process_submodules(ud, ud.clonedir, download_submodule, d)
>>   
>> +    def clean(self, ud, d):
>> +        def clean_submodule(ud, url, module, modpath, d):
>> +            try:
>> +                newfetch = Fetch([url], d, cache=False)
>> +                newfetch.clean()
>> +            except Exception as e:
>> +                logger.warn('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e)))
>> +
>> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
>> +        Git.clean(self, ud, d)
>> +
> 
> I don't see 'clean_submodule' implemented in this patch.  The process_submodules
> will simply call that function, if it's not defined it will error or not do
> anything if it finds a submodule.
> 
>>       def unpack(self, ud, destdir, d):
>>           def unpack_submodules(ud, url, module, modpath, d):
>>               url += ";bareclone=1;nobranch=1"
>>
> 
>
Mark Hatle March 15, 2019, 3:44 p.m.
On 3/14/19 9:50 PM, Robert Yang wrote:
> Hi,
> 
> On 3/14/19 8:34 PM, Lisicki, Raphael wrote:
>> I have not fully understand the logic but I am wondering what happens here is a submodule is located in two different repositories? Wouldn't a clean of one module delete submodules of the other one as well?
> 
> Here are more explanations about the logic, for example, repo_A has two
> submodules repo_B and repo_C, then after do_fetch,
> they will be"
> DL_DIR/repo_A
> DL_DIR/repo_B
> DL_DIR/repo_C
> 
> Without this patch, do_cleanall only removes DL_DIR/repo_A, but leaves
> repo_B and repo_C there, this patch makes it remove repo_B and C.

The issue Raphael is mentioning is when you have multiple projects that use the
same submodules.  I.e.:

recipeA:
 DL_DIR/repo_A
 DL_DIR/repo_B
 DL_DIR/repo_C

recipeB:
 DL_DIR/repo_D
 DL_DIR/repo_C

If you do a clean of A, you will end up removing repo_C which both items end up
using.

I don't know any way to resolve this, since the dependency is implicit, not
explicit -- so the system won't have any idea that the same thing is used in
multiple places.

> 
> For your question, if repo_B is submodule of both repo_A and repo_D, it
> is like foo and foo-native use the same git source, when you run
> "bitbake foo -cccleanall", both foo and foo-native's source will be
> removed since they use the same source, the similar to gitsm, so it is not
> a problem, the source will be re-downloaded automatically when needed.

I'm not sure if this situation will work, since the recipeB will have a stamp
that says the download is complete, even though part of it is now missing.

--Mark

> // Robert
> 
>>
>> Best regards
>> Raphael
>>
>>> -----Original Message-----
>>> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>>> bounces@lists.openembedded.org> On Behalf Of Robert Yang
>>> Sent: Thursday, March 14, 2019 10:28 AM
>>> To: bitbake-devel@lists.openembedded.org
>>> Subject: [bitbake-devel] [PATCH 4/5] fetch2/gitsm: Add clean function
>>>
>>> The git's clean can only remove parent git repo from DL_DIR, but doesn't
>>> remove
>>> submodules, this patch fixes the problem.
>>>
>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>> ---
>>>   bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>>>   1 file changed, 13 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>>> index b21fed2..bb61093 100644
>>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>>> @@ -76,8 +76,8 @@ class GitSM(Git):
>>>           for name in ud.names:
>>>               try:
>>>                   gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>>> -            except:
>>> -                # No submodules to update
>>> +            except Exception as esc:
>>> +                logger.info("No submodules found: %s" % workdir)
>>>                   continue
>>>
>>>               for m, md in parse_gitmodules(gitmodules).items():
>>> @@ -164,6 +164,17 @@ class GitSM(Git):
>>>           Git.download(self, ud, d)
>>>           self.process_submodules(ud, ud.clonedir, download_submodule, d)
>>>
>>> +    def clean(self, ud, d):
>>> +        def clean_submodule(ud, url, module, modpath, d):
>>> +            try:
>>> +                newfetch = Fetch([url], d, cache=False)
>>> +                newfetch.clean()
>>> +            except Exception as e:
>>> +                logger.warn('gitsm: submodule clean failed: %s %s' %
>>> (type(e).__name__, str(e)))
>>> +
>>> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
>>> +        Git.clean(self, ud, d)
>>> +
>>>       def unpack(self, ud, destdir, d):
>>>           def unpack_submodules(ud, url, module, modpath, d):
>>>               url += ";bareclone=1;nobranch=1"
>>> --
>>> 2.7.4
>>>
>>> --
>>> _______________________________________________
>>> bitbake-devel mailing list
>>> bitbake-devel@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Mark Hatle March 15, 2019, 3:46 p.m.
On 3/14/19 9:52 PM, Robert Yang wrote:
> Hi Mark,
> 
> On 3/14/19 10:41 PM, Mark Hatle wrote:
>> On 3/14/19 4:28 AM, Robert Yang wrote:
>>> The git's clean can only remove parent git repo from DL_DIR, but doesn't remove
>>> submodules, this patch fixes the problem.
>>>
>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>> ---
>>>   bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>>>   1 file changed, 13 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>>> index b21fed2..bb61093 100644
>>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>>> @@ -76,8 +76,8 @@ class GitSM(Git):
>>>           for name in ud.names:
>>>               try:
>>>                   gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>>> -            except:
>>> -                # No submodules to update
>>> +            except Exception as esc:
>>> +                logger.info("No submodules found: %s" % workdir)
>>
>> We don't want the log message.  git submodules are processed recursively, and
>> the end of the recursion will -always- have 'gitsm' items that have no
>> submodules.  This is also why it was a generic except, because we really don't
>> care WHAT the exception is, only that if one happens -- we know the command
>> failed and to simply ignore it and finish the iteration.
>>
>>                    continue
>>>   
>>>               for m, md in parse_gitmodules(gitmodules).items():
>>> @@ -164,6 +164,17 @@ class GitSM(Git):
>>>           Git.download(self, ud, d)
>>>           self.process_submodules(ud, ud.clonedir, download_submodule, d)
>>>   
>>> +    def clean(self, ud, d):
>>> +        def clean_submodule(ud, url, module, modpath, d):
>>> +            try:
>>> +                newfetch = Fetch([url], d, cache=False)
>>> +                newfetch.clean()
>>> +            except Exception as e:
>>> +                logger.warn('gitsm: submodule clean failed: %s %s' % (type(e).__name__, str(e)))
>>> +
>>> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
>>> +        Git.clean(self, ud, d)
>>> +
>>
>> I don't see 'clean_submodule' implemented in this patch.  The process_submodules
>> will simply call that function, if it's not defined it will error or not do
>> anything if it finds a submodule.
> 
> 
> The clean_submodule is inside of 'def clean', right under 'def clean':
> 
> +    def clean(self, ud, d):
> +        def clean_submodule(ud, url, module, modpath, d):

Apparently I'm blind.  :)

--Mark

> // Robert
> 
>>
>>>       def unpack(self, ud, destdir, d):
>>>           def unpack_submodules(ud, url, module, modpath, d):
>>>               url += ";bareclone=1;nobranch=1"
>>>
>>
>>
Robert Yang March 19, 2019, 6:41 a.m.
Hi Mark,

On 3/15/19 11:44 PM, Mark Hatle wrote:
> On 3/14/19 9:50 PM, Robert Yang wrote:
>> Hi,
>>
>> On 3/14/19 8:34 PM, Lisicki, Raphael wrote:
>>> I have not fully understand the logic but I am wondering what happens here is a submodule is located in two different repositories? Wouldn't a clean of one module delete submodules of the other one as well?
>>
>> Here are more explanations about the logic, for example, repo_A has two
>> submodules repo_B and repo_C, then after do_fetch,
>> they will be"
>> DL_DIR/repo_A
>> DL_DIR/repo_B
>> DL_DIR/repo_C
>>
>> Without this patch, do_cleanall only removes DL_DIR/repo_A, but leaves
>> repo_B and repo_C there, this patch makes it remove repo_B and C.
> 
> The issue Raphael is mentioning is when you have multiple projects that use the
> same submodules.  I.e.:
> 
> recipeA:
>   DL_DIR/repo_A
>   DL_DIR/repo_B
>   DL_DIR/repo_C
> 
> recipeB:
>   DL_DIR/repo_D
>   DL_DIR/repo_C
> 
> If you do a clean of A, you will end up removing repo_C which both items end up
> using.
> 
> I don't know any way to resolve this, since the dependency is implicit, not
> explicit -- so the system won't have any idea that the same thing is used in
> multiple places.
> 
>>
>> For your question, if repo_B is submodule of both repo_A and repo_D, it
>> is like foo and foo-native use the same git source, when you run
>> "bitbake foo -cccleanall", both foo and foo-native's source will be
>> removed since they use the same source, the similar to gitsm, so it is not
>> a problem, the source will be re-downloaded automatically when needed.
> 
> I'm not sure if this situation will work, since the recipeB will have a stamp
> that says the download is complete, even though part of it is now missing.


It's a generic issue when the same sources are use across recipes, e.g.:

# Cleanup everything
$ bitbake gnu-config gnu-config-native -ccleanall

# Fetch
$ bitbake gnu-config gnu-config-native -cfetch

# Cleanup the git repo
$ bitbake gnu-config -ccleanall

# Unpack gnu-config-native
$ bitbake gnu-config-native -cunpack

ERROR: gnu-config-native-20181128+gitAUTOINC+058639be22-r0 do_unpack: Unpack 
failure for URL: 'git://git.savannah.gnu.org/config.git'. No up to date source 
found: clone directory not available or not up to date: 
/buildarea1/lyang1/rebase-work/test_fetch/downloads/git2/git.savannah.gnu.org.config.git; 
shallow clone not enabled
ERROR: gnu-config-native-20181128+gitAUTOINC+058639be22-r0 do_unpack:
ERROR: gnu-config-native-20181128+gitAUTOINC+058639be22-r0 do_unpack: Function 
failed: base_do_unpack

So the users should handle it themselves in such a case.

// Robert

> 
> --Mark
> 
>> // Robert
>>
>>>
>>> Best regards
>>> Raphael
>>>
>>>> -----Original Message-----
>>>> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>>>> bounces@lists.openembedded.org> On Behalf Of Robert Yang
>>>> Sent: Thursday, March 14, 2019 10:28 AM
>>>> To: bitbake-devel@lists.openembedded.org
>>>> Subject: [bitbake-devel] [PATCH 4/5] fetch2/gitsm: Add clean function
>>>>
>>>> The git's clean can only remove parent git repo from DL_DIR, but doesn't
>>>> remove
>>>> submodules, this patch fixes the problem.
>>>>
>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>> ---
>>>>    bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>>>>    1 file changed, 13 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>>>> index b21fed2..bb61093 100644
>>>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>>>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>>>> @@ -76,8 +76,8 @@ class GitSM(Git):
>>>>            for name in ud.names:
>>>>                try:
>>>>                    gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>>>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>>>> -            except:
>>>> -                # No submodules to update
>>>> +            except Exception as esc:
>>>> +                logger.info("No submodules found: %s" % workdir)
>>>>                    continue
>>>>
>>>>                for m, md in parse_gitmodules(gitmodules).items():
>>>> @@ -164,6 +164,17 @@ class GitSM(Git):
>>>>            Git.download(self, ud, d)
>>>>            self.process_submodules(ud, ud.clonedir, download_submodule, d)
>>>>
>>>> +    def clean(self, ud, d):
>>>> +        def clean_submodule(ud, url, module, modpath, d):
>>>> +            try:
>>>> +                newfetch = Fetch([url], d, cache=False)
>>>> +                newfetch.clean()
>>>> +            except Exception as e:
>>>> +                logger.warn('gitsm: submodule clean failed: %s %s' %
>>>> (type(e).__name__, str(e)))
>>>> +
>>>> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
>>>> +        Git.clean(self, ud, d)
>>>> +
>>>>        def unpack(self, ud, destdir, d):
>>>>            def unpack_submodules(ud, url, module, modpath, d):
>>>>                url += ";bareclone=1;nobranch=1"
>>>> --
>>>> 2.7.4
>>>>
>>>> --
>>>> _______________________________________________
>>>> bitbake-devel mailing list
>>>> bitbake-devel@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
> 
>