| Message ID | 20240219155513.76738-1-luca.ceresoli@bootlin.com |
|---|---|
| State | New |
| Headers | show |
| Series | ref-manual: tasks: do_cleanall: mention it is not allowed with a shared DL_DIR | expand |
Hi Luca, On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote: > [You don't often get email from luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] > > do_cleanall can produce failures if using a shared download directory. This > is why it is forbidden when writing tests that will run on the autobuilders > (https://docs.yoctoproject.org/test-manual/intro.html?highlight=cleanall#considerations-when-writing-tests). > > Reported-by: Sam Liddicott > Link: https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335 > Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> > --- > documentation/ref-manual/tasks.rst | 13 +++++++++++++ > 1 file changed, 13 insertions(+) > > diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst > index 0db960b22f80..c754eb15514b 100644 > --- a/documentation/ref-manual/tasks.rst > +++ b/documentation/ref-manual/tasks.rst > @@ -474,6 +474,19 @@ Typically, you would not normally use the :ref:`ref-tasks-cleanall` task. Do so > if you want to start fresh with the :ref:`ref-tasks-fetch` > task. > > +.. note:: > + > + Never run the :ref:`ref-tasks-cleanall` task if your :term:`DL_DIR` is This also applies if SSTATE_DIR is shared I believe? > + shared. This would break in this scenario:: > + > + dir1$ bitbake <recipe> -c fetch > + dir2$ bitbake <recipe> -c cleanall > + dir1$ bitbake <recipe> -c unpack > + > + because before step 3 in dir1 there is a stamp file for the > + :ref:`ref-tasks-fetch` task, so the unpack task will try to extract the > + downloaded archive and fail as it has been deleted in step 2. > + > .. _ref-tasks-cleansstate: > > ``do_cleansstate`` We would need the same warning here for a shared SSTATE_DIR I believe. Cheers, Quentin
Hi Quentin, On Mon, 19 Feb 2024 17:12:46 +0100 Quentin Schulz <quentin.schulz@theobroma-systems.com> wrote: > Hi Luca, > > On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote: > > [You don't often get email from luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] > > > > do_cleanall can produce failures if using a shared download directory. This > > is why it is forbidden when writing tests that will run on the autobuilders > > (https://docs.yoctoproject.org/test-manual/intro.html?highlight=cleanall#considerations-when-writing-tests). > > > > Reported-by: Sam Liddicott > > Link: https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335 > > Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> > > --- > > documentation/ref-manual/tasks.rst | 13 +++++++++++++ > > 1 file changed, 13 insertions(+) > > > > diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst > > index 0db960b22f80..c754eb15514b 100644 > > --- a/documentation/ref-manual/tasks.rst > > +++ b/documentation/ref-manual/tasks.rst > > @@ -474,6 +474,19 @@ Typically, you would not normally use the :ref:`ref-tasks-cleanall` task. Do so > > if you want to start fresh with the :ref:`ref-tasks-fetch` > > task. > > > > +.. note:: > > + > > + Never run the :ref:`ref-tasks-cleanall` task if your :term:`DL_DIR` is > > This also applies if SSTATE_DIR is shared I believe? I don't think so. As far as I understand the sstate cache, a cache entry is either found or not found, while for the downloads there is the strange state of having a do_fetch stamp but no downloaded archive. However I may be wrong. Do you have an example command sequence where do_cleansstate can create a build failure? Luca
On Tue, 2024-02-20 at 08:58 +0100, Luca Ceresoli via lists.yoctoproject.org wrote: > Hi Quentin, > > On Mon, 19 Feb 2024 17:12:46 +0100 > Quentin Schulz <quentin.schulz@theobroma-systems.com> wrote: > > > Hi Luca, > > > > On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote: > > > [You don't often get email from > > > luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this > > > is important at https://aka.ms/LearnAboutSenderIdentification ] > > > > > > do_cleanall can produce failures if using a shared download > > > directory. This > > > is why it is forbidden when writing tests that will run on the > > > autobuilders > > > ( > > > https://docs.yoctoproject.org/test-manual/intro.html?highlight=cle > > > anall#considerations-when-writing-tests). > > > > > > Reported-by: Sam Liddicott > > > Link: > > > https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335 > > > Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> > > > --- > > > documentation/ref-manual/tasks.rst | 13 +++++++++++++ > > > 1 file changed, 13 insertions(+) > > > > > > diff --git a/documentation/ref-manual/tasks.rst > > > b/documentation/ref-manual/tasks.rst > > > index 0db960b22f80..c754eb15514b 100644 > > > --- a/documentation/ref-manual/tasks.rst > > > +++ b/documentation/ref-manual/tasks.rst > > > @@ -474,6 +474,19 @@ Typically, you would not normally use the > > > :ref:`ref-tasks-cleanall` task. Do so > > > if you want to start fresh with the :ref:`ref-tasks-fetch` > > > task. > > > > > > +.. note:: > > > + > > > + Never run the :ref:`ref-tasks-cleanall` task if your > > > :term:`DL_DIR` is > > > > This also applies if SSTATE_DIR is shared I believe? > > I don't think so. As far as I understand the sstate cache, a cache > entry is either found or not found, while for the downloads there is > the strange state of having a do_fetch stamp but no downloaded > archive. > > However I may be wrong. Do you have an example command sequence where > do_cleansstate can create a build failure? The builds check sstate "up front" then download later so it if is deleted in the meantime, it will cause an error but not a total failure as it will rebuild it. We therefore don't recommend cleansstate on a shared sstate directory. Instead you can use -f to invalidate a task. Cheers, Richard
Hi all, On 2/20/24 10:01, Richard Purdie wrote: > On Tue, 2024-02-20 at 08:58 +0100, Luca Ceresoli via > lists.yoctoproject.org wrote: >> Hi Quentin, >> >> On Mon, 19 Feb 2024 17:12:46 +0100 >> Quentin Schulz <quentin.schulz@theobroma-systems.com> wrote: >> >>> Hi Luca, >>> >>> On 2/19/24 16:55, Luca Ceresoli via lists.yoctoproject.org wrote: >>>> [You don't often get email from >>>> luca.ceresoli=bootlin.com@lists.yoctoproject.org. Learn why this >>>> is important at https://aka.ms/LearnAboutSenderIdentification ] >>>> >>>> do_cleanall can produce failures if using a shared download >>>> directory. This >>>> is why it is forbidden when writing tests that will run on the >>>> autobuilders >>>> ( >>>> https://docs.yoctoproject.org/test-manual/intro.html?highlight=cle >>>> anall#considerations-when-writing-tests). >>>> >>>> Reported-by: Sam Liddicott >>>> Link: >>>> https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335 >>>> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> >>>> --- >>>> documentation/ref-manual/tasks.rst | 13 +++++++++++++ >>>> 1 file changed, 13 insertions(+) >>>> >>>> diff --git a/documentation/ref-manual/tasks.rst >>>> b/documentation/ref-manual/tasks.rst >>>> index 0db960b22f80..c754eb15514b 100644 >>>> --- a/documentation/ref-manual/tasks.rst >>>> +++ b/documentation/ref-manual/tasks.rst >>>> @@ -474,6 +474,19 @@ Typically, you would not normally use the >>>> :ref:`ref-tasks-cleanall` task. Do so >>>> if you want to start fresh with the :ref:`ref-tasks-fetch` >>>> task. >>>> >>>> +.. note:: >>>> + >>>> + Never run the :ref:`ref-tasks-cleanall` task if your >>>> :term:`DL_DIR` is >>> >>> This also applies if SSTATE_DIR is shared I believe? >> >> I don't think so. As far as I understand the sstate cache, a cache >> entry is either found or not found, while for the downloads there is >> the strange state of having a do_fetch stamp but no downloaded >> archive. >> >> However I may be wrong. Do you have an example command sequence where >> do_cleansstate can create a build failure? > > The builds check sstate "up front" then download later so it if is > deleted in the meantime, it will cause an error but not a total failure > as it will rebuild it. > > We therefore don't recommend cleansstate on a shared sstate directory. > Instead you can use -f to invalidate a task. > And considering that cleanall also calls cleansstate (or at least does the same), it applies and we should document it :) Cheers, Quentin
diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst index 0db960b22f80..c754eb15514b 100644 --- a/documentation/ref-manual/tasks.rst +++ b/documentation/ref-manual/tasks.rst @@ -474,6 +474,19 @@ Typically, you would not normally use the :ref:`ref-tasks-cleanall` task. Do so if you want to start fresh with the :ref:`ref-tasks-fetch` task. +.. note:: + + Never run the :ref:`ref-tasks-cleanall` task if your :term:`DL_DIR` is + shared. This would break in this scenario:: + + dir1$ bitbake <recipe> -c fetch + dir2$ bitbake <recipe> -c cleanall + dir1$ bitbake <recipe> -c unpack + + because before step 3 in dir1 there is a stamp file for the + :ref:`ref-tasks-fetch` task, so the unpack task will try to extract the + downloaded archive and fail as it has been deleted in step 2. + .. _ref-tasks-cleansstate: ``do_cleansstate``
do_cleanall can produce failures if using a shared download directory. This is why it is forbidden when writing tests that will run on the autobuilders (https://docs.yoctoproject.org/test-manual/intro.html?highlight=cleanall#considerations-when-writing-tests). Reported-by: Sam Liddicott Link: https://bootlin.com/blog/yocto-sharing-the-sstate-cache-and-download-directories/#comment-2650335 Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> --- documentation/ref-manual/tasks.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+)