| Submitter | Richard Purdie |
|---|---|
| Date | Nov. 16, 2012, 3:32 p.m. |
| Message ID | <1353079960.3709.93.camel@ted> |
| Download | mbox | patch |
| Permalink | /patch/39151/ |
| State | Accepted |
| Commit | b43faba37816817edc5240a139361d16e07c6131 |
| Headers | show |
Comments
On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > This is a first attempt at logic to determine when a sstate dependency > needs > to be installed and when it does not. Its a start at the logic and errs on > the > s/errs/errors/ > side of caution, as it gets wider testing, we can refine the logic as > needed. > > This code should allow a significant performance speedup to certain > workflows, for > example "bitbake xxx-image -c rootfs" will not populate the target sysroot. > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > index c085c62..470e0ee 100644 > --- a/meta/classes/sstate.bbclass > +++ b/meta/classes/sstate.bbclass > @@ -620,3 +620,55 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, > sq_hashfn, d): > > return ret > > +BB_SETSCENE_DEPVALID = "setscene_depvalid" > + > +def setscene_depvalid(task, taskdependees, notneeded, d): > + # taskdependees is a dict of tasks which depend on task, each being a > 3 item list of [PN, TASKNAME, FILENAME] > + # task is included in taskdependees too > + > + bb.debug(2, "Considering setscene task: %s" % > (str(taskdependees[task]))) > + > + def isNative(x): > + return x.endswith("-native") > + def isNativeCross(x): > + return x.endswith("-native") or x.endswith("-cross") or > x.endswith("-cross-initial") > + def isSafeDep(x): > + if x in ["quilt-native", "autoconf-native", "automake-native", > "gnu-config-native", "libtool-native", "pkgconfig-native", "gcc-cross", > "binutils-cross"]: > + return True > + return False > + > + # We can skip these "safe" dependencies since the aren't runtime > dependencies, just build time > + if isSafeDep(taskdependees[task][0]) and taskdependees[task][1] == > "do_populate_sysroot": > + return True > + > + # We only need to trigger populate_lic through direct dependencies > + if taskdependees[task][1] == "do_populate_lic": > + return True > + > + for dep in taskdependees: > + bb.debug(2, " considering dependency: %s" % > (str(taskdependees[dep]))) > + if task == dep: > + continue > + if dep in notneeded: > + continue > + # do_package_write_* and do_package doesn't need do_package > + if taskdependees[task][1] == "do_package" and > taskdependees[dep][1] in ['do_package', 'do_package_write_deb', > 'do_package_write_ipk', 'do_package_write_rpm']: > + continue > I understand why you does this but it would be better if we had a central place to have the do_package_write_$pkg recorded or when adding new type this could be forgotten. > + # do_package_write_* and do_package doesn't need > do_populate_sysroot > + if taskdependees[task][1] == "do_populate_sysroot" and > taskdependees[dep][1] in ['do_package', 'do_package_write_deb', > 'do_package_write_ipk', 'do_package_write_rpm']: > + continue > + # Native/Cross packages don't exist and are noexec anyway > + if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] > in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']: > + continue > + # Native/Cross populate_sysroot need their dependencies > + if isNativeCross(taskdependees[task][0]) and > isNativeCross(taskdependees[dep][0]) and taskdependees[task][1] == > 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot': > + return False > + # Target populate_sysroot do not need their dependencies > + if taskdependees[task][1] == 'do_populate_sysroot' and > taskdependees[dep][1] == 'do_populate_sysroot': > + continue > + > + # Safe fallthrough default > + bb.debug(2, " Default setscene dependency fall through due to > dependency: %s" % (str(taskdependees[dep]))) > + return False > + return True > + >
On 16 November 2012 15:58, Otavio Salvador <otavio@ossystems.com.br> wrote: >> Its a start at the logic and errs on the > > s/errs/errors/ http://www.thefreedictionary.com/errs+on+the+side Ross
On Fri, 2012-11-16 at 13:58 -0200, Otavio Salvador wrote: > On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > This is a first attempt at logic to determine when a sstate > dependency needs > to be installed and when it does not. Its a start at the logic > and errs on the > > s/errs/errors/ It is actually meant to be errs :) > + # do_package_write_* and do_package doesn't need > do_package > + if taskdependees[task][1] == "do_package" and > taskdependees[dep][1] in ['do_package', > 'do_package_write_deb', 'do_package_write_ipk', > 'do_package_write_rpm']: > + continue > > I understand why you does this but it would be better if we had a > central place to have the do_package_write_$pkg recorded or when > adding new type this could be forgotten. Its a nice idea but I can't see any practical way to do this at the moment. Cheers, Richard
On Fri, Nov 16, 2012 at 2:09 PM, Burton, Ross <ross.burton@intel.com> wrote: > On 16 November 2012 15:58, Otavio Salvador <otavio@ossystems.com.br> > wrote: > >> Its a start at the logic and errs on the > > > > s/errs/errors/ > > http://www.thefreedictionary.com/errs+on+the+side Cool!
On Fri, Nov 16, 2012 at 2:12 PM, Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Fri, 2012-11-16 at 13:58 -0200, Otavio Salvador wrote: > > On Fri, Nov 16, 2012 at 1:32 PM, Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > This is a first attempt at logic to determine when a sstate > > dependency needs > > to be installed and when it does not. Its a start at the logic > > and errs on the > > > > s/errs/errors/ > > It is actually meant to be errs :) > > > + # do_package_write_* and do_package doesn't need > > do_package > > + if taskdependees[task][1] == "do_package" and > > taskdependees[dep][1] in ['do_package', > > 'do_package_write_deb', 'do_package_write_ipk', > > 'do_package_write_rpm']: > > + continue > > > > I understand why you does this but it would be better if we had a > > central place to have the do_package_write_$pkg recorded or when > > adding new type this could be forgotten. > > Its a nice idea but I can't see any practical way to do this at the > moment. > We might hava a variable that provide all package backends names? Like: PACKAGE_BACKENDS = "ipk deb rpm" ?
On Fri, 2012-11-16 at 14:22 -0200, Otavio Salvador wrote: > On Fri, Nov 16, 2012 at 2:12 PM, Richard Purdie > > + # do_package_write_* and do_package doesn't > need > > do_package > > + if taskdependees[task][1] == "do_package" > and > > taskdependees[dep][1] in ['do_package', > > 'do_package_write_deb', 'do_package_write_ipk', > > 'do_package_write_rpm']: > > + continue > > > > I understand why you does this but it would be better if we > had a > > central place to have the do_package_write_$pkg recorded or > when > > adding new type this could be forgotten. > > Its a nice idea but I can't see any practical way to do this > at the > moment. > > We might hava a variable that provide all package backends names? > Like: > > PACKAGE_BACKENDS = "ipk deb rpm" That works for some usages but not where the expressions are in calls to things like addtask. I think there are other problems worth spending time on rather that this which at best will probably cover 50% of the usages. Cheers, Richard
Patch
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index c085c62..470e0ee 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -620,3 +620,55 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d): return ret +BB_SETSCENE_DEPVALID = "setscene_depvalid" + +def setscene_depvalid(task, taskdependees, notneeded, d): + # taskdependees is a dict of tasks which depend on task, each being a 3 item list of [PN, TASKNAME, FILENAME] + # task is included in taskdependees too + + bb.debug(2, "Considering setscene task: %s" % (str(taskdependees[task]))) + + def isNative(x): + return x.endswith("-native") + def isNativeCross(x): + return x.endswith("-native") or x.endswith("-cross") or x.endswith("-cross-initial") + def isSafeDep(x): + if x in ["quilt-native", "autoconf-native", "automake-native", "gnu-config-native", "libtool-native", "pkgconfig-native", "gcc-cross", "binutils-cross"]: + return True + return False + + # We can skip these "safe" dependencies since the aren't runtime dependencies, just build time + if isSafeDep(taskdependees[task][0]) and taskdependees[task][1] == "do_populate_sysroot": + return True + + # We only need to trigger populate_lic through direct dependencies + if taskdependees[task][1] == "do_populate_lic": + return True + + for dep in taskdependees: + bb.debug(2, " considering dependency: %s" % (str(taskdependees[dep]))) + if task == dep: + continue + if dep in notneeded: + continue + # do_package_write_* and do_package doesn't need do_package + if taskdependees[task][1] == "do_package" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']: + continue + # do_package_write_* and do_package doesn't need do_populate_sysroot + if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']: + continue + # Native/Cross packages don't exist and are noexec anyway + if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']: + continue + # Native/Cross populate_sysroot need their dependencies + if isNativeCross(taskdependees[task][0]) and isNativeCross(taskdependees[dep][0]) and taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot': + return False + # Target populate_sysroot do not need their dependencies + if taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot': + continue + + # Safe fallthrough default + bb.debug(2, " Default setscene dependency fall through due to dependency: %s" % (str(taskdependees[dep]))) + return False + return True +
This is a first attempt at logic to determine when a sstate dependency needs to be installed and when it does not. Its a start at the logic and errs on the side of caution, as it gets wider testing, we can refine the logic as needed. This code should allow a significant performance speedup to certain workflows, for example "bitbake xxx-image -c rootfs" will not populate the target sysroot. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> ---