Patchwork [1/4] linux-yocto: make KBRANCH the exception and not the rule

login
register
mail settings
Submitter Bruce Ashfield
Date Aug. 15, 2012, 8:06 p.m.
Message ID <9adc11ebd961b769e39d0ac664998a2de014b058.1345059176.git.bruce.ashfield@windriver.com>
Download mbox | patch
Permalink /patch/34667/
State Accepted
Commit 3cac3ce65abae9dc253641a2004440a2b38fd44d
Headers show

Comments

Bruce Ashfield - Aug. 15, 2012, 8:06 p.m.
The kernel branch is no longer required by the yocto-kern-tools
to locate BSP feature descriptions (it is the MACHINE:KTYPE
descriptor), so we no longer require that the BSP branch be
explicitly set.

If a kernel branch is explicitly set, it is now used to trigger
a checks to ensure that the branch really is being built.
Otherwise the branch that the machine description creates will
be built (just as it always was).

This further simplies the use and configuration of a linux-yocto
based kernel recipe.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/classes/kernel-yocto.bbclass                  |   88 ++++++++++++++------
 .../kern-tools/kern-tools-native_git.bb            |    2 +-
 2 files changed, 63 insertions(+), 27 deletions(-)
Richard Purdie - Aug. 18, 2012, 12:07 p.m.
On Wed, 2012-08-15 at 16:06 -0400, Bruce Ashfield wrote:
> The kernel branch is no longer required by the yocto-kern-tools
> to locate BSP feature descriptions (it is the MACHINE:KTYPE
> descriptor), so we no longer require that the BSP branch be
> explicitly set.
> 
> If a kernel branch is explicitly set, it is now used to trigger
> a checks to ensure that the branch really is being built.
> Otherwise the branch that the machine description creates will
> be built (just as it always was).
> 
> This further simplies the use and configuration of a linux-yocto
> based kernel recipe.

[...]

>  
> -	# We have SRCREVs and we have branches so validation can continue!
> -	current=`git branch |grep \*|sed 's/^\* //'`
> -	if [ -n "$target_branch_head" ] && [ "$branch_head" != "$target_branch_head" ] &&
> -           [ "$target_branch_head" != "AUTOINC" ]; then
> -		ref=`git show ${target_branch_head} 2>&1 | head -n1 || true`
> -		if [ "$ref" = "fatal: bad object ${target_meta_head}" ]; then
> -			echo "ERROR ${target_branch_head} is not a valid commit ID."
> -			echo "The kernel source tree may be out of sync"
> -			exit 1
> -		else
> -			echo "Forcing branch $current to ${target_branch_head}"
> -			git branch -m $current $current-orig
> -			git checkout -b $current ${target_branch_head}
> -		fi
> +	containing_branches=`git branch --contains $target_branch_head | sed 's/^..//'`
> +	if [ -z "$containing_branches" ]; then
> +		echo "ERROR: SRCREV was set to \"$target_branch_head\", but no branches"
> +		echo "       contain this commit"
> +		exit 1
> +	fi
> +	ref=`git show ${target_branch_head} 2>&1 | head -n1 || true`
> +	if [ "$ref" = "fatal: bad object ${target_meta_head}" ]; then
> +	    echo "ERROR ${target_branch_head} is not a valid commit ID."
> +	    echo "The kernel source tree may be out of sync"
> +	    exit 1
>  	fi

There is a bit of uncertainty flying around at the moment about why
these changes are failing on the autobuilder. For example:

http://autobuilder.yoctoproject.org:8010/builders/nightly-mips/builds/560/steps/shell_138/logs/stdio

Log data follows:
| DEBUG: Executing shell function do_validate_branches
| ERROR: SRCREV was set to "1c17c082b6ee565acc176cde5be835ac4269817b", but no branches
|        contain this commit
| ERROR: Function failed: do_validate_branches

It looks like:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/fetch2/git.py?id=64662290d3e7deb0b6093b3959c3f3eddb873893

doesn't totally fix the problem. It allows the correct revisions to be
found and sets alternatives correctly but imports the wrong branch
config.

I don't really want to have to require a patched git binary so we may
end up having to rebuild the branch structure manually.

I guess the older code cared less about the branch name and more about
the revisions being present.

Cheers,

Richard
Bruce Ashfield - Aug. 18, 2012, 2:33 p.m.
On Sat, Aug 18, 2012 at 8:07 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2012-08-15 at 16:06 -0400, Bruce Ashfield wrote:
>> The kernel branch is no longer required by the yocto-kern-tools
>> to locate BSP feature descriptions (it is the MACHINE:KTYPE
>> descriptor), so we no longer require that the BSP branch be
>> explicitly set.
>>
>> If a kernel branch is explicitly set, it is now used to trigger
>> a checks to ensure that the branch really is being built.
>> Otherwise the branch that the machine description creates will
>> be built (just as it always was).
>>
>> This further simplies the use and configuration of a linux-yocto
>> based kernel recipe.
>
> [...]
>
>>
>> -     # We have SRCREVs and we have branches so validation can continue!
>> -     current=`git branch |grep \*|sed 's/^\* //'`
>> -     if [ -n "$target_branch_head" ] && [ "$branch_head" != "$target_branch_head" ] &&
>> -           [ "$target_branch_head" != "AUTOINC" ]; then
>> -             ref=`git show ${target_branch_head} 2>&1 | head -n1 || true`
>> -             if [ "$ref" = "fatal: bad object ${target_meta_head}" ]; then
>> -                     echo "ERROR ${target_branch_head} is not a valid commit ID."
>> -                     echo "The kernel source tree may be out of sync"
>> -                     exit 1
>> -             else
>> -                     echo "Forcing branch $current to ${target_branch_head}"
>> -                     git branch -m $current $current-orig
>> -                     git checkout -b $current ${target_branch_head}
>> -             fi
>> +     containing_branches=`git branch --contains $target_branch_head | sed 's/^..//'`
>> +     if [ -z "$containing_branches" ]; then
>> +             echo "ERROR: SRCREV was set to \"$target_branch_head\", but no branches"
>> +             echo "       contain this commit"
>> +             exit 1
>> +     fi
>> +     ref=`git show ${target_branch_head} 2>&1 | head -n1 || true`
>> +     if [ "$ref" = "fatal: bad object ${target_meta_head}" ]; then
>> +         echo "ERROR ${target_branch_head} is not a valid commit ID."
>> +         echo "The kernel source tree may be out of sync"
>> +         exit 1
>>       fi
>
> There is a bit of uncertainty flying around at the moment about why
> these changes are failing on the autobuilder. For example:
>
> http://autobuilder.yoctoproject.org:8010/builders/nightly-mips/builds/560/steps/shell_138/logs/stdio
>
> Log data follows:
> | DEBUG: Executing shell function do_validate_branches
> | ERROR: SRCREV was set to "1c17c082b6ee565acc176cde5be835ac4269817b", but no branches
> |        contain this commit
> | ERROR: Function failed: do_validate_branches
>
> It looks like:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/fetch2/git.py?id=64662290d3e7deb0b6093b3959c3f3eddb873893
>
> doesn't totally fix the problem. It allows the correct revisions to be
> found and sets alternatives correctly but imports the wrong branch
> config.

bugger. eh. But that's what I saw on the autobuilder as well. There are
two repos, one is being updated, another isn't and the older/non updated
one isn't being updated.

But what's causing the inconsistency in the .git and no ".git" repos ? i.e
aren't all the SRC_URIs now the same ? So why would one be fetched
into in the downloads, and the other used for the build ?

>
> I don't really want to have to require a patched git binary so we may
> end up having to rebuild the branch structure manually.

Where would we have to do the branch rebuild ? From what I saw, in the
broken repo, the blobs aren't even present .. so even if you know the
branch names, you can't set the SRCREV and just dump it in the branch
ref (which I've tried in the past, and it just breaks in equally impressive
ways as).

>
> I guess the older code cared less about the branch name and more about
> the revisions being present.

And the old code was wrong .. or at least not very good. It misses scenarios
where new BSPs are created, or other branches selected to be built by the
board descriptions. Human's working on a kernel, work on branches, not
git hashes and floating branches .. so this actually allows a BSP developer to
control the content, and have the build system build what he wants.

I can say is that this works fine here, has been working for over two months
within Wind River and never a failure seen, but the fetch infrastructure is
configured differently here.

We can skip the changes for 1.3, but they are a big step forward in the tools
and flexibility for working with the tree .. which was one of the macro goals
that I for 1.3.

Cheers,

Bruce

>
> Cheers,
>
> Richard
>
>
>
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Richard Purdie - Aug. 18, 2012, 3:29 p.m.
On Sat, 2012-08-18 at 10:33 -0400, Bruce Ashfield wrote:
> On Sat, Aug 18, 2012 at 8:07 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > There is a bit of uncertainty flying around at the moment about why
> > these changes are failing on the autobuilder. For example:
> >
> > http://autobuilder.yoctoproject.org:8010/builders/nightly-mips/builds/560/steps/shell_138/logs/stdio
> >
> > Log data follows:
> > | DEBUG: Executing shell function do_validate_branches
> > | ERROR: SRCREV was set to "1c17c082b6ee565acc176cde5be835ac4269817b", but no branches
> > |        contain this commit
> > | ERROR: Function failed: do_validate_branches
> >
> > It looks like:
> >
> > http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/fetch2/git.py?id=64662290d3e7deb0b6093b3959c3f3eddb873893
> >
> > doesn't totally fix the problem. It allows the correct revisions to be
> > found and sets alternatives correctly but imports the wrong branch
> > config.
> 
> bugger. eh. But that's what I saw on the autobuilder as well. There are
> two repos, one is being updated, another isn't and the older/non updated
> one isn't being updated.

In GITDIR, yes.

> But what's causing the inconsistency in the .git and no ".git" repos ? i.e
> aren't all the SRC_URIs now the same ? So why would one be fetched
> into in the downloads, and the other used for the build ?

Its a problem in git itself. It uses one for part of the operation and
the other repository for other bits of the operation. Yes, really :/.
You can see it under strace.

> > I don't really want to have to require a patched git binary so we may
> > end up having to rebuild the branch structure manually.
> 
> Where would we have to do the branch rebuild ? From what I saw, in the
> broken repo, the blobs aren't even present

When it runs the clone command, it does a clone of a hybrid of both
repositories. The alternates link points to the correct repo but it has
the branch and heads of the "bad" one. So the blobs are there in the
clone in WORKDIR, the heads and branches are not.

>  .. so even if you know the
> branch names, you can't set the SRCREV and just dump it in the branch
> ref (which I've tried in the past, and it just breaks in equally impressive
> ways as).
> 
> >
> > I guess the older code cared less about the branch name and more about
> > the revisions being present.
> 
> And the old code was wrong .. or at least not very good. It misses scenarios
> where new BSPs are created, or other branches selected to be built by the
> board descriptions. Human's working on a kernel, work on branches, not
> git hashes and floating branches .. so this actually allows a BSP developer to
> control the content, and have the build system build what he wants.
> 
> I can say is that this works fine here, has been working for over two months
> within Wind River and never a failure seen, but the fetch infrastructure is
> configured differently here.
> 
> We can skip the changes for 1.3, but they are a big step forward in the tools
> and flexibility for working with the tree .. which was one of the macro goals
> that I for 1.3.

If you have git > 1.7.9.2 you won't see the problem. You also won't see
the problem if you don't have two confusing directories in GITDIR from
some of the legacy SRC_URI problems we had (these things come back when
we build denzil for example).

Bottom line, I figured out a workaround for the fetcher. Its evil and
involves magic symlink but should stop this occurring. We can remove
that in a while when git 1.7.9.2 or later is more standard.

I've pushed the workaround into bitbake master, we now need to test your
branch again on the autobuilder and see where we are. Unforunately this
has eaten most of my day :(.

Cheers,

Richard

Patch

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index f09d503..10a8d40 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -80,8 +80,12 @@  do_patch() {
 		done
 	fi
 
+	if [ "${kbranch}" != "${KBRANCH_DEFAULT}" ]; then
+		updateme_flags="--branch ${kbranch}"
+	fi
+
 	# updates or generates the target description
-	updateme --branch ${kbranch} -DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \
+	updateme ${updateme_flags} -DKDESC=${KMACHINE}:${LINUX_KERNEL_TYPE} \
                            ${addon_features} ${ARCH} ${KMACHINE} ${sccs} ${patches}
 	if [ $? -ne 0 ]; then
 		echo "ERROR. Could not update ${kbranch}"
@@ -91,7 +95,19 @@  do_patch() {
 	# executes and modifies the source tree as required
 	patchme ${KMACHINE}
 	if [ $? -ne 0 ]; then
-		echo "ERROR. Could not modify ${kbranch}"
+		echo "ERROR. Could not apply updates for ${KMACHINE}"
+		exit 1
+	fi
+
+	# Perform a final check. If something other than the default kernel
+	# branch was requested, and that's not where we ended up, then we 
+	# should thrown an error, since we aren't building what was expected
+	final_branch="$(git symbolic-ref HEAD 2>/dev/null)"
+	final_branch=${final_branch##refs/heads/}
+	if [ "${kbranch}" != "${KBRANCH_DEFAULT}" ] &&
+	   [ "${final_branch}" != "${kbranch}" ]; then
+		echo "ERROR: branch ${kbranch} was requested, but was not properly"
+		echo "       configured to be built. The current branch is ${final_branch}"
 		exit 1
 	fi
 }
@@ -199,10 +215,9 @@  python do_kernel_configcheck() {
     bb.plain( "%s" % result )
 }
 
-
 # Ensure that the branches (BSP and meta) are on the locations specified by
 # their SRCREV values. If they are NOT on the right commits, the branches
-# are reset to the correct commit.
+# are corrected to the proper commit.
 do_validate_branches() {
 	cd ${S}
 
@@ -213,39 +228,57 @@  do_validate_branches() {
 		return
 	fi
 
-	# if the branches do not exist, then there's nothing to check either
+	# If something other than the default branch was requested, it must
+	# exist in the tree, and it's a hard error if it wasn't
 	git show-ref --quiet --verify -- "refs/heads/${KBRANCH}"
 	if [ $? -eq 1 ]; then
-		return
+		if [ -n "${KBRANCH_DEFAULT}" ] && 
+                      [ "${KBRANCH}" != "${KBRANCH_DEFAULT}" ]; then
+			echo "ERROR: branch ${KBRANCH} was set for kernel compilation, "
+			echo "       but it does not exist in the kernel repository."
+			echo "       Check the value of KBRANCH and ensure that it describes"
+			echo "       a valid banch in the source kernel repository"
+			exit 1
+		fi
 	fi
 
- 	branch_head=`git show-ref -s --heads ${KBRANCH}`
 	if [ -z "${SRCREV_machine}" ]; then
 		target_branch_head="${SRCREV}"
 	else
 	 	target_branch_head="${SRCREV_machine}"
 	fi
 
+	# $SRCREV could have also been AUTOINC, so check again
 	if [ "${target_branch_head}" = "AUTOINC" ]; then
 		return
 	fi
 
-	# We have SRCREVs and we have branches so validation can continue!
-	current=`git branch |grep \*|sed 's/^\* //'`
-	if [ -n "$target_branch_head" ] && [ "$branch_head" != "$target_branch_head" ] &&
-           [ "$target_branch_head" != "AUTOINC" ]; then
-		ref=`git show ${target_branch_head} 2>&1 | head -n1 || true`
-		if [ "$ref" = "fatal: bad object ${target_meta_head}" ]; then
-			echo "ERROR ${target_branch_head} is not a valid commit ID."
-			echo "The kernel source tree may be out of sync"
-			exit 1
-		else
-			echo "Forcing branch $current to ${target_branch_head}"
-			git branch -m $current $current-orig
-			git checkout -b $current ${target_branch_head}
-		fi
+	containing_branches=`git branch --contains $target_branch_head | sed 's/^..//'`
+	if [ -z "$containing_branches" ]; then
+		echo "ERROR: SRCREV was set to \"$target_branch_head\", but no branches"
+		echo "       contain this commit"
+		exit 1
+	fi
+	ref=`git show ${target_branch_head} 2>&1 | head -n1 || true`
+	if [ "$ref" = "fatal: bad object ${target_meta_head}" ]; then
+	    echo "ERROR ${target_branch_head} is not a valid commit ID."
+	    echo "The kernel source tree may be out of sync"
+	    exit 1
 	fi
 
+	# force the SRCREV in each branch that contains the specified
+	# SRCREV (if it isn't the current HEAD of that branch)
+	git checkout -q master
+	for b in $containing_branches; do
+		branch_head=`git show-ref -s --heads ${b}`
+		if [ "$branch_head" != "$target_branch_head" ]; then
+			echo "[INFO] Setting branch $b to ${target_branch_head}"	      
+			git branch -D $b > /dev/null
+			git branch $b $target_branch_head > /dev/null
+		fi
+	done
+
+	## KMETA branch validation
  	meta_head=`git show-ref -s --heads ${KMETA}`
  	target_meta_head="${SRCREV_meta}"
 	git show-ref --quiet --verify -- "refs/heads/${KMETA}"
@@ -264,18 +297,21 @@  do_validate_branches() {
 			echo "The kernel source tree may be out of sync"
 			exit 1
 		else
-			echo "Forcing branch meta to ${target_meta_head}"
+			echo "[INFO] Setting branch ${KMETA} to ${target_meta_head}"
 			git branch -m ${KMETA} ${KMETA}-orig
-			git checkout -b ${KMETA} ${target_meta_head}
+			git checkout -q -b ${KMETA} ${target_meta_head}
 			if [ $? -ne 0 ];then
-				echo "ERROR: could not checkout meta branch from known hash ${target_meta_head}"
+				echo "ERROR: could not checkout ${KMETA} branch from known hash ${target_meta_head}"
 				exit 1
 			fi
 		fi
 	fi
 
-	# restore the branch for builds
-	git checkout -f ${KBRANCH}
+	git show-ref --quiet --verify -- "refs/heads/${KBRANCH}"
+	if [ $? -eq 0 ]; then
+		# restore the branch for builds
+		git checkout -q -f ${KBRANCH}
+	fi
 }
 
 # Many scripts want to look in arch/$arch/boot for the bootable
diff --git a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
index f47262f..0cb111c 100644
--- a/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
+++ b/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb
@@ -4,7 +4,7 @@  LIC_FILES_CHKSUM = "file://git/tools/kgit;beginline=5;endline=9;md5=d8d1d729a70c
 
 DEPENDS = "git-native guilt-native"
 
-SRCREV = "b8dfd3d641400a8dfbf16868ee64f524508c80b7"
+SRCREV = "12c39b76eca4ed993b5ffb38cbe89e0608b216c3"
 PR = "r12"
 PV = "0.1+git${SRCPV}"