[yocto-autobuilder-helper] scripts: add a pair of scripts to set up and run Auto Upgrade Helper

Submitted by Alexander Kanavin on May 17, 2020, 3:21 p.m. | Patch ID: 172643

Details

Message ID 20200517152109.23350-1-alex.kanavin@gmail.com
State New
Headers show

Commit Message

Alexander Kanavin May 17, 2020, 3:21 p.m.
This allows automating its setup and execution on all autobuilder worker machines;
previously there was a static setup on a dedicated machine, which wasn't
great from maintenance perspective.

To use:

scripts/setup-auh target_dir
scripts/run-auh target_dir

(run-auh can be run several times in a directory that
was previously set up)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
---
 scripts/auh-config/local.conf.append   |  3 +++
 scripts/auh-config/upgrade-helper.conf | 33 ++++++++++++++++++++++++++
 scripts/run-auh                        | 32 +++++++++++++++++++++++++
 scripts/setup-auh                      | 26 ++++++++++++++++++++
 4 files changed, 94 insertions(+)
 create mode 100644 scripts/auh-config/local.conf.append
 create mode 100644 scripts/auh-config/upgrade-helper.conf
 create mode 100755 scripts/run-auh
 create mode 100755 scripts/setup-auh

Patch hide | download patch | download mbox

diff --git a/scripts/auh-config/local.conf.append b/scripts/auh-config/local.conf.append
new file mode 100644
index 0000000..9628737
--- /dev/null
+++ b/scripts/auh-config/local.conf.append
@@ -0,0 +1,3 @@ 
+
+INHERIT += "buildhistory"
+LICENSE_FLAGS_WHITELIST = "commercial"
diff --git a/scripts/auh-config/upgrade-helper.conf b/scripts/auh-config/upgrade-helper.conf
new file mode 100644
index 0000000..fbf5d8a
--- /dev/null
+++ b/scripts/auh-config/upgrade-helper.conf
@@ -0,0 +1,33 @@ 
+[maintainer_override]
+# mails for recipe upgrades will go to john.doe instead of jane.doe, etc
+#ross.burton@intel.com=anibal.limon@linux.intel.com
+
+[settings]
+# recipes in blacklist will be skipped
+blacklist=linux-libc-headers linux-yocto alsa-utils-scripts build-appliance-image
+#blacklist=python python3 glibc gcc linux-libc-headers linux-yocto-rt linux-yocto linux-yocto-dev linux-yocto-tiny qt4-x11-free qt4-embedded qt4-x11-free qt4e-demo-image gnome-common gnome-desktop3 gnome-desktop-testing adt-installer build-appliance-image
+# only recipes belonging to maintainers in whitelist will be attempted
+#maintainers_whitelist=anibal.limon@linux.intel.com
+# SMTP server
+smtp=smtp1.yoctoproject.org:25
+# from whom should the mails arrive
+from=auh@auh.yoctoproject.org
+# who should get the status mail with statistics, at the end
+status_recipients=openembedded-core@lists.openembedded.org
+# clean sstate directory before upgrading
+#clean_sstate=yes
+# clean tmp directory before upgrading
+#clean_tmp=yes
+# machines to test build with
+#machines=qemux86 qemux86-64 qemuarm qemumips qemuppc
+#machines=qemux86
+
+buildhistory=yes
+#testimage=yes
+#testimage_name=core-image-minimal
+
+#workdir=/home/auh/work/
+#publish_work_url=https://logs.yoctoproject.org/auh/
+
+commit_revert_policy=all
+
diff --git a/scripts/run-auh b/scripts/run-auh
new file mode 100755
index 0000000..29a8044
--- /dev/null
+++ b/scripts/run-auh
@@ -0,0 +1,32 @@ 
+#!/bin/bash
+# Run Auto Upgrade Helper in a directory set up by setup_auh.
+#
+# Called with $1 - the directory where the setup was created
+
+if [ -z $1 ]; then
+  echo "Use: $0 auh_setup_dir"
+  exit 1
+fi
+
+full_dir=$(readlink -e $1)
+
+auh_dir=$full_dir/auto-upgrade-helper
+poky_dir=$full_dir/poky
+build_dir=$full_dir/build
+sstate_dir=$full_dir/build/sstate-cache
+
+pushd $poky_dir
+
+# Base the upgrades on poky master
+git fetch origin
+git checkout -B tmp-auh-upgrades origin/master
+
+source $poky_dir/oe-init-build-env $build_dir
+$auh_dir/upgradehelper.py -e all
+
+# clean up to avoid the disk filling up
+rm -rf $build_dir/tmp/
+rm -rf $build_dir/workspace/sources/*
+find $sstate_dir -atime +10 -delete
+
+popd
diff --git a/scripts/setup-auh b/scripts/setup-auh
new file mode 100755
index 0000000..23f3d44
--- /dev/null
+++ b/scripts/setup-auh
@@ -0,0 +1,26 @@ 
+#!/bin/bash
+# Initialize Auto Upgrade Helper in a directory.
+#
+# Called with $1 - the directory to place the setup
+CONFIG_DIR=`dirname $0`/auh-config
+
+if [ -z $1 ]; then
+  echo "Use: $0 target_dir"
+  exit 1
+fi
+
+mkdir -p $1
+pushd $1
+
+git clone git://git.yoctoproject.org/poky
+pushd poky
+git config user.email auh@auh.yoctoproject.org
+git config user.name "Auto Upgrade Helper"
+popd
+git clone git://git.yoctoproject.org/auto-upgrade-helper
+source poky/oe-init-build-env build
+mkdir -p upgrade-helper
+popd
+
+cp $CONFIG_DIR/upgrade-helper.conf $1/build/upgrade-helper
+cat $CONFIG_DIR/local.conf.append >> $1/build/conf/local.conf

Comments

Nicolas Dechesne May 18, 2020, 7:02 a.m.
On Sun, May 17, 2020 at 5:21 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> This allows automating its setup and execution on all autobuilder worker
> machines;
> previously there was a static setup on a dedicated machine, which wasn't
> great from maintenance perspective.
>
> To use:
>
> scripts/setup-auh target_dir
> scripts/run-auh target_dir
>
> (run-auh can be run several times in a directory that
> was previously set up)
>
> Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
> ---
>  scripts/auh-config/local.conf.append   |  3 +++
>  scripts/auh-config/upgrade-helper.conf | 33 ++++++++++++++++++++++++++
>  scripts/run-auh                        | 32 +++++++++++++++++++++++++
>  scripts/setup-auh                      | 26 ++++++++++++++++++++
>  4 files changed, 94 insertions(+)
>  create mode 100644 scripts/auh-config/local.conf.append
>  create mode 100644 scripts/auh-config/upgrade-helper.conf
>  create mode 100755 scripts/run-auh
>  create mode 100755 scripts/setup-auh
>
> diff --git a/scripts/auh-config/local.conf.append
> b/scripts/auh-config/local.conf.append
> new file mode 100644
> index 0000000..9628737
> --- /dev/null
> +++ b/scripts/auh-config/local.conf.append
> @@ -0,0 +1,3 @@
> +
> +INHERIT += "buildhistory"
> +LICENSE_FLAGS_WHITELIST = "commercial"
> diff --git a/scripts/auh-config/upgrade-helper.conf
> b/scripts/auh-config/upgrade-helper.conf
> new file mode 100644
> index 0000000..fbf5d8a
> --- /dev/null
> +++ b/scripts/auh-config/upgrade-helper.conf
> @@ -0,0 +1,33 @@
> +[maintainer_override]
> +# mails for recipe upgrades will go to john.doe instead of jane.doe, etc
> +#ross.burton@intel.com=anibal.limon@linux.intel.com


unrelated.. but I just spotted this ^. I am cc'ing Anibal new email
address. it's in a few other places.


>
> +
> +[settings]
> +# recipes in blacklist will be skipped
> +blacklist=linux-libc-headers linux-yocto alsa-utils-scripts
> build-appliance-image
> +#blacklist=python python3 glibc gcc linux-libc-headers linux-yocto-rt
> linux-yocto linux-yocto-dev linux-yocto-tiny qt4-x11-free qt4-embedded
> qt4-x11-free qt4e-demo-image gnome-common gnome-desktop3
> gnome-desktop-testing adt-installer build-appliance-image
> +# only recipes belonging to maintainers in whitelist will be attempted
> +#maintainers_whitelist=anibal.limon@linux.intel.com
> +# SMTP server
> +smtp=smtp1.yoctoproject.org:25
> +# from whom should the mails arrive
> +from=auh@auh.yoctoproject.org
> +# who should get the status mail with statistics, at the end
> +status_recipients=openembedded-core@lists.openembedded.org
> +# clean sstate directory before upgrading
> +#clean_sstate=yes
> +# clean tmp directory before upgrading
> +#clean_tmp=yes
> +# machines to test build with
> +#machines=qemux86 qemux86-64 qemuarm qemumips qemuppc
> +#machines=qemux86
> +
> +buildhistory=yes
> +#testimage=yes
> +#testimage_name=core-image-minimal
> +
> +#workdir=/home/auh/work/
> +#publish_work_url=https://logs.yoctoproject.org/auh/
> +
> +commit_revert_policy=all
> +
> diff --git a/scripts/run-auh b/scripts/run-auh
> new file mode 100755
> index 0000000..29a8044
> --- /dev/null
> +++ b/scripts/run-auh
> @@ -0,0 +1,32 @@
> +#!/bin/bash
> +# Run Auto Upgrade Helper in a directory set up by setup_auh.
> +#
> +# Called with $1 - the directory where the setup was created
> +
> +if [ -z $1 ]; then
> +  echo "Use: $0 auh_setup_dir"
> +  exit 1
> +fi
> +
> +full_dir=$(readlink -e $1)
> +
> +auh_dir=$full_dir/auto-upgrade-helper
> +poky_dir=$full_dir/poky
> +build_dir=$full_dir/build
> +sstate_dir=$full_dir/build/sstate-cache
> +
> +pushd $poky_dir
> +
> +# Base the upgrades on poky master
> +git fetch origin
> +git checkout -B tmp-auh-upgrades origin/master
> +
> +source $poky_dir/oe-init-build-env $build_dir
> +$auh_dir/upgradehelper.py -e all
> +
> +# clean up to avoid the disk filling up
> +rm -rf $build_dir/tmp/
> +rm -rf $build_dir/workspace/sources/*
> +find $sstate_dir -atime +10 -delete
> +
> +popd
> diff --git a/scripts/setup-auh b/scripts/setup-auh
> new file mode 100755
> index 0000000..23f3d44
> --- /dev/null
> +++ b/scripts/setup-auh
> @@ -0,0 +1,26 @@
> +#!/bin/bash
> +# Initialize Auto Upgrade Helper in a directory.
> +#
> +# Called with $1 - the directory to place the setup
> +CONFIG_DIR=`dirname $0`/auh-config
> +
> +if [ -z $1 ]; then
> +  echo "Use: $0 target_dir"
> +  exit 1
> +fi
> +
> +mkdir -p $1
> +pushd $1
> +
> +git clone git://git.yoctoproject.org/poky
> +pushd poky
> +git config user.email auh@auh.yoctoproject.org
> +git config user.name "Auto Upgrade Helper"
> +popd
> +git clone git://git.yoctoproject.org/auto-upgrade-helper
> +source poky/oe-init-build-env build
> +mkdir -p upgrade-helper
> +popd
> +
> +cp $CONFIG_DIR/upgrade-helper.conf $1/build/upgrade-helper
> +cat $CONFIG_DIR/local.conf.append >> $1/build/conf/local.conf
> --
> 2.26.2
>
> 
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#49434): https://lists.yoctoproject.org/g/yocto/message/49434
Mute This Topic: https://lists.yoctoproject.org/mt/74271373/3617530
Group Owner: yocto+owner@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-
Aníbal Limón May 27, 2020, 2:19 a.m.
On Mon, 18 May 2020 at 02:03, Nicolas Dechesne <nicolas.dechesne@linaro.org>
wrote:

>
>
> On Sun, May 17, 2020 at 5:21 PM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> This allows automating its setup and execution on all autobuilder worker
>> machines;
>> previously there was a static setup on a dedicated machine, which wasn't
>> great from maintenance perspective.
>>
>> To use:
>>
>> scripts/setup-auh target_dir
>> scripts/run-auh target_dir
>>
>> (run-auh can be run several times in a directory that
>> was previously set up)
>>
>> Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
>> ---
>>  scripts/auh-config/local.conf.append   |  3 +++
>>  scripts/auh-config/upgrade-helper.conf | 33 ++++++++++++++++++++++++++
>>  scripts/run-auh                        | 32 +++++++++++++++++++++++++
>>  scripts/setup-auh                      | 26 ++++++++++++++++++++
>>  4 files changed, 94 insertions(+)
>>  create mode 100644 scripts/auh-config/local.conf.append
>>  create mode 100644 scripts/auh-config/upgrade-helper.conf
>>  create mode 100755 scripts/run-auh
>>  create mode 100755 scripts/setup-auh
>>
>> diff --git a/scripts/auh-config/local.conf.append
>> b/scripts/auh-config/local.conf.append
>> new file mode 100644
>> index 0000000..9628737
>> --- /dev/null
>> +++ b/scripts/auh-config/local.conf.append
>> @@ -0,0 +1,3 @@
>> +
>> +INHERIT += "buildhistory"
>> +LICENSE_FLAGS_WHITELIST = "commercial"
>> diff --git a/scripts/auh-config/upgrade-helper.conf
>> b/scripts/auh-config/upgrade-helper.conf
>> new file mode 100644
>> index 0000000..fbf5d8a
>> --- /dev/null
>> +++ b/scripts/auh-config/upgrade-helper.conf
>> @@ -0,0 +1,33 @@
>> +[maintainer_override]
>> +# mails for recipe upgrades will go to john.doe instead of jane.doe, etc
>> +#ross.burton@intel.com=anibal.limon@linux.intel.com
>
>
> unrelated.. but I just spotted this ^. I am cc'ing Anibal new email
> address. it's in a few other places.
>

Right, Looks good to me since Alexander cleaned-up the AUH after some
tinfoil improvements, he is the new maintainer :).

Cheers,
Anibal


>
>
>>
>> +
>> +[settings]
>> +# recipes in blacklist will be skipped
>> +blacklist=linux-libc-headers linux-yocto alsa-utils-scripts
>> build-appliance-image
>> +#blacklist=python python3 glibc gcc linux-libc-headers linux-yocto-rt
>> linux-yocto linux-yocto-dev linux-yocto-tiny qt4-x11-free qt4-embedded
>> qt4-x11-free qt4e-demo-image gnome-common gnome-desktop3
>> gnome-desktop-testing adt-installer build-appliance-image
>> +# only recipes belonging to maintainers in whitelist will be attempted
>> +#maintainers_whitelist=anibal.limon@linux.intel.com
>> +# SMTP server
>> +smtp=smtp1.yoctoproject.org:25
>> +# from whom should the mails arrive
>> +from=auh@auh.yoctoproject.org
>> +# who should get the status mail with statistics, at the end
>> +status_recipients=openembedded-core@lists.openembedded.org
>> +# clean sstate directory before upgrading
>> +#clean_sstate=yes
>> +# clean tmp directory before upgrading
>> +#clean_tmp=yes
>> +# machines to test build with
>> +#machines=qemux86 qemux86-64 qemuarm qemumips qemuppc
>> +#machines=qemux86
>> +
>> +buildhistory=yes
>> +#testimage=yes
>> +#testimage_name=core-image-minimal
>> +
>> +#workdir=/home/auh/work/
>> +#publish_work_url=https://logs.yoctoproject.org/auh/
>> +
>> +commit_revert_policy=all
>> +
>> diff --git a/scripts/run-auh b/scripts/run-auh
>> new file mode 100755
>> index 0000000..29a8044
>> --- /dev/null
>> +++ b/scripts/run-auh
>> @@ -0,0 +1,32 @@
>> +#!/bin/bash
>> +# Run Auto Upgrade Helper in a directory set up by setup_auh.
>> +#
>> +# Called with $1 - the directory where the setup was created
>> +
>> +if [ -z $1 ]; then
>> +  echo "Use: $0 auh_setup_dir"
>> +  exit 1
>> +fi
>> +
>> +full_dir=$(readlink -e $1)
>> +
>> +auh_dir=$full_dir/auto-upgrade-helper
>> +poky_dir=$full_dir/poky
>> +build_dir=$full_dir/build
>> +sstate_dir=$full_dir/build/sstate-cache
>> +
>> +pushd $poky_dir
>> +
>> +# Base the upgrades on poky master
>> +git fetch origin
>> +git checkout -B tmp-auh-upgrades origin/master
>> +
>> +source $poky_dir/oe-init-build-env $build_dir
>> +$auh_dir/upgradehelper.py -e all
>> +
>> +# clean up to avoid the disk filling up
>> +rm -rf $build_dir/tmp/
>> +rm -rf $build_dir/workspace/sources/*
>> +find $sstate_dir -atime +10 -delete
>> +
>> +popd
>> diff --git a/scripts/setup-auh b/scripts/setup-auh
>> new file mode 100755
>> index 0000000..23f3d44
>> --- /dev/null
>> +++ b/scripts/setup-auh
>> @@ -0,0 +1,26 @@
>> +#!/bin/bash
>> +# Initialize Auto Upgrade Helper in a directory.
>> +#
>> +# Called with $1 - the directory to place the setup
>> +CONFIG_DIR=`dirname $0`/auh-config
>> +
>> +if [ -z $1 ]; then
>> +  echo "Use: $0 target_dir"
>> +  exit 1
>> +fi
>> +
>> +mkdir -p $1
>> +pushd $1
>> +
>> +git clone git://git.yoctoproject.org/poky
>> +pushd poky
>> +git config user.email auh@auh.yoctoproject.org
>> +git config user.name "Auto Upgrade Helper"
>> +popd
>> +git clone git://git.yoctoproject.org/auto-upgrade-helper
>> +source poky/oe-init-build-env build
>> +mkdir -p upgrade-helper
>> +popd
>> +
>> +cp $CONFIG_DIR/upgrade-helper.conf $1/build/upgrade-helper
>> +cat $CONFIG_DIR/local.conf.append >> $1/build/conf/local.conf
>> --
>> 2.26.2
>>
>> 
>>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#49492): https://lists.yoctoproject.org/g/yocto/message/49492
Mute This Topic: https://lists.yoctoproject.org/mt/74271373/3617530
Group Owner: yocto+owner@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [oe-patchwork@oe-patch.openembedded.org]
-=-=-=-=-=-=-=-=-=-=-=-