Patchwork [V3,0/5] manifest.py/package_manager.py/rootfs.py: support ipk incremental image generation

login
register
mail settings
Submitter Hongxu Jia
Date Feb. 21, 2014, 6:38 a.m.
Message ID <cover.1392963956.git.hongxu.jia@windriver.com>
Download mbox
Permalink /patch/67089/
State New
Headers show

Pull-request

git://git.pokylinux.org/poky-contrib hongxu/ipk-inc-gen

Comments

Hongxu Jia - Feb. 21, 2014, 6:38 a.m.
Change in V2:
- Move dump_install_solution to Manifest class as create_full;
- Move load_old_install_solution to Manifest class as parse_full_manifest;
- Handle the output of dummy install in Python rather than shell;
- Fix typos
- Rename _file_duplicate with _file_equal to avoid confusion;

Change in V3:
- While create_full calls dummy_install, pass an instance to PM
  as an argument rather than instantiate another PM object;
- Rename _create_incremental with _remove_extra_packages;
- Move the call of _remove_extra_packages below the call of
  self.pm.handle_bad_recommendations();
- Remove redundant self.pm.update in _remove_extra_packages;
- Rename _remove_existing_image with _remove_old_rootfs;

Test Cases

---------------------------------------------
Case 1: Rebuild in place
1. Edit local.conf, enable multilib and ipk incremental image generation
...
 46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257 
258 IMAGE_INSTALL_append = " lib32-gzip"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262 
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, add some spaces in IMAGE_INSTALL_append
vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip "
...

3. bb core-image-minimal, the image should not be rebuilt
vim log.do_rootfs:
...
 16 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
 17 Package opkg-collateral (1.0-r2) installed in root is up to date.
 18 Package run-postinsts (1.0-r9) installed in root is up to date.
 19 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 20 Package opkg (1:0.2.0-r0) installed in root is up to date.
 21 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
 ...
 53 NOTE: Package lib32-gzip (1.6-r7) installed in root is up to date.
...

vim installed_pkgs.txt
...
lib32-gzip x86
...

4. runqemu qemux86-64, ssh could work and lib32-gzip is installed
On host:
runqemu qemux86-64
ssh root@192.168.7.2 and run 'opkg-cl list-installed'
...
lib32-gzip - 1.6-r7
...

---------------------------------------------
Case 2: Decremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
 46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257 
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262 
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, remove 'lib32-gzip openssl'
vim local.conf
...
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
...

3. bb core-image-minimal, lib32-gzip and openssl should be removed.
vim log.do_rootfs:
...
 20 NOTE: Removing package lib32-eglibc from root...
 21 Removing package lib32-gzip from root...
 22 Removing package lib32-update-alternatives-opkg from root...
 23 Removing package libcrypto1.0.0 from root...
 24 Removing package libssl1.0.0 from root...
 25 Removing package openssl from root...
 26 Removing package openssl-conf from root...
 ...
 31 Package opkg-collateral (1.0-r2) installed in root is up to date.
 32 Package run-postinsts (1.0-r9) installed in root is up to date.
 33 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 34 Package opkg (1:0.2.0-r0) installed in root is up to date.
 35 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...

---------------------------------------------
Case 3: Incremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
 46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257 
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262 
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, append 'lib32-gzip openssl' to IMAGE_INSTALL
vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
...

3. bb core-image-minimal, lib32-gzip and openssl should be added.
vim log.do_rootfs:
...
 20 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
 21 Package opkg-collateral (1.0-r2) installed in root is up to date.
 22 Package run-postinsts (1.0-r9) installed in root is up to date.
 23 Installing openssl (1.0.1e-r15.0) to root...
 ...
 31 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 32 Package opkg (1:0.2.0-r0) installed in root is up to date.
 33 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
 ...
 54 NOTE: Installing the following packages: lib32-gzip
...

---------------------------------------------
Case 4: Rebuild in place and Upgrade package
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " gzip"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, upgrade gzip's PR
...
258 PR_pn-gzip = "r1"
...

3. bb core-image-minimal, gzip should be upgraded to r1
vim log.do_rootfs:
...
 21 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
 22 Package opkg-collateral (1.0-r2) installed in root is up to date.
 23 Package run-postinsts (1.0-r9) installed in root is up to date.
 24 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 25 Package opkg (1:0.2.0-r0) installed in root is up to date.
 26 Upgrading gzip on root from 1.6-r0 to 1.6-r1...
...

---------------------------------------------
Case 5: Test BAD_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

The util-linux-fdisk is in util-linux's RRECOMMENDS
vim installed_pkgs.txt
...
41 util-linux-fdisk core2-64
...

2. Edit local.conf, add some spaces to IMAGE_INSTALL and add
util-linux-fdisk to core-image-minimal's BAD_RECOMMENDATIONS
...
260 IMAGE_INSTALL_append = " util-linux  "
265 BAD_RECOMMENDATIONS_pn-core-image-minimal += "util-linux-fdisk"
...

3. bb core-image-minimal, util-linux-fdisk should not be installed
vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
    ...
119 util-linux: ignoring recommendation for util-linux-fdisk at user request
120 util-linux: unsatisfied recommendation for util-linux-fdisk
...

The util-linux-fdisk doesn't exist in installed_pkgs.txt any more.

---------------------------------------------
Case 6: Test NO_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

The util-linux-fdisk util-linux-cfdisk util-linux-sfdisk
util-linux-mount util-linux-readprofile util-linux-mkfs are
in util-linux's RRECOMMENDS

vim installed_pkgs.txt
...
4 util-linux-mkfs core2-64
18 util-linux-sfdisk core2-64
20 util-linux-mount core2-64
38 util-linux-cfdisk core2-64
41 util-linux-fdisk core2-64
46 util-linux-readprofile core2-64
...

2. Edit local.conf, add some spaces to IMAGE_INSTALL and
   assign core-image-minimal's NO_RECOMMENDATIONS with 1
...
260 IMAGE_INSTALL_append = " util-linux  "
267 NO_RECOMMENDATIONS_pn-core-image-minimal = "1"
...

3. bb core-image-minimal, The six packages should not be installed
vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
    ...
113 util-linux: ignoring recommendation for util-linux-sfdisk at user request
114 util-linux: unsatisfied recommendation for util-linux-sfdisk
115 util-linux: ignoring recommendation for util-linux-cfdisk at user request
116 util-linux: unsatisfied recommendation for util-linux-cfdisk
117 util-linux: ignoring recommendation for util-linux-mount at user request
118 util-linux: unsatisfied recommendation for util-linux-mount
119 util-linux: ignoring recommendation for util-linux-fdisk at user request
120 util-linux: unsatisfied recommendation for util-linux-fdisk
121 util-linux: ignoring recommendation for util-linux-mkfs at user request
122 util-linux: unsatisfied recommendation for util-linux-mkfs
123 util-linux: ignoring recommendation for util-linux-readprofile at user request
124 util-linux: unsatisfied recommendation for util-linux-readprofile
...

They don't exist in installed_pkgs.txt any more.

//Hongxu

The following changes since commit bb0c26960d3b05070346cdfdfd05d6fd4ad0a7b1:

  wic: Make exec_native_command() fail if a command isn't found (2014-02-20 14:28:13 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/ipk-inc-gen
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/ipk-inc-gen

Hongxu Jia (5):
  manifest.py: add create_full for OpkgManifest class
  package_manager.py: support ipk incremental image generation
  rootfs.py: support ipk incremental image generation
  rootfs.py: support BAD_RECOMMENDATIONS for ipk incremental image
    generation
  rootfs.py: tweak _multilib_sanity_test for ipk incremental image
    generation

 meta/lib/oe/manifest.py        |  57 +++++++++++++++++-
 meta/lib/oe/package_manager.py |  81 ++++++++++++++++++++++++--
 meta/lib/oe/rootfs.py          | 128 +++++++++++++++++++++++++++++++++++++++--
 3 files changed, 254 insertions(+), 12 deletions(-)
Laurentiu Palcu - Feb. 21, 2014, 8:24 a.m.
Reviewed-by: Laurentiu Palcu <laurentiu.palcu.intel.com>

laurentiu

On Fri, Feb 21, 2014 at 02:38:16PM +0800, Hongxu Jia wrote:
> Change in V2:
> - Move dump_install_solution to Manifest class as create_full;
> - Move load_old_install_solution to Manifest class as parse_full_manifest;
> - Handle the output of dummy install in Python rather than shell;
> - Fix typos
> - Rename _file_duplicate with _file_equal to avoid confusion;
> 
> Change in V3:
> - While create_full calls dummy_install, pass an instance to PM
>   as an argument rather than instantiate another PM object;
> - Rename _create_incremental with _remove_extra_packages;
> - Move the call of _remove_extra_packages below the call of
>   self.pm.handle_bad_recommendations();
> - Remove redundant self.pm.update in _remove_extra_packages;
> - Rename _remove_existing_image with _remove_old_rootfs;
> 
> Test Cases
> 
> ---------------------------------------------
> Case 1: Rebuild in place
> 1. Edit local.conf, enable multilib and ipk incremental image generation
> ...
>  46 MACHINE ?= "qemux86-64"
> 254 require conf/multilib.conf
> 255 MULTILIBS = "multilib:lib32"
> 256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
> 257 
> 258 IMAGE_INSTALL_append = " lib32-gzip"
> 260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
> 262 
> 263 INC_IPK_IMAGE_GEN = "1"
> ...
> bb core-image-minimal
> 
> 2. Edit local.conf, add some spaces in IMAGE_INSTALL_append
> vim local.conf
> ...
> 258 IMAGE_INSTALL_append = " lib32-gzip "
> ...
> 
> 3. bb core-image-minimal, the image should not be rebuilt
> vim log.do_rootfs:
> ...
>  16 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
>  17 Package opkg-collateral (1.0-r2) installed in root is up to date.
>  18 Package run-postinsts (1.0-r9) installed in root is up to date.
>  19 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
>  20 Package opkg (1:0.2.0-r0) installed in root is up to date.
>  21 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
>  ...
>  53 NOTE: Package lib32-gzip (1.6-r7) installed in root is up to date.
> ...
> 
> vim installed_pkgs.txt
> ...
> lib32-gzip x86
> ...
> 
> 4. runqemu qemux86-64, ssh could work and lib32-gzip is installed
> On host:
> runqemu qemux86-64
> ssh root@192.168.7.2 and run 'opkg-cl list-installed'
> ...
> lib32-gzip - 1.6-r7
> ...
> 
> ---------------------------------------------
> Case 2: Decremental Rebuild
> 1. Edit local.conf, enable multilib and ipk incremental image generation
> ...
>  46 MACHINE ?= "qemux86-64"
> 254 require conf/multilib.conf
> 255 MULTILIBS = "multilib:lib32"
> 256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
> 257 
> 258 IMAGE_INSTALL_append = " lib32-gzip openssl"
> 260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
> 262 
> 263 INC_IPK_IMAGE_GEN = "1"
> ...
> bb core-image-minimal
> 
> 2. Edit local.conf, remove 'lib32-gzip openssl'
> vim local.conf
> ...
> 258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
> ...
> 
> 3. bb core-image-minimal, lib32-gzip and openssl should be removed.
> vim log.do_rootfs:
> ...
>  20 NOTE: Removing package lib32-eglibc from root...
>  21 Removing package lib32-gzip from root...
>  22 Removing package lib32-update-alternatives-opkg from root...
>  23 Removing package libcrypto1.0.0 from root...
>  24 Removing package libssl1.0.0 from root...
>  25 Removing package openssl from root...
>  26 Removing package openssl-conf from root...
>  ...
>  31 Package opkg-collateral (1.0-r2) installed in root is up to date.
>  32 Package run-postinsts (1.0-r9) installed in root is up to date.
>  33 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
>  34 Package opkg (1:0.2.0-r0) installed in root is up to date.
>  35 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
> ...
> 
> ---------------------------------------------
> Case 3: Incremental Rebuild
> 1. Edit local.conf, enable multilib and ipk incremental image generation
> ...
>  46 MACHINE ?= "qemux86-64"
> 254 require conf/multilib.conf
> 255 MULTILIBS = "multilib:lib32"
> 256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
> 257 
> 258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
> 260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
> 262 
> 263 INC_IPK_IMAGE_GEN = "1"
> ...
> bb core-image-minimal
> 
> 2. Edit local.conf, append 'lib32-gzip openssl' to IMAGE_INSTALL
> vim local.conf
> ...
> 258 IMAGE_INSTALL_append = " lib32-gzip openssl"
> ...
> 
> 3. bb core-image-minimal, lib32-gzip and openssl should be added.
> vim log.do_rootfs:
> ...
>  20 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
>  21 Package opkg-collateral (1.0-r2) installed in root is up to date.
>  22 Package run-postinsts (1.0-r9) installed in root is up to date.
>  23 Installing openssl (1.0.1e-r15.0) to root...
>  ...
>  31 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
>  32 Package opkg (1:0.2.0-r0) installed in root is up to date.
>  33 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
>  ...
>  54 NOTE: Installing the following packages: lib32-gzip
> ...
> 
> ---------------------------------------------
> Case 4: Rebuild in place and Upgrade package
> 1. Edit local.conf, enable ipk incremental image generation
> ...
> 258 IMAGE_INSTALL_append = " gzip"
> 263 INC_IPK_IMAGE_GEN = "1"
> ...
> bb core-image-minimal
> 
> 2. Edit local.conf, upgrade gzip's PR
> ...
> 258 PR_pn-gzip = "r1"
> ...
> 
> 3. bb core-image-minimal, gzip should be upgraded to r1
> vim log.do_rootfs:
> ...
>  21 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
>  22 Package opkg-collateral (1.0-r2) installed in root is up to date.
>  23 Package run-postinsts (1.0-r9) installed in root is up to date.
>  24 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
>  25 Package opkg (1:0.2.0-r0) installed in root is up to date.
>  26 Upgrading gzip on root from 1.6-r0 to 1.6-r1...
> ...
> 
> ---------------------------------------------
> Case 5: Test BAD_RECOMMENDATIONS
> 1. Edit local.conf, enable ipk incremental image generation
> ...
> 258 IMAGE_INSTALL_append = " util-linux"
> 263 INC_IPK_IMAGE_GEN = "1"
> ...
> bb core-image-minimal
> 
> The util-linux-fdisk is in util-linux's RRECOMMENDS
> vim installed_pkgs.txt
> ...
> 41 util-linux-fdisk core2-64
> ...
> 
> 2. Edit local.conf, add some spaces to IMAGE_INSTALL and add
> util-linux-fdisk to core-image-minimal's BAD_RECOMMENDATIONS
> ...
> 260 IMAGE_INSTALL_append = " util-linux  "
> 265 BAD_RECOMMENDATIONS_pn-core-image-minimal += "util-linux-fdisk"
> ...
> 
> 3. bb core-image-minimal, util-linux-fdisk should not be installed
> vim log.do_rootfs:
> ...
> 34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
>     ...
> 119 util-linux: ignoring recommendation for util-linux-fdisk at user request
> 120 util-linux: unsatisfied recommendation for util-linux-fdisk
> ...
> 
> The util-linux-fdisk doesn't exist in installed_pkgs.txt any more.
> 
> ---------------------------------------------
> Case 6: Test NO_RECOMMENDATIONS
> 1. Edit local.conf, enable ipk incremental image generation
> ...
> 258 IMAGE_INSTALL_append = " util-linux"
> 263 INC_IPK_IMAGE_GEN = "1"
> ...
> bb core-image-minimal
> 
> The util-linux-fdisk util-linux-cfdisk util-linux-sfdisk
> util-linux-mount util-linux-readprofile util-linux-mkfs are
> in util-linux's RRECOMMENDS
> 
> vim installed_pkgs.txt
> ...
> 4 util-linux-mkfs core2-64
> 18 util-linux-sfdisk core2-64
> 20 util-linux-mount core2-64
> 38 util-linux-cfdisk core2-64
> 41 util-linux-fdisk core2-64
> 46 util-linux-readprofile core2-64
> ...
> 
> 2. Edit local.conf, add some spaces to IMAGE_INSTALL and
>    assign core-image-minimal's NO_RECOMMENDATIONS with 1
> ...
> 260 IMAGE_INSTALL_append = " util-linux  "
> 267 NO_RECOMMENDATIONS_pn-core-image-minimal = "1"
> ...
> 
> 3. bb core-image-minimal, The six packages should not be installed
> vim log.do_rootfs:
> ...
> 34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
>     ...
> 113 util-linux: ignoring recommendation for util-linux-sfdisk at user request
> 114 util-linux: unsatisfied recommendation for util-linux-sfdisk
> 115 util-linux: ignoring recommendation for util-linux-cfdisk at user request
> 116 util-linux: unsatisfied recommendation for util-linux-cfdisk
> 117 util-linux: ignoring recommendation for util-linux-mount at user request
> 118 util-linux: unsatisfied recommendation for util-linux-mount
> 119 util-linux: ignoring recommendation for util-linux-fdisk at user request
> 120 util-linux: unsatisfied recommendation for util-linux-fdisk
> 121 util-linux: ignoring recommendation for util-linux-mkfs at user request
> 122 util-linux: unsatisfied recommendation for util-linux-mkfs
> 123 util-linux: ignoring recommendation for util-linux-readprofile at user request
> 124 util-linux: unsatisfied recommendation for util-linux-readprofile
> ...
> 
> They don't exist in installed_pkgs.txt any more.
> 
> //Hongxu
> 
> The following changes since commit bb0c26960d3b05070346cdfdfd05d6fd4ad0a7b1:
> 
>   wic: Make exec_native_command() fail if a command isn't found (2014-02-20 14:28:13 +0000)
> 
> are available in the git repository at:
> 
>   git://git.pokylinux.org/poky-contrib hongxu/ipk-inc-gen
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/ipk-inc-gen
> 
> Hongxu Jia (5):
>   manifest.py: add create_full for OpkgManifest class
>   package_manager.py: support ipk incremental image generation
>   rootfs.py: support ipk incremental image generation
>   rootfs.py: support BAD_RECOMMENDATIONS for ipk incremental image
>     generation
>   rootfs.py: tweak _multilib_sanity_test for ipk incremental image
>     generation
> 
>  meta/lib/oe/manifest.py        |  57 +++++++++++++++++-
>  meta/lib/oe/package_manager.py |  81 ++++++++++++++++++++++++--
>  meta/lib/oe/rootfs.py          | 128 +++++++++++++++++++++++++++++++++++++++--
>  3 files changed, 254 insertions(+), 12 deletions(-)
> 
> -- 
> 1.8.1.2
>