4372 2013-04-22 09:03:31 +0000 [Performance] make hardlinks for files in DEPLOY_DIR when possible 2013-06-21 12:49:42 +0000 1 1 7 Build System, Metadata & Runtime OE-Core deployment 1.5 x86 Multiple VERIFIED FIXED Medium+ enhancement 1.5 M1 4643 1 liezhi.yang liezhi.yang mark.hatle mihai.lindner richard.purdie sgw --- --- --- oldest_to_newest 32434 0 liezhi.yang 2013-04-22 09:03:31 +0000 The rpm file (suppose PACKAGE_CLASSES = "package_rpm") in ${WORK_DIR}/deploy-rpms (AKA PKGWRITEDIRRPM) is the same as the one in DEPLOY_DIR_RPM (e.g., tmp/deploy/rpm), if we make symlink from DEPLOY_DIR_RPM to ${WORK_DIR}/deploy-rpms for the rpm file, it would save a lot time and disk space, or can we have a configuration variable to control this ? We copy the files by default, but can use the symlink when the user requires. 32444 1 liezhi.yang 2013-04-22 10:28:25 +0000 (In reply to comment #0) > The rpm file (suppose PACKAGE_CLASSES = "package_rpm") in > ${WORK_DIR}/deploy-rpms (AKA PKGWRITEDIRRPM) is the same as the one in > DEPLOY_DIR_RPM (e.g., tmp/deploy/rpm), if we make symlink from > DEPLOY_DIR_RPM to ${WORK_DIR}/deploy-rpms for the rpm file, it would save a > lot time and disk space, or can we have a configuration variable to control > this ? We copy the files by default, but can use the symlink when the user > requires. Or we can just move the rpm files from ${WORK_DIR}/deploy-rpms to tmp/deploy/rpm which would also save the build time and disk space. 32468 2 richard.purdie 2013-04-22 13:59:29 +0000 I thought we were already hardlinking these files. It appears we don't so I'd accept a patch to hardlink them instead of copying. 32469 3 mark.hatle 2013-04-22 14:10:43 +0000 Just an implementation note. Can deploy-dir be set to another disk? If so, we need to make sure that if a hardlink can NOT be generated, a copy is still made. 32472 4 richard.purdie 2013-04-22 14:36:09 +0000 In theory it can, in practise I don't want to support it and I'm not aware of anyone doing that despite various other hardlinks we use in the system. 32573 5 liezhi.yang 2013-04-24 10:22:20 +0000 (In reply to comment #4) > In theory it can, in practise I don't want to support it and I'm not aware > of anyone doing that despite various other hardlinks we use in the system. Hi Richard, I set the DEPLOY_DIR to another filesystem, it worked well, and the bitbake.conf said: # Setting DEPLOY_DIR outside of TMPDIR is helpful, when you are using # packaged staging and/or multimachine. DEPLOY_DIR ?= "${TMPDIR}/deploy" DEPLOY_DIR_TAR = "${DEPLOY_DIR}/tar" DEPLOY_DIR_IPK = "${DEPLOY_DIR}/ipk" DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm" So it seems that we'd better copy when can't hardlink, I've made a patch for it, and it is in testing now, I will send it to the oe-core mailing list later: diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 67ddc46..3565627 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -198,7 +198,14 @@ def sstate_install(ss, d): # Run the actual file install for state in ss['dirs']: if os.path.exists(state[1]): - oe.path.copytree(state[1], state[2]) + tgt_devno = None + if os.path.exists(state[2]): + tgt_devno = os.stat(state[2]) + if (os.stat(state[1]).st_dev == tgt_devno): + oe.path.copyhardlinktree(state[1], state[2]) + else: + oe.path.copytree(state[1], state[2]) + for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split(): bb.build.exec_func(postinst, d) 32597 6 richard.purdie 2013-04-24 16:25:25 +0000 btw, patch looks good but can you please put the: if (os.stat(state[1]).st_dev == tgt_devno): into the oe.path.copyhardlinktree() function as it belongs there, not in this code. 32884 7 liezhi.yang 2013-05-04 07:34:16 +0000 Merged into master: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=c51e8b4532361b11fe73c28613653c0f5aaf566b And great thanks for RP's fix: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=61823f61948df24c3329de2766c715e562711a94 34047 8 mihai.lindner 2013-06-21 12:49:42 +0000 It works. # ls -lai tmp/deploy/rpm/x86_64/at-3.1.13-r5.x86_64.rpm tmp/work/x86_64-poky-linux/at/3.1.13-r5/deploy-rpms/x86_64/at-3.1.13-r5.x86_64.rpm 51123552 -rw-r--r-- 2 mihai users 34989 Jun 20 19:26 tmp/deploy/rpm/x86_64/at-3.1.13-r5.x86_64.rpm 51123552 -rw-r--r-- 2 mihai users 34989 Jun 20 19:26 tmp/work/x86_64-poky-linux/at/3.1.13-r5/deploy-rpms/x86_64/at-3.1.13-r5.x86_64.rpm