| Summary: | [Performance] make hardlinks for files in DEPLOY_DIR when possible | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Robert Yang <liezhi.yang> |
| Component: | deployment | Assignee: | Robert Yang <liezhi.yang> |
| Status: | VERIFIED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | Medium+ | CC: | mark.hatle, mihai.lindner, richard.purdie, sgw |
| Version: | 1.5 | ||
| Target Milestone: | 1.5 M1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 4643 | ||
(In reply to comment #0) > The rpm file (suppose PACKAGE_CLASSES = "package_rpm") in > ${WORK_DIR}/deploy-rpms (AKA PKGWRITEDIRRPM) is the same as the one in > DEPLOY_DIR_RPM (e.g., tmp/deploy/rpm), if we make symlink from > DEPLOY_DIR_RPM to ${WORK_DIR}/deploy-rpms for the rpm file, it would save a > lot time and disk space, or can we have a configuration variable to control > this ? We copy the files by default, but can use the symlink when the user > requires. Or we can just move the rpm files from ${WORK_DIR}/deploy-rpms to tmp/deploy/rpm which would also save the build time and disk space. I thought we were already hardlinking these files. It appears we don't so I'd accept a patch to hardlink them instead of copying. Just an implementation note. Can deploy-dir be set to another disk? If so, we need to make sure that if a hardlink can NOT be generated, a copy is still made. In theory it can, in practise I don't want to support it and I'm not aware of anyone doing that despite various other hardlinks we use in the system. (In reply to comment #4) > In theory it can, in practise I don't want to support it and I'm not aware > of anyone doing that despite various other hardlinks we use in the system. Hi Richard, I set the DEPLOY_DIR to another filesystem, it worked well, and the bitbake.conf said: # Setting DEPLOY_DIR outside of TMPDIR is helpful, when you are using # packaged staging and/or multimachine. DEPLOY_DIR ?= "${TMPDIR}/deploy" DEPLOY_DIR_TAR = "${DEPLOY_DIR}/tar" DEPLOY_DIR_IPK = "${DEPLOY_DIR}/ipk" DEPLOY_DIR_RPM = "${DEPLOY_DIR}/rpm" So it seems that we'd better copy when can't hardlink, I've made a patch for it, and it is in testing now, I will send it to the oe-core mailing list later: diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index 67ddc46..3565627 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -198,7 +198,14 @@ def sstate_install(ss, d): # Run the actual file install for state in ss['dirs']: if os.path.exists(state[1]): - oe.path.copytree(state[1], state[2]) + tgt_devno = None + if os.path.exists(state[2]): + tgt_devno = os.stat(state[2]) + if (os.stat(state[1]).st_dev == tgt_devno): + oe.path.copyhardlinktree(state[1], state[2]) + else: + oe.path.copytree(state[1], state[2]) + for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split(): bb.build.exec_func(postinst, d) btw, patch looks good but can you please put the: if (os.stat(state[1]).st_dev == tgt_devno): into the oe.path.copyhardlinktree() function as it belongs there, not in this code. Merged into master: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=c51e8b4532361b11fe73c28613653c0f5aaf566b And great thanks for RP's fix: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=61823f61948df24c3329de2766c715e562711a94 It works. # ls -lai tmp/deploy/rpm/x86_64/at-3.1.13-r5.x86_64.rpm tmp/work/x86_64-poky-linux/at/3.1.13-r5/deploy-rpms/x86_64/at-3.1.13-r5.x86_64.rpm 51123552 -rw-r--r-- 2 mihai users 34989 Jun 20 19:26 tmp/deploy/rpm/x86_64/at-3.1.13-r5.x86_64.rpm 51123552 -rw-r--r-- 2 mihai users 34989 Jun 20 19:26 tmp/work/x86_64-poky-linux/at/3.1.13-r5/deploy-rpms/x86_64/at-3.1.13-r5.x86_64.rpm |
The rpm file (suppose PACKAGE_CLASSES = "package_rpm") in ${WORK_DIR}/deploy-rpms (AKA PKGWRITEDIRRPM) is the same as the one in DEPLOY_DIR_RPM (e.g., tmp/deploy/rpm), if we make symlink from DEPLOY_DIR_RPM to ${WORK_DIR}/deploy-rpms for the rpm file, it would save a lot time and disk space, or can we have a configuration variable to control this ? We copy the files by default, but can use the symlink when the user requires.