9189 2016-03-01 09:35:08 +0000 [meta-swupd] Creates a lot of duplicate files 2016-12-08 17:44:06 +0000 1 1 7 Build System, Metadata & Runtime Other YP Layers meta-swupd 2.2 x86 Multiple RESOLVED FIXED Medium enhancement 2.3 M1 1 joshuagloe patrick.ohly igor.stoppa patrick.ohly poky.bs.watcher poky.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 59565 0 joshuagloe 2016-03-01 09:35:08 +0000 Due to the way swupd works by processing chroot-like bundle directories on each os release we potentially end up carrying a lot of duplicate files in DEPLOY_DIR_SWUPD. We should look at using the hardlink program (or possibly fdupes) to replace duplicate files in that directory with hard links in order to save disk space. 62094 1 igor.stoppa 2016-05-11 11:14:28 +0000 Could this be a requirement for swupd? Anyway it seems like an optional step. If one makes lots of throwaway builds, it might be faster to jsut bear with the duplicated files. I think it would be ok to have it enabled by default only if the overhead of identifying duplicates is minimum (could the hash from the manifests of previous releases help?) 62114 2 joshuagloe 2016-05-11 18:46:49 +0000 (In reply to comment #1) > Could this be a requirement for swupd? At least partially, yes. In fact swupd-server introduced a feature to use hardlinks when possible during swupd_make_pack https://github.com/clearlinux/swupd-server/releases/tag/v3.1.2 I haven't had any chance to investigate it yet, nor try to determine whether it needs to be enabled or whether it requires extra utilities on the host (i.e. a hardlink-native). > Anyway it seems like an optional step. > If one makes lots of throwaway builds, it might be faster to jsut bear with > the duplicated files. > > I think it would be ok to have it enabled by default only if the overhead of > identifying duplicates is minimum (could the hash from the manifests of > previous releases help?) This is useful feedback, thanks. ISTR the request for hardlink support came from a colleague on the Ostro team? If the demand isn't still there I shall push this out to a later milestone. 67863 3 patrick.ohly 2016-11-02 14:45:21 +0000 The right approach for avoiding duplicate files in the www directory is to create updates from the last build: swupd_create_update then automatically reuses existing, unmodified files. As an added bonus, that also reduces time spent in swupd_make_fullfiles because only new or modified files need to be handled. I'm currently working on some code which does that. Note that this is different from computing deltas. 67864 4 patrick.ohly 2016-11-02 14:46:20 +0000 Let's pull this in. Concern about disk space in the update repo was also mentioned by other parties interested in using meta-swupd. 68958 5 patrick.ohly 2016-12-08 17:44:06 +0000 Current meta-swupd master (today's commit 07ad71b78) supports incremental builds, so no files get duplicated anymore.