For example, this is libffi_3.1.bb's SRC_URI: SRC_URI = "ftp://sourceware.org/pub/libffi/${BP}.tar.gz" and It would fetch ${BP}.tar.gz to DL_DIR, it would help us to cleanup the DL_DIR if we can save a log like this: /path/to/libffi_3.1.bb fetches ${BP}.tar.gz Then we can cleanup the DL_DIR according to this log, maybe we can save the log to the DL_DIR or tmp/log/cooker
I'm a little worried this is the wrong way to solve this problem. The fetcher has code which knows how to clean a given recipe so it would be straightforward to also make it generate a list of files a given fetch could touch. We could therefore have a "fetchall" type task which would list all the DL_DIR files known to the current metadata. The implication would then be that any files not known would no longer be required. Would that work as a proposal instead of this?
(In reply to comment #1) > I'm a little worried this is the wrong way to solve this problem. The > fetcher has code which knows how to clean a given recipe so it would be > straightforward to also make it generate a list of files a given fetch could > touch. We could therefore have a "fetchall" type task which would list all > the DL_DIR files known to the current metadata. The implication would then > be that any files not known would no longer be required. > > Would that work as a proposal instead of this? Thanks, Yes, I think that it should work. We already have the fetchall task in meta/classes/utility-tasks.bbclass: addtask fetchall after do_fetch do_fetchall[recrdeptask] = "do_fetchall do_fetch" do_fetchall[recideptask] = "do_${BB_DEFAULT_TASK}" do_fetchall() { : } It seems that now what we need is let do_fetch save the list of what has been fetched ? for example: python base_do_fetch() { [snip] if saved_fetch_list: save_fetch_list to ${T}/fetch_list [snip] } It would be better if we can save the list to TMPDIR or DL_DIR rather than T so that we can easily find the list and cleanup the DL_DIR according to the list.
Just a spontaneous idea: What's about adding this file to the build history?
(In reply to comment #3) > Just a spontaneous idea: What's about adding this file to the build history? Sounds good to me.
If the actual goal is to clean up a DL_DIR, why not just delete all files that haven't been accessed for a year?
We talked about the defect on the YP call. Richard seemed to think that giving out the info could be problematic. Instead, we need to define the desired change in behaviour and extend the fetcher and other code to implement that. As Ross say, if it's just general disk space savings deleting things that are too old works fine. I suspect that this is part of you rebase work where you want to be sure that all downloaded source has been added to our local repo that the fetcher uses. Robert can you explain what workflow you are trying to fix?
Created attachment 4543 [details] fetch_helper.bbclass
Yes, this is is for rebase work, we need cleanup old source (tarball) and new one when a recipe is upgraded, or remove the source (tarball and git) when a recipe is removed, I've written a local bbclass to do it, please see the attachment, fetch_helper.bbclass.
This came up again in bug triage. I was told this was to create a list of sources that could be placed in a clean DL_DIR to ship with a product. To do that I'd much prefer an approach where a new DL_DIR is set, the previous DL_DIR is set as a mirror and fetches are run over the combinations of metadata you want to ship sources for. If alternatively this is an issue trying to clean up DL_DIR, as ross says, removing files not touch for X months should work just as well?
Hi RP, We cleanup the files twice a week, so X months is a little long. I was thinking that others may also have similar requirements so I've file this enhancement, but seems not, I'm learning to close this ER if no objections.
At WR, when we update, we just add new tarballs or git repos and then at a later time, we remove unused source archives.