Bug 8422

Summary: we currently don't allow setting of sstate_dir or dl_dir
Product: [Build System, Metadata & Runtime] Toaster Reporter: brian avery <brian.avery>
Component: toasterAssignee: Sujith H <sujith.h>
Status: RESOLVED FIXED QA Contact: Mihail Stanciu <stanciux.mihail>
Severity: enhancement    
Priority: Medium CC: belen.barros.pena, brian.avery, eduard.bartosh, jessica.zhang, stanciux.mihail, sujith.h
Version: 2.0   
Target Milestone: 2.1   
Hardware: x86   
OS: Multiple   
Whiteboard: GUI design available
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)
Attachments:
Description Flags
Design - Setting DL_DIR and SSTATE_DIR
none
Design - Setting DL_DIR and SSTATE_DIR none

Description brian avery 2015-09-30 21:04:19 UTC
For local host builds, allowing people to set these directories on a per project basis would help speed up their builds considerably.  Right now, toaster is implicitly enforcing that the user download everything from scratch each time they setup a new instance. The same holds for not being able to leverage a host or network sstate mirror.
Comment 1 brian avery 2015-11-12 22:26:30 UTC
Since we will move to Paul's suggestion to use a different build dir per project, we would need to be able to share the sstate and downloads to ease the pain/size issue.  Otherwise we would end up rebuilding or redownloading too many items. So I moved it back to 2.1.  Since sstate is quite happy to be mixed together, we can do it in the background w/o ui changes initially, similar to how we handle the _toaster_clone_layers.
Comment 2 Belen Barros Pena 2015-11-13 09:55:51 UTC
Yes, we could do that. But to be honest I rather take the time to implement the UI for this, since it should be fairly easy and will give people control over these things.

We just need to set a sane default (for those who don't want to mess with it), and provide a way to change that default for those who do want to mess with it.

Maybe it could work like this. By default, both sstate and downloads directories will be set to a shared directory across projects. This can be overridden on a per-project basis when you create the project and via the 'BitBake variables' page in the project configuration after the project is created.

If you want to change the default location of the share directories for the toaster instance, you should be able to do so via a variable and/or via the Django admin interface.

We won't provide an option to delete the contents of those directories from Toaster. And if any of the directories is deleted under the hood, Toaster should create them in the specified location when a build starts. 

If this sounds ok to everybody, I should be able to get a design ready fairly quickly.

The only question in my head is that this effectively changes the default structure of the poky directory, so the moment you start Toaster that directory will undergo some changes in its structure (a common sstate and downloads directory will be created outside the build directory, and new build directories will be created). I am not sure if this is a problem or not: just bringing it up.
Comment 3 brian avery 2015-11-13 18:42:19 UTC
so, a couple of things:
1) If you can do the UI to make these (SSTATE_DIR & DL_DIR) settable for each project then that would be gr8!
2) overriding them for a whole instance should probably be done in the toasteroconf.json where other variables like this are set.
3) for 2.1 I will definitely encourage people in the manual to *not* run toaster from within poky. but, we are changing the structure anyway since we will have build_cli build_proj1 build_proj2 in order to accommodate the bug 7880.  It is much safer to put it outside the git tree so an inadvertent git clean doesn't wipe all your data.
4) if toaster set;s the variables, bitbake will recreate the directories as needed if someone removed them under the hood so I think leaving this up to the command line is fine by me
Comment 4 Belen Barros Pena 2015-11-18 11:05:07 UTC
Created attachment 2853 [details]
Design - Setting DL_DIR and SSTATE_DIR
Comment 5 Belen Barros Pena 2015-11-19 16:12:21 UTC
Created attachment 2858 [details]
Design - Setting DL_DIR and SSTATE_DIR

Just a small correction on page 6: validation should happen on key up, not on focus out
Comment 6 Sujith H 2016-02-04 10:29:33 UTC
Taking up this task and start working on it.
Comment 7 Sujith H 2016-02-17 16:53:37 UTC
The work is already in progress. Will send patch for review soon.
Comment 9 Ed Bartosh 2016-04-07 09:55:30 UTC
The implementation has been accepted to bitbake master: http://cgit.openembedded.org/bitbake/commit/?id=bd9f8973d4c9c0722874a058466b1b911112500e