When you retag, did you git clone your new repo and do a clean build?

Can you capture your "bitbake -e" output and your local.conf for both 0.1.3 and 0.1.4 build and attach to the bug.  It seems the way you setup your SDK_VERSION and DISTRO_VERSION are not quite right.  Please take a look at how poky setup SDK_VERSION and DISTRO_VERSION, they're all under meta-yocto/conf/poky.conf.  If you model after that, it should trigger regenerate the SDK once you change your distro version, but by looking at your build log, that didn't happen to your layer after change the version to 0.1.4

(In reply to comment #1)
> When you retag, did you git clone your new repo and do a clean build?

The tagging I refer to is in the repo itself so a clone was not needed. If I do do a clean build ( as in "rm -rf tmp/ sstate_cache/ downloads" and then rebuild) then the expected file is generated with the new tag and it extracts into only one directory with the correct version. To continue the example from before, the file  eglibc-x86_64-i586-toolchain-v0.1.4.tar.bz2 is created and it extracts only to /opt/concordia/v0.1.4

Created attachment 524 [details]
output of bitbake -e when tag is v0.1.3

Created attachment 525 [details]
output of bitbake -e when tag is v0.1.4

Created attachment 526 [details]
local.conf

Created attachment 527 [details]
distro configuration

(In reply to comment #2)
> Can you capture your "bitbake -e" output and your local.conf for both 0.1.3
> and 0.1.4 build and attach to the bug.  It seems the way you setup your
> SDK_VERSION and DISTRO_VERSION are not quite right.  Please take a look at
> how poky setup SDK_VERSION and DISTRO_VERSION, they're all under
> meta-yocto/conf/poky.conf.  If you model after that, it should trigger
> regenerate the SDK once you change your distro version, but by looking at
> your build log, that didn't happen to your layer after change the version to
> 0.1.4

Yes, the original settings are in meta-yocto/conf/poky.conf, which is included in conf/distro/poky-tiny.conf, which is included in our own distro (concordia's configuration), which is attached also. It is in this new distro configuration that I override the settings of SDK_VERSION and DISTRO_VERSION, not in local.conf. I do attach local.conf also though.

Could you please point out where I am setting the values incorrectly?

another question is if you just rebuild your image, not the sdk, do you have similar issue for 0.1.4?  The problem is somehow when you rebuild 0.1.4, it didn't trigger rebuild the packages for 0.1.4 which it should, instead, it just reuse what's build for 0.1.3

I've checked with Beth our release gal, and here's our procedure for yocto releases, can you follow the same step to see whether still have issue?

02:18:03 PM) eflanagan: ok, I think I see why I'm not seeing it
(02:18:20 PM) eflanagan: I run with a clean build directory but with a non-clean SSTATE_DIR
(02:19:20 PM) eflanagan: jzhang: So I would never hit what they're hitting since I rm TMPDIR and build with SSTATE_DIR pointed to the nas

(In reply to comment #9)
> another question is if you just rebuild your image, not the sdk, do you have
> similar issue for 0.1.4?  The problem is somehow when you rebuild 0.1.4, it
> didn't trigger rebuild the packages for 0.1.4 which it should, instead, it
> just reuse what's build for 0.1.3

rebuilding the image does not have this issue ... but from what I understand the image does not seem to use the version number internally though, just the filename, which seems to be ok for the sdk also

When I try to reproduce with version v0.1.t first and then create new version v0.1.x I see the following in my working directory:

$ find . | grep environment-setup-core2-ccd-linux
./core2-ccd-linux/meta-toolchain-1.0-r7/sdk/image/opt/concordia/v0.1.t/environment-setup-core2-ccd-linux
./core2-ccd-linux/meta-toolchain-1.0-r7/sdk/image/opt/concordia/v0.1.x/environment-setup-core2-ccd-linux
./x86_64-nativesdk-ccdsdk-linux/meta-environment-i586-1.0-r8/sdk/image/opt/concordia/v0.1.t/environment-setup-core2-ccd-linux
./x86_64-nativesdk-ccdsdk-linux/meta-environment-i586-1.0-r8/package/opt/concordia/v0.1.t/environment-setup-core2-ccd-linux
./x86_64-nativesdk-ccdsdk-linux/meta-environment-i586-1.0-r8/image/opt/concordia/v0.1.t/environment-setup-core2-ccd-linux
./x86_64-nativesdk-ccdsdk-linux/meta-environment-i586-1.0-r8/packages-split/meta-environment-i586/opt/concordia/v0.1.t/environment-setup-core2-ccd-linux

(In reply to comment #10)
> I've checked with Beth our release gal, and here's our procedure for yocto
> releases, can you follow the same step to see whether still have issue?
> 
> 02:18:03 PM) eflanagan: ok, I think I see why I'm not seeing it
> (02:18:20 PM) eflanagan: I run with a clean build directory but with a
> non-clean SSTATE_DIR
> (02:19:20 PM) eflanagan: jzhang: So I would never hit what they're hitting
> since I rm TMPDIR and build with SSTATE_DIR pointed to the nas

Just a note to let you know that I am planning to test this. I completed a clean build and was about to test this when you arrived in my cube and we went through the steps to reproduce the issue instead. At the moment our build machine is undergoing an OS upgrade. When this is complete I will redo a clean build and then try your suggestion. Will let you know the results asap.

here's what I got when I tried to reproduce just using Yocto by following your steps, the following is the start of build output after I bump poky.conf distro_version from 1.2 to 1.3:

jzhang@jzhang-desktop:~/poky-master-build$ bitbake meta-toolchain
WARNING: Host distribution "Ubuntu 10.04.1 LTS" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Parsing recipes: 100% |#########################################| Time: 00:00:11
Parsing of 846 .bb files complete (0 cached, 846 parsed). 1144 targets, 19 skipped, 0 masked, 0 errors.

Build Configuration:
BB_VERSION        = "1.15.2"
TARGET_ARCH       = "i586"
TARGET_OS         = "linux"
MACHINE           = "qemux86"
DISTRO            = "poky"
DISTRO_VERSION    = "1.3+snapshot-20120523"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              
meta-yocto        = "master-next:b254f398c908250d23faef3804401b85e04b5caf"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Running task 167 of 1854 (ID: 919, /home/jzhang/poky-master/meta/recipes-devtools/binutils/binutils-crosssdk_2.22.bb, do_configure)
NOTE: Running task 178 of 1854 (ID: 957, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_3.2.bb, do_configure)
NOTE: Running task 191 of 1854 (ID: 1300, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-core/eglibc/eglibc-initial_2.15.bb, do_unpack)
NOTE: Running task 202 of 1854 (ID: 501, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-core/eglibc/eglibc_2.15.bb, do_unpack)
NOTE: Running task 1002 of 1854 (ID: 1089, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_configure)
NOTE: Running noexec task 1059 of 1854 (ID: 725, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_configure)
NOTE: Running noexec task 1060 of 1854 (ID: 726, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_compile)
NOTE: Running task 1134 of 1854 (ID: 727, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_generate_content)
NOTE: Running task 1175 of 1854 (ID: 154, /home/jzhang/poky-master/meta/recipes-core/tasks/task-cross-canadian.bb, do_configure)
NOTE: package linux-libc-headers-nativesdk-3.2-r1: task do_configure: Started
NOTE: package eglibc-initial-nativesdk-2.15-r7+svnr17386: task do_unpack: Started
NOTE: package eglibc-nativesdk-2.15-r7+svnr17386: task do_unpack: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_configure: Started
NOTE: package meta-environment-i586-1.0-r8: task do_generate_content: Started
NOTE: package task-cross-canadian-i586-1.0-r0: task do_configure: Started
NOTE: package binutils-crosssdk-2.22-r2: task do_configure: Started
NOTE: package meta-environment-i586-1.0-r8: task do_generate_content: Succeeded
NOTE: package task-cross-canadian-i586-1.0-r0: task do_configure: Succeeded
NOTE: Running task 1293 of 1854 (ID: 721, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_install)
NOTE: Running task 1294 of 1854 (ID: 155, /home/jzhang/poky-master/meta/recipes-core/tasks/task-cross-canadian.bb, do_compile)
NOTE: package task-cross-canadian-i586-1.0-r0: task do_compile: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_configure: Succeeded
NOTE: Running task 1295 of 1854 (ID: 1090, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_compile)
NOTE: package task-cross-canadian-i586-1.0-r0: task do_compile: Succeeded
NOTE: Running task 1296 of 1854 (ID: 150, /home/jzhang/poky-master/meta/recipes-core/tasks/task-cross-canadian.bb, do_install)
NOTE: package meta-environment-i586-1.0-r8: task do_install: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_compile: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_compile: Succeeded
NOTE: package meta-environment-i586-1.0-r8: task do_install: Succeeded
NOTE: package task-cross-canadian-i586-1.0-r0: task do_install: Started
NOTE: Running task 1297 of 1854 (ID: 729, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_package)
NOTE: Running noexec task 1298 of 1854 (ID: 722, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_populate_sysroot)
NOTE: Running task 1299 of 1854 (ID: 1085, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_install)
NOTE: package task-cross-canadian-i586-1.0-r0: task do_install: Succeeded
NOTE: Running task 1300 of 1854 (ID: 157, /home/jzhang/poky-master/meta/recipes-core/tasks/task-cross-canadian.bb, do_package)
NOTE: Running task 1301 of 1854 (ID: 151, /home/jzhang/poky-master/meta/recipes-core/tasks/task-cross-canadian.bb, do_populate_sysroot)
NOTE: package meta-environment-i586-1.0-r8: task do_package: Started
NOTE: package task-cross-canadian-i586-1.0-r0: task do_package: Started
NOTE: package task-cross-canadian-i586-1.0-r0: task do_populate_sysroot: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_install: Started
NOTE: package task-cross-canadian-i586-1.0-r0: task do_populate_sysroot: Succeeded
NOTE: package gnu-config-nativesdk-20111111-r1: task do_install: Succeeded
NOTE: Running task 1302 of 1854 (ID: 1092, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_package)
NOTE: Running task 1303 of 1854 (ID: 1086, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_populate_sysroot)
NOTE: package gnu-config-nativesdk-20111111-r1: task do_package: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_populate_sysroot: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_populate_sysroot: Succeeded
NOTE: package task-cross-canadian-i586-1.0-r0: task do_package: Succeeded
NOTE: package meta-environment-i586-1.0-r8: task do_package: Succeeded
NOTE: Running task 1304 of 1854 (ID: 731, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_package_write_ipk)
NOTE: package gnu-config-nativesdk-20111111-r1: task do_package: Succeeded
NOTE: Running task 1305 of 1854 (ID: 1094, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_package_write_ipk)
NOTE: package meta-environment-i586-1.0-r8: task do_package_write_ipk: Started
NOTE: package gnu-config-nativesdk-20111111-r1: task do_package_write_ipk: Started
NOTE: package meta-environment-i586-1.0-r8: task do_package_write_ipk: Succeeded
NOTE: Running noexec task 1306 of 1854 (ID: 728, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_package_write)
NOTE: Running noexec task 1307 of 1854 (ID: 724, /home/jzhang/poky-master/meta/recipes-core/meta/meta-environment.bb, do_build)
NOTE: package gnu-config-nativesdk-20111111-r1: task do_package_write_ipk: Succeeded
NOTE: Running noexec task 1308 of 1854 (ID: 1091, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_package_write)
NOTE: Running noexec task 1309 of 1854 (ID: 1088, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-devtools/gnu-config/gnu-config_20111111.bb, do_build)
NOTE: package eglibc-initial-nativesdk-2.15-r7+svnr17386: task do_unpack: Succeeded
NOTE: Running task 1310 of 1854 (ID: 1301, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-core/eglibc/eglibc-initial_2.15.bb, do_patch)
NOTE: package eglibc-initial-nativesdk-2.15-r7+svnr17386: task do_patch: Started
NOTE: package linux-libc-headers-nativesdk-3.2-r1: task do_configure: Succeeded
NOTE: Running task 1311 of 1854 (ID: 958, virtual:nativesdk:/home/jzhang/poky-master/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_3.2.bb, do_compile)
NOTE: package linux-libc-headers-nativesdk-3.2-r1: task do_compile: Started
NOTE: package linux-libc-headers-nativesdk-

Created attachment 533 [details]
do_install log of eglibc-initial-nativesdk-2.15-r6+svnr17386

(In reply to comment #10)
> I've checked with Beth our release gal, and here's our procedure for yocto
> releases, can you follow the same step to see whether still have issue?
> 
> 02:18:03 PM) eflanagan: ok, I think I see why I'm not seeing it
> (02:18:20 PM) eflanagan: I run with a clean build directory but with a
> non-clean SSTATE_DIR
> (02:19:20 PM) eflanagan: jzhang: So I would never hit what they're hitting
> since I rm TMPDIR and build with SSTATE_DIR pointed to the nas

I did a clean build (starting with rm -rf tmp/ downloads/ sstate-cache/) using a tag v0.1.t as DISTRO_VERSION and SDK_VERSION. After this I edited a file (README) that does not form part of any layer, commited it and retagged the repo to v0.1.x, thus setting DISTRO_VERSION and SDK_VERSION to v0.1.x. After this I only did a "rm -rf tmp" and then tried to rebuild. Unfortunately this failed while in do_install of eglibc-initial-nativesdk-2.15-r6+svnr17386. See attachment for complete log

The more I try to figure out what is going on here the more I think that a change in SDK_VERSION should invalidate the entire SDK build. Since DISTRO_VERSION and SDK_VERSION is similar I thus assume a change here should invalidate _all_ builds then, images also. That is, nothing from sstate cache should be re-used. At the moment the actual value of SDKPATH is not in the siginfo of the sstate cache so if it (value of SDKPATH) changes the sstate scripts will not rebuild anything. Even so, many packages for which there is sstate data have the SDKPATH hardcoded in their output. I tried a systemwide override in my local.conf of:

do_populate_sysroot[vardeps] += "${SDKPATH}"
 
This seemed to have helped a little bit but it seems that SDKPATH is buried even deeper in many more tasks than do_populate_sysroot ... for example, gcc-crosssdk-initial_4.6 actually uses the binaries found in SDKPATH in its do_compile task.

I think this may finally be addressed with the following commit now in master:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=8d3285f99b685eb08399b037b49fe5d8270fd589

I don't think so since this is what I have been doing all along (see bug 2342 that prompted this patch). In my setup I set DISTRO_VERSION and SDK_VERSION to same value so replacing one with the other should not change behavior. Even so, maybe with this change you will be able to reproduce issue easier? If you create a build with SDK_VERSION set to one value and then modify SDK_VERSION and recreate the SDK ... does it create all the artifacts with the new version and extract to directories with the correct name? I have been forced to do a clean build every time to obtain the expected results.

I just tried this with recent master and it worked; changing SDK_VERSION triggered re-running a number of tasks and when meta-toolchain finished building, the new version was reflected in the output file name and the path within the SDK itself (i.e. the default installation path, since this is now relocatable).

With the dylan branch (what will become 1.4.1) the same test succeeded but the output filename did not change since DISTRO_VERSION is still being used there. The build failed during do_install for several recipes when I tried the same with the danny branch (what will become 1.3.2) however.

(In reply to comment #19)
> I just tried this with recent master and it worked; changing SDK_VERSION
> triggered re-running a number of tasks and when meta-toolchain finished
> building, the new version was reflected in the output file name and the path
> within the SDK itself (i.e. the default installation path, since this is now
> relocatable).
> 
> With the dylan branch (what will become 1.4.1) the same test succeeded but
> the output filename did not change since DISTRO_VERSION is still being used
> there. The build failed during do_install for several recipes when I tried
> the same with the danny branch (what will become 1.3.2) however.

I see now. It thus seems that more than just the patch mentioned in comment 17 is needed to address this issue. We are in the process of upgrading from Yocto 1.2 (which this issue was reported against) to Yocto 1.4. On previous occasion I tried to work with master branch but ran into issues since it is a moving target in many aspects. Do you think that testing with Yocto 1.4 would be ok if I just apply patch from comment 17 on top? Alternatively, could you please recommend the id of the commit on master that I should be testing with?

I also ran into issues with "do_install" (see comment 15) (still Yocto 1.2)

I'd say if you want to work with something stable I'd recommend going with 1.4, perhaps with a move to 1.4.1 when it gets released which will be fairly soon. I'll talk to Richard and see if we can get the aforementioned SDK_VERSION instead of DISTRO_VERSION patch in for 1.4.1 as well.

Marking as resolved as I think we've addressed this now; if not please re-open. (The aforementioned patch is now in the dylan branch and will be part of 1.4.1.)

I am currently running with it and tried to verify it by building an SDK from scratch, making some modifications and then trying to rebuild the SDK. The filename of the SDK as well as the default path it extracts to changed as expected. 

While testing this I encountered an issue during extracting of rebuilt SDK:
$ sudo ./concordia-eglibc-x86_64-i586-toolchain-v0.11-rc0-1-g73d41d0.sh 
Enter target directory for SDK (default: /opt/concordia/v0.11-rc0-1-g73d41d0): 
You are about to install the SDK to "/opt/concordia/v0.11-rc0-1-g73d41d0". Proceed[Y/n]?Y
Extracting SDK...done
Setting it up...find: `/opt/concordia/v0.11-rc0-1-g73d41d0/sysroots/x86_64-ccdsdk-linux/lib': No such file or directory
SDK could not be set up. Relocate script unable to find ld-linux.so. Abort!

We are currently building all our images from scratch (ensure tmp, downloads, as well as sstate is clear before a build). We thus do not set ourselves up to encounter this issue.

This latest result is with 1.4 + the aforementioned patch, is that correct?

(In reply to comment #24)
> This latest result is with 1.4 + the aforementioned patch, is that correct?

Correct. I am running with one more patch to 1.4 that is trying to resolve another issue I am encountering. To see exactly what I am running you can take a look at tag ccd-v0.11-rc0 on git://otcgit.jf.intel.com/rchatre/ccd-yocto.git ... or as a shortcut you can just view the tree on gitweb:

http://otcgit.jf.intel.com/?p=rchatre/ccd-yocto.git;a=summary