Bug 3071

After the recent bug fix, sstate files at least represent the configuration they were built against. The issue is therefore that the COUNT numbers may differ between builders and the sstate cache only applies if they match.

At least on the Yocto Autobuilders, we persist the bb_persist_data.sqlite3 between build so we don't see this problem. Only one builder can run a given build at a time and the file is copied to/from a shared disk.

Whether this process can be improved is an open question, I suspect the answer is yes, perhaps using the functionality in the network PR server since its a similar syncronisation issue.

Given that solutions do exist and builds do not break, I'm making this a 1.4 enhancement request.

Agreed, the issue was most important when gcc was built from git and when builder populating SSTATE_MIRROR was using different LOCALCOUNT then all packages depending on gcc-* weren't reused from SSTATE_MIRROR but built again on host just because of different LOCALCOUNT.

I've given this a lot of thought and I think we need to move the "increment" mechanism into the PR service and do this well, in one place. The proposal:

http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-November/003878.html

This bug has been fixed using the PRService.

Cheers,
Constantin

Can you reply on this:
http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/31052
?

I still see only git0+hash 
$ ls tmp-eglibc/deploy/ipk/*/*git*
tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dbg_0.1.8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dev_0.1.8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
tmp-eglibc/deploy/ipk/armv5te/opkg-utils_0.1.8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk

when higher "LOCALCOUNT" should be used:
sqlite> select * from BB_URI_LOCALCOUNT where key like '%opkg-utils%';
git:git.yoctoproject.org.opkg-utilsmaster-opkg-utils_rev|f1a9d6701993a6f6b1930b4c2bdb71525aa25320
git:git.yoctoproject.org.opkg-utilsmaster-opkg-utils_count|1

Does it import cache/bb_persist_data.sqlite3 automatically or can you extend PR_service wiki page with example how to migrate from LOCALCOUNTs to AUTOINCs?

(In reply to comment #5)
> Can you reply on this:
> http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/31052
> ?
> 
> I still see only git0+hash 
> $ ls tmp-eglibc/deploy/ipk/*/*git*
> tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dbg_0.1.
> 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dev_0.1.
> 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> tmp-eglibc/deploy/ipk/armv5te/opkg-utils_0.1.
> 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> 
> when higher "LOCALCOUNT" should be used:
> sqlite> select * from BB_URI_LOCALCOUNT where key like '%opkg-utils%';
> git:git.yoctoproject.org.opkg-utilsmaster-opkg-
> utils_rev|f1a9d6701993a6f6b1930b4c2bdb71525aa25320
> git:git.yoctoproject.org.opkg-utilsmaster-opkg-utils_count|1
> 
> Does it import cache/bb_persist_data.sqlite3 automatically or can you extend
> PR_service wiki page with example how to migrate from LOCALCOUNTs to
> AUTOINCs?

The idea is that we use the PRService API for associating a number with a (PV, SRCREV) tuple. When the sha1 changes the number increments.

The tmp/cache/prserv.sqlite3 database contains a table PRMAIN_nohist which has entries like:
AUTOINC-opkg-utils-0.1.8+git|i586|AUTOINC+f1a9d6701993a6f6b1930b4c2bdb71525aa25320|0

If you want to migrate LOCALCOUNT to AUTOINCs you have to add the corresponding entries.

I will update the wiki with additional examples.

Cheers,
Constantin

(In reply to comment #6)
> (In reply to comment #5)
> > Can you reply on this:
> > http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/31052
> > ?
> > 
> > I still see only git0+hash 
> > $ ls tmp-eglibc/deploy/ipk/*/*git*
> > tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dbg_0.1.
> > 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> > tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dev_0.1.
> > 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> > tmp-eglibc/deploy/ipk/armv5te/opkg-utils_0.1.
> > 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> > 
> > when higher "LOCALCOUNT" should be used:
> > sqlite> select * from BB_URI_LOCALCOUNT where key like '%opkg-utils%';
> > git:git.yoctoproject.org.opkg-utilsmaster-opkg-
> > utils_rev|f1a9d6701993a6f6b1930b4c2bdb71525aa25320
> > git:git.yoctoproject.org.opkg-utilsmaster-opkg-utils_count|1
> > 
> > Does it import cache/bb_persist_data.sqlite3 automatically or can you extend
> > PR_service wiki page with example how to migrate from LOCALCOUNTs to
> > AUTOINCs?
> 
> The idea is that we use the PRService API for associating a number with a
> (PV, SRCREV) tuple. When the sha1 changes the number increments.
> 
> The tmp/cache/prserv.sqlite3 database contains a table PRMAIN_nohist which
> has entries like:
> AUTOINC-opkg-utils-0.1.
> 8+git|i586|AUTOINC+f1a9d6701993a6f6b1930b4c2bdb71525aa25320|0
> 
> If you want to migrate LOCALCOUNT to AUTOINCs you have to add the
> corresponding entries.

You mean manually? :))

> I will update the wiki with additional examples.

Any ETA on that?

bitbake commit which breaks upgrade path was commited 14+ days ago, at least I've noticed it in time and didn't upgrade bitbake on servers used to populate binary feed..

(In reply to comment #7)
> (In reply to comment #6)
> > (In reply to comment #5)
> > > Can you reply on this:
> > > http://permalink.gmane.org/gmane.comp.handhelds.openembedded.core/31052
> > > ?
> > > 
> > > I still see only git0+hash 
> > > $ ls tmp-eglibc/deploy/ipk/*/*git*
> > > tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dbg_0.1.
> > > 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> > > tmp-eglibc/deploy/ipk/armv5te/opkg-utils-dev_0.1.
> > > 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> > > tmp-eglibc/deploy/ipk/armv5te/opkg-utils_0.1.
> > > 8+git0+f1a9d6701993a6f6b1930b4c2bdb71525aa25320-r0.0_armv5te.ipk
> > > 
> > > when higher "LOCALCOUNT" should be used:
> > > sqlite> select * from BB_URI_LOCALCOUNT where key like '%opkg-utils%';
> > > git:git.yoctoproject.org.opkg-utilsmaster-opkg-
> > > utils_rev|f1a9d6701993a6f6b1930b4c2bdb71525aa25320
> > > git:git.yoctoproject.org.opkg-utilsmaster-opkg-utils_count|1
> > > 
> > > Does it import cache/bb_persist_data.sqlite3 automatically or can you extend
> > > PR_service wiki page with example how to migrate from LOCALCOUNTs to
> > > AUTOINCs?
> > 
> > The idea is that we use the PRService API for associating a number with a
> > (PV, SRCREV) tuple. When the sha1 changes the number increments.
> > 
> > The tmp/cache/prserv.sqlite3 database contains a table PRMAIN_nohist which
> > has entries like:
> > AUTOINC-opkg-utils-0.1.
> > 8+git|i586|AUTOINC+f1a9d6701993a6f6b1930b4c2bdb71525aa25320|0
> > 
> > If you want to migrate LOCALCOUNT to AUTOINCs you have to add the
> > corresponding entries.
> 
> You mean manually? :))

I don't have any script that migrates LOCALCOUNT to AUTOINCs. Sorry.
> 
> > I will update the wiki with additional examples.
> 
> Any ETA on that?

I will try these days to update the wiki.

> 
> bitbake commit which breaks upgrade path was commited 14+ days ago, at least
> I've noticed it in time and didn't upgrade bitbake on servers used to
> populate binary feed..

No upgrade path from LOCALCOUNT to AUTOINC for git recipe isn't acceptable.

I can write such script for myself, but we cannot assume that other builders will migrate their bb_persist_data.sqlite3.

And there wasn't big fat warning that binary feeds will get broken when that bitbake patch was merged..

(In reply to comment #9)
> No upgrade path from LOCALCOUNT to AUTOINC for git recipe isn't acceptable.
> 
> I can write such script for myself, but we cannot assume that other builders
> will migrate their bb_persist_data.sqlite3.
> 
> And there wasn't big fat warning that binary feeds will get broken when that
> bitbake patch was merged..

Ok. I will write a script that does the migration.

Cheers,
Constantin

Patches sent for the migration script feature.

Cheers,
Constantin

(In reply to comment #11)
> Patches sent for the migration script feature.
> 
> Cheers,
> Constantin

The patches have been merged. (commit 8d6e55bf2192524bda84138b5356a2791adbbe61)

Cheers,
Constantin

(In reply to comment #12)
> (In reply to comment #11)
> > Patches sent for the migration script feature.
> > 
> > Cheers,
> > Constantin
> 
> The patches have been merged. (commit
> 8d6e55bf2192524bda84138b5356a2791adbbe61)

Thanks!

Please update that wiki page.

Migration does not work correctly with non-default PE.
http://lists.linuxtogo.org/pipermail/openembedded-core/2013-January/034950.html

Please update
https://wiki.yoctoproject.org/wiki/index.php?title=PR_Service
with information about changed PRSERV_HOST and migration script before resolving this.

(In reply to comment #14)
> Migration does not work correctly with non-default PE.
> http://lists.linuxtogo.org/pipermail/openembedded-core/2013-January/034950.
> html
> 
Patch for fixing the non-default PE issues sent to the mailing list.

> Please update
> https://wiki.yoctoproject.org/wiki/index.php?title=PR_Service
> with information about changed PRSERV_HOST and migration script before
> resolving this.

Right, I will update it today. I have waited for the patches to be merged.

Btw, thanks for sending the fix for the 'host variable undefined' issue.

Cheers,
Constantin

(In reply to comment #15)
> (In reply to comment #14)
> > Migration does not work correctly with non-default PE.
> > http://lists.linuxtogo.org/pipermail/openembedded-core/2013-January/034950.
> > html
> > 
> Patch for fixing the non-default PE issues sent to the mailing list.
> 
> > Please update
> > https://wiki.yoctoproject.org/wiki/index.php?title=PR_Service
> > with information about changed PRSERV_HOST and migration script before
> > resolving this.
> 
> Right, I will update it today. I have waited for the patches to be merged.
> 
> Btw, thanks for sending the fix for the 'host variable undefined' issue.
> 
> Cheers,
> Constantin

I don't have wiki access yet. I will update the wiki page as soon as I receive
access.

Cheers,
Constantin

Patch merged.

Verified on master branch.
Commit ID:05985a7c22655a2f35ff0fcec723173c404c7993
This feature is covered by the following Testopia testcase:
https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=147