Bug 7981 - Set up procedure for layer index database upgrades
Summary: Set up procedure for layer index database upgrades
Status: RESOLVED FIXED
Alias: None
Product: Layer Index
Classification: Yocto Project Subprojects
Component: Layer Index Metadata (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 2.2 M1
Assignee: Michael Halstead
QA Contact: Libertad
URL:
Whiteboard:
Depends on:
Blocks: 6618 7855
  Show dependency tree
 
Reported: 2015-07-09 16:48 UTC by Paul Eggleton
Modified: 2016-07-14 17:10 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Update process outline (2.41 KB, text/plain)
2015-08-20 20:01 UTC, Alex Franco
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Eggleton 2015-07-09 16:48:57 UTC
When we make changes to the layer index we usually have to upgrade the database; there is a risk that the upgrade will fail and thus we need to do a backup of the database first, then do the upgrade, and then restore the backup if it fails. There's the added complication that the database user used by the layer index application normally has restricted access to the database, which is not sufficient for the upgrade.

What we need is a documented procedure to follow for doing the upgrade on our own instance (this is not really user documentation because it's specific to our setup).
Comment 1 Belen Barros Pena 2015-07-14 09:39:56 UTC
This kind of blocks the Toaster image customisation feature ... Paul disagrees, but I still think it does ;)
Comment 2 Alex Franco 2015-08-20 20:01:26 UTC
Created attachment 2670 [details]
Update process outline

For the full document contact Alex Franco
Comment 3 Alex Franco 2015-08-20 20:04:04 UTC
Need more info from Michael Halstead: (1) production set up using docker, (2) feedback on update process - stripped down version attached.
Comment 4 Michael Wood 2015-09-23 16:01:53 UTC
Any update on this, we're getting close to requiring the blocking bug #7575 fixed for information needed in live toaster instances. Is there an intermediate / less complicated solution that can be done in the mean time to unblock this bug?
Comment 5 Michael Halstead 2015-10-06 03:45:09 UTC
We are very close currently the migration fails. I've been in touch with Alex Franco about this. 

I see that the update script is currently run with just the -q switch while the update also uses the -r switch to reload the data.

Does -r only needs to be set on the first update or is this required on all calls to update.py now?
Comment 6 Alex Franco 2015-10-06 22:59:39 UTC
Been in touch with Michael about this, now since this is an enhancement it probably won't go into 2.0 so I am moving this to 2.1 as we continue working on it
Comment 7 Belen Barros Pena 2015-10-07 08:14:19 UTC
(In reply to comment #6)
> Been in touch with Michael about this, now since this is an enhancement it
> probably won't go into 2.0 so I am moving this to 2.1 as we continue working
> on it

We really need this in Toaster for 2.0. We have split the image recipes from all other recipes as a usability enhancement and in preparation for image customisation. But Toaster's ability to properly identify image recipes is severely flawed without the parsing information the layer index provides.

We have been asking for this since the beginning of this release. Can we please (please, please) get this done within 2.0?
Comment 8 Michael Halstead 2015-10-07 09:15:47 UTC
I sat down with Paul and he fixed out the fix. The migration revision history is missing on production so we ran the following commands to set the versions manually. Now we're moving forward again.

python /opt/layerindex/manage.py migrate captcha 0001 --fake
python /opt/layerindex/manage.py migrate reversion 0005 --fake

If all goes well we should have this done today.
Comment 9 Belen Barros Pena 2015-10-07 09:30:36 UTC
 
> If all goes well we should have this done today.

Remind me to buy you a drink when I get to Dublin :)

Thank you!
Comment 10 Michael Halstead 2015-10-09 11:28:15 UTC
The new version based on the paule/inherits2 layer is now running in production. Please review the site and report any errors.

I did have to make several changes to the Dockerfile and support files to make it production ready. I'll push these to my own branch soon if that's acceptable.
Comment 11 Belen Barros Pena 2015-10-09 11:39:12 UTC
The layer index looked ok to me and just tested with Toaster, and things seem to be working as expected.

Once again, thank you so much for getting this done!
Comment 12 Alex Franco 2015-10-19 22:58:20 UTC
7575 was closed, so for this we are now discussing a public DB snapshot, beyond that it can probably be closed unless Dockerfile changes need to happen as well, discussion ongoing with Michael and Paul.
Comment 13 Paul Eggleton 2016-05-30 02:05:45 UTC
I think this one is now resolved - Michael has put in place procedures for upgrades and we've been through at least two upgrades with that now.