Bug 7932 - Can not migrate orm (mysql, virtualenv) master branch (2015-06-29) #2578ff3
Summary: Can not migrate orm (mysql, virtualenv) master branch (2015-06-29) #2578ff3
Status: VERIFIED FIXED
Alias: None
Product: Toaster
Classification: Build System, Metadata & Runtime
Component: toaster (show other bugs)
Version: unspecified
Hardware: x86 x86_64
: Medium normal
Target Milestone: 1.9 M3
Assignee: Elliot Smith
QA Contact: Mihail Stanciu
URL:
Whiteboard: GUI design available
Depends on:
Blocks:
 
Reported: 2015-06-29 12:47 UTC by Daniel Kuhne
Modified: 2016-02-29 14:52 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments
Design - Showing the default project storing the Knotty builds (1.46 MB, application/pdf)
2015-08-26 13:25 UTC, Belen Barros Pena
no flags Details
Design - Showing the default project storing the Knotty builds (1.56 MB, application/pdf)
2015-08-28 10:36 UTC, Belen Barros Pena
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Kuhne 2015-06-29 12:47:03 UTC
Error: RuntimeError: Cannot reverse this migration. 'Project.bitbake_version' and its values cannot be restored.

Steps to reproduce: 

0. cd opt/ 
A. git clone git://git.yoctoproject.org/poky
B. virtualenv venv
C. source venv/bin/activate
D. pip install -r /opt/poky/bitbake/toaster-requirements.txt
E. pip install mysql-python
F. create a mysql database and setup access in /opt/poky/bitbake/lib/toaster/toastermain/settings.py
G. /opt/poky/bitbake/lib/toaster/manage.py syncdb
H. /opt/poky/bitbake/lib/toaster/manage.py migrate orm
Comment 1 Michael Wood 2015-06-30 14:20:48 UTC
If you're following the instructions for the poky 1.8 release "fido" please make sure to switch this after cloning. (or specify the branch in the clone command git clone -b fido git://git.yoctoproject.org/poky.git )

We need to update the documentation to fix these instructions as it seems to be missing.
Comment 2 Michael Wood 2015-06-30 14:25:56 UTC
References to "git clone" should also include the corresponding release of poky that the documentation is written for.

http://www.yoctoproject.org/docs/1.8/toaster-manual/toaster-manual.html#setting-up-a-hosted-service-and-running-in-build-mode

Instances of:

" $ git clone git://git.yoctoproject.org/poky"

Should be:

" $ git clone -b fido git://git.yoctoproject.org/poky"

Where "fido" is the release for which the documentation is written for.
Comment 3 Elliot Smith 2015-08-13 15:43:53 UTC
I see the same issue on poky master @ a533776. The error doesn't occur if using SQLite.

I'm running MySQL as part of XAMPP. `/opt/lampp/bin/mysql --version` gives:

```
/opt/lampp/bin/mysql  Ver 14.14 Distrib 5.6.16, for Linux (x86_64) using  EditLine wrapper
```

`toastermain/settings.py` is like this:

```
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'toaster',
        'USER': 'toaster',
        'PASSWORD': '*******',
        'HOST': '/opt/lampp/var/mysql/mysql.sock',
        'PORT': ''
    }
}
```

I did the setup of the basic database structure using the mysql command line:

```
$ /opt/lampp/bin/mysql -u root

> CREATE USER 'toaster'@'localhost' IDENTIFIED BY '*******';
> CREATE DATABASE toaster;
> GRANT ALL PRIVILEGES ON toaster.* TO 'toaster'@'localhost';
```

Note that the MySQL server is configured to use InnoDB tables by default, which support transactions (the error I was getting suggested lack of DDL support, so I thought the use of MyISAM tables might be causing my error). I confirmed InnoDB support with:

```
> SELECT TABLE_NAME, ENGINE FROM information_schema.TABLES where TABLE_SCHEMA = 'toaster';
+--------------------------------+--------+
| TABLE_NAME                     | ENGINE |
+--------------------------------+--------+
| auth_group                     | InnoDB |
| auth_group_permissions         | InnoDB |
...
```

I then started toaster in managed mode. When the migrations run (against a clean db) the migrations fail with this error:

```
$ TOASTER_MANAGED=1 source toaster start
The system will start.
Syncing...
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_admin_log
Creating table south_migrationhistory

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): no
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

Synced:
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.messages
 > django.contrib.sessions
 > django.contrib.admin
 > django.contrib.staticfiles
 > django.contrib.humanize
 > south

Not synced (use migrations):
 - orm
 - bldcontrol
(use ./manage.py migrate to migrate these)
Running migrations for orm:
 - Migrating forwards to 0024_auto__add_field_recipe_is_image.
 > orm:0001_initial
 > orm:0002_auto__add_field_build_timespent
 > orm:0003_timespent
 - Migration 'orm:0003_timespent' is marked for no-dry-run.
 > orm:0004_auto__add_field_package_installed_name
 > orm:0005_auto__add_target_image_file__add_target_file__add_field_variablehistor
 > orm:0006_auto__add_field_target_image_size__add_field_target_license_manifest_p
 > orm:0007_auto__add_helptext
 > orm:0008_auto__chg_field_variablehistory_operation__chg_field_recipe_descriptio
 > orm:0009_auto__add_projectvariable__add_projectlayer__add_projecttarget__add_pr
 > orm:0010_auto__add_field_project_branch__add_field_project_short_description__a
 > orm:0011_auto__add_field_projectlayer_dirpath
 > orm:0012_auto__add_field_projectlayer_optional__add_field_projecttarget_task
 > orm:0013_auto__add_release__add_layerversiondependency__add_unique_layerversion
 > orm:0014_auto__chg_field_package_summary__chg_field_layer_summary__chg_field_re
 > orm:0015_auto__add_field_layer_vcs_web_url__add_field_layer_vcs_web_tree_base_u
 > orm:0016_auto__add_field_release_helptext__chg_field_release_branch__add_index_
 > orm:0017_auto__del_toastersettingdefaultlayer__add_releaselayersourcepriority__
 > orm:0018_auto__add_field_layer_version_project
 > orm:0019_auto__add_buildartifact
 > orm:0020_auto__add_field_layer_version_local_path__add_field_recipe_pathflags__
 > orm:0021_auto__chg_field_build_project__chg_field_project_bitbake_version__chg_
Traceback (most recent call last):
  File "/home/ell/dev/toaster/poky/bitbake/bin/../lib/toaster/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
    ignore_ghosts = ignore_ghosts,
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 254, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 329, in migrate_many
    result = self.migrate(migration, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 133, in migrate
    result = self.run(migration, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
    return self.run_migration(migration, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 93, in run_migration
    print(self.run_migration_error(migration))
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 323, in run_migration_error
    (self.format_backwards(migration), extra_info))
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 310, in format_backwards
    self.backwards(migration)()
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 60, in <lambda>
    return (lambda: direction(orm))
  File "/home/ell/dev/toaster/poky/bitbake/lib/toaster/orm/migrations/0021_auto__chg_field_build_project__chg_field_project_bitbake_version__chg_.py", line 33, in backwards
    raise RuntimeError("Cannot reverse this migration. 'Project.bitbake_version' and its values cannot be restored.")
RuntimeError: Cannot reverse this migration. 'Project.bitbake_version' and its values cannot be restored.

Error on migration, trying to recover...
Running migrations for orm:
 - Migrating backwards to just after 0001_initial.
 < orm:0020_auto__add_field_layer_version_local_path__add_field_recipe_pathflags__
   (faked)
 < orm:0019_auto__add_buildartifact
   (faked)
 < orm:0018_auto__add_field_layer_version_project
   (faked)
 < orm:0017_auto__del_toastersettingdefaultlayer__add_releaselayersourcepriority__
   (faked)
 < orm:0016_auto__add_field_release_helptext__chg_field_release_branch__add_index_
   (faked)
 < orm:0015_auto__add_field_layer_vcs_web_url__add_field_layer_vcs_web_tree_base_u
   (faked)
 < orm:0014_auto__chg_field_package_summary__chg_field_layer_summary__chg_field_re
   (faked)
 < orm:0013_auto__add_release__add_layerversiondependency__add_unique_layerversion
   (faked)
 < orm:0012_auto__add_field_projectlayer_optional__add_field_projecttarget_task
   (faked)
 < orm:0011_auto__add_field_projectlayer_dirpath
   (faked)
 < orm:0010_auto__add_field_project_branch__add_field_project_short_description__a
   (faked)
 < orm:0009_auto__add_projectvariable__add_projectlayer__add_projecttarget__add_pr
   (faked)
 < orm:0008_auto__chg_field_variablehistory_operation__chg_field_recipe_descriptio
   (faked)
 < orm:0007_auto__add_helptext
   (faked)
 < orm:0006_auto__add_field_target_image_size__add_field_target_license_manifest_p
   (faked)
 < orm:0005_auto__add_target_image_file__add_target_file__add_field_variablehistor
   (faked)
 < orm:0004_auto__add_field_package_installed_name
   (faked)
 < orm:0003_timespent
   (faked)
 < orm:0002_auto__add_field_build_timespent
   (faked)
Running migrations for orm:
 - Migrating forwards to 0024_auto__add_field_recipe_is_image.
 > orm:0002_auto__add_field_build_timespent
FATAL ERROR - The following SQL query failed: ALTER TABLE `orm_build` ADD COLUMN `timespent` integer NOT NULL DEFAULT 0;
The error was: (1060, "Duplicate column name 'timespent'")
 ! Error found during real run of migration! Aborting.

 ! Since you have a database that does not support running
 ! schema-altering statements in transactions, we have had
 ! to leave it in an interim state between migrations.

! You *might* be able to recover with:   - no dry run output for delete_foreign_key() due to dynamic DDL, sorry
   = ALTER TABLE `orm_build` DROP COLUMN `timespent` CASCADE; []

 ! The South developers regret this has happened, and would
 ! like to gently persuade you to consider a slightly
 ! easier-to-deal-with DBMS (one that supports DDL transactions)
 ! NOTE: The error which caused the migration to fail is further up.
Error in migration: orm:0002_auto__add_field_build_timespent
Traceback (most recent call last):
  File "/home/ell/dev/toaster/poky/bitbake/bin/../lib/toaster/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
    ignore_ghosts = ignore_ghosts,
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 254, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 329, in migrate_many
    result = self.migrate(migration, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 133, in migrate
    result = self.run(migration, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
    return self.run_migration(migration, database)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 84, in run_migration
    migration_function()
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/migration/migrators.py", line 60, in <lambda>
    return (lambda: direction(orm))
  File "/home/ell/dev/toaster/poky/bitbake/lib/toaster/orm/migrations/0002_auto__add_field_build_timespent.py", line 14, in forwards
    keep_default=False)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/db/generic.py", line 47, in _cache_clear
    return func(self, table, *args, **opts)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/db/generic.py", line 418, in add_column
    self.execute(sql)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/south/db/generic.py", line 282, in execute
    cursor.execute(sql, params)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/db/backends/util.py", line 69, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/home/ell/toaster-run/toaster-env/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
django.db.utils.OperationalError: (1060, "Duplicate column name 'timespent'")
```
Comment 4 Elliot Smith 2015-08-14 08:18:00 UTC
This is a blocker for me, as I need it fixed to be able to run Toaster (I can't get it to work with SQLite, as described in bug 8147).

Which branch of poky should be referenced in the docs change?
Comment 5 Elliot Smith 2015-08-14 08:20:26 UTC
I think this is just an issue with MySQL performing a dry run of the SQL statements, which throws an error with some of the migrations as they can't be run backwards. (When using SQLite, migrations aren't dry run: see http://south.aeracode.org/wiki/Tutorial3.)
Comment 6 Belen Barros Pena 2015-08-26 13:25:06 UTC
Created attachment 2686 [details]
Design - Showing the default project storing the Knotty builds
Comment 7 Elliot Smith 2015-08-26 13:47:16 UTC
Do we need to consider other mechanisms for triggering builds apart from the command line? Just wondering whether they should be called "Command line builds" or something like "External builds" instead. (I don't think "External builds" is correct either, but there could be a way to trigger a bitbake build, but not from the command line.)
Comment 8 Belen Barros Pena 2015-08-26 13:54:13 UTC
(In reply to comment #7)
> Do we need to consider other mechanisms for triggering builds apart from the
> command line? Just wondering whether they should be called "Command line
> builds" or something like "External builds" instead. (I don't think
> "External builds" is correct either, but there could be a way to trigger a
> bitbake build, but not from the command line.)

The only case I can think of is something like the Autobuilder (CI sending information about its builds to a Toaster instance).

An easy workaround would be to allow people to change the name of the project. 'Command line builds' reflects what we believe is the most likely use case for this project. If it doesn't fit your use case, change the project name! :)

Would that work?
Comment 9 Elliot Smith 2015-08-26 14:32:57 UTC
Yes, I think that would be fine. Should the UI therefore retain the option to edit the "default project"'s name?

Note that I've mostly implemented a better default project.

I have tried adding the default project to the db during initial setup by migration, rather than at runtime when the first command line build appears. However, this has the effect of putting a project into the database, which means the landing page is skipped (because there's a project).

Should I keep the old behaviour for now (where the default project is created at runtime), even though it feels incorrect?
Comment 10 Elliot Smith 2015-08-26 14:34:37 UTC
Alternatively, I could modify the landing page (to only redirect if there is a project other than the default one) and modify the projects page to only show projects with a build count > 0.
Comment 11 Belen Barros Pena 2015-08-26 14:37:10 UTC
(In reply to comment #9)
> Yes, I think that would be fine. Should the UI therefore retain the option
> to edit the "default project"'s name?

yes, please!

> 
> Note that I've mostly implemented a better default project.
> 
> I have tried adding the default project to the db during initial setup by
> migration, rather than at runtime when the first command line build appears.
> However, this has the effect of putting a project into the database, which
> means the landing page is skipped (because there's a project).
> 
> Should I keep the old behaviour for now (where the default project is
> created at runtime), even though it feels incorrect?
Comment 12 Belen Barros Pena 2015-08-26 14:40:46 UTC
(In reply to comment #10)
> Alternatively, I could modify the landing page (to only redirect if there is
> a project other than the default one) 

This sounds better to me, particularly if creating the project at runtime is the wrong thing to do.

and modify the projects page to only
> show projects with a build count > 0.

This will be strange for normal projects. It means that if I've just created a project, and then go to the all projects page, the project I just created won't be listed there, which will make me think that something has gone wrong. What's worse: there would be no way for me to access the project, since the only entry point for a project with no builds is the list of all projects.

I'm afraid we'll need to do something like "if this is the default project and the number of builds in it is 0, do now show it".
Comment 13 Belen Barros Pena 2015-08-28 10:36:10 UTC
Created attachment 2693 [details]
Design - Showing the default project storing the Knotty builds

Adding some additional help text based on Tiago's feedback (pages 5 and 8)
Comment 14 Elliot Smith 2015-08-28 13:20:06 UTC
Belen, could we move the new design for the command line builds out of this bug and into a fresh one? It would be good if we could just get the "default project" working close to how it works with SQLite right now, and think about a new design for that part of the application in future.
Comment 15 Belen Barros Pena 2015-09-01 09:44:45 UTC
GUI work is now covered by bug 8231
Comment 16 Elliot Smith 2015-09-04 08:54:03 UTC
To test, set up Toaster with a MySQL database, as per instructions in http://www.yoctoproject.org/docs/1.8/toaster-manual/toaster-manual.html#setting-up-a-hosted-service-and-running-in-analysis-mode.
Comment 17 Mihail Stanciu 2016-02-29 14:52:54 UTC
Verified while installing MySQL instance for GDC.