Bug 16507 - layerindex-web: 31 database tables still use latin1, so text outside Latin-1 fails to save
Summary: layerindex-web: 31 database tables still use latin1, so text outside Latin-1 ...
Status: NEW
Alias: None
Product: Layer Index
Classification: Yocto Project Subprojects
Component: Layer Index (show other bugs)
Version: unspecified
Hardware: All Multiple
: Medium normal
Target Milestone: 6.2 M2
Assignee: Tim Orling
QA Contact: apoorv sangal
URL:
Whiteboard:
Depends on:
Blocks: 16476
  Show dependency tree
 
Reported: 2026-11-15 00:30 UTC by Tim Orling
Modified: 2026-11-15 01:00 UTC (History)
0 users

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Orling 2026-11-15 00:30:46 UTC
In a copy of the layers.openembedded.org database (on the test instance
layers.moto-timo.net, MariaDB), 31 tables have a latin1_swedish_ci
default collation, with 65 latin1 text columns between them. The rest
of the database is utf8mb4. The latin1 tables include every RRS table
(rrs_maintainer, rrs_recipeupgrade, rrs_recipesymbol,
rrs_recipemaintainerhistory, rrs_recipeupstream, rrs_recipedistro,
rrs_milestone, rrs_release, ...) and several layer index ones
(layerindex_incfile, layerindex_staticbuilddep,
layerindex_dynamicbuilddep, layerindex_extendedprovide,
layerindex_packageconfig, layerindex_patchdisposition,
layerindex_userprofile, layerindex_securityquestion(answer),
layerindex_ypcompatibleversion) plus the axes tables. They were probably
created before the server default changed.

Text that can't be represented in Latin-1 then fails. For example,
rrs_upgrade_history.py fails on commits whose recipe maintainer has such
a name, in Maintainer.create_or_update():

  MySQLdb.OperationalError: (1267, "Illegal mix of collations
  (latin1_swedish_ci,IMPLICIT) and (utf8mb4_uca1400_ai_ci,COERCIBLE)
  for operation '='")

The run logs the error and carries on, so the upgrade record for that
commit is silently missing. The production schema has not been checked
directly yet.

Fix: convert these tables to utf8mb4 (ALTER TABLE ... CONVERT TO
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, after a backup), or
handle it as part of the move to Postgres (bug 16476).
Comment 1 Tim Orling 2026-11-15 01:00:35 UTC
Correction to the description: the rest of the database isn't all
utf8mb4. On the test instance's copy of the production database the 73
tables are: latin1_swedish_ci 31 (65 columns), utf8mb3_unicode_ci 35
(104 columns), utf8mb4_unicode_ci 5 (11 columns), utf8mb4_0900_ai_ci 2
(5 columns).

Fixed on the test instance (layers.moto-timo.net) on 2026-11-15:
- Backed up the database with mariadb-dump --single-transaction.
- Merged two pairs of rrs_maintainer rows that collide in the unique
  index on name under utf8mb4_unicode_ci (latin1_swedish_ci treats o and
  ö as different letters). Each pair is the same person with the same
  email: "Enrico Jorns"/"Enrico Jörns" and "Zoltan Boszormenyi"/"Zoltán
  Böszörményi". Their RecipeUpgrade and RecipeMaintainerHistory
  references were moved to the row the maintainers file uses.
- Converted the 31 tables with ALTER TABLE ... CONVERT TO CHARACTER SET
  utf8mb4 COLLATE utf8mb4_unicode_ci. No latin1 columns are left.

A maintainer name such as "Łukasz Ślęzak 测试" now saves, and
rrs_upgrade_history.py runs without the "Illegal mix of collations"
error. Production has not been changed yet. The scripts and steps are
kept with the layer index handoff notes.