Bug 9651

Summary: Error report web: Optimize searching times
Product: [Yocto Project Subprojects] Error Reporting tool Reporter: Aníbal Limón <anibal.limon>
Component: Error Reporting ToolAssignee: Tim Orling <tim.orling>
Status: NEW --- QA Contact:
Severity: enhancement    
Priority: Medium CC: belen.barros.pena, benjamin.esquivel, changqing.li, juan.m.cruz.alcaraz, randy.macleod, richard.purdie, tim.orling
Version: unspecified   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
mysql slow query log none

Description Aníbal Limón 2016-05-17 22:22:13 UTC
When you made a search in error report web it takes too much ~1 minute depending on the search.

This is because the tool is searching over the log without an index [1], currently MySQL supports Full Text Search but Django ORM didn't expose directly.

The work is to create an full text search index for the log field and modify error report web code to use a custom MySQL query when MySQL is available.

[1] http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/tree/Post/views.py#n202
Comment 1 Richard Purdie 2016-05-19 14:52:28 UTC
We did tweak some things like the link used from the autobuilder so its no longer a full text search but just searching for specific revisions. Does that help your usecase or not? We're struggling to figure out the priority of this without knowing what usecases its impacting. If there is a significant impact we could make it a Medium+. We could also assign to someone on the GDC team, depending on how Michael and GDC want to handle this.
Comment 2 Aníbal Limón 2016-05-19 14:57:04 UTC
(In reply to comment #1)
> We did tweak some things like the link used from the autobuilder so its no
> longer a full text search but just searching for specific revisions. Does
> that help your usecase or not? We're struggling to figure out the priority
> of this without knowing what usecases its impacting. If there is a
> significant impact we could make it a Medium+. We could also assign to
> someone on the GDC team, depending on how Michael and GDC want to handle
> this.

We can workaround the search over log using only the link from auto builder for access the failures also using specific revisions but this not solve the current searching problem if the search functionality isn't needed may be is better to remove it instead of leave as is and only using specific revisions and AB link.
Comment 3 Aníbal Limón 2016-05-19 20:11:58 UTC
Created attachment 3182 [details]
mysql slow query log

When did a search of smart, in the current db.
Comment 4 Aníbal Limón 2016-05-19 20:36:11 UTC
(In reply to comment #3)
> Created attachment 3182 [details]
> mysql slow query log
> 
> When did a search of smart, in the current db.

As summary the query took 440s,

# Query_time: 440.112315  Lock_time: 0.000082  Rows_sent: 1  Rows_examined: 79488
Comment 5 Michael Wood 2016-05-19 21:57:49 UTC
Fine to take this bug, pretty easy to implement. Note that we already have a bug #6541 to improve the search on the UI side which may help where people haven't discovered out how to do the correct query.
Comment 6 Michael Wood 2016-10-05 14:53:42 UTC
We would need to special case when the backend database selection is mysql and create a full text index on the log field. We would also need to check which backend database is in use to be able to switch between the querying method.

https://docs.djangoproject.com/en/1.8/ref/models/querysets/#search
Comment 7 Randy MacLeod 2022-03-31 15:08:21 UTC
Still slow...
Comment 8 Tim Orling 2026-04-23 15:01:04 UTC
Still valid.