Bug 16525 - [layerindex-web] REST API: an unpaginated list request builds the whole table in memory
Summary: [layerindex-web] REST API: an unpaginated list request builds the whole table...
Status: RESOLVED FIXED
Alias: None
Product: Layer Index
Classification: Yocto Project Subprojects
Component: Layer Index (show other bugs)
Version: 6.1
Hardware: All Multiple
: Medium+ major
Target Milestone: 6.2 M1
Assignee: Tim Orling
QA Contact: apoorv sangal
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2026-11-16 20:18 UTC by Tim Orling
Modified: 2026-11-17 19:33 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
layerindex/restviews.py: stream unpaginated JSON lists (on master-next 1befc98) (8.65 KB, patch)
2026-11-16 22:39 UTC, Tim Orling
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Orling 2026-11-16 20:18:02 UTC
Every REST API list except recipesExtended is unpaginated, and the response is rendered into one string before it is sent. A request that matches most of a table therefore holds all of it in a worker's memory at once.

Trigger: /layerindex/api/recipes/ unfiltered, or with a ?search= or filter that matches most rows. The JSON is about 300 MB (about 380,000 recipes). No login is needed.

Measured on a test server with a copy of the production database:
- one such request took a worker to 1.59 GB RSS for 31.6 s
- whole-table ?search= requests took about 95 s, past the proxy timeout

layers.openembedded.org has 22 GiB of RAM, no swap, and shares the host with other services, so a few concurrent requests could get a worker or the database killed by the OOM killer.

The unfiltered recipes list is also ordinary traffic (about 70 requests a day, from python-requests and BitBake's layerindexlib), and BitBake's client, frozen in every released branch, expects a bare JSON array. So paginating these endpoints or rejecting unfiltered requests would break clients.

Fix, not yet posted: stream unpaginated JSON lists a chunk of 2000 objects at a time. Output is byte-identical. On the test server the worker peak drops to 112 MB with PostgreSQL and 595 MB with MariaDB (MySQLdb buffers every row before the first chunk). Each whole-table request still costs about 30 s of CPU.

Please keep this bug private until the fix is deployed on layers.openembedded.org.
Comment 1 Tim Orling 2026-11-16 22:39:32 UTC
Created attachment 5260 [details]
layerindex/restviews.py: stream unpaginated JSON lists (on master-next 1befc98)

Proposed fix. JSON list responses are sent 2000 objects at a time; output is byte-identical. Paginated lists and the browsable API are unchanged.