Bug 16525

Summary: [layerindex-web] REST API: an unpaginated list request builds the whole table in memory
Product: [Yocto Project Subprojects] Layer Index Reporter: Tim Orling <tim.orling>
Component: Layer IndexAssignee: Tim Orling <tim.orling>
Status: RESOLVED FIXED QA Contact: apoorv sangal <apoorvsangal>
Severity: major    
Priority: Medium+ CC: mhalstead, paul, piotr
Version: 6.1   
Target Milestone: 6.2 M1   
Hardware: All   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
layerindex/restviews.py: stream unpaginated JSON lists (on master-next 1befc98) none

Description Tim Orling 2026-11-16 20:18:02 UTC
Every REST API list except recipesExtended is unpaginated, and the response is rendered into one string before it is sent. A request that matches most of a table therefore holds all of it in a worker's memory at once.

Trigger: /layerindex/api/recipes/ unfiltered, or with a ?search= or filter that matches most rows. The JSON is about 300 MB (about 380,000 recipes). No login is needed.

Measured on a test server with a copy of the production database:
- one such request took a worker to 1.59 GB RSS for 31.6 s
- whole-table ?search= requests took about 95 s, past the proxy timeout

layers.openembedded.org has 22 GiB of RAM, no swap, and shares the host with other services, so a few concurrent requests could get a worker or the database killed by the OOM killer.

The unfiltered recipes list is also ordinary traffic (about 70 requests a day, from python-requests and BitBake's layerindexlib), and BitBake's client, frozen in every released branch, expects a bare JSON array. So paginating these endpoints or rejecting unfiltered requests would break clients.

Fix, not yet posted: stream unpaginated JSON lists a chunk of 2000 objects at a time. Output is byte-identical. On the test server the worker peak drops to 112 MB with PostgreSQL and 595 MB with MariaDB (MySQLdb buffers every row before the first chunk). Each whole-table request still costs about 30 s of CPU.

Please keep this bug private until the fix is deployed on layers.openembedded.org.
Comment 1 Tim Orling 2026-11-16 22:39:32 UTC
Created attachment 5260 [details]
layerindex/restviews.py: stream unpaginated JSON lists (on master-next 1befc98)

Proposed fix. JSON list responses are sent 2000 objects at a time; output is byte-identical. Paginated lists and the browsable API are unchanged.