Every REST API list except recipesExtended is unpaginated, and the response is rendered into one string before it is sent. A request that matches most of a table therefore holds all of it in a worker's memory at once. Trigger: /layerindex/api/recipes/ unfiltered, or with a ?search= or filter that matches most rows. The JSON is about 300 MB (about 380,000 recipes). No login is needed. Measured on a test server with a copy of the production database: - one such request took a worker to 1.59 GB RSS for 31.6 s - whole-table ?search= requests took about 95 s, past the proxy timeout layers.openembedded.org has 22 GiB of RAM, no swap, and shares the host with other services, so a few concurrent requests could get a worker or the database killed by the OOM killer. The unfiltered recipes list is also ordinary traffic (about 70 requests a day, from python-requests and BitBake's layerindexlib), and BitBake's client, frozen in every released branch, expects a bare JSON array. So paginating these endpoints or rejecting unfiltered requests would break clients. Fix, not yet posted: stream unpaginated JSON lists a chunk of 2000 objects at a time. Output is byte-identical. On the test server the worker peak drops to 112 MB with PostgreSQL and 595 MB with MariaDB (MySQLdb buffers every row before the first chunk). Each whole-table request still costs about 30 s of CPU. Please keep this bug private until the fix is deployed on layers.openembedded.org.
Created attachment 5260 [details] layerindex/restviews.py: stream unpaginated JSON lists (on master-next 1befc98) Proposed fix. JSON list responses are sent 2000 objects at a time; output is byte-identical. Paginated lists and the browsable API are unchanged.