Bug 1174 - [zypper/rpm] package install/removal costs lots of disk space
Summary: [zypper/rpm] package install/removal costs lots of disk space
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.2
Assignee: Mei Lei
QA Contact:
URL:
Whiteboard: Fix rpm log infinite increase issue, ...
Depends on:
Blocks:
 
Reported: 2011-06-16 19:34 UTC by Jiajun Xu
Modified: 2011-11-21 21:55 UTC (History)
10 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jiajun Xu 2011-06-16 19:34:17 UTC 
Tree/Branch: Poky/master
Commit: e5f3cc34df74c08c8cab3e1cf9207fafaec6cb9a

When using zypper/rpm install/removal package, database files and log files cost a lot of disk space and may cause target image out of space.

The first time when we use RPM, a lot of database files will be created, larger than 10MB.
###
root@qemux86:~# ls -l /var/lib/rpm/
-rw-r--r--    1 root     root         12288 Jun 13 18:51 Arch
-rw-r--r--    1 root     root        237568 Jun 13 18:51 Basenames
-rw-r--r--    1 root     root          8192 Jun 13 18:51 Conflictname
-rw-rw-rw-    1 root     root           477 Jun 13 18:51 DB_CONFIG
-rw-r--r--    1 root     root         57344 Jun 13 18:51 Dirnames
-rw-r--r--    1 root     root        172032 Jun 13 18:51 Filedigests
-rw-r--r--    1 root     root        618496 Jun 13 18:51 Filepaths
-rw-r--r--    1 root     root         16384 Jun 13 18:51 Group
-rw-r--r--    1 root     root         12288 Jun 13 18:51 Installtid
-rw-r--r--    1 root     root         24576 Jun 13 18:51 Name
-rw-r--r--    1 root     root         32768 Jun 13 18:51 Nvra
-rw-r--r--    1 root     root          8192 Jun 13 18:51 Obsoletename
-rw-r--r--    1 root     root         12288 Jun 13 18:51 Os
-rw-r--r--    1 root     root         16384 Jun 13 18:51 Packagecolor
-rw-r--r--    1 root     root       2265088 Jun 13 18:51 Packages
-rw-r--r--    1 root     root         61440 Jun 13 18:51 Providename
-rw-r--r--    1 root     root         12288 Jun 13 18:51 Pubkeys
-rw-r--r--    1 root     root         12288 Jun 13 18:51 Release
-rw-r--r--    1 root     root         94208 Jun 13 18:51 Requirename
-rw-r--r--    1 root     root          8192 Jun 13 18:51 Seqno
-rw-r--r--    1 root     root         45056 Jun 13 18:51 Sha1header
-rw-r--r--    1 root     root         24576 Jun 13 18:51 Sigmd5
-rw-r--r--    1 root     root          8192 Jun 13 18:51 Sourcepkgid
-rw-r--r--    1 root     root          8192 Jun 13 18:51 Triggername
-rw-r--r--    1 root     root         16384 Jun 13 18:51 Version
-rw-r--r--    1 root     root         32768 Jun 13 18:54 __db.001
-rw-r--r--    1 root     root      16392192 Jun 13 18:54 __db.002
-rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.003
-rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.004
-rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.005
-rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.006
-rw-r--r--    1 root     root        163840 Jun 13 18:54 __db.007
-rw-r--r--    1 root     root       9101312 Jun 13 18:54 __db.008
-rw-r--r--    1 root     root         40960 Jun 13 18:54 __db.009
drwxrwxrwx    2 root     root          1024 Jun 13 18:51 log
###

And under /var/lib/rpm/log, log file for rpm will be saved and continuously increase. Each log file is 10MB and a new log will be generated when old files are fullfilled.
####
root@qemux86:/var/lib/rpm/log# ls -l
total 41837
-rw-r--r-- 1 root root 10485760 Jun 17 16:24 log.0000000001
-rw-r--r-- 1 root root 10485760 Jun 17 16:30 log.0000000002
-rw-r--r-- 1 root root 10485760 Jun 17 16:43 log.0000000003
-rw-r--r-- 1 root root 10485760 Jun 17 17:13 log.0000000004
-rw-r--r-- 1 root root 10485760 Jun 17 17:13 log.0000000005
####
Comment 1 Mark Hatle 2011-06-21 08:44:58 UTC 
This is an artifact of using BerkleyDB.  I will investigate and attempt to find the settings to flush the journals.
Comment 2 wenhua.fan 2011-06-27 20:00:37 UTC 
Tree/Branch: Poky/master
Commit: a1f79a7896b6411669b3ccada6204d2695e80fc5

Qemux86-sato and Qemux86-64-sato-sdk have the same issue in Yocto 1.1 20110624 build.
Comment 3 wenhua.fan 2011-07-11 01:46:02 UTC 
Do't reappear in Yocto 1.1_M2_RC2 testing.
Comment 4 wenhua.fan 2011-07-11 01:47:55 UTC 
Verify this bug.

Tree/Brahch : Poky/1.1_M2
Poky Commit id: 8df2a558b4d61778a669705b867969abe3514846
Meta Branch 1.1_M2
Meta Commit id : 92fc07a5f1b98779806cdcc2341487ff5ea5a238
Comment 5 wenhua.fan 2011-07-11 02:22:28 UTC 
Sorry for this verification, because of the log and large database files still created, this bug still exist in Yocto1.1_M2_RC2 testing.
Comment 6 Saul Wold 2011-08-24 14:02:42 UTC 
Can you provide additional infomration on how to tune BerkeleyDB, we might find someone to take a closer look at this.
Comment 7 Mei Lei 2011-09-04 20:26:17 UTC 
(In reply to comment #1)
> This is an artifact of using BerkleyDB.  I will investigate and attempt to find
> the settings to flush the journals.
Hi Mark, do you have any clue about this bug? I find the Mandriva distribution also report this bug, and they fixed it by enable perl-URPM 4.12:https://qa.mandriva.com/show_bug.cgi?id=62271
But yocto didn't include this package, do you think we need to enable perl-URPM for this bug?
Comment 8 Mei Lei 2011-11-16 23:53:37 UTC 
I have fixed the rpm log grows indefinitly bug:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lmei3/bug1174&id=ea4995cfbec9d75978131ba7566e30d3a20f99fa

(In reply to comment #0)
> Tree/Branch: Poky/master
> Commit: e5f3cc34df74c08c8cab3e1cf9207fafaec6cb9a
> When using zypper/rpm install/removal package, database files and log files
> cost a lot of disk space and may cause target image out of space.
> The first time when we use RPM, a lot of database files will be created, larger
> than 10MB.
> ###
> root@qemux86:~# ls -l /var/lib/rpm/
> -rw-r--r--    1 root     root         12288 Jun 13 18:51 Arch
> -rw-r--r--    1 root     root        237568 Jun 13 18:51 Basenames
> -rw-r--r--    1 root     root          8192 Jun 13 18:51 Conflictname
> -rw-rw-rw-    1 root     root           477 Jun 13 18:51 DB_CONFIG
> -rw-r--r--    1 root     root         57344 Jun 13 18:51 Dirnames
> -rw-r--r--    1 root     root        172032 Jun 13 18:51 Filedigests
> -rw-r--r--    1 root     root        618496 Jun 13 18:51 Filepaths
> -rw-r--r--    1 root     root         16384 Jun 13 18:51 Group
> -rw-r--r--    1 root     root         12288 Jun 13 18:51 Installtid
> -rw-r--r--    1 root     root         24576 Jun 13 18:51 Name
> -rw-r--r--    1 root     root         32768 Jun 13 18:51 Nvra
> -rw-r--r--    1 root     root          8192 Jun 13 18:51 Obsoletename
> -rw-r--r--    1 root     root         12288 Jun 13 18:51 Os
> -rw-r--r--    1 root     root         16384 Jun 13 18:51 Packagecolor
> -rw-r--r--    1 root     root       2265088 Jun 13 18:51 Packages
> -rw-r--r--    1 root     root         61440 Jun 13 18:51 Providename
> -rw-r--r--    1 root     root         12288 Jun 13 18:51 Pubkeys
> -rw-r--r--    1 root     root         12288 Jun 13 18:51 Release
> -rw-r--r--    1 root     root         94208 Jun 13 18:51 Requirename
> -rw-r--r--    1 root     root          8192 Jun 13 18:51 Seqno
> -rw-r--r--    1 root     root         45056 Jun 13 18:51 Sha1header
> -rw-r--r--    1 root     root         24576 Jun 13 18:51 Sigmd5
> -rw-r--r--    1 root     root          8192 Jun 13 18:51 Sourcepkgid
> -rw-r--r--    1 root     root          8192 Jun 13 18:51 Triggername
> -rw-r--r--    1 root     root         16384 Jun 13 18:51 Version
> -rw-r--r--    1 root     root         32768 Jun 13 18:54 __db.001
> -rw-r--r--    1 root     root      16392192 Jun 13 18:54 __db.002
> -rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.003
> -rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.004
> -rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.005
> -rw-r--r--    1 root     root       9740288 Jun 13 18:54 __db.006
> -rw-r--r--    1 root     root        163840 Jun 13 18:54 __db.007
> -rw-r--r--    1 root     root       9101312 Jun 13 18:54 __db.008
> -rw-r--r--    1 root     root         40960 Jun 13 18:54 __db.009
> drwxrwxrwx    2 root     root          1024 Jun 13 18:51 log
> ###
> And under /var/lib/rpm/log, log file for rpm will be saved and continuously
> increase. Each log file is 10MB and a new log will be generated when old files
> are fullfilled.
> ####
> root@qemux86:/var/lib/rpm/log# ls -l
> total 41837
> -rw-r--r-- 1 root root 10485760 Jun 17 16:24 log.0000000001
> -rw-r--r-- 1 root root 10485760 Jun 17 16:30 log.0000000002
> -rw-r--r-- 1 root root 10485760 Jun 17 16:43 log.0000000003
> -rw-r--r-- 1 root root 10485760 Jun 17 17:13 log.0000000004
> -rw-r--r-- 1 root root 10485760 Jun 17 17:13 log.0000000005
> ####
Comment 9 Mei Lei 2011-11-17 00:42:52 UTC 
I filed a new bug 1769 for database capacity issue.
http://bugzilla.pokylinux.org/show_bug.cgi?id=1769
Comment 10 Jiajun Xu 2011-11-21 21:55:14 UTC 
verify the bug as fixed with commit f2d077255d9ddf9b87e9a781b989bcce69047cf5