<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>797</bug_id>
          
          <creation_ts>2011-03-03 16:00:33 +0000</creation_ts>
          <short_desc>Possible race condition during do_rootfs</short_desc>
          <delta_ts>2011-05-13 11:15:51 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>Meta-yocto</product>
          <component>meta-yocto</component>
          <version>1.0</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>High</priority>
          <bug_severity>major</bug_severity>
          <target_milestone>1.1</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Beth Flanagan">elizabeth.flanagan</reporter>
          <assigned_to name="Mark Hatle">mark.hatle</assigned_to>
          <cc>bluelightning</cc>
    
    <cc>dongxiao.xu</cc>
    
    <cc>jessica.zhang</cc>
    
    <cc>josh</cc>
    
    <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
    
    <cc>richard.purdie</cc>
    
    <cc>sgw</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>---</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>12246</commentid>
    <comment_count>0</comment_count>
    <who name="Beth Flanagan">elizabeth.flanagan</who>
    <bug_when>2011-03-03 16:00:33 +0000</bug_when>
    <thetext>Autobuilder reports:

ERROR: Function &apos;do_rootfs&apos; failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-poky-linux/poky-image-lsb-1.0-r0/temp/log.do_rootfs.28453 for further information)
ERROR: Logfile of failure stored in: /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-poky-linux/poky-image-lsb-1.0-r0/temp/log.do_rootfs.28453
Log data follows:
| ERROR: Function &apos;do_rootfs&apos; failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-poky-linux/poky-image-lsb-1.0-r0/temp/log.do_rootfs.28453 for further information)
| Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/qemux86...
|    total:               1      0.000000 MB      2.016852 secs
|    fingerprint:      1146      0.008542 MB      0.062726 secs
|    install:           382      0.000000 MB      0.438539 secs
|    dbadd:             382      0.000000 MB      0.427742 secs
|    dbget:            2505      0.000000 MB      0.005212 secs
|    dbput:             382      1.720148 MB      0.353341 secs
|    readhdr:          3821      3.376208 MB      1.108379 secs
|    hdrload:          1910      5.010692 MB      0.009065 secs
|    hdrget:          65200      0.000000 MB      0.055676 secs
| Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586...
| error: open of /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586/task-poky-sdk-1.0-r7.i586.rpm failed: No such file or directory
| rpm.real: ./rpmio_internal.h:190: fdGetOPath: Assertion `fd != ((void *)0) &amp;&amp; fd-&gt;magic == 0x04463138&apos; failed.
| /builds2/pok

Upon investigation we see that:


eflanagan@autobuilder03:~&gt; ls -al /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586/task-poky-sdk-1.0-r7.i586.rpm 

-rwxr--r-- 1 pokybuild users 3000 2011-03-03 13:47 /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/i586/task-poky-sdk-1.0-r7.i586.rpm

This has also occurred elsewhere:

ERROR: Function &apos;do_rootfs&apos; failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-64-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.20074 for further information)
ERROR: Logfile of failure stored in: /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-64-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.20074
Log data follows:
| Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/qemux86-64...
|    total:               1      0.000000 MB      5.299116 secs
|    fingerprint:      1158      0.008484 MB      0.213973 secs
|    install:           386      0.000000 MB      0.790160 secs
|    dbadd:             386      0.000000 MB      0.771915 secs
|    dbget:            2469      0.000000 MB      0.007738 secs
|    dbput:             386      1.734856 MB      0.665613 secs
|    readhdr:          3861      3.402560 MB      3.726118 secs
|    hdrload:          1936      5.067812 MB      0.012818 secs
|    hdrget:          65936      0.000000 MB      0.072568 secs
| Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64...
| error: open of /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64/libtiffxx-dev-3.8.2-r2.x86_64.rpm failed: No such file or directory
|    total:               1      0.000000 MB     17.470722 secs
|    fingerprint:      3918      0.070208 MB      0.596252 secs
|    install:          1306      0.000000 MB      2.646070 secs
|    dbadd:            1306      0.000000 MB      2.610542 secs
|    dbget:           24728      0.000000 MB      0.028352 secs
|    dbput:            1306      7.334516 MB      2.321668 secs
|    readhdr:         13061     14.552984 MB     12.311993 secs
|    hdrload:          7526     27.525252 MB      0.044016 secs
|    hdrget:         235962      0.000000 MB      0.221356 secs
| Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/all...
|    total:               1      0.000000 MB      1.034876 secs
|    fingerprint:       126      0.003376 MB      0.101518 secs
|    install:            42      0.000000 MB      0.063101 secs
|    dbadd:              42      0.000000 MB      0.059886 secs
|    dbget:             639      0.000000 MB      0.000695 secs
|    dbput:              42      0.268924 MB      0.035743 secs
|    readhdr:           421      0.534456 MB      0.806430 secs
|    hdrload:           210      0.793876 MB      0.001077 secs
|    hdrget:           6939      0.000000 MB      0.005857 secs
| Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64-nativesdk...
|    total:               1      0.000000 MB     13.518695 secs
|    fingerprint:      1830      0.028198 MB      0.162148 secs
|    install:           610      0.000000 MB      1.221415 secs
|    dbadd:             610      0.000000 MB      1.205607 secs
|    dbget:           10548      0.000000 MB      0.012174 secs
|    dbput:             610      4.071680 MB      1.103758 secs
|    readhdr:          6101      8.095502 MB     11.510949 secs
|    hdrload:          3053     12.045458 MB      0.014814 secs
|    hdrget:          86330      0.000000 MB      0.076150 secs
| Generating solve db for /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/all...
| Processing task-poky-boot...
| error: Failed dependencies:
| 	tinylogin is needed by task-poky-boot-1.0-r7.qemux86
| 	busybox is needed by task-poky-boot-1.0-r7.qemux86
| 	modutils-initscripts is needed by task-poky-boot-1.0-r7.qemux86
| 	initscripts is needed by task-poky-boot-1.0-r7.qemux86
| 	update-alternatives-cworth is needed by task-poky-boot-1.0-r7.qemux86
| 	udev is needed by task-poky-boot-1.0-r7.qemux86
| 	sysvinit is needed by task-poky-boot-1.0-r7.qemux86
| 	base-passwd is needed by task-poky-boot-1.0-r7.qemux86
| 	libc6 &gt;= 2.12 is needed by v86d-0.1.9-r0.qemux86
| 	/bin/sh is needed by v86d-0.1.9-r0.qemux86
| 	/bin/sh is needed by netbase-4.44-r0.qemux86
| 	/bin/sh is needed by kernel-module-uvesafb-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6e3c1-r16.qemux86
| 	module-init-tools-depmod is needed by update-modules-1.0-r8.all
| 	/bin/sh is needed by update-modules-1.0-r8.all
| 	/bin/sh is needed by kernel-module-cfbfillrect-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6e3c1-r16.qemux86
| 	/bin/sh is needed by kernel-module-cfbimgblt-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6eERROR: Function &apos;do_rootfs&apos; failed (see /builds2/pokybuilds/nightly-internal/build/build/tmp/work/qemux86-64-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.20074 for further information)
| 3c1-r16.qemux86
| 	/bin/sh is needed by kernel-module-cfbcopyarea-2.6.37+git1+e2737075b79e4fc682e41051cf1c0bc47a47d502_1+2b412826bbeb4a16abe2ea74f2456ab880c6e3c1-r16.qemux86



eflanagan@autobuilder03:~&gt; ls -al /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64/libtiffxx-dev-3.8.2-r2.x86_64.rpm

-rwxr--r-- 1 pokybuild users 21049 2011-03-03 14:02 /builds2/pokybuilds/nightly-internal/build/build/tmp/deploy/rpm/x86_64/libtiffxx-dev-3.8.2-r2.x86_64.rpm</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12367</commentid>
    <comment_count>1</comment_count>
    <who name="Mark Hatle">mark.hatle</who>
    <bug_when>2011-03-08 13:32:58 +0000</bug_when>
    <thetext>We need to add the rpm lock file to the package generation.  The concern though is that we can generate all of the rpm packages we want, we just can&apos;t generate packages during a rootfs build..

Need some further information on how to do this.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12462</commentid>
    <comment_count>2</comment_count>
    <who name="Mark Hatle">mark.hatle</who>
    <bug_when>2011-03-10 16:22:37 +0000</bug_when>
    <thetext>pull request for a fix has been sent.  The fix uses shared locks vs exclusive locks.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12484</commentid>
    <comment_count>3</comment_count>
    <who name="Saul Wold">sgw</who>
    <bug_when>2011-03-10 21:59:32 +0000</bug_when>
    <thetext>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=c180383885e32aed83c4f8691115859966fb53be</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12595</commentid>
    <comment_count>4</comment_count>
    <who name="Dongxiao Xu">dongxiao.xu</who>
    <bug_when>2011-03-14 19:34:31 +0000</bug_when>
    <thetext>*** Bug 867 has been marked as a duplicate of this bug. ***</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12596</commentid>
    <comment_count>5</comment_count>
    <who name="Dongxiao Xu">dongxiao.xu</who>
    <bug_when>2011-03-14 19:36:18 +0000</bug_when>
    <thetext>The issue still exists in my build.

Commit 43a2d098008eee711399b8d64594d84ae034b9bf.

1) Run &quot;bitbake poky-image-sato-live&quot;. Build the full sato-live image until it
is successful.
2) Bump connman&apos;s PR (or some other sato recipe&apos;s PR).
3) Rebuild the sato live image by &quot;bitbake poky-image-sato-live&quot;.

Sometimes, I meet build error like the following, however it does not happen
every time.

-------------------------------------------------------
Generating solve db for
/distro/dongxiao/build-qemux86/tmp/deploy/rpm/qemux86...
   total:               1      0.000000 MB      1.424841 secs
   fingerprint:      1020      0.006796 MB      0.033057 secs
   install:           340      0.000000 MB      0.371773 secs
   dbadd:             340      0.000000 MB      0.362746 secs
   dbget:            2196      0.000000 MB      0.004278 secs
   dbput:             340      1.504908 MB      0.308950 secs
   readhdr:          3401      2.961280 MB      0.005603 secs
   hdrload:          1700      4.389932 MB      0.007001 secs
   hdrget:          57535      0.000000 MB      0.043769 secs
Generating solve db for /distro/dongxiao/build-qemux86/tmp/deploy/rpm/i586...
error: open of
/distro/dongxiao/build-qemux86/tmp/deploy/rpm/i586/connman-plugin-ethernet-0.65-r4.i586.rpm
failed: No such file or directory
rpm.real: ./rpmio_internal.h:190: fdGetOPath: Assertion `fd != ((void *)0) &amp;&amp;
fd-&gt;magic == 0x04463138&apos; failed.
/distro/dongxiao/build-qemux86/tmp/work/qemux86-poky-linux/poky-image-minimal-initramfs-1.0-r0/temp/run.do_rootfs.468:
line 375:   669 Aborted                 rpm --dbpath /var/lib/rpm
--define=&apos;_openall_before_chroot 1&apos; -i --replacepkgs --replacefiles
--oldpackage -D &quot;_dbpath $pkgdir/solvedb&quot; --justdb --noaid --nodeps --noorder
--noscripts --notriggers --noparentdirs --nolinktos --stats --ignoresize
--nosignature --nodigest -D &quot;__dbi_txn create nofsync&quot; $pkgdir/solvedb/manifest
ERROR: Function &apos;do_rootfs&apos; failed (see
/distro/dongxiao/build-qemux86/tmp/work/qemux86-poky-linux/poky-image-minimal-initramfs-1.0-r0/temp/log.do_rootfs.468
for further information)
-------------------------------------------------------

The root cause for this issue should be, poky-image-minimal-initramfs&apos;s
do_rootfs task doesn&apos;t have dependency on connman, thus their tasks will be run
simultaneously. Poky-image-minimal-initramfs&apos;s do_rootfs will call
&quot;rootfs_rpm_do_rootfs&quot; --&gt; &quot;package_update_index_rpm&quot;, where it will update all
the packages depsolver db in ${DEPLOY_DIR_RPM}.

When the package_update_index_rpm function is handling connman&apos;s rpm package,
and at the same time, connman is removing old rpm and trying to generate a new
one (e.x, from r4 to r5), then the build error will occur, saying that it could
not find r4 version of connman-plugin-ethernet...</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12629</commentid>
    <comment_count>6</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2011-03-15 08:46:24 +0000</bug_when>
    <thetext>This fix was applied: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c180383885e32aed83c4f8691115859966fb53be

but I think the lock is taken in the wrong place. The files are not placed into tmp/deploy/ at this point but later in the sstate code. Its that later point where we need to have the lock held (and also at the point any sstate package is uninstalled.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12661</commentid>
    <comment_count>7</comment_count>
    <who name="Dongxiao Xu">dongxiao.xu</who>
    <bug_when>2011-03-15 18:39:26 +0000</bug_when>
    <thetext>Locks (rpm.lock) are already taken in python function sstate_clean and sstate_install. But the lock is not taken in &quot;do_rootfs&quot; --&gt; &quot;rootfs_rpm_do_rootfs&quot; --&gt; &quot;&quot;package_update_index_rpm&quot;, where it operates in &quot;${DEPLOY_DIR_RPM}&quot;. 

I am in confuse that the &quot;do_rootfs&quot; is written in shell, and how can we add lock for it or its sub-functions?</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12666</commentid>
    <comment_count>8</comment_count>
    <who name="Mark Hatle">mark.hatle</who>
    <bug_when>2011-03-15 19:09:11 +0000</bug_when>
    <thetext>There is a global exclusive lock in the rootfs_rpm.bbclass:


# RPM doesn&apos;t work with multiple rootfs generation at once due to collisions in the use of files
# in ${DEPLOY_DIR_RPM}. This can be removed if package_update_index_rpm can be called concurrently
do_rootfs[lockfiles] += &quot;${DEPLOY_DIR_RPM}/rpm.lock&quot;



This is the same lock file that should be done whenever the tmp/deploy/rpm/* is being modified.


The intention is to use an exclusive lock in the do_rootfs, but use a shared lock during RPM copying into the deploy directory.  We can manipulate multiple RPMs at the same time, but we must block when doing all of the rootfs operations.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12722</commentid>
    <comment_count>9</comment_count>
    <who name="Mark Hatle">mark.hatle</who>
    <bug_when>2011-03-16 18:56:20 +0000</bug_when>
    <thetext>I&apos;ve pushed a possible fix to poky-contrib  mhatle/bernard/fixes, commit id:

81c5ab86c5c0697cdee3958f69d4ad6251083a9d


Please try that and see if it corrects the issue you were seeing.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12733</commentid>
    <comment_count>10</comment_count>
    <who name="Dongxiao Xu">dongxiao.xu</who>
    <bug_when>2011-03-16 19:16:35 +0000</bug_when>
    <thetext>Sorry I still see the same error after applied your latest fix on mhatle/bernard/fixes.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12734</commentid>
    <comment_count>11</comment_count>
    <who name="Mark Hatle">mark.hatle</who>
    <bug_when>2011-03-16 19:49:34 +0000</bug_when>
    <thetext>Please refresh from poky-contrib mhatle/bernard/fixes and try again.

The top commit: 3182ada41f06e20c325d0f63ae9b211523490a60  takes the drastic approach of locking/serializing the whole system during the RPM package creation.

I&apos;m curious if this resolves the problem or not.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12735</commentid>
    <comment_count>12</comment_count>
    <who name="Mark Hatle">mark.hatle</who>
    <bug_when>2011-03-16 19:54:56 +0000</bug_when>
    <thetext>Sorry, I just noticed I changed the wrong variable, the right commit has now been pushed:

c0edbfdd704a8fcb6f07e3b5059bec6d3b7f22ab

difference is affecting &quot;sstate-lockfile&quot; vs &quot;lockfile&quot;..</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12736</commentid>
    <comment_count>13</comment_count>
    <who name="Mark Hatle">mark.hatle</who>
    <bug_when>2011-03-16 20:13:11 +0000</bug_when>
    <thetext>ONE more time, the -right- commit for head is: 8824e8130cac060e9d06881aa98f8aba7bf4002b

Sorry for any confusion.  Again the key line that should be there:


+do_package_write_rpm[sstate-lockfiles] += &quot;${DEPLOY_DIR_RPM}/rpm.lock&quot;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12737</commentid>
    <comment_count>14</comment_count>
    <who name="Dongxiao Xu">dongxiao.xu</who>
    <bug_when>2011-03-16 20:21:06 +0000</bug_when>
    <thetext>I am aware of that change and is now testing against the latest fix.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>12738</commentid>
    <comment_count>15</comment_count>
    <who name="Dongxiao Xu">dongxiao.xu</who>
    <bug_when>2011-03-16 20:31:24 +0000</bug_when>
    <thetext>Sorry this bug still exists after applying the fix of the following. But it seems that the rate to reproduce the issue is lower, now I can encounter this issue within about 5~10 trials.




diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 4647116..56e8308 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -777,16 +777,10 @@ python do_package_rpm () {
        cmd = cmd + &quot; --define &apos;debug_package %{nil}&apos;&quot;
        cmd = cmd + &quot; -bb &quot; + outspecfile

-       # Take a shared lock, we can write multiple packages at the same time...
-       # but we need to stop the rootfs/solver from running while we do...
-       lf = bb.utils.lockfile(bb.data.expand(&quot;${DEPLOY_DIR_RPM}/rpm.lock&quot;, d), True)
-
        # Build the rpm package!
        bb.data.setVar(&apos;BUILDSPEC&apos;, cmd + &quot;\n&quot;, d)
        bb.data.setVarFlag(&apos;BUILDSPEC&apos;, &apos;func&apos;, &apos;1&apos;, d)
        bb.build.exec_func(&apos;BUILDSPEC&apos;, d)
-
-       bb.utils.unlockfile(lf)
 }

 python () {
@@ -803,6 +797,7 @@ SSTATETASKS += &quot;do_package_write_rpm&quot;
 do_package_write_rpm[sstate-name] = &quot;deploy-rpm&quot;
 do_package_write_rpm[sstate-inputdirs] = &quot;${PKGWRITEDIRRPM}&quot;
 do_package_write_rpm[sstate-outputdirs] = &quot;${DEPLOY_DIR_RPM}&quot;
+do_package_write_rpm[sstate-lockfiles] += &quot;${DEPLOY_DIR_RPM}/rpm.lock&quot;

 python do_package_write_rpm_setscene () {
        sstate_setscene(d)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>13021</commentid>
    <comment_count>16</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2011-03-29 03:54:14 +0000</bug_when>
    <thetext>I&apos;ve noticed it should be &quot;sstate-lockfile&quot;, not sstate-lockfiles. I&apos;m guessing this is why the patch we expected to fix this didn&apos;t work...

I&apos;m going to add shared lockfile support to sstate, then we can use that...</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>13024</commentid>
    <comment_count>17</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2011-03-29 06:27:27 +0000</bug_when>
    <thetext>Fixed in master in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=68baef906513c1abef5372d92c769dc5d695e054</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>13815</commentid>
    <comment_count>18</comment_count>
    <who name="Saul Wold">sgw</who>
    <bug_when>2011-05-13 11:15:51 +0000</bug_when>
    <thetext>Bernard

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=3d2c481ab0047a0432b61ec226aa7da2e64bdde1</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>