| Summary: | [zypper] search failure on qemuppc (nightly build 20110212-1) | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Meta-yocto | Reporter: | liang li <liang.li2> |
| Component: | meta-yocto | Assignee: | Qing He <qing.he> |
| Status: | VERIFIED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | elizabeth.flanagan, jiajun.xu, ke.yu, liangliang.wang, mark.hatle, poky.bs.watcher, poky.watcher, richard.purdie, sgw, yi.zhao |
| Version: | 1.0 | ||
| Target Milestone: | 1.0 M4 | ||
| Hardware: | x86 | ||
| OS: | ppc | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
|
Description
liang li
2011-02-14 17:55:56 UTC
I've tried several ways to debug this issues, but gdb together with -dbg packages failed surprisingly in several different ways:
a. gdb -core core /usr/lib/zypper
gdb outputs "Backtrace stopped: frame did not save the PC" when bt
b. running /usr/bin/.debug/zypper
line 1: syntax error: unexpected "("
c. running zypper with debug version of librpm* and libzypp*
segmentation fault at ld.so stage
d. gdb --args zypper search avahi
warning: Unable to find dynamic linker breakpoint function
Not only that, different arch and image types show completely different errors, as we can see the ppc sato, ppc sdk and x86_64 fails (Bug 721) in very different ways, while at least x86 works.
The backtrace shows that qemuppc sato failures happens in the destruction of the following class:
map<std::string, ZypperCommand::Command>
while in the source, the member of this data structure is fully statically constructed (and nothing looks architecture related).
Various aspects shows that ppc arch may be a little special and maybe even beyond only zypper: 1. The exact issue is reported only on ppc 2. different image types have different status 3. The problem submerged and emerged over time, see Bug 443 and Bug 722 - bug reported: 20101018, 20110114, 20110213 - not reported: 20101113, 20110122 4. other random bugs exists on qemu as well that binaries may segfault, see Bug 417 These make it very difficult to root cause the problem, and several other problems make it even harder: 1. zypper is written in C++ and boost (boost is difficult to debug...) 2. qemu is relatively slow for debugging (1 min+ to load a gdb session) 3. -sato or even -sato-sdk images are still somewhat restricted for a full development environment I'll continue to look into the bug, but am suggesting to lower the priority of ppc related zypper bugs for the above concerns. Since the problem also happens on qemux86-64 could we debug it there to help eliminate some of the potentially ppc related issues? For reference, you should never run the binary in .debug. Using the main binary is fine and the symbols will automatically be picked up from the extra file in .debug as its part of gdb's default search path (well, that from libbfd). Figured I should add the comments to the bug as well: While Paul was investigating he determined that the sizeof a malloc in rpm was incorrect. This sizeof indicated that the 'pcre' version of the regex_t structure was being used. (Which is correct -- RPM should ONLY be using pcre regex, as the normal system regex has limitations RPM needs to avoid.) Looking through the Zypper code, I find no invocations of the regex using the system regex. Looking at libzypp, I find a single call to the "#include <regex.h>". I've changed this to match rpm's pcreposix.h usage. This may fix the problem. The one confusing part in this there should not be a direct interface for Zypper to pass a regex_t structure into RPM for it to go through the "miRE" processing within RPM. So somewhere either Zypper is using a private interface, RPM is exposing more then it intended to, or ??? (My guess is RPM creates the memory with the pcre regex_t size, then passes it into libzypp, which tries to process it using the system regex_t instead -- causing the memory overrun.) So the patch that I provided attempts to synchronize libzypp and RPM to both be using pcre as -the- regex. Assuming this resolves the memory issue, we can worry about tracing how the problem occurred later... if it doesn't fix the problem, we need to start with Zypper/libzypp and trace into the RPM call structure and figure out how these data structures are created, used and destroyed. (What Paul pointed back to earlier today was the creation -- it's the usage that is where the memory overrun happens..) Hope this helps explain what I think the problem is, even if it doesn't really explain how to solve it. (One possible solution if we need to have an export of the regex_t data structure and avoid passing around raw memory... then zypper can do what it needs to using it's own regex functions...) -- I have pushed my changes to mhatle/zypper on the poky-contrib tree. (but I've been unable to test them) Basically what I did was poison the one instance of mire.h where it might have included "regex.h". This is likely always a bad thing to do. Then in the libzypp, I changed the one instance of including regex.h to include pcreposix.h instead. I THINK this will load in the correct regex processing functions -- but I'm not completely sure. (There might need to be an associated link line change to libzypp -- but so far it seems to link fine..) In jbj's latest reply in rpm-devel:
http://marc.info/?l=rpm-devel&m=129900937327748&w=2
He mentioned --with-pcre=internal, that's what we used in rpm-5.1.10,
but with rpm-5.4.0, internal pcre is not available, so external pcre is
used. I'd first guess that the pcre is a patched version, so I did a quick
diff, the diff shows the interesting differences:
-------
@@ -133,14 +130,19 @@
/* The functions */
-PCREPOSIX_EXP_DECL int regcomp(regex_t *, const char *, int);
-PCREPOSIX_EXP_DECL int regexec(const regex_t *, const char *, size_t,
+PCREPOSIX_EXP_DECL int pcreposix_regcomp(regex_t *, const char *, int);
+PCREPOSIX_EXP_DECL int pcreposix_regexec(const regex_t *, const char *, size_t,
regmatch_t *, int);
-PCREPOSIX_EXP_DECL size_t regerror(int, const regex_t *, char *, size_t);
-PCREPOSIX_EXP_DECL void regfree(regex_t *);
+PCREPOSIX_EXP_DECL size_t pcreposix_regerror(int, const regex_t *, char *, size_t);
+PCREPOSIX_EXP_DECL void pcreposix_regfree(regex_t *);
#ifdef __cplusplus
} /* extern "C" */
#endif
+#define regcomp pcreposix_regcomp
+#define regexec pcreposix_regexec
+#define regerror pcreposix_regerror
+#define regfree pcreposix_regfree
+
#endif /* End of pcreposix.h */
-------
together with the following definition in <pcreposix.h>:
-------
typedef struct {
void *re_pcre;
size_t re_nsub;
size_t re_erroffset;
} regex_t;
-------
which is exactly 24 bytes on x86_64, it's beginning to be clear what is
happened.
That is, the external libpcre.so has the symbol "regcomp", at complie
time, rpm is built and supposed to be linked against it, but the name
unfortunately collides with the "standard regex" which is in libc.so.
Although there is no "#include <regex.h>" in the source file, at
runtime, it's unintentionally linked to the libc version, the regcomp of
libc is called instead the pcre one using pcre's data structure...
that looks like a disaster.
After rebuilding libpcre with the patch, zypper invocation seems to work
well, at least on x86_64, no crash is observed.
I checked several pcre versions, from 8.02 to 8.12, they all have the
same issue, and debian actually has a patch against 8.02 to address it.
this also happend on mpc8315e-rdb sato-sdk images (nightly build 20110301-4): http://autobuilder.pokylinux.org/nightly/20110301-4/machines/mpc8315e-rdb/ppc/poky-image-sato-sdk-mpc8315e-rdb-20110302163011.rootfs.tar.gz verified on nightly build 20110305-4 : sato images in http://autobuilder.pokylinux.org/nightly/20110305-4/machines/qemu sato-sdk images in http://autobuilder.pokylinux.org/nightly/20110305-4/machines/mpc8315e-rdb/ppc/ git info : c192fefd868be53c20dfd8bc44e3e815cf53a10a |