256 2010-09-02 02:52:25 +0000 Sync up RPM version policy with rest of system (including support '~' in version strings) 2017-06-13 10:53:16 +0000 1 1 7 Build System, Metadata & Runtime OE-Core deployment 1.6 x86 Multiple RESOLVED FIXED Moved from Laurentiu to Unassigned Medium enhancement Future 1 richard.purdie unassigned jessica.zhang julie.n.fleischer kevin.tian mark.hatle n3npq pinto.elia poky.bs.watcher poky.watcher richard.purdie ross.burton sgw shane.wang --- --- Don't know oldest_to_newest 9543 0 richard.purdie 2010-09-02 02:52:25 +0000 It would be desirable to support the ~ notation in version strings for prerelease versions, e.g. 2.6.34~rc1 instead of 2.6.33+2.6.34-rc1. To make this change, bitbake and opkg need to both support the syntax in their verson comparision functions, we need to upgrade the dpkg/apt tools to a version which supports it and check how rpm handles it. 10812 1 n3npq 2010-10-27 11:59:31 +0000 Tracked at https://bugs.launchpad.net/rpm/+bug/633742 15209 2 mark.hatle 2011-08-04 07:40:18 +0000 To understand how this may or may not workin the scope of RPM. Can you give some examples on how the comparison logic should work with tilde within the version? 15981 3 richard.purdie 2011-09-02 14:36:21 +0000 For Mark's benefit, the following is an incremental list of versions illustrating what tilde means: 2.6.24~rc1 2.6.24~rc2 2.6.24 2.6.24.1~rc1 2.6.24.1 2.6.25~rc1 2.6.25~rc2 2.6.25 2.6.25.1 Marking as post 1.1 as this isn't likely to happen in the 1.1 timeframe at this point. 15982 4 n3npq 2011-09-02 14:45:18 +0000 The sequence you have given is insufficient to determine what the collation sequence SHOULD be even if tilde-in-version is included. rpmvercmp does segmented (i.e. alphas/digits/other are split into classes) comparisons The comparison issues start to creep-in with (arbitrarily defined) mixed-mode comparisons. I.e. How should digits be compared to alphas or punctuation when found in a comparison sequence? Your sequence is too predictable to capture the problematic mixed-mode comparison issues. There are historical issues that prevent ASCII or LC_COLLATE being simply re-used. The intent of tilde-in-version is very well known: how to implement mixed-mode comparison in rpm is what isn't obvious. 15984 5 richard.purdie 2011-09-02 14:53:59 +0000 The general rule is that for version comparison, we tend to follow the debian scheme (http://www.debian.org/doc/debian-policy/ch-controlfields.html). I don't know the rpm version comparison code well enough to know how this will translate there. What examples would you want to see added to the sequence to 100% clarify the situation? 15986 6 n3npq 2011-09-02 15:12:53 +0000 You can plug whatever versioning scheme you wish, with whatever collate sequence you desire. But the mixed mode comparison pathologies need to be pinned down precisely in explicit test cases before anything can be "supported". Saying "Just like Debian!" isn't sufficient either, even though the Debian comparison (or at least my rewrite of dpkgcmp code, none has ever bothered to confirm correctness) has been in RPM for years. The hysterical issues in rpmvercmp are largely 1) comparison between, say, digit <-> punctualtion is surprising the actual comparison is with "" (or missing or empty) and the (arbitrarily defined) comparison result isn't what people expect (which is ASCII or float based) 2) digit padding means that 1.0000000002 > 1.1 3) strcmp (as in glibc) means that case isn't distinguished. 4) (mandriva and disttag related) many distros wish a "branding" identifier like foo-1.2-3mdv2011 Sadly the distro "branding" bigots also wish the explicitly "branded" version to be "newer" than the unbranded version, which breaks the natural ASCII (or LC_COLLATE but lets ignore encoding issues) collation 5) the '-' character is forbidden in version strings. and ':' is almost forbidden for similar reasons: rpm MUST be able to split "E:V-R" strings like Requires: foo = E:V-R into a tuple of {E,V-R} on known boundaries (both '-' and ':' are boundaries) There's also the additional twist(s) of twiddle-in-epoch and twiddle-in-release that twiddle-in-version will surely lead to RFE's for. So tests cases to handle missed-class comparisons, and with missing values, and '~' in other locations, possibly multiple, with regression tests are needed to be acceptable for inclusion in RPM itself. Again, all the RPM itself needs is a boolean return from an opaque comparison. But "RPM sux! because it doesn't do what Debian does!" (and interoperability in general) isn't enhanced by opaque pluggable comparisons. 15987 7 n3npq 2011-09-02 15:19:31 +0000 I would point out that RPM already permits a trailing * character in {V,R} strings which will match the remainder of the string (by exiting immediately with the current comparison result) is already implemented. In some ways the trailing '*' character accomplishes what tilde-in-version accomplishes with rather simpler semantics. Just not what is in Debian ... 15989 8 richard.purdie 2011-09-02 15:33:50 +0000 I haven't said RPM sux. We have the situation that OE started with opkg, added dpkg, then rpm support. opkg has a defined approach to version numbers which is to follow debian unless it doesn't make sense for embedded use. We can therefore choose to follow the debian approach, or, if I hear good arguments otherwise, we could patch dpkg and do something different. I'd prefer not to but its possible. The one thing I want to see is that bitbake, opkg, dpkg and rpm have some kind of similar ideas about how to use the versions, at least in the environment we're using. I've seen what happens when they disagree and we're not doing it again ;-). In summary I agree we need a good set of definitions and a good test set and those need to be created as part of this work. I worry if we're waiting on rpm to write those definitions, we may have a long wait as I see links with discussions back five years ago and little movement :(. 15990 9 richard.purdie 2011-09-02 15:36:43 +0000 If * can be made to work and we can come up with some stable translation, that is perfectly fine. We already allow "-" in V but have translations for rpm to deal with that... 15991 10 n3npq 2011-09-02 15:43:29 +0000 The lack of "forward"motion isn't from me. SHow me test cases or set a goal and you will have an implementation within a month. The general implementation is harder because of rampant "RPM sux because ..." comments (not from you). Again, you can implement whatever version comparison you wish to see in RPM by plugging in a (*rpmvercmp) vector. There's a similar vector for the higher level triple of {E,V,R} that can be plugged as well. And there's a *RE pattern that is used to split on boundaries that can be re-written to do whatever you chosse. Meanwhile RPM is expected to be compatible with itself: that isn't your opkg/bitbake/ipkg/dpkg/... usage case. And its not RPM, but rather tools that don't use rpm lib interfaces correctly, that is going to break. I've seen 5-6 tool bugs over the last few months with Mandriva adding Distepoch/Disttag (which the *RE parser handles perfectly, including ignoring if/when the additionl info isn't needed/useful). So tools like zypper and/or smart and/or yum are what is going to break because not correctly using rpm lib comparison functions. Happy to help with whatever you desire in Poky: its just a collation sequence *shrug*. 15992 11 n3npq 2011-09-02 15:51:05 +0000 Instead of transforms, there's already an excluded character list in version comparison. All excluded characters are treated equivalently (originally, and still in @rpm.org code, all "other" punctuation characters are treated equivalently: so 1.2_3 = 1.2|3 See rpmdb/rpmevr.c (where the comparison routines live) for two quirks: /* XXX Force digits to beat alphas. See bugzilla #50977. */ /*@unchecked@*/ #if defined(RPM_VENDOR_MANDRIVA) || defined(RPMVERCMP_DIGITS_BEAT_ALPHA) /* old-comparision-behaviour */ static int _invert_digits_alphas_comparison = -1; #else static int _invert_digits_alphas_comparison = 1; #endif /* XXX Punctuation characters that are not treated as alphas */ /*@unchecked@*/ /*@observer@*/ static const char * _rpmnotalpha = ".:-"; Unless you need to handle "branding" in release with the expectation of "newer", don't do what Mandriva is doing. And add any characters you wish to the excluded character _rpmnotalpha list. The far harder issue (which likely necessitates awkward/complex transforms always) is tools like zipper not using rpmlib API calls for version comparison. 15993 12 n3npq 2011-09-02 15:59:17 +0000 One last related note is the EVRD parsing. This is the split into a N-tuple (currently up to 4) and twiddle-in-version is adding another element to the tuple with speshul comparison semantics. The *RE parsing is quite general and can be used to split dependency strings on newer boundaries, and can drop fields as well as permute the precedence of the tuple comparisons, each of which is the (*rpmvercmp) segmented string comparison applied sequentially to strings in the tuple. Here is the configuration (from /usr/lib/rpm/macros) with the explicit usage cases that match: #============================================================================== # ---- EVRD... dependency tuple comparison macros. # # STEP 1: Match the string and capture regex parts # 1 2 3 4 # X ":" X "-"X ":"X %evr_tuple_match ^(?:([^:-]+):)?([^:-]+)(?:-([^:-]+))?(?::([^:-]+))?$ # STEP 2: Assemble <E,V,R,D> tuple from regex parts # <E,V,R,D> case 1: case 2: case 3: case 4: case 5: case 6: # X:X-X:X X:X-X X:X X: X-X X # %evr_tuple_select 1234 123_ 12__ 1___ _23_ _2__ %evr_tuple_select 123_ # STEP 3: Configure the comparison order of the <E,V,R,D> tuple elements %evr_tuple_order EVR SO what is in RPM is quite general even if there isn't the conventionalized and tightly administratively controlled narros usage of twiddle-in-version as in Debian. I'm sure that whatever Poky wishes to do with RPM version comparison can be accommodated. 20933 13 richard.purdie 2012-05-01 16:19:48 +0000 The rest of the system (bitbake, opkg, opkg-utils, dpkg) is updated to a shared debian version policy including support for "~". The only missing piece would be having RPM following that policy which this bug now reflects. 21190 14 mark.hatle 2012-05-08 15:39:14 +0000 Discussions on this have occurred with the upstream RPM maintainer. It's not completely clear yet how easy it would be to make this change, or what ramifications it may have to compatibility with other RPM releases. Estimate includes significant testing requirements. 21201 15 n3npq 2012-05-08 17:21:03 +0000 The implementation cost for adding twiddle in version for RPM version compare is perhaps a day. Devising an adequate test case for twiddle-in-version is trickier because of the number of unknown corner cases that need to be identified to ensure RPM is consistent with itself. There's also a huge hidden cost as RPM tool chains adjust to a new feature (which will likely take years, sigh). Meanwhile -- for yocto with both dpkg+rpm -- perhaps your interests are better served by _ALWAYS_ using Debian version comparison. This is basically replacing a single vector in RPM with a reimplementation (was straight recoding from dpkgcmp(1) at the time). There's still testing needed (noone has ever verified what I recoded has anything to do with what Debian does) but that is a much easier/passive/lazy test paradigm: Parallel dpkg/rpm implementations and add an assertion that the same result _ALWAYS_ is returned. I can go either way: its the years of explaining as the new (and mostly unnecessary) twiddle-in-version implementation is incorporated that I'm not looking forward too. The twiddle-in-version proposal was made like 3 years ago and is just barely starting to get acceptance ... 21301 16 richard.purdie 2012-05-11 09:13:54 +0000 I have to admit at this point I favour redirecting rpm's version comparision function to match the rest of the OE system in our usage of rpm. This is likely simpler and also less likely to result in a mismatch between what bitbake does, what another package backend does and what rpm does. This would still allow people to customise the system and use the normal rpm algorithm if desired and appropriate for their use case. 21302 17 richard.purdie 2012-05-11 09:18:43 +0000 I'd also note that we've come at this from a very specific set of directions to end up in this position. We had a number of different version comparision algorithms in different pieces of the system and were hitting bugs due to them doing slightly different things. Opkg was always intended to follow debian so making a code change there for the ~ support and aligning with dpkg/debian made sense and actually fixed a number of other inconsistencies. Bitbake itself had two algorithms (yes, nuts), one heavily used and a rather strange older one, possibly from portage with a ton of hardcoded ~ like modifiers. Again, I'm very pleased to throw most of that away and use one single consistent algorithm. 21702 18 pinto.elia 2012-05-29 16:09:42 +0000 Just for info this is the oldest thread on this issue on the, now, defunct duke rpm-devel mailing list. https://lists.dulug.duke.edu/pipermail/rpm-devel/2007-April/002360.html This type of argument makes it seem very animated discussions. 41870 19 ross.burton 2014-03-27 16:29:53 +0000 FWIW, rpm4 supported this in 2012: http://rpm.org/gitweb?p=rpm.git;a=commitdiff;h=db28221a4a48f6ee3c804c92314330637c808638 41878 20 n3npq 2014-03-27 18:27:57 +0000 And twiddle-in-version was implemented the same day @rpm5.org (with reservations about the usefulness) https://www.mail-archive.com/rpm-devel@rpm5.org/msg05282.html 41879 21 mark.hatle 2014-03-27 18:33:41 +0000 FYI, this is still open simply because the amount of testing to verify the implementation that is in RPM5 so far hasn't been worth the effort. At a point we can verify that the behavior is the same between the three package management systems (or at least document the difference is acceptable for the purpose of OE) then we can reconcile this defect. I do hope to spend more time when YP 1.7 rolls around on RPM issues, including this one. --Mark 43041 22 ross.burton 2014-04-28 14:45:37 +0000 I propose that we aim to close this in 1.7. 43043 23 n3npq 2014-04-28 15:37:19 +0000 +1 74049 24 ross.burton 2017-06-13 10:53:16 +0000 All of the package managers as of pyro/2.3 now support ~ with the same semantics, tested with a selftest. Closing.