Bug 256

Tracked at https://bugs.launchpad.net/rpm/+bug/633742

To understand how this may or may not workin the scope of RPM.  Can you give some examples on how the comparison logic should work with tilde within the version?

For Mark's benefit, the following is an incremental list of versions illustrating what tilde means:

2.6.24~rc1
2.6.24~rc2
2.6.24
2.6.24.1~rc1
2.6.24.1
2.6.25~rc1
2.6.25~rc2
2.6.25
2.6.25.1

Marking as post 1.1 as this isn't likely to happen in the 1.1 timeframe at this point.

The sequence you have given is insufficient to determine
what the collation sequence SHOULD be even if tilde-in-version
is included.

rpmvercmp does segmented (i.e. alphas/digits/other are split into classes) comparisons

The comparison issues start to creep-in with (arbitrarily defined) mixed-mode comparisons.
I.e. How should digits be compared to alphas or punctuation when found in
a comparison sequence?

Your sequence is too predictable to capture the problematic mixed-mode
comparison issues. There are historical issues that prevent ASCII
or LC_COLLATE being simply re-used.

The intent of tilde-in-version is very well known: how to implement
mixed-mode comparison in rpm is what isn't obvious.

The general rule is that for version comparison, we tend to follow the debian scheme (http://www.debian.org/doc/debian-policy/ch-controlfields.html). I don't know the rpm version comparison code well enough to know how this will translate there.

What examples would you want to see added to the sequence to 100% clarify the situation?

You can plug whatever versioning scheme you
wish, with whatever collate sequence you desire.

But the mixed mode comparison pathologies
need to be pinned down precisely in explicit
test cases before anything can be "supported".

Saying "Just like Debian!" isn't sufficient either,
even though the Debian comparison (or at least my
rewrite of dpkgcmp code, none has ever bothered
to confirm correctness) has been in RPM for years.

The hysterical issues in rpmvercmp are largely
    1) comparison between, say, digit <-> punctualtion is surprising
    the actual comparison is with "" (or missing or empty) and
    the (arbitrarily defined) comparison result isn't what people
    expect (which is ASCII or float based)

    2) digit padding means that 1.0000000002 > 1.1

    3) strcmp (as in glibc) means that case isn't distinguished.

    4) (mandriva and disttag related) many distros wish a "branding"
    identifier like
            foo-1.2-3mdv2011
    Sadly the distro "branding" bigots also wish the explicitly "branded"
    version to be "newer" than the unbranded version, which breaks
    the natural ASCII (or LC_COLLATE but lets ignore encoding issues) collation

   5) the '-' character is forbidden in version strings. and ':' is almost forbidden
   for similar reasons: rpm MUST be able to split "E:V-R" strings like
        Requires: foo = E:V-R
   into a tuple of {E,V-R} on known boundaries (both '-' and ':' are boundaries)

There's also the additional twist(s) of twiddle-in-epoch and twiddle-in-release
that twiddle-in-version will surely lead to RFE's for.

So tests cases to handle missed-class comparisons, and with missing values,
and '~' in other locations, possibly multiple, with regression tests are needed
to be acceptable for inclusion in RPM itself.

Again, all the RPM itself needs is a boolean return from an opaque comparison.
But "RPM sux! because it doesn't do what Debian does!" (and interoperability
in general) isn't enhanced by opaque pluggable comparisons.

I would point out that RPM already permits a trailing *
character in {V,R} strings which will match the remainder of the string
(by exiting immediately with the current comparison result)
is already implemented.

In some ways the trailing '*' character accomplishes what tilde-in-version
accomplishes with rather simpler semantics. Just not what is in Debian ...

I haven't said RPM sux. We have the situation that OE started with opkg, added dpkg, then rpm support. opkg has a defined approach to version numbers which is to follow debian unless it doesn't make sense for embedded use.

We can therefore choose to follow the debian approach, or, if I hear good arguments otherwise, we could patch dpkg and do something different. I'd prefer not to but its possible.

The one thing I want to see is that bitbake, opkg, dpkg and rpm have some kind of similar ideas about how to use the versions, at least in the environment we're using. I've seen what happens when they disagree and we're not doing it again ;-).

In summary I agree we need a good set of definitions and a good test set and those need to be created as part of this work. I worry if we're waiting on rpm to write those definitions, we may have a long wait as I see links with discussions back five years ago and little movement :(.

If * can be made to work and we can come up with some stable translation, that is perfectly fine. We already allow "-" in V but have translations for rpm to deal with that...

The lack of "forward"motion isn't from me.

SHow me test cases or set a goal and you will have
an implementation within a month.

The general implementation is harder because of
rampant "RPM sux because ..." comments (not from you).

Again, you can implement whatever version comparison
you wish to see in RPM by plugging in a (*rpmvercmp)
vector. There's a similar vector for the higher level triple
of {E,V,R} that can be plugged as well.

And there's a *RE pattern that is used to split on boundaries
that can be re-written to do whatever you chosse.

Meanwhile RPM is expected to be compatible with itself: that
isn't your opkg/bitbake/ipkg/dpkg/... usage case.

And its not RPM, but rather tools that don't use rpm lib interfaces
correctly, that is going to break. I've seen 5-6 tool bugs over
the last few months with Mandriva adding Distepoch/Disttag
(which the *RE parser handles perfectly, including ignoring
if/when the additionl info isn't needed/useful). So tools like
zypper and/or smart and/or yum are what is going to break
because not correctly using rpm lib comparison functions.

Happy to help with whatever you desire in Poky: its just a collation sequence *shrug*.

Instead of transforms, there's already an excluded character
list in version comparison. All excluded characters are treated
equivalently (originally, and still in @rpm.org code, all "other"
punctuation characters are treated equivalently: so
    1.2_3 = 1.2|3

See rpmdb/rpmevr.c (where the comparison routines live)
for two quirks:

/* XXX Force digits to beat alphas. See bugzilla #50977. */
/*@unchecked@*/
#if defined(RPM_VENDOR_MANDRIVA)  || defined(RPMVERCMP_DIGITS_BEAT_ALPHA) /* old-comparision-behaviour */
static int _invert_digits_alphas_comparison = -1;
#else
static int _invert_digits_alphas_comparison = 1;
#endif

/* XXX Punctuation characters that are not treated as alphas */
/*@unchecked@*/ /*@observer@*/
static const char * _rpmnotalpha = ".:-";


Unless you need to handle "branding" in release with the expectation of "newer",
don't do what Mandriva is doing.

And add any characters you wish to the excluded character _rpmnotalpha list.
The far harder issue (which likely necessitates awkward/complex transforms always)
is tools like zipper not using rpmlib API calls for version comparison.

One last related note is the EVRD parsing. This is the split
into a N-tuple (currently up to 4) and twiddle-in-version is
adding another element to the tuple with speshul comparison semantics.

The *RE parsing is quite general and can be used to split dependency
strings on newer boundaries, and can drop fields as well as permute
the precedence of the tuple comparisons, each of which is the (*rpmvercmp)
segmented string comparison applied sequentially to strings in the tuple.

Here is the configuration (from /usr/lib/rpm/macros) with the explicit
usage cases that match:


#==============================================================================
# ---- EVRD... dependency tuple comparison macros.
#

# STEP 1: Match the string and capture regex parts
#                      1          2           3             4
#                      X     ":"  X        "-"X          ":"X
%evr_tuple_match  ^(?:([^:-]+):)?([^:-]+)(?:-([^:-]+))?(?::([^:-]+))?$

# STEP 2: Assemble <E,V,R,D> tuple from regex parts
# <E,V,R,D>         case 1:  case 2:  case 3:  case 4:  case 5:  case 6:
#                   X:X-X:X  X:X-X    X:X      X:       X-X      X
# %evr_tuple_select 1234     123_     12__     1___     _23_     _2__
%evr_tuple_select 123_

# STEP 3: Configure the comparison order of the <E,V,R,D> tuple elements
%evr_tuple_order  EVR


SO what is in RPM is quite general even if there isn't the conventionalized
and tightly administratively controlled narros usage of twiddle-in-version
as in Debian. I'm sure that whatever Poky wishes to do with RPM version
comparison can be accommodated.

The rest of the system (bitbake, opkg, opkg-utils, dpkg) is updated to a shared debian version policy including support for "~". The only missing piece would be having RPM following that policy which this bug now reflects.

Discussions on this have occurred with the upstream RPM maintainer.  It's not completely clear yet how easy it would be to make this change, or what ramifications it may have to compatibility with other RPM releases.

Estimate includes significant testing requirements.

The implementation cost for adding twiddle in
version for RPM version compare is perhaps
a day.

Devising an adequate test case for twiddle-in-version
is trickier because of the number of unknown
corner cases that need to be identified
to ensure RPM is consistent with itself. There's
also a huge hidden cost as RPM tool chains adjust
to a new feature (which will likely take years, sigh).

Meanwhile -- for yocto with both dpkg+rpm -- perhaps
your interests are better served by _ALWAYS_ using
Debian version comparison. This is basically replacing
a single vector in RPM with a reimplementation (was
straight recoding from dpkgcmp(1) at the time). There's
still testing needed (noone has ever verified what I
recoded has anything to do with what Debian does) but
that is a much easier/passive/lazy test paradigm:
     Parallel dpkg/rpm implementations and add an
     assertion that the same result _ALWAYS_ is returned.

I can go either way: its the years of explaining as
the new (and mostly unnecessary) twiddle-in-version
implementation is incorporated that I'm not looking
forward too. The twiddle-in-version proposal was made
like 3 years ago and is just barely starting to get
acceptance ...

I have to admit at this point I favour redirecting rpm's version comparision function to match the rest of the OE system in our usage of rpm. This is likely simpler and also less likely to result in a mismatch between what bitbake does, what another package backend does and what rpm does.

This would still allow people to customise the system and use the normal rpm algorithm if desired and appropriate for their use case.

I'd also note that we've come at this from a very specific set of directions to end up in this position. We had a number of different version comparision algorithms in different pieces of the system and were hitting bugs due to them doing slightly different things. Opkg was always intended to follow debian so making a code change there for the ~ support and aligning with dpkg/debian made sense and actually fixed a number of other inconsistencies. Bitbake itself had two algorithms (yes, nuts), one heavily used and a rather strange older one, possibly from portage with a ton of hardcoded ~ like modifiers. Again, I'm very pleased to throw most of that away and use one single consistent algorithm.

Just for info this is the oldest thread on this issue on
the, now, defunct duke rpm-devel mailing list.

https://lists.dulug.duke.edu/pipermail/rpm-devel/2007-April/002360.html

This type of argument makes it seem very animated discussions.

FWIW, rpm4 supported this in 2012:

http://rpm.org/gitweb?p=rpm.git;a=commitdiff;h=db28221a4a48f6ee3c804c92314330637c808638

And twiddle-in-version was implemented the same day @rpm5.org
(with reservations about the usefulness)

    https://www.mail-archive.com/rpm-devel@rpm5.org/msg05282.html

FYI, this is still open simply because the amount of testing to verify the implementation that is in RPM5 so far hasn't been worth the effort.

At a point we can verify that the behavior is the same between the three package management systems (or at least document the difference is acceptable for the purpose of OE) then we can reconcile this defect.

I do hope to spend more time when YP 1.7 rolls around on RPM issues, including this one.

--Mark

I propose that we aim to close this in 1.7.

+1

All of the package managers as of pyro/2.3 now support ~ with the same semantics, tested with a selftest.  Closing.