From discussion at http://lists.openembedded.org/pipermail/openembedded-core/2016-May/122003.html Oe-core is currently using versions of smartpm and createrepo with the following issues: 1) createrepo's upstream version is 10 years old and doesn't support Python 3 - we can't update to a newer one because it's been rewritten on top of yum python modules 2) smartpm has been abandoned upstream, and isn't supporting Python 3 either. I see two alternatives: 1) Introduce yum (or its 'next generation' variant called dnf) to oe-core as smartpm's replacement, update createrepo to the current version. 2) Fork, port to py3, and maintain createrepo and smartpm.
Have we made a decision on this yet?
On the bug review call on July 7, 2016, the consensus was to delay changing the package manager until 2.3. We'll investigate porting to python3 since requiring python2 would be annoying considering that the rest of oe-core has made the switch.
(In reply to comment #2) > On the bug review call on July 7, 2016, the consensus was to delay changing > the package manager until 2.3. We'll investigate porting to python3 since > requiring python2 would be annoying considering that the rest of oe-core has > made the switch. I think the agreement was to get a python 3 version of createrepo integrated (presumably by porting the version we use to python3?) but to stick with smart and requiring python 2 on rpm and package management using targets for 2.2 In the 2.3 timeframe we should address the RPM package manager issue, my personal opinion is that dnf is the only valid option (as yum and zypper have already burned the project and smart has well known issues documented by its creator and is no longer actively maintained) but we'll likely need to do some work to support rpm5 with that package management stack.
Update: I'm making good progress, but the task is large and has a lot of loose ends. Here's where the work is published: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=akanavin/dnf-rpm4 Note: rpm5 is completely removed and replaced by rpm4. I understand this is controversial, but I believe this is the only sane way forward and out of the dead-end mess we got ourselves into with rpm5 and smart.
Alex, I have not done an exhaustive review.. but a few comments: In the rpm4 recipe: +# We need to configure rpm to use our primary package architecture as the installation architecture, +# and to make it compatible with other package architectures that we use. +# Otherwise it will refuse to proceed with packages installation. +do_install_append() { + mkdir -p ${D}${sysconfdir}/rpm + echo "${MACHINE_ARCH}-pc-linux" > ${D}${sysconfdir}/rpm/platform + echo "arch_compat: ${MACHINE_ARCH}: ${PACKAGE_ARCHS}" > ${D}${sysconfdir}/rpmrc +} You can't do this, as you have just made the rpm program MACHINE specific. (Worse even machine and configuration specific.) Any files written with installation knowledge need to be done as part of the rootfs setup. --- The chroot processing for the scriptlets needs to be more controlled: ++ rpmChrootOut(); + rc = runExtScript(plugins, prefixes, script->descr, lvl, scriptFd, &args, script->body, arg1, arg2, &script->nextFileFunc); ++ rpmChrootIn(); That will unconditionally pop out and back in to the chroot. This may work fine in the cross-install, but if you try to run an installer on a target system build with Yocto Project components it will fail. The corresponding change we had in rpm5 to this: http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/rpm/rpm/rpm-scriptletexechelper.patch Can be used as a model for this. Effectively we introduce a new variable: _cross_scriptlet_wrapper If this is defined, then we call that (instead of going into the chroot). This also allows us to set specific parameters to the script being called to indicate the proper chroot and associated elements, completely hiding the install behavior so RPM doesn't know we've had to intercept it and do something else. The script also takes the task of dealing with install scripts that can NOT run at cross-install time. The current approach can not deal with those cases, as the package will simply fail to install. --- I believe the changes from: 0001-Do-not-read-config-files-from-HOME.patch, are only partial. RPM hard codes the location of all of those configuration files, which will prevent RPM from working from an sstate-cache or in an eSDK or regular SDK. The startup of RPM needs to be captured and the paths to all hard coded elements handled via a dynamic processing aspect. Code for doing this was merged into RPM5 (under VENDOR_WINDRIVER if I remember correctly.) When the time comes I can help dig out the version we created there if it would be helpful. --- I see BerkleyDB - 6 was removed, only leaving 5. Does rpm 4 not work with DB 6, or is there another reason why this is necessary? My concern is there may be other programs that are expecting DB 6 and now there is no way to implement the specific version required for rpm4. (I don't object to removing DB6, but my concern is around DB 5 being called 'db' then.) --- noarch replacing all. When the RPM5 work was originally implemented, I had proposed this and was very clearly told by the OE community at large this was no acceptable. (Maybe attitudes have changed since then?) I do think we need to support the specific variants that OE has. In the RPM5 world, this was automatic as the package arch had no significance during package process, the package manager would interrogate the contents of the packages and different rules applied to ELF and non-elf files. A package with all non-elf files was effectively an all or noarch package for processing. The other concern I have with this, it can break upgrading from an older to a newer system. (Yes I know RPM itself changing pretty much breaks this -- but there are external items handling RPM package naming, manifests, etc that could be broken by this change.) --- In the (not yet completed) commit, rootfs.py: def _cleanup(self): change. You've stopped removing the __db.000 file. We should check if this is still necessary. Normally, BerkleyDB will put cache information in the __db.* files. This cached information includes full path names, lock file types, and other host specific information. Without erasing this file, when you try to run RPM on the target numerous BerkleyDB errors will occur. Something to watch out for. (I never liked just deleting these files, but it may still be necessary... it's also possible the filenames have changed.)
(In reply to comment #5) > You can't do this, as you have just made the rpm program MACHINE specific. > (Worse even machine and configuration specific.) > > Any files written with installation knowledge need to be done as part of the > rootfs setup. Thanks, this has been fixed now. I moved this code to meta/lib/oe/package_manager.py and changed rpm to read the location prefixes for configuration files from environment variables. And then rpm is directed to the target rootfs directory via setting those. > The chroot processing for the scriptlets needs to be more controlled: > > ++ rpmChrootOut(); > + rc = runExtScript(plugins, prefixes, script->descr, lvl, scriptFd, > &args, script->body, arg1, arg2, &script->nextFileFunc); > ++ rpmChrootIn(); > > That will unconditionally pop out and back in to the chroot. This may work > fine in the cross-install, but if you try to run an installer on a target > system build with Yocto Project components it will fail. Can you be more specific please, and describe the steps I need to take to make it fail (or confirm that it works). I cannot fix something blindly. > The corresponding change we had in rpm5 to this: > > http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/rpm/ > rpm/rpm-scriptletexechelper.patch > > Can be used as a model for this. Effectively we introduce a new variable: > _cross_scriptlet_wrapper > > If this is defined, then we call that (instead of going into the chroot). > This also allows us to set specific parameters to the script being called to > indicate the proper chroot and associated elements, completely hiding the > install behavior so RPM doesn't know we've had to intercept it and do > something else. > > The script also takes the task of dealing with install scripts that can NOT > run at cross-install time. The current approach can not deal with those > cases, as the package will simply fail to install. Yes, I didn't reimplement the _cross_scriptlet_wrapper on purpose, because it duplicates existing functionality. Oe-core has a generic facility (postinst_intercept) that does exactly same things, but does not require invasive (and hard to understand tbh) rpm patching. I've verified that it correctly executes install scripts in a controlled environment outside of rpm and defers them to first boot if they fail. > I believe the changes from: 0001-Do-not-read-config-files-from-HOME.patch, > are only partial. > > RPM hard codes the location of all of those configuration files, which will > prevent RPM from working from an sstate-cache or in an eSDK or regular SDK. > > The startup of RPM needs to be captured and the paths to all hard coded > elements handled via a dynamic processing aspect. Configuration file locations are now dynamic, see above. Now, it's possible the native version of rpm is still somehow contaminated with specific host paths in a way that matters for our purposes, but let's deal with that when we have a confirmed failure. > I see BerkleyDB - 6 was removed, only leaving 5. Does rpm 4 not work with > DB 6, or is there another reason why this is necessary? The reason is that open source community has rejected db6 and is sticking with db5. Everything in Fedora and Debian is compiled against it, and db6 isn't even packaged. I don't see why we should provide db6, considering its hostile license, lack of adoption by the community and the fact that rpm4 is tested against db5. We can pick up Fedora/Debian's patches for db5 as needed. > My concern is there may be other programs that are expecting DB 6 and now > there is no way to implement the specific version required for rpm4. (I > don't object to removing DB6, but my concern is around DB 5 being called > 'db' then.) There needs to be specific evidence for this being an actual problem. > noarch replacing all. When the RPM5 work was originally implemented, I had > proposed this and was very clearly told by the OE community at large this > was no acceptable. (Maybe attitudes have changed since then?) Yes, this is far easier than teaching the entire dnf/rpm4 stack to accept 'all'. I need to have actual reasons why 'all' == good, 'noarch' == bad which are something else than 'we have custom code with this assumption that will have to be fixed'. > The other concern I have with this, it can break upgrading from an older to > a newer system. (Yes I know RPM itself changing pretty much breaks this -- > but there are external items handling RPM package naming, manifests, etc > that could be broken by this change.) The upgrade path is something that worries me as well, because I don't have any idea how an existing, working system could be updated live via package feeds when rpms in those feeds are suddenly re-generated by rpm4 instead of rpm5. I need community input on this. Maybe the only sane way is reflashing a new image. > In the (not yet completed) commit, rootfs.py: def _cleanup(self): change. > You've stopped removing the __db.000 file. > > We should check if this is still necessary. Normally, BerkleyDB will put > cache information in the __db.* files. This cached information includes > full path names, lock file types, and other host specific information. > Without erasing this file, when you try to run RPM on the target numerous > BerkleyDB errors will occur. > Something to watch out for. (I never liked just deleting these files, but > it may still be necessary... it's also possible the filenames have changed.) The way I read rpm4 FAQ, if those files are present, then there was a problem with one of the programs using rpm libraries. Deleting those files is hiding the problem under the carpet; in any case, I've yet to see them appear when running tools on the host machine.
>> The chroot processing for the scriptlets needs to be more controlled: >> >> ++ rpmChrootOut(); >> + rc = runExtScript(plugins, prefixes, script->descr, lvl, scriptFd, >> &args, script->body, arg1, arg2, &script->nextFileFunc); >> ++ rpmChrootIn(); >> This code will unconditionally drop out of the chroot, run the script, then come back in. With an -on-target- installation (via anaconda, or even just using DNF), we need the system to chroot into the install environment and run the script. We will not have any external controlling application to assist, nor do we want one. >> The script also takes the task of dealing with install scripts that can NOT >> run at cross-install time. The current approach can not deal with those >> cases, as the package will simply fail to install. > >Yes, I didn't reimplement the _cross_scriptlet_wrapper on purpose, because it >duplicates existing functionality. Oe-core has a generic facility >(postinst_intercept) that does exactly same things, but does not require invasive >(and hard to understand tbh) rpm patching. I've verified that it correctly executes >install scripts in a controlled environment outside of rpm and defers them to first >boot if they fail. RPM by default (unless RPM4 has changed the behavior recently) will take an exit 1 (failure) as a package installation failure. The wrapper script, if defined, will run and invoke the scriptlet. It then captures the return code and handles if this is a deferred or successful run. If deferred it stages it into the first-boot initscripts. However is this working in RPM4? >Configuration file locations are now dynamic, see above. Now, it's possible the >native version of rpm is still somehow contaminated with specific host paths in a >way that matters for our purposes, but let's deal with that when we have a >confirmed failure. The path names to the configuration files to load were set via a define. The define hard coded the host path to each of these things. If you have a patch to adjust these components, that should help resolve it. The easiest way to check if things are broken. Build a system at one path. Re-usse the sstate-cache on a second path as a -different user- that does not have permissions of the first user. (So you will get errors about permissions vs not available.) In the past various files such as the popt macros, the macros, rpmrc, etc were all hard coded -- and in turn the sub-programs of rpm ended up be called in the hard coded location. It was a pretty catestrophic failure condition with sstate-cache and moving paths between users. In regards to the BerkleyDB versions. We have things (primarily in external layers) that use BerkleyDB. If they were build prior to this change, they would have been linked to BerkleyDB 6, with a simple dependency on 'db'. If we now remove berkleyDB, bitbake may not see that they have to be recompiled (or they could recompile and fail before they do not support the older BerkleyDB.) It would be better if RPM requiring a specific version of BerkelyDB could be made exact. I.e. remove the 'db' recipe, and replace it with a 'db51' recipe. Then set rpm4 to require 'db51'. >Yes, this is far easier than teaching the entire dnf/rpm4 stack to accept 'all'. I >need to have actual reasons why 'all' == good, 'noarch' == bad which are something >else than 'we have custom code with this assumption that will have to be fixed'. I have not looked at the DNF code. BUT why is it so difficult to teach DNF about alternative architecture names? The all vs noarch is very much an OpenEmbedded thing. When I proposed using noarch back in the early Yocto Project (0.9) days, I was more or less told no.. We will use 'all' because that is how OE works. If you can convince the OE folks 'noarch' is fine... then I can live with it -- but it was a big problem for the OE community before.
re noarch vs. all, looking at the patch we are translating all to noarch at the boundary between us and generating the rpms. Surely that's not a major issue? We do the same for deb. I can understand people being up in arms if we wanted to move to "noarch" in PACKAGE_ARCH, but that's not what we're doing. What was proposed (and understood by the folks that rejected the proposal) earlier?
My memory (perhaps flawed) when I brought this up last time, is that OE users were expecting a specific syntax associated with the arch in the page for the feed control. So they wanted any particular set of arbitrary architecture names, including all, any, and noarch [for compatibility]. When I suggested translating to noarch (for RPM only) the concern was there was a clear difference in behavior now between the other package formats and RPM. (Remember there was a pretty strong anti-rpm bias at the time as well.)
(In reply to comment #7) > >> The chroot processing for the scriptlets needs to be more controlled: > >> > >> ++ rpmChrootOut(); > >> + rc = runExtScript(plugins, prefixes, script->descr, lvl, scriptFd, > >> &args, script->body, arg1, arg2, &script->nextFileFunc); > >> ++ rpmChrootIn(); > >> > > This code will unconditionally drop out of the chroot, run the script, then > come back in. > > With an -on-target- installation (via anaconda, or even just using DNF), we > need the system to chroot into the install environment and run the script. > > We will not have any external controlling application to assist, nor do we > want one. The part I'm not getting is: is the filesystem that you are installing into different than the filesystem you are running the installation tool from? If they're the same, then chroot-in and -out are no-ops - / is same as /. If they're different, what is the broader use case/scenario? > RPM by default (unless RPM4 has changed the behavior recently) will take an > exit 1 (failure) as a package installation failure. The wrapper script, if > defined, will run and invoke the scriptlet. It then captures the return > code and handles if this is a deferred or successful run. If deferred it > stages it into the first-boot initscripts. > > However is this working in RPM4? Here's how postinst_intercept works - I want to emphasize that this is not specific to any packaging backend, not written by me as a replacement for rpm5 wrapper, and it has been available in oe-core for several years: 1. If the install script can be written in a way that is guaranteed not to fail in the host cross-install environment, then it's simply executed directly by rpm. Any script failure halts bitbake and is a bug in the recipe to be fixed. 2. If the install script needs to do things that are not possible to do on the host or may fail (for example, running target binaries via qemu, or interacting with target hardware), it's written differently: - the script checks if it's running on host or target - if it's running on target, then it simply does what needs to be done - if it's running on host, it executes postinst_intercept script with a few parameters. This is guaranteed to succeed, and will queue the host-specific action, which will run after rpm has finished. If that action fails, the install script will be extracted from .rpm and placed on target rootfs so that it will be executed on first boot. > The easiest way to check if things are broken. Build a system at one path. > Re-usse the sstate-cache on a second path as a -different user- that does > not have permissions of the first user. (So you will get errors about > permissions vs not available.) > > In the past various files such as the popt macros, the macros, rpmrc, etc > were all hard coded -- and in turn the sub-programs of rpm ended up be > called in the hard coded location. It was a pretty catestrophic failure > condition with sstate-cache and moving paths between users. Thanks, this is something I need to try. > In regards to the BerkleyDB versions. We have things (primarily in external > layers) that use BerkleyDB. If they were build prior to this change, they > would have been linked to BerkleyDB 6, with a simple dependency on 'db'. > > If we now remove berkleyDB, bitbake may not see that they have to be > recompiled (or they could recompile and fail before they do not support the > older BerkleyDB.) The version of db would go from 0:6.something to 1:5.something, so bitbake will recompile any dependencies. There is a small chance someone somewhere has a recipe that won't work with 5.x for API incompatibility reasons, but I think that's pretty unlikely, given db6's lack of adoption. They can pull db6 into their layer if truly needed. > It would be better if RPM requiring a specific version of BerkelyDB could be > made exact. I.e. remove the 'db' recipe, and replace it with a 'db51' > recipe. Then set rpm4 to require 'db51'. This would break everything across all layers that depends on db. Perl and Python, for starters. > >Yes, this is far easier than teaching the entire dnf/rpm4 stack to accept 'all'. I > >need to have actual reasons why 'all' == good, 'noarch' == bad which are something > >else than 'we have custom code with this assumption that will have to be fixed'. > > I have not looked at the DNF code. BUT why is it so difficult to teach DNF > about alternative architecture names? Basically, because 'noarch' is scattered all over the place in dnf, libsolv, rpm and possibly elsewhere. I don't want to produce patches via search-and-replace for all components when the alternative fix in oe-core is a one-liner.
>The part I'm not getting is: is the filesystem that you are installing into different >than the filesystem you are running the installation tool from? If they're the same, >then chroot-in and -out are no-ops - / is same as /. If they're different, what is the >broader use case/scenario? (This is on the target) Often our customers boot to one partition. They then run an installer, which formats (mkfs), downloads and installs a system to another partition. This can be for initial device setup --or-- disaster recovery. So the --root= functionality MUST work in RPM. This is the reason why we have the -optional- wrapper intercept on RPM5. If the user is not running the intercept, we -always- chroot. If we are running the intercept, then it's the wrapper script that is responsible for deciding when and if we chroot. (As well as dealing with first-boot scripts and such..) >Here's how postinst_intercept works - I want to emphasize that this is not specific >to any packaging backend, not written by me as a replacement for rpm5 wrapper, and >it has been available in oe-core for several years: We're disconnected here. The debian, opkg and RPM's all behave the same conceptually: We install the packages, we look for scripts that exit with a non-0 error code (can't do a cross install) and then copy that script to the first-boot only init script system. Both debian and opkg software has the ability to return a list of scripts that have failed, so this can be taken -after- RPM has run. RPM on the otherhand, ALWAYS expects the install script to pass, no exceptions. If it does not pass -- then that package has failed to install. You must have a way to intercept to emulate the debian/opkg behavior. >1. If the install script can be written in a way that is guaranteed not to fail in >the host cross-install environment, then it's simply executed directly by rpm. Any >script failure halts bitbake and is a bug in the recipe to be fixed. Simply put, that is not possible. In many cases you can do this either via emulation or via special programming. But emulation does not work everywhere, and it's not always possible to run a host application to perform a target task. (Especially if it involves some type of machine identification, endian or LSB specific action, etc.) I've found many examples, this is just the first one that came to mind... Look at: http://git.openembedded.org/meta-openembedded/tree/meta-oe/recipes-support/gradm/gradm_3.1.bb lines 44-49: pkg_postinst_${PN}() { # make sure running on the target if [ x"$D" != "x" ]; then exit 1 fi /bin/mknod -m 0622 /dev/grsec c 1 13 } This is a simple post install script. But the first thing it does is check if we are running on the host or the target. If we are running on the -host-, it does an exit 1. With the existing wrapper script, that is intercepted, and the contents of this are staged and run on the target devices first-boot. >2. If the install script needs to do things that are not possible to do on the host >or may fail (for example, running target binaries via qemu, or interacting with >target hardware), it's written differently: >- the script checks if it's running on host or target >- if it's running on target, then it simply does what needs to be done >- if it's running on host, it executes postinst_intercept script with a few >parameters. This is guaranteed to succeed, and will queue the host-specific >action, which will run after rpm has finished. If that action fails, the install >script will be extracted from .rpm and placed on target rootfs so that it will be >executed on first boot. This is not existing behavior. See the example above. That is the existing system behavior. If exit 1 -- we stage for the first boot. If not exit 1, package was installed and finished -- we're good. I don't think it's reasonable to change the system wide packaging behavior to avoid a simply patch to RPM. It's really simple to look to see if a variable is defined, and if it is simply call that script (outside of the chroot) instead of directly invoking the scriptlet. >> It would be better if RPM requiring a specific version of BerkelyDB could be >> made exact. I.e. remove the 'db' recipe, and replace it with a 'db51' >> recipe. Then set rpm4 to require 'db51'. > >This would break everything across all layers that depends on db. Perl and Python, >for starters. No. Simple way to do this: git mv .../db_5.3.28.bb .../db5_5.3.28.bb Anything that depends on 'db' should already be using 'virtual/db'. (perl is already doing this for example.) Something that needs a specific version of db (5) could require db5. The reason is, I expect that people will need to provide db6 for their own applications -- and typically the oe upgrade policy is move to the latest version. So I would expect db5 to clearly be locked down on the '5' version, while 'db' could be db 6, 7, 8, 9 or whatever as time goes on. (This is not saying that 'db' would be an oe-core package, but it's certainly reasonable to put into meta-openembedded somewhere. Finally re noarch vs all/any. I still believe this is an architectural change that needs to be discussed on the oe-core/meta-openembedded and oe-architecture lists (likely starting with the later first.) As I've said before.. it was very clear in discussions when I first did the RPM5 implementation, anything other then all/any was not acceptable to the community at large.
(In reply to comment #11) > (This is on the target) Often our customers boot to one partition. They > then run an installer, which formats (mkfs), downloads and installs a system > to another partition. This can be for initial device setup --or-- disaster > recovery. I see. Note that this is not possible to figure out from contents of oe-core: there is no target test for this use case, or any mention of this scenario anywhere else in the tree. If you want this to not get broken, it needs to be tested from meta/lib/oeqa/runtime/rpm.py. > So the --root= functionality MUST work in RPM. This is the reason why we > have the -optional- wrapper intercept on RPM5. If the user is not running > the intercept, we -always- chroot. If we are running the intercept, then > it's the wrapper script that is responsible for deciding when and if we > chroot. (As well as dealing with first-boot scripts and such..) I'll add a check for an environment variable before the chroot out and in that the script run is currently sandwiched between. Then on target the chroot will happen as expected before the scripts are run, because the env var is not set. > pkg_postinst_${PN}() { > # make sure running on the target > if [ x"$D" != "x" ]; then > exit 1 > fi > /bin/mknod -m 0622 /dev/grsec c 1 13 > } > I don't think it's reasonable to change the system wide packaging behavior to > avoid a simply patch to RPM. > > It's really simple to look to see if a variable is defined, and if it is > simply call that script (outside of the chroot) instead of directly invoking > the scriptlet. The existing syntax and behavior can be improved, and now is the chance to do it. It's totally unclear that the check for $D and subsequent 'exit 1' means 'delay to first boot' (yes, there is a whole chapter about it in the manual somewhere, but if you need to explain it that way, that means the design is flawed), the rpm wrapper is causing functionality duplication with postinst_intercept (the bits about running the script and after-failure copying to firstboot directory in lib/oe/rootfs.py are basically identical), and I really don't want to patch any upstream project in a non-upstreamable way, unless absolutely necessary. I'd like to deprecate this approach (with a warning during image build - I concur that breaking recipes is bad), and make the following the preferred way (with postinst_intercept doing the delay to first boot if necessary behind the scenes): pkg_runontarget_${PN}() { /bin/mknod -m 0622 /dev/grsec c 1 13 } > No. Simple way to do this: > > git mv .../db_5.3.28.bb .../db5_5.3.28.bb > > Anything that depends on 'db' should already be using 'virtual/db'. (perl > is already doing this for example.) Python, ruby and apt do not. And same in meta-oe: for each recipe requiring virtual/db there's several that require simply db. > Something that needs a specific version of db (5) could require db5. No, this is backwards. db5 is what open source projects are developed and tested against, and *everything* we ship is already proven to work with db5 in Fedora and Debian. If someone needs db6 (for which there is still no evidence - we need to see community reaction), then the recipe should be called db6, and dependencies adjusted accordingly. > The reason is, I expect that people will need to provide db6 for their own > applications -- and typically the oe upgrade policy is move to the latest > version. Not in this case. Using db6 by default is counter-productive as it's not used or tested by distros outside of Yocto, and you've actually hit some pretty nasty bugs in it, if I remember right. Not to mention the 'gplv3 contamination' db6 is causing, and the potential for license clash with gplv2-only projects that rely on db. This needs to be brought up on the mailing lists. > Finally re noarch vs all/any. I still believe this is an architectural > change that needs to be discussed on the oe-core/meta-openembedded and > oe-architecture lists (likely starting with the later first.) As I've said > before.. it was very clear in discussions when I first did the RPM5 > implementation, anything other then all/any was not acceptable to the > community at large. I'll add this change to the cover letter for the initial patchset. Need to sit down and write one, but still so many other things to fix.
Patchset posted for review/testing: http://lists.openembedded.org/pipermail/openembedded-core/2017-February/132632.html
I've written the companion email to the oe-architecture list, here it is: http://lists.openembedded.org/pipermail/openembedded-architecture/2017-February/000419.html Flame away!
Patchset has been merged to oe-core master. Documentation needs to be updated, here's a starting point for that: http://lists.openembedded.org/pipermail/openembedded-architecture/2017-February/000419.html
Hi, The information is long and technical in this bug and seems to be mostly concerned with how things work under the hood. I read through the write up at http://lists.openembedded.org/pipermail/openembedded-architecture/2017-February/000419.html and don't understand the ramifications except that it looks like a huge entry for the migration section in the 2.3 YP doc set. I am going to need help deciphering all this if it is not just an entry in the migration section. I am reopening the bug for documentation purposes and will need to work with people that know what the hell is going on with this :) So, my first question does the documentation addition for this bug fall outside a migration section? One thing that I determined is that of all the various items discussed in this thread, only 'smartpm' is mentioned once. And, that was in a migration note in the "Removals and Renames" section talking about migrating to YP 1.4. The terms 'createrepo', 'rpm4', 'rpm5', and 'BerkleyDB' currently exist nowhere in our YP user doc set. We do use 'dnf' as the method for installing packages on Fedora. Thanks, Scott
Setting to 2.3 M4 and to NEEDINFO
Sometimes 'smartpm' is referred to as simply 'smart', can you check that that is not mentioned in the docs either?
Alexander, Here is a list of every place that "Smart" is referenced or discussed in the YP documentation.... We have a section in the dev-manual (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#using-runtime-package-management) called "Using Runtime Package Management". A subsection of that section is called "Using RPM" (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#runtime-package-management-target-rpm), which talks about setting up the target for package management. That subsection does go into "Smart" quite a bit. "Smart" is also mentioned in the third bullet of the http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu-image-enabling-tests section, which has to do with enabling tests. Smart is mentioned later and used in various examples in the "Running Tests" and "Writing Tests" subsections (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu-image-running-tests and http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu-image-writing-tests-class-attributes) In the ref-manual in "Image Generation" (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#image-generation-dev-environment), "Smart" is mentioned there. Also in the ref-manual's migration section for moving to YP 1.4, Smart is mentioned (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration-target-package-management-with-rpm) in the "Target Package Management with RPM" section. Again, in the migration section in "Removals and Renames" for moving to YP 1.4 (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration-1.4-removals-and-renames). Again, in the migration section for moving to YP 2.2 (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration-2.2-target-python-recipes-switched-to-python-3) in the "Target Python Recipes Switched to Python 3" section. In the ref-manual in the "package.bbclass" section (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#ref-classes-package), "Smart" is mentioned. In the variable glossary for the TEST_SERVER_IP variable (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#var-TEST_SERVER_IP), "Smart" appears in a note.
Thanks Scott, I'll look at all these, and rewrite them as needed.
Alexander, You can just give me suggestions or provide some facts around items that need to change. Don't think that you have to try and rewrite them all. Thanks, Scott
(In reply to comment #19) > We have a section in the dev-manual > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#using- > runtime-package-management) called "Using Runtime Package Management". A > subsection of that section is called "Using RPM" > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#runtime- > package-management-target-rpm), which talks about setting up the target for > package management. That subsection does go into "Smart" quite a bit. Update this one as follows: ====== 5.18.4.3.1. Using RPM The dnf application performs runtime package management of RPM packages. You must perform an initial setup for dnf on the target machine if the PACKAGE_FEED_ARCHS, PACKAGE_FEED_BASE_PATHS, and PACKAGE_FEED_URIS variables have not been set or the target image was built before the variables were set. As an example, assume the target is able to use the following package databases: all, i586, and qemux86 from a server named my.server. You must inform dnf of the availability of these databases by creating a /etc/yum.repos.d/oe-packages.repo file with the following content: [oe-packages] baseurl="http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all" From the target machine, fetch the repository: # dnf makecache After everything is set up, dnf is able to find, install, and upgrade packages from the specified repository. Dnf documentation is available here: http://dnf.readthedocs.io/en/latest/ ======= Section 5.18.5.2 should be adjusted to say that signing rpm feeds is currently not supported - there is a bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=11209 to fix that: ====================== In addition to being able to sign RPM packages, you can also enable the OpenEmbedded build system to be able to handle previously signed package feeds for IPK packages. Note The OpenEmbedded build system does not currently support signed DPKG or RPM package feeds. ====================== > "Smart" is also mentioned in the third bullet of the > http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu-image- > enabling-tests section, which has to do with enabling tests. Smart is > mentioned later and used in various examples in the "Running Tests" and > "Writing Tests" subsections > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu- > image-running-tests and > http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu-image- > writing-tests-class-attributes) Replace ========== Some of the tests (in particular smart tests) start an HTTP server on a random high number port, which is used to serve files to the target. The smart module serves ${DEPLOY_DIR}/rpm so it can run smart channel commands. =========== with ========= Some of the tests (in particular dnf tests) start an HTTP server on a random high number port, which is used to serve files to the target. The smart module serves ${WORKDIR}/oe-rootfs-repo so it can run dnf channel commands. ========= Elsewhere, simply replace smart with dnf. > In the ref-manual in "Image Generation" > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#image- > generation-dev-environment), "Smart" is mentioned there. Replace smart with dnf. > Also in the ref-manual's migration section for moving to YP 1.4, Smart is > mentioned > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration- > target-package-management-with-rpm) in the "Target Package Management with > RPM" section. > > Again, in the migration section in "Removals and Renames" for moving to YP > 1.4 > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration- > 1.4-removals-and-renames). > > Again, in the migration section for moving to YP 2.2 > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration- > 2.2-target-python-recipes-switched-to-python-3) in the "Target Python > Recipes Switched to Python 3" section. Those don't need to be changed, but the Migration to YP 2.3 section should contain a section of the implications of moving to dnf (I provided a link above that you can largely copy-paste). > In the ref-manual in the "package.bbclass" section > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#ref- > classes-package), "Smart" is mentioned. Replace with dnf. > In the variable glossary for the TEST_SERVER_IP variable > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#var- > TEST_SERVER_IP), "Smart" appears in a note. Replace with: The TEST_SERVER_IP variable is only used for a small number of tests such as the "dnf" test suite, which needs to download packages from WORKDIR/oe-rootfs-repo.
(In reply to comment #16) > The information is long and technical in this bug and seems to be mostly > concerned with how things work under the hood. I read through the write up > at > http://lists.openembedded.org/pipermail/openembedded-architecture/2017- > February/000419.html and don't understand the ramifications except that it > looks like a huge entry for the migration section in the 2.3 YP doc set. Actually, that email is not entirely accurate anymore. Here's what you should use: 1. Smart package manager is replaced by Dnf package manager. Smart has become unmaintained upstream, is not ported to Python 3.x and we need to find a replacement. Dnf is the only feasible candidate. The change in functionality is that the on-target runtime package management from remote package feeds is now done with a different tool that has a different set of command line options. If you have scripts that call the tool directly, or use its api (not sure if smart has an api), they need to be fixed. Dnf documentation: http://dnf.readthedocs.io/en/latest/ 2. Rpm 5.x is replaced with rpm 4.x. This is done for two major reasons: - DNF is API incompatible with Rpm 5.x and porting it and maintaining the port is non-trivial. - rpm 5.x itself is more or less unmaintained both in the upstream and in Yocto. There is no community around it, and Yocto is the sole remaining user. I think this is a stronger argument for moving away from it, than any technically superior features rpm 5.x may have over 4.x (not sure what those would be). 3. Berkeley db 6.x is removed; Berkeley db 5.x becomes default Version 6.x of Berkeley DB has been rejected by open source community due to its hostile AGPLv3 license; both Fedora and Debian are sticking with db 5.x - and by extension,all the open source projects are still developed and tested with db 5.x In oe-core the only thing that was requiring db 6.x was rpm 5.x, and so there's no reason to continue carrying db 6.x in oe-core. If someone needs API features that are only available in db 6.x, it can be re-added to meta-oe. 4. Createrepo is replaced with createrepo_c createrepo_c is the current incarnation of the tool that generates remote repository metadata; it's written in C (createrepo was in Python), is faster, and is maintained. 5. architecture-independent .rpm packages are "noarch" instead of "all" Too many places in dnf/rpm4 stack make that assumption; let's not fight against it. It's only about the filenames and the architecture tag in them; nothing else has changed in oe-core system, particularly allarch.bbclass is untouched. 6. Signing of remote package feeds using PACKAGE_FEED_SIGN is not supported, but there is a bug to fix that: https://bugzilla.yoctoproject.org/show_bug.cgi?id=11209
It appears the question has been answered.
(In reply to comment #22) > (In reply to comment #19) > > We have a section in the dev-manual > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#using- > > runtime-package-management) called "Using Runtime Package Management". A > > subsection of that section is called "Using RPM" > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#runtime- > > package-management-target-rpm), which talks about setting up the target for > > package management. That subsection does go into "Smart" quite a bit. > > Update this one as follows: > > ====== > 5.18.4.3.1. Using RPM > > The dnf application performs runtime package management of RPM packages. You > must perform an initial setup for dnf on the target machine if the > PACKAGE_FEED_ARCHS, PACKAGE_FEED_BASE_PATHS, and PACKAGE_FEED_URIS variables > have not been set or the target image was built before the variables were > set. > > As an example, assume the target is able to use the following package > databases: all, i586, and qemux86 from a server named my.server. You must > inform dnf of the availability of these databases by creating a > /etc/yum.repos.d/oe-packages.repo file with the following content: > > [oe-packages] > baseurl="http://my.server/rpm/i586 http://my.server/rpm/qemux86 > http://my.server/rpm/all" > > > From the target machine, fetch the repository: > > # dnf makecache > > > After everything is set up, dnf is able to find, install, and upgrade > packages from the specified repository. > > Dnf documentation is available here: > http://dnf.readthedocs.io/en/latest/ See http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#runtime-package-management-target-rpm for the changes. > ======= > > Section 5.18.5.2 should be adjusted to say that signing rpm feeds is > currently not supported - there is a bug > https://bugzilla.yoctoproject.org/show_bug.cgi?id=11209 to fix that: > > ====================== > In addition to being able to sign RPM packages, you can also enable the > OpenEmbedded build system to be able to handle previously signed package > feeds for IPK packages. > Note > The OpenEmbedded build system does not currently support signed DPKG or RPM > package feeds. See http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#processing-package-feeds for the change in 5.18.5.2. > ====================== > > > "Smart" is also mentioned in the third bullet of the > > http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu-image- > > enabling-tests section, which has to do with enabling tests. Smart is > > mentioned later and used in various examples in the "Running Tests" and > > "Writing Tests" subsections > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu- > > image-running-tests and > > http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#qemu-image- > > writing-tests-class-attributes) > > Replace > ========== > Some of the tests (in particular smart tests) start an HTTP server on a > random high number port, which is used to serve files to the target. The > smart module serves ${DEPLOY_DIR}/rpm so it can run smart channel commands. > =========== > with > ========= > Some of the tests (in particular dnf tests) start an HTTP server on a random > high number port, which is used to serve files to the target. The smart > module serves ${WORKDIR}/oe-rootfs-repo so it can run dnf channel commands. See the third bullet in http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#qemu-image-enabling-tests for this change. > ========= > > Elsewhere, simply replace smart with dnf. I found a couple instances here as I noted and changed them. One in the "default tests" bullet of here - http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#qemu-image-running-tests Also for "server_ip" in http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#qemu-image-writing-tests-class-attributes > > > In the ref-manual in "Image Generation" > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#image- > > generation-dev-environment), "Smart" is mentioned there. > > Replace smart with dnf. A single occurrence in the paragraph following the variable descriptions here - http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#image-generation-dev-environment > > > Also in the ref-manual's migration section for moving to YP 1.4, Smart is > > mentioned > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration- > > target-package-management-with-rpm) in the "Target Package Management with > > RPM" section. > > > > Again, in the migration section in "Removals and Renames" for moving to YP > > 1.4 > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration- > > 1.4-removals-and-renames). > > > > Again, in the migration section for moving to YP 2.2 > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#migration- > > 2.2-target-python-recipes-switched-to-python-3) in the "Target Python > > Recipes Switched to Python 3" section. > > Those don't need to be changed, but the Migration to YP 2.3 section should > contain a section of the implications of moving to dnf (I provided a link > above that you can largely copy-paste). ok - I left these. I usually work with Paul Eggleton on creating the migration section for each release. I will try to remind him of the moving to dnf issue for that new section for YP 2.3. > > > In the ref-manual in the "package.bbclass" section > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#ref- > > classes-package), "Smart" is mentioned. > > Replace with dnf. See the third paragraph in the package.bbclass (http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#ref-classes-package) description. > > > In the variable glossary for the TEST_SERVER_IP variable > > (http://www.yoctoproject.org/docs/2.3/mega-manual/mega-manual.html#var- > > TEST_SERVER_IP), "Smart" appears in a note. > > Replace with: > > The TEST_SERVER_IP variable is only used for a small number of tests such as > the "dnf" test suite, which needs to download packages from > WORKDIR/oe-rootfs-repo. See the variable description at http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#var-TEST_SERVER_IP for the change.
Hi Alexander, I addressed the individual changes and noted them inline in your Comment 22 (see my Comment 25). Regarding Comment 23... This is what looks like a migration section... correct? So take a look at the changes I made for replacing or rewriting "smart" stuff with "dnf" stuff and let me know if that is okay. And if you can confirm that the information in that email you referenced in Comment 23 is indeed a migration topic for YP 2.3. Thanks, Scott
(In reply to comment #26) > I addressed the individual changes and noted them inline in your Comment 22 > (see my Comment 25). Regarding Comment 23... This is what looks like a > migration section... correct? Correct. The text in comment 23 is for migration section and release notes, so please make sure it's in both. > So take a look at the changes I made for replacing or rewriting "smart" > stuff with "dnf" stuff and let me know if that is okay. And if you can > confirm that the information in that email you referenced in Comment 23 is > indeed a migration topic for YP 2.3. Here: http://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#qemu-image-enabling-tests where it says " The smart module serves", that should be dnf instead of smart. Otherwise, looks good to me.
Hi, I have documented that email about all the replaced things in the 2.3 migration notes (see http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#moving-to-the-yocto-project-2.3-release). So, I am marking this bug as RESOLVED and setting the doc flag to "done." Thanks, Scott