Bug 1882

I'll take this bug to start with and then hand-off to Scott R as:
a) Debian package backend needs work (see bug #1881)
b) We need to gather instructions for Scott to work with before he can write any docs

We have a wiki page on Zypper repository setup:
https://wiki.yoctoproject.org/wiki/Zypper_Repository_Setup

The commit adding the package-index target also discusses how to use it:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=9fbd9b93aea780c8168f6de2692148b14dd9d6a7

For Zypper read Smart here. There is a new page with instructions on how to set up a repo for that:

https://wiki.yoctoproject.org/wiki/Smart_Repository_Setup

I need to get you more information on both debian packaging, now that it works, 
I am just figuring it out myself and for ipkg.

Hi, 

Here is a link to the updated section for package feeds:

 * http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-build

The section did previously exist.  However, Daniela Placencia contributed to updating the sections.  Please look it over and be sure that things are up to par.  The bug is quite old.

Thanks,
Scott

I looked over this and it looks great, a few minor comments and a slight tangent follow.

One pet peeve of mine is that the "Host or Server Machine Setup" tells the user three ways to serve the packages.

In my opinion as a general rule we should recommend one, simple, method in the document and then either a) include other options in an appendix b) suggest alternatives and let the user find appropriate documentation for their choice.

The three different ways we recommend restarting the web server are prime examples of why we should try and avoid this kind of documentation. As a Fedora user I haven't used the service command for several years… (I use systemctl restart httpd).

Do we really want to document how to use 3 different web servers across all of the host distributions we support?

Apologies for the slight tangent, on to the rest:

In "5.16.4.3.1. Using RPM" I don't know what the sentence "This application is aware of every package database you want to use." is trying to say and it seems superfluous, can we remove it?

That same sentence is used in "5.16.4.3.2. Using IPK" and "5.16.4.3.3. Using DEB". Could we remove it from each section?

All three "Using X" sections propose a list of assumptions about the feeds for the sake of demonstrating syntax and commands, but "Using IPK" lists them as a bullet list whilst the other two list the assumptions in a sentence (i.e. "As an example, suppose you are serving packages from a deb/ directory containing the i586, all, and qemux86 databases through an HTTP server named my.server."). 

Could we pick one style and be consistent?

Minor typo, please add a space between # and opkg in "Using IPK" to be consistent with previous examples of commands

"#opkg update" → "# opkg update"

In "5.16.4.3.3. Using DEB" we say "You must perform an initial setup for opkg on the target machine." when we're talking about apt, not opkg.

"You must perform an initial setup for opkg on the target machine." → "You must perform an initial setup for apt on the target machine."

Please add a space between # and apt-get in "Using DEB" to be consistent with previous examples of commands

"#apt-get update" → "# apt-get update"

(In reply to comment #6)
> I looked over this and it looks great, a few minor comments and a slight
> tangent follow.
> 
> One pet peeve of mine is that the "Host or Server Machine Setup" tells the
> user three ways to serve the packages.
> 
> In my opinion as a general rule we should recommend one, simple, method in
> the document and then either a) include other options in an appendix b)
> suggest alternatives and let the user find appropriate documentation for
> their choice.
> 
> The three different ways we recommend restarting the web server are prime
> examples of why we should try and avoid this kind of documentation. As a
> Fedora user I haven't used the service command for several years… (I use
> systemctl restart httpd).
> 
> Do we really want to document how to use 3 different web servers across all
> of the host distributions we support?
> 
> Apologies for the slight tangent, on to the rest:

I could have won money on this comment :)  I am fairly sure that there is an ancient bug out there against the docs for running eclipse where you made the same comment regarding steps supporting the most recent version of Eclipse and some back versions - haha. Good point though.  

So, what we have is three servers (Apache 2, lighttpd, and SimpleHTTPServer).  For Apache 2, depending on the distro, there are three ways to reload the Apache configuration.  For the lighttpd server, it seems to just take a "most common" method regardless of any distribution.  And then for SimpleHTTPServer, there seems to be just a single set of commands.  

So if we streamline the Apache 2 method... what is the preferred method?


> 
> In "5.16.4.3.1. Using RPM" I don't know what the sentence "This application
> is aware of every package database you want to use." is trying to say and it
> seems superfluous, can we remove it?

Removed.

> 
> That same sentence is used in "5.16.4.3.2. Using IPK" and "5.16.4.3.3. Using
> DEB". Could we remove it from each section?

Yes - removed throughout

> 
> All three "Using X" sections propose a list of assumptions about the feeds
> for the sake of demonstrating syntax and commands, but "Using IPK" lists
> them as a bullet list whilst the other two list the assumptions in a
> sentence (i.e. "As an example, suppose you are serving packages from a deb/
> directory containing the i586, all, and qemux86 databases through an HTTP
> server named my.server."). 
> 
> Could we pick one style and be consistent?

Good observation - I fixed them.  

> 
> Minor typo, please add a space between # and opkg in "Using IPK" to be
> consistent with previous examples of commands
> 
> "#opkg update" → "# opkg update"

Fixed.

> 
> In "5.16.4.3.3. Using DEB" we say "You must perform an initial setup for
> opkg on the target machine." when we're talking about apt, not opkg.
> 
> "You must perform an initial setup for opkg on the target machine." → "You
> must perform an initial setup for apt on the target machine."

Fixed.

> 
> Please add a space between # and apt-get in "Using DEB" to be consistent
> with previous examples of commands
> 
> "#apt-get update" → "# apt-get update"

Fixed.

Just commenting on the http servers, I believe we can stick with SimpleHTTPServer, I personally don't think its the best option to use for a package feed, although I believe it is the simplest to explain and configure, and we can assume that whoever wants to use something different such as Apache, Lighttpd, Nginx or whatever else, knows what they're doing.

Forgot to comment on this last night, but in "5.16.4.1. Build Considerations" we say "As described below in the "Using IPK" section, if you are using IPK as your package format, you can make use of the distro-feed-configs recipe provided by meta-oe in order to configure your target to use your IPK databases." but we don't mention distro-feed-configs in the "Using IPK" section.

We do however have a mechanism in OE-Core to automatically configure package feeds on a target which we should discuss in this section of the manual; PACKAGE_FEED_URIS, PACKAGE_FEED_BASE_PATHS and PACKAGE_FEED_ARCHS variables as already discussed in the variable reference section of the ref manual:

http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#var-PACKAGE_FEED_URIS

(In reply to comment #8)
> Just commenting on the http servers, I believe we can stick with
> SimpleHTTPServer, I personally don't think its the best option to use for a
> package feed, although I believe it is the simplest to explain and
> configure, and we can assume that whoever wants to use something different
> such as Apache, Lighttpd, Nginx or whatever else, knows what they're doing.

I agree, we can document how to use SimpleHTTPServer to share package feeds from the developers machine and be sure to emphasise that this isn't a production solution and that people should look to Apache, Lighttpd and Nginx for that.

For the SimpleHTTPServer use case, I'm not sure the instructions are correct. Calling `python -m SimpleHTTPServer` will serve cwd, our steps should be:

cd {TMPDIR}/deploy/\\packageformat\\
python -m SimpleHTTPServer

Hey, 

So we have several things going on at once here.  I want to attack them one at a time so they don't get to complicated for me :)

I put in the changes that reduce the setup steps down to just SimpleHTTPServer.  Please look this over.  Once that is good... I will look at the "Using IPK" issue and the use of some stuff regarding the variables mentioned in Comment 9.

http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-server

Thanks, 
Scott

(In reply to comment #11) 
> I put in the changes that reduce the setup steps down to just
> SimpleHTTPServer.  Please look this over.  Once that is good... I will look
> at the "Using IPK" issue and the use of some stuff regarding the variables
> mentioned in Comment 9.
> 
> http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-
> package-management-server

I think that's probably more complicated than we need.

From within the build directory where I just built an RPM based image I can run:

$ cd tmp/deploy/rpm
$ python -m SimpleHTTPServer

rpm could be deb or ipk, depending on the PACKAGE_CLASSES setting

Ok, 

Here is a simple version, which apparently we probably only need here.  

Scott

http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-server

(In reply to comment #13)
> 
> Here is a simple version, which apparently we probably only need here.  
> 

Looks good to me, thanks Scott.

Joshua, 

I am setting the doc flag to done on this.  I guess I will leave the Status to you since you own the bug.  These types of situations are a bit confusing for me.  I never feel comfortable jacking with the status of someone else's bug.  However, the doc flag tends to get forgotten in the process so since you signed off on that change... I am clearing it. 

Thanks, 
Scott

(In reply to comment #9)
> Forgot to comment on this last night, but in "5.16.4.1. Build
> Considerations" we say "As described below in the "Using IPK" section, if
> you are using IPK as your package format, you can make use of the
> distro-feed-configs recipe provided by meta-oe in order to configure your
> target to use your IPK databases." but we don't mention distro-feed-configs
> in the "Using IPK" section.
> 
> We do however have a mechanism in OE-Core to automatically configure package
> feeds on a target which we should discuss in this section of the manual;
> PACKAGE_FEED_URIS, PACKAGE_FEED_BASE_PATHS and PACKAGE_FEED_ARCHS variables
> as already discussed in the variable reference section of the ref manual:
> 
> http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#var-
> PACKAGE_FEED_URIS

Scott, I think we haven't yet addressed this portion of my feedback?

Joshua, 

Ok - resetting the doc flag.

Hi, 

I am trying to learn a bit more about Joshua Lock's Comment 16.  Right now, we note to use the distro-feed-config recipe rather than the deprecated poky-feed-config-opkg recipe (see http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#migration-1.8-removed-recipes).  That is in a migration section.

We do have a "use" case described in the variable's glossary for the PACKAGE_FEED_URIS variable (see http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-PACKAGE_FEED_URIS).  I was thinking that we could take that use case and place it in the "Using IPK" section (http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-target-ipk).  But, I think distro-feed-config is a different beast.  The distro-feed-configs recipe is mentioned in the "Build Considerations" section (see http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-build).  

So I am unclear on a couple things... 1) exactly how do we show how to use distro-feed-configs since we obviously reference it and point the reader to that "Using IPK" section and there is no mention of it there. 2) In looking this over, I discovered that we have two references to this recipe.  One is "distro-feed-config" and the other is "distro-feed-configs".  Which one should I be using?

Scott

It's distro-feed-configs but I think it's irrelevant. I've done a bit of investigation and it seems this recipe, and recommendations to use it, pre-date the addition of PACKAGE_FEED_URIS and friends.

Therefore I don't think we should be recommending the use of distro-feed-configs, it's an Ipk specific implementation whereas the PACKAGE_FEED_URIS and related variables work for all supported package backends.

We should definitely mention those variables in the "Using runtime package management" section of the manual as they remove the need for manual configuration on the target.

We could maybe mention them in the build considerations section, as the variables need to be set before building the image in order to produce a correctly configured image, then in the "Using <backend>" sections we can suggest that the manual steps for configuring the target are only required if the variables were not used, or the target image was built before the variables were set?

Joshua, 

Good digging... that makes it much easier and cleaner all around.  I will fix things up on the manual side of things.

Scott

Joshua, 

Look in http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-build and the "Using <backend> sections for the changes.  Let me know of any changes.

http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-target-rpm

http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-target-ipk

http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-target-deb

Thanks,
Scott

Getting there, thanks Scott.

Could we clarify this part of the "build considerations" section a little bit

"Several packaging variables exist that help define the packages. You can use the PACKAGE_FEED_ARCHS, PACKAGE_FEED_BASE_PATHS, and PACKAGE_FEED_URIS variables to define the package feed URIs. If you do not define these variables, then manual steps as described in the subsequent sections are necessary to configure the target. You should set these variables before building the image in order to produce a correctly configured image."

That first sentence doesn't add any value and is a little muddled, how about:

"You can use the PACKAGE_FEED_ARCHS, PACKAGE_FEED_BASE_PATHS, and PACKAGE_FEED_URIS variables to pre-configure target images to use a package feed. If you do not define these variables, then manual steps as described in the subsequent sections are necessary to configure the target. You should set these variables before building the image in order to produce a correctly configured image."

Joshua, 

I implemented your suggestion verbatim.  See the section at http://www.yoctoproject.org/docs/2.2/dev-manual/dev-manual.html#runtime-package-management-build for the changes. 

Scott

Thanks Scott, I think we might be done with this one?

Marking resolved, Scott has set Docs flag to done and I've reviewed the changes.