Bug 8584

I finally found the bug I thought already covered this - it's a bit short on details, but it's bug 1619.

*** Bug 1619 has been marked as a duplicate of this bug. ***

The gtk-doc patchset has been sent for review, but there's still significant amount of work to do (doxygen in particular requires lots of small tweaks to recipes), so the whole thing is postponed to 2.3.

I've finished the survey of doxygen support in upstream projects, and here are the findings:

1) doxygen api documentation generation is supported in some way by 67 recipes

2) of those, only 9 support doxygen in a way that is distribution-friendly, specifically:
- there is a configure switch to enable/disable usage of doxygen
- generation of html documentation happens with make's default target, and installation happens with 'make install'

Those recipes are dbus, flac, libexif, libfakekey, libvorbis, libxcb, libxcbcommon, mc, source-highlight.

3) The rest of the recipes do things in various different ways:

- the most common way is that there is a separate make target to build doxygen (make apidocs, make doxygen, make docs, make html and so on - everyone names it differently), and no configure switches or installation targets. 

- another common approach is to ship only doxyfile (the config file for doxygen) and provide no makefile support at all.

4) The conclusion I have is that upstreams, by and large, don't see doxygen documentation as something that distributions should package or worry about. To upstream projects, doxygen support is their own internal thing, that they use to put the html output on their websites. I'd say we should not struggle against this by doing a ton of recipe fixing and tweaking - there are more beneficial things to do.

Okay, now onto manpages. A large amount of upstream projects ship and install ready-to-use manpages, for them nothing needs to be done. Then there projects where manpages can be conditionally built. A few of those can be found by looking for "--enable/disable-manpages" in recipe directory, but there might be others lurking - so I'll look for "enable-man" and its variants in the build tree as well.

I first wanted to be more generic and look for variants of "manpage" in the source tree, but quickly realized that it's a road to insanity, caused by monotonous, astronomical amount of false positives.

Then building of manpages will be conditionally enabled if 'api-documentation' is in distro features, as it tends to pull in the whole xml stack and other specialized tools.

The patchset for manpages is now available here (will submit for review after 2 weeks of holiday):
https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=akanavin/enable-documentation

I've also added patches that remove the aging, difficult to maintain SGML stack to the same branch. Amazingly, nothing seems to need it in meta-oe, so it doesn't have to be moved there (unless someone using a 3rd party layer complains).

Patches sent for review:
http://lists.openembedded.org/pipermail/openembedded-core/2016-October/127647.html

Paul, I'm now ready to enable adding the generated documentation into SDKs. Can you point me to the files where SDKs generation and configuration happens, so I don't need to look for it with grep etc.? :)

I'm not sure much needs to be done - for bitbake -c populate_sdk <image> you will get documentation packages in the SDK right now if you add doc-pkgs to SDKIMAGE_FEATURES.

(In reply to comment #9)
> I'm not sure much needs to be done - for bitbake -c populate_sdk <image> you
> will get documentation packages in the SDK right now if you add doc-pkgs to
> SDKIMAGE_FEATURES.

Actually, also 'api-documentation' must be in DISTRO_FEATURES so that additional manpages and gtk-doc based documentation is produced, so where should I place that? It's not enabled by default as it adds significantly to the build time for common images (core-image-sato etc.).

I'm not sure I fully understand the question. These are two different things:

* 'api-documentation' in DISTRO_FEATURES enables building the documentation
* 'doc-pkgs' in IMAGE_FEATURES / SDKIMAGE_FEATURES enables installing all related documentation (API or otherwise) into the image / SDK respectively.

You might perhaps trigger doc-pkgs from api-documentation I suppose, but it's possible that's not always going to be what the user wants.

   (In reply to comment #11)
> I'm not sure I fully understand the question. These are two different things:
> 
> * 'api-documentation' in DISTRO_FEATURES enables building the documentation
> * 'doc-pkgs' in IMAGE_FEATURES / SDKIMAGE_FEATURES enables installing all
> related documentation (API or otherwise) into the image / SDK respectively.

The question is, for every SDK that gets generated, how do we ensure that 'api-documentation' was enabled in DISTRO_FEATURES?

The process that constructs an SDK cannot control DISTRO_FEATURES, so I can't see how that would be possible.

(In reply to comment #13)
> The process that constructs an SDK cannot control DISTRO_FEATURES, so I
> can't see how that would be possible.

The SDK building could refuse to continue if DISTRO_FEATURES doesn't have the needed entry?

I really don't think that's appropriate. There will be quite a number of people out there who don't care to have the documentation included in their SDK.

(In reply to comment #15)
> I really don't think that's appropriate. There will be quite a number of
> people out there who don't care to have the documentation included in their
> SDK.

In that case I think the work is basically complete here (when the patchset for manpages gets merged). I'll file a separate bug for changing the site configuration so that when Yocto SDKs are generated, the api-documentation distro feature is in use.

Ross, any comment?

There's outstanding issues with adding doc-pkgs to SDK images (as they get giant) but Alex is aware of that.  I think we should also add doc-pkgs to SDKIMAGE_FEATURES (maybe if api-documentation is enabled in DISTRO_FEATURES) so that generated SDKs (not SDK images) also get the documentation (I think that docs in standalone SDKs is more important than in SDK images) but apart from that we're done.

'api-documentation' DISTRO_FEATURE needs to be documented. Something like:

When this feature is in use, generation of API documentation during recipe builds is enabled, and the resulting documentation will be added to SDK tarballs produced by 'bitbake -c populate_sdk'.

Regarding documenting api-documentation, here is a section in the ref-manual that lists the DISTRO_FEATURES available in YP: http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-features-distro.  Would adding an entry for api-documentation cover this issue?

Note that the glossary entry for DISTRO_FEATURES does not provide an exhaustive list but rather highlights some stuff - http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-DISTRO_FEATURES.

In the SDK manual we have a chapter here (http://www.yoctoproject.org/docs/2.2/sdk-manual/sdk-manual.html#sdk-appendix-customizing) that talks about customizing an SDK.  Should we create a subsection on how to add API documentation?

Setting this to ACCEPTED since I own it now.

Scott

(In reply to comment #19)
> Regarding documenting api-documentation, here is a section in the ref-manual
> that lists the DISTRO_FEATURES available in YP:
> http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#ref-features-
> distro.  Would adding an entry for api-documentation cover this issue?

Yes.

> In the SDK manual we have a chapter here
> (http://www.yoctoproject.org/docs/2.2/sdk-manual/sdk-manual.html#sdk-
> appendix-customizing) that talks about customizing an SDK.  Should we create
> a subsection on how to add API documentation?

Yes please.

Hi, 

This api-documentation fix... Did it make it into Morty?  I am guessing that it did not.  Correct?

Thanks, 
Scott

Hi,  

The fix is in for this now.  There were several areas that were touched.  Here they are:

 * http://www.yoctoproject.org/docs/2.3/sdk-manual/sdk-manual.html#sdk-appendix-customizing-standard - this is a new appendix in the SDK manual for customizing the standard SDK.  This is where we talk about adding API documentation to the SDK.  I had to put in another subsection in this appendix to justify creating a whole new appendix.  That other section was on how to add individual packages to the SDK.

 * http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#var-TOOLCHAIN_HOST_TASK - I added a cross-reference from this variable description to the new section on adding packages to a standard SDK.

 * http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#var-TOOLCHAIN_TARGET_TASK - same thing here.  Cross-reference to new section.

 * http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#ref-features-distro - Added the api-documentation feature to this list.

Let me know of any changes or modifications. 

Thanks, 
Scott

(In reply to comment #21)
> This api-documentation fix... Did it make it into Morty?  I am guessing that
> it did not.  Correct?

No, it didn't. Only gtk-doc support made it to morty, not all the other stuff, so morty documentation doesn't need to be touched.

> http://www.yoctoproject.org/docs/2.3/sdk-manual/sdk-manual.html#sdk-appendix-
> customizing-standard - this is a new appendix in the SDK manual for
> customizing the standard SDK.  This is where we talk about adding API
> documentation to the SDK.  I had to put in another subsection in this
> appendix to justify creating a whole new appendix.  That other section was
> on how to add individual packages to the SDK.

This says:

" You might want to include documentation as part of the standard SDK. For example, you might want the API documentation to be built and included with the SDK. You can include API documentation as well as any other documentation with the standard SDK by setting the SDKIMAGE_FEATURES and DISTRO_FEATURES variables:

     SDKIMAGE_FEATURES = "doc-pkgs"
     DISTRO_FEATURES = "api-documentation"
        

Setting these variables as shown here causes the OpenEmbedded build system to build the documentation and then include it in the standard SDK."

You can drop the first two sentences, they're just repeating the same thing. And also changing SDKIMAGE_FEATURES is not needed, just adding to DISTRO_FEATURES is enough.

Here's how it should be fixed:

"You can include API documentation as well as any other documentation provided by recipes with the standard SDK by adding 'api-documentation' to the DISTRO_FEATURE variable:

     DISTRO_FEATURES_append = " api-documentation"

Setting this variable as shown here causes the OpenEmbedded build system to build the documentation and then include it in the standard SDK."


> Let me know of any changes or modifications. 

Looks ok otherwise.

Hi, 

See http://www.yoctoproject.org/docs/2.3/sdk-manual/sdk-manual.html#adding-api-documentation-to-the-standard-sdk for the changes.

Scott

Seems fine to me now.

Thanks, 

I am setting to RESOLVED and marking the doc flag to "done".

Scott

Changing to verified since the documentation is here:
http://www.yoctoproject.org/docs/2.3/sdk-manual/sdk-manual.html#adding-api-documentation-to-the-standard-sdk