Bug 10172 - Suggested new section to clarify stamps, input checksums, and the sstate cache
Summary: Suggested new section to clarify stamps, input checksums, and the sstate cache
Status: RESOLVED FIXED
Alias: None
Product: Reference
Classification: Documentation
Component: handbook (show other bugs)
Version: 2.2
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.2 M4
Assignee: Scott Rifenbark
QA Contact:
URL:
Whiteboard: 01 Sept 2016: RESOLVED
Depends on:
Blocks:
 
Reported: 2016-08-23 10:25 UTC by ulfalizer
Modified: 2016-09-01 18:02 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Done (doc changes complete)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ulfalizer 2016-08-23 10:25:26 UTC
Hello,

One thing I think trips many people up and that also confused me at first is
input signatures and stamp files. It's easy to get the impression that input
signatures are only used for the sstate cache and nothing else, and stamp files
become very hazy then too.

I suggest adding the following section before the existing '3.5.7. Setscene
Tasks and Shared State' section. It tries to straighten the concepts out.

  3.5.7. Stamp Files and Rerunning of Tasks
  
  For each task that completes successfully, BitBake writes a stamp file into the
  [STAMPS_DIR] directory. The beginning of the stamp file's filename is
  determined by the [STAMP] variable, and the end consists of the task's name and
  current [input checksum](https://www.yoctoproject.org/docs/2.2/bitbake-user-manual/bitbake-user-manual.html#checksums).
  (This assumes that
  [BB_SIGNATURE_HANDLER](https://www.yoctoproject.org/docs/2.2/bitbake-user-manual/bitbake-user-manual.html#var-BB_SIGNATURE_HANDLER)
  is "OEBasicHash", but this is almost always the case in modern OpenEmbedded.)
  
  To determine if a task needs to be rerun, BitBake checks if a stamp file with a
  matching input checksum exists for the task. If such a stamp file exists, the
  task's output is assumed to exist and still be valid, and otherwise the task is
  rerun.
  
  Note {
  The stamp mechanism is more general than the shared state (sstate) cache
  mechanism described below. BitBake avoids rerunning any task that has a valid
  stamp file, not just tasks that can be accelerated through the sstate cache.
  
  Note however that stamp files only serve as a marker that some work has been
  done and do not record task output. The actual task output would usually be
  somewhere in [TMPDIR], e.g. in some recipe's [WORKDIR]. What the sstate cache
  mechanism adds is a way to cache task output that can then be shared between
  build machines.
  }
  
  Since STAMPS_DIR is usually a subdirectory of TMPDIR, removing TMPDIR will also
  remove STAMPS_DIR, meaning tasks will properly be rerun to repopulate TMPDIR.
  
  If you want some task to always be considered "out of date", you can mark it
  with the
  [[nostamp]](https://www.yoctoproject.org/docs/2.2/bitbake-user-manual/bitbake-user-manual.html#variable-flags)
  flag. Note that if some other task depends on such a task, then that task will
  also always be considered out of date, which might not be what you want.


To make the new section easier to discover, some links could be added:

 * At the end of 6.2.19:

     See [3.5.7. Stamp Files and Rerunning of Tasks] for more information.

 * In the glossary entry for STAMP, in a separate paragraph before "See
   STAMPS_DIR, MULTIMACH_TARGET_SYS, ...":

     See [3.5.7. Stamp Files and Rerunning of Tasks] for more information.


Cheers,
Ulf
Comment 1 Scott Rifenbark 2016-09-01 17:12:19 UTC
Ulf, 

I am back working now....  YP only allows me a certain number of hours per month and I had run out for the month of July...

For this bug, see the following:

 * http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#stamp-files-and-the-rerunning-of-tasks

 * http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#structure-build-tmp-stamps

 * http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#var-STAMP

Let me know of any corrections.

Scott
Comment 2 ulfalizer 2016-09-01 17:44:49 UTC
Hello,

No worries. Don't get stomach ulcers for my sake. There's seldom any super panic to get stuff in. :)

There might be a missing "that" in the sentence below. Other than that, it looks fine to me.

  However, you should realize that stamp files only serve as a marker that some
  work has been done and <that> these files do not record task output.

Cheers,
Ulf
Comment 3 Scott Rifenbark 2016-09-01 18:02:45 UTC
Ulf, 

I won't get any ulcers :)

See http://www.yoctoproject.org/docs/2.2/ref-manual/ref-manual.html#stamp-files-and-the-rerunning-of-tasks.

Since that was the only item I am going to set this to RESOLVED now and turn the doc flag to "done."

Thanks