We need to identify all features relevant to Yocto and test them. This will take some time. This will not block the feature merge into master though.

While testing the bastille package from the poky-contrib adinux/meta-security branch I found the following issues:

1) bastille did not make any changes to permissions on /sbin/runlevel even though I selected "Yes" when asked to do so (question nr. 3)

Steps to reproduce:
-issue 'bastille -c'
-select 'Yes' to question nr. 3: Q: Would you like to set more restrictive permissions on the administration utilities?
-select 'No' for all other questions
-issue:
"
root@qemux86-64:~# ls -l /sbin/runlevel
lrwxrwxrwx 1 root root 23 Jul 19 11:26 /sbin/runlevel -> /sbin/runlevel.sysvinit
root@qemux86-64:~# ls -l /sbin/runlevel.sysvinit
-rwxr-xr-x 1 root root 5000 Jun 18 21:47 /sbin/runlevel.sysvinit
"

Notice that permissions have not changed. According to Bastille description, this binary should not be world-executable anymore.

2) /etc/Bastille/config can not be changed or re-created using the latest answers given in interactive mode(bastille -c). Bastille should populate this file with the answers from the interactive mode.

3) the file /etc/Bastille/config that the image comes with does is incomplete:
root@qemux86-64:~# bastille -b -f /etc/Bastille/config
NOTE:    Entering Critical Code Execution.
         Bastille has disabled keyboard interrupts.


NOTE:    Bastille is scanning the system configuration...

FATAL:   A fatal error has occurred.  Not all of the questions
         that pertain to this system have been answered.  Rerun
         the interactive portion of Bastille on this system.
         MODULE.QUESTION=Patches.spc_run

4) 'bastille -l' does not identify /etc/Bastille/conf as a valid configuration file

I could not make Bastille make any changes as it is now.

For testing I used an older 1.5 master commit. When I tried to make a core-image-sato using meta-security and poky 1.5 M1 RC1(master:a62aed41f2d8f874f7ae24d0e5be5dbc66ea2199) I got this warning:

WARNING: QA Issue: bastille: Files/directories were installed but not shipped
  /run
  /run/lock
  /run/lock/subsys
  /run/lock/subsys/bastille

When I tried with the latest master:31e6eee860b5f9f4ac9ef0889bcff5648de6e3f9 I got this error:

ERROR: QA Issue: bastille: Files/directories were installed but not shipped
  /run
  /run/lock
  /run/lock/subsys
  /run/lock/subsys/bastille
ERROR: QA run found fatal errors. Please consider fixing them.
ERROR: Function failed: do_package_qa
ERROR: Logfile of failure stored in: /home/cornel/SSD/poky/build/tmp/work/all-poky-linux/bastille/3.2.1-r0/temp/log.do_package.28495
ERROR: Task 338 (/home/cornel/SSD/poky/meta-security/recipes-security/bastille/bastille_3.2.1.bb, do_package) failed with exit code '1'

core-image-sato can be successfully built using latest master and git://git.yoctoproject.org/meta-security .

Some of these problems and also #3867 problems addressed in:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=mulhern/bastille-final.

Comment (1)
Now, a record of these decisions can be maintained simply by editing the config file in the files directory.

Comment (2)
(1) Bastille is comprehensively broken. It can't reliably be expected to make _any_ changes based on user responses while run in Interactive mode. Currently, InteractiveBastille makes a system call to BackEndBastille which seems to immediately fail. The call fails because it does not grant BackEndBastille the useful information it needs to run correctly. The screen flickers briefly between the last screen and the credits, I believe that that is an error message for the console's benefit.

(2) This works correctly, now.

(3) This is fixed, because now the questions answered in the configuration file match the questions that Bastille expect for the Yocto distribution. Unfortunately, that's not a lot of progress, since Bastille has great difficulty applying the changes.

(4) This is fixed. Bastille could locate the config file, but previously it refused to list it because there were no log messages indicating that Bastille had been run already. Now it lists config files regardless of whether it believes it has already been run.

There do not seem to be any errors during baking for me, either.

Moving to 1.5 M5 because of install issue with bastille (bug 5177)

Since we have removed bastille from the 1.5 requirements, this tracking QA bug should be closed and QA should test the buck security or checksecurity packages.

Are buck-security and checksecurity requirements for 1.5?

The points that mulhern mentioned are working as described. Postponing this feature as per Saul's comment.