Bug 8385

I added a new Test buildstep to a certain buildset, which was not picked up by the autobuilder.
For the sake of simplicity the new buildstep will be executed after the GetDistroVersion and before CreateAutoConf, say the buildset contains something like this:

steps: [{'SetDest':{}},
        {'CheckOutLayers': {}},
        {'RunPreamble': {}},
        {'GetDistroVersion' : {'distro': 'poky'}},
        {'Test' :{}},
        {'CreateAutoConf': {'machine': 'qemux86', 'SDKMACHINE' : 'i686',
                            'distro': 'poky', 'buildhistory' : True}},
...

I was able to trace this issue back to the ConfigParser library, it is invoked  by lib/python2.7/site-packages/autobuilder/Autobuilder.py on the function ParseConfig like the following:

import ConfigParser  
...
self.config = ConfigParser.ConfigParser()  
self.configdir=os.path.dirname(self.cfile)  
self.configfiles = [ join(self.configdir,f) for f in listdir(self.configdir) if isfile(join(self.configdir,f)) and (f != 'autobuilder.conf') ]  
    try:  
        self.config.read(self.configfiles)
...
self.buildsets=self.config.sections()  
self.configdict = {}  
for section in self.buildsets:  
    self.configdict[section]= dict(self.config.items(section))

Basically, after parsing the self.configfiles it creates a dictionary for every section (see https://docs.python.org/2/library/configparser.html), when reading that dictionary we can see that it is already missing the new buildstep:

[{'SetDest': {}}, {'CheckOutLayers': {}}, {'RunPreamble': {}}, {'GetDistroVersion': {'distro': 'poky'}}, {'CreateAutoConf': {'machine': 'qemux86', 'SDKMACHINE' : 'i686','distro': 'poky', 'buildhistory' : True}}}

This means the data is coming incomplete from ConfigParser, which comes from the HOST, we need to find out how it is caching the data (I didn't see any documentation regarding ConfigParser cache) and also find a way to force it to reparse the config files more easily.

Well, I figured out what the problem was, the issue is that emacs creates temporary files ending with a "~", so for buildset_test.conf we also have buildset_test.conf~ which is also parsed by ConfigParser, and apparently, if it ends up being parsed after the original file, it overrides the values obtained from the original configuration, so if a temporary file is old enough, it won't have the latest changes of the original file, which will cause the issue.

We can either assume the user won't have these temporary files, or maybe ignore them, I will send a patch to ignore these files since I feel a more robust system is a better solution.

Alejandro, have you seen this issue recur? I have not seen it in awhile and haven't been able to replicate what I was seeing. (Similar issue with no temp files.)

Currently reproducing

Tested with several conf files in buildset-config, which contain the same buildset (with a different branch, to see the changes) but have different names, including filenames ending in ~

e.g:

MORE_3~.conf
nightly-x86-64.conf
~nightly-x86-64.conf
nightly-x86-64.conf~
z_anotherfile.conf~

The behavior I observed:

The AB config parser will read all files ending in .conf (it skips other cases, including files ending in .conf~) regardless of the rest of the file names, and read the buildsets in them. It will match the names found under the section [BuildSets] to buildset names created as sections in those config files (e.g. [nightly-x86-64]).

I can not reproduce the AB reading config from files ending in ~

I propose adding a WARNING when a buildset section is found more than once

Also it seems this has been resolved by Beth Flanagan's commit:

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder/commit/lib/python2.7/site-packages/autobuilder/Autobuilder.py?id=4b9bfc3c0b429589714cfc157532559ebd73ef43