Bug 11508 - Bugs in MultiBranch GitPoller example code and limitation in buildbot library
Summary: Bugs in MultiBranch GitPoller example code and limitation in buildbot library
Status: RESOLVED OBSOLETE
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.5
Assignee: Joshua Lock
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-05-12 16:17 UTC by Johannes Schrimpf
Modified: 2018-03-22 08:49 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments
Patch that includes branch names in GitPoller name to make it possible to use several pollers for same repo with different branches (1.93 KB, application/octet-stream)
2017-05-12 16:17 UTC, Johannes Schrimpf
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Johannes Schrimpf 2017-05-12 16:17:15 UTC
Created attachment 3763 [details]
Patch that includes branch names in GitPoller name to make it possible to use several pollers for same repo with different branches

There are several bugs in the code in example of MultiBranch GitPoller in the documentation and a related limitation in the buildbot library that is used:

1) In both schedulers there is a '}' after 3600 in the line for the interval

Solution for 1:
Remove the '}'s

After doing this, autobuilder managed to start but in Changesources in the webinterface only one GitPoller is visible.

2) The second scheduler seems to overwrite the first one. I could not verify with same repository and different branches but I could verify it happens with different repositories)

Solution for 2:
Add the two schedulers to the same scheduler list
scheduler:[{....},
           {....}]

That leads to the following problem:
3) The two pollers are not allowed to have the same name

Solution for 3:
Change the names for the two schedulers to be unique

Doing these three steps, I was able to add scheduler with multiple different repositories with a similar buildset config, but still did not get two GitPollers to work on the same repository. This is a problem in gitpoller.py.

The problem is the following:
- Two GitPollers with the same repository but different branches will get the same name which is a problem for twisted. To prevent this, gitpoller.py has to be changed. The applies patch is a quick fix that appends the list of branches as string in the name of the service. It works in my local setup, together with my suggested patch in ID 11507.


Autobuilder version: commit 867b093d540fcd51d821adcdd8184668ab23b40e
Comment 1 Joshua Lock 2017-06-07 15:29:01 UTC
As this bug is in buildbot core, have you considered submitting your fix to the upstream buildbot project? https://github.com/buildbot/buildbot

Over the course of the next development cycle or so I'd really like to move to a newer version of buildbot and ideally no longer carry a fork of the code in the yocto-autobuilder repository.
Comment 2 Johannes Schrimpf 2017-06-09 08:32:25 UTC
I had a brief look at the upstream repo, but since autobuilder uses a fork in an older version,  I did not look more into it.

Also, the use of gitpoller in autobuilder is quite specific and traceable and I don't know how this change would impact other software using buildbot. I don't have enough insight in the mainline buildbot setup to know how gitpoller is intended to be used outside autobuilder, and I could imagine this specific change could potentially break compatibility with other software.

I think moving to a mainline buildbot version would be very good and I guess then this issue should be addressed then in regard to the new version. If still present, it should definitively be reported to the buildbot developers, but I suggest to apply the patch until then to make autobuilder work in this setup.
Comment 3 Joshua Lock 2018-03-22 08:49:21 UTC
Resolved by the new infrastructure architecture of helper scripts (yocto-autobuilder-helper) with a more recent buildbot with minimal configuration (yocto-autobuilder2).