During a Buildbot build step you might need to have a list of files which have changed as a result of a commit to your repository. A quick way to get this information during a build step is to subclass the build step class of your choice and overload its start() function.

A build step’s start() function is responsible for telling the slave to start executing the command. Each step class has a member variable called “build” of type “buildbot.process.base.Build” which it inherits from its main parent class “buildbot.process.buildstep.BuildStep”. Here is a quick example of getting file change information during a ShellCommand step in your master.cfg file:

from twisted.python import log
from buildbot.steps import shell
from buildbot.process.buildstep import RemoteShellCommand

# Executes a remote command with changed files appended onto the end
class ShellCommandChangeList(shell.ShellCommand):
  def start(self):
    # Substitute build properties into command
    command = self._interpolateProperties(self.command)
    # fail assert if command is not of correct type
    assert isinstance(command, (list, tuple, str))

    # Get changed file list from the build which invoked this step
    files = self.build.allFiles()
    # Log the changed files to twistd.log
    log.msg("Build Files: %s" % files)
    # Now we can do whatever we want with the list of changed files.
    # I will just append them to the end of the command.

    # Convert file list so it can be appended to the command's type
    if isinstance(command, str):
        files = " ".join(files)
    elif isinstance(command, tuple):
        files = tuple(files)

    # append files to end of command
    command += files

    # We have created the final command string
    # so we can fill out the arguments for a RemoteShellCommand
    # using our new command string
    kwargs = self.remote_kwargs
    kwargs['command'] = command
    kwargs['logfiles'] = self.logfiles

    # Create the RemoteShellCommand and start it
    cmd = RemoteShellCommand(**kwargs)
    self.setupEnvironment(cmd)
    self.checkForOldSlaveAndLogfiles()
    self.startCommand(cmd)

Here is a short contrived example of using the new ShellCommandChangeList build step with a BuildFactory to call a remote command for a fictional program named `checksum`. Let’s say that `checksum` is a program which takes filenames as input, computes the checksum of those files and prints those checksums to standard out.

from buildbot.steps import source, shell
from buildbot.process import factory

f = factory.BuildFactory()
f.addStep(source.SVN, svnurl="http://svn.example.org/Trunk/")
f.addStep(ShellCommandChangeList, command=["checksum"])

You should note that the way your VC system works and how your repository is set up in terms of file location and branches may affect how the file change paths are presented to you. You can easily see the list of file changes by using log.msg in your start() function to print them to the twistd.log file.

This is a quick update to my last article on Running Buildbot as a Windows XP Service. There was an interesting comment posted by jdpipe regarding the previous article . He offered a nice solution for some issues I was having with environment variables and stopping the Buildbot service. You can check out the article he wrote about it at https://pse.cheme.cmu.edu/wiki/view/Ascend/BuildBot.

He recommends using a python script instead of a batch file to ensure the build environment is setup correctly. By using a python script the service now launches pythonw.exe directly. This solves the Buildbot service issue where it’s not able to stop or restart properly as a result of being launched from a batch file. The article by jdpipe has excellent information on setting this up along with some pictures to help illustrate the process.

I have been working on setting up the new automated build system. We were having some hardware problems with the new server which was very unfortunate. Denrei tracked it down to the new gigabit NIC I installed. For now, we are going to revert back to the 100 Mbit onboard NIC.

Despite the problems, the build system seems to be working. The server seems stable, so I am finishing the automatic game source asset creation. Our build system notifies developers, via email, about updates to the distributed files. This is a good method for keeping developers up to date on the status of distributed files, but it is far too slow. If a developer is using files that are changing, they need to know immediately when those files change. As a result, we created a small utility called SNTask, that sits in the task bar. It shows developers the status of the build system in real-time, so they know immediately when files have changed.

One part of our next milestone consists of extending our automated build system. We already use Buildbot to automate most of our builds and testing. Every time source code is checked into the repository it is compiled and all unit tests are run. If the compile or unit tests fail the entire team is notified via email.

Our next milestone requires us to take our automated build system one step further. Source code and game source assets will be automatically built and distributed for all to use when checked into the repository. Any changes to the source code or game assets will result in all developers automatically using the updated version. If bad source code is checked in (it doesn’t compile or fails a unit test) then the build will fail, the team is notified and the broken version is not automatically deployed. There are many other cool extras we get for free when using this system. For example the ability to automatically deploy any version of the repository. If something went horribly wrong with the latest checked in version we can instantly build and deploy any prior version of the repository and all developers will automatically be using that version. Developers can deploy and work on different versions of the repository at the same time. This can also extend to branches and tags. We can have one part of our team working on an unstable branch of the engine while the other part is working on the main trunk of the repository which is stable. Each team works on their own branch and any changes they make are automatically deployed to the others using that same branch. When the unstable branch becomes stable and is merged into the main trunk then the new changes are instantly deployed to the group working with the main trunk.

This will also be very useful for game assets. The artist never has to compile or generate game assets from their source assets and then check the generated game assets into the repository. They just check in their source assets which are then automatically built into game assets and deployed to all developers. To test the new automated build system we decided to make another small game. Gavin and I are going to do most of the modeling so we can become acquainted with Maya. The game is a simple target range where you can shoot at various objects. As a result I felt compelled to name our current milestone “Hogan’s Alley”.

We finally got internet at our new place today. The release of our Pub Crawl milestone should happen shortly. We also replaced our old 100 Mbit networking hardware with new 1 Gbit switches. I was worried about the cost being too much for the extra speed. I was also worried that we wouldn’t see a big difference in our transfer rates. After hooking up the new switches my worries were quickly put to rest. The new transfer speeds over the network are wonderful and it was worth every penny. If you are deciding whether or not to switch to Gigabit Ethernet I highly recommend you do. If your computers don’t have Gigabit Ethernet cards you can pick them up pretty cheap and the money saved by using 100 Mbit hardware will not justify the slower transfer rates.

This is an in progress tutorial on how to run buildbot as a Windows XP service. Most of the information comes from these three articles by Grig Gheorghiu:

continuous-integration-with-buildbot
running-buildbot-on-various-platforms
running-python-script-as-windows

At Stolen Notebook we use Buildbot for our automated testing. Buildbot is a system that automates project builds and tests. It helps our team quickly identify errors in our code before they become problems. Buildbot runs after every repository check-in and compiles the code with tests. If the build or tests fail our team receives an email indicating who checked in code last and why the build failed. This aids in catching small problems such as forgetting to add a new file to the repository. A missing file is a very simple problem which can hold up other developers in a big way. Buildbot can also immediately catch larger issues such as tests failing due to new code changes. If you are new to buildbot there is a nice article by Grig Gheorghiu on setting up buildbot at continuous-integration-with-buildbot. If you are setting up buildslaves on multiple platforms he also has a nice article at running-buildbot-on-various-platforms.

Once you get your buildbot farm going you might want to run your Windows NT/XP/2003 buildslaves as windows services. Our WindowsXP buildslave serves multiple purposes. Different users log in and out of it on a regular basis. Running buildbot from the command line doesn’t work very well in this situation. The easiest solution is to run buildbot as a Windows Service. This allows buildbot to start running when Windows begins. Buildbot then runs in the background despite which users log in or out of the machine. Some small issues may arise when turning buildbot into a service so it is best to make sure buildbot is working from the command line first.

Here is an article about running a python script as a windows service running-python-script-as-windows

I created a buildslave user and got my slave working on the command line under that user’s account. I then created a service called BuildSlave and set it to run under the buildslave user. You can do this by going to Control Panel-> Administrative Tools->Services. Right click on the Buildslave service and select Properites. Go to the “Log On” tab and under “Log on as:” select “This account:” Now type in the buildslave username (I named mine buildslave) and password. The service will now run as the buildslave user. Some environmental variables might not get set. See the Problem Solving section below for a more detailed description.

There are different methods for launching the python script. The one mentioned in the article uses a batch file.

UPDATE: A comment by jdpipe mentions a better way to launch buildbot by using a python script instead of a windows batch file. Check out his article at http://ascendwiki.cheme.cmu.edu/BuildBot.

For buildbot you would make a batch file with the contents:

C:\Python24\python C:\buildbot\bin\buildbot start c:\buildslave\game

Make sure to substitute the correct paths for python, buildbot, and the buildslave directory. I named my batch file startslave.bat.

It should be mentioned that if the batch file method is used then stopping the service might not kill the python.exe process. You will have to kill it manually.

Problem Solving:

Buildslave stays in ACTIVE state:
A problem I had was when a buildslave would do a build and then never return to the IDLE state. On the next build the buildmaster would go through the buildslaves looking for IDLE ones but the buildslave would still be in the ACTIVE state and get skipped. Since it would never return to the IDLE state it would never do another build. I believe this problem was caused by not calling a superclass’s __init__ function from a subclass in my master.cfg file. If you have a configuration where you are subclassing a buildbot class(Such as subclassing step.ShellCommand to change the name of a build step) make sure you call the superclass’s __init__ function from the subclass’s __init__ function. If you don’t the effects could be small and unnoticeable at first but cause the buildslaves or buildmaster to behave in very strange ways because certain classes were not initialized properly.

Buildslave hangs or doesn’t start a build when running as a service:
When buildbot is started as a service the windows environment may be different even if you run the service as the same user you ran buildbot as from the command line. Certain environment variables which are set and valid when you run your slave from the command line might be invalid or not even exists when the slave is run as a service. Two big variables are HOMEPATH and HOMEDRIVE. If your buildbot setup requires some programs which rely on config files in the buildslave user’s home directory they might use those environment variables to find where those files reside. If the variables don’t exist those programs might use default options which cause them to just sit there and do nothing. When buildbot receives a command it dumps the environment just before running the command. A good idea is you run the buildslave from the command line and take a look at the environment. Then run the buildslave as a service and compare the two environments. You can find all logged messages in the twistd.log file which resides in your buildslave’s configuration directory. After comparing environments I found the service environment was missing both the HOMEPATH and HOMEDRIVE variables. My ssh setup required config files in those directories and was unable to find them. I then set those variables in my startslave.bat(which is launched by the BuildSlave service) prior to launching buildbot. It now looks like this:

set HOMEDRIVE = "C:"
set HOMEPATH = "\Documents and Settings\buildslave"
C:\Python24\python C:\buildbot\bin\buildbot start c:\buildslave\game