Tony's Blog

During a Buildbot build step you might need to have a list of files which have changed as a result of a commit to your repository. A quick way to get this information during a build step is to subclass the build step class of your choice and overload its start() function.

A build step’s start() function is responsible for telling the slave to start executing the command. Each step class has a member variable called “build” of type “buildbot.process.base.Build” which it inherits from its main parent class “buildbot.process.buildstep.BuildStep”. Here is a quick example of getting file change information during a ShellCommand step in your master.cfg file:

from twisted.python import log
from buildbot.steps import shell
from buildbot.process.buildstep import RemoteShellCommand
 
# Executes a remote command with changed files appended onto the end
class ShellCommandChangeList(shell.ShellCommand):
  def start(self):
    # Substitute build properties into command
    command = self._interpolateProperties(self.command)
    # fail assert if command is not of correct type
    assert isinstance(command, (list, tuple, str))
 
    # Get changed file list from the build which invoked this step
    files = self.build.allFiles()
    # Log the changed files to twistd.log
    log.msg("Build Files: %s" % files)
    # Now we can do whatever we want with the list of changed files.
    # I will just append them to the end of the command.
 
    # Convert file list so it can be appended to the command's type
    if isinstance(command, str):
        files = " ".join(files)
    elif isinstance(command, tuple):
        files = tuple(files)
 
    # append files to end of command
    command += files
 
    # We have created the final command string
    # so we can fill out the arguments for a RemoteShellCommand
    # using our new command string
    kwargs = self.remote_kwargs
    kwargs['command'] = command
    kwargs['logfiles'] = self.logfiles
 
    # Create the RemoteShellCommand and start it
    cmd = RemoteShellCommand(**kwargs)
    self.setupEnvironment(cmd)
    self.checkForOldSlaveAndLogfiles()
    self.startCommand(cmd)

You can use the new ShellCommandChangeList class in the same way you use the ShellCommand class. The only difference being the ShellCommandChangeList class will append a list of changed files to the end of the step’s shell command. We use a similar class and method at Stolen Notebook to build large source asset files (models, levels, art, etc.) automatically with buildbot when they are checked into our repository.

Here is a short contrived example of using the new ShellCommandChangeList build step with a BuildFactory to call a remote command for a fictional program named `checksum`. Let’s say that `checksum` is a program which takes filenames as input, computes the checksum of those files and prints those checksums to standard out.

from buildbot.steps import source, shell
from buildbot.process import factory
 
f = factory.BuildFactory()
f.addStep(source.SVN, svnurl="http://svn.example.org/Trunk/")
f.addStep(ShellCommandChangeList, command=["checksum"])

Lets say two files named main.cpp and main.h are changed in the src directory of our svnurl and submitted to the repository which triggers this build. The normal ShellCommand step would just call “checksum” remotely. The ShellCommandChangeList step on the other hand would call “checksum src/main.cpp src/main.h”

You should note that the way your VC system works and how your repository is set up in terms of file location and branches may affect how the file change paths are presented to you. You can easily see the list of file changes by using log.msg in your start() function to print them to the twistd.log file.