Cruise Notes - A readers guide to Cruise Control

26^th October 2008 Final Draft

From http://cruisecontrolrb.thoughtworks.com:

CruiseControl.rb is a continuous integration tool. Its basic purpose in life is to alert members of a software project when one of them checks something into source control that breaks the build.

Cruise control consists of a set of scripts for adding new projects and building existing ones, and a Dashboard application to display the results of builds

This guide describes cruise control 1.3.0

Dashboard Rails App

The Dashboard is a very simple Rails application, it has skinny controllers and well-factored views, and makes use of rails ajax capabilties, rails caching, builder and erb.

As well as controllers for Projects and Builds, cruise control also serves its own documentation from the webapp.

DocumentationController

The DocumentationController is a simple static controller it performs basic routing and caching

Documentation Routes : routes.rb

ActionController::Routing::Routes.draw do |map|

  ...

  map.plugin_doc_list 'documentation/plugins', :controller => 'documentation', :action => 'plugins'
  map.plugin_doc 'documentation/plugins/:type/:name', :controller => 'documentation', :action => 'plugins', :name => /[^\/]+/
  map.document 'documentation/*path', :controller => 'documentation', :action => 'get'

  ...

end

Documentation Controller : documentation_controller.rb

class DocumentationController < ApplicationController
  caches_page :get

  def get
    path = File.join('documentation', params[:path])

    if path == "documentation/plugin_repositories"
      render :template => path, :layout => false
    elsif template_exists?(path)
      render :template => path
    elsif template_exists?(path + '/index')
      render :template => path + '/index'
    else
      render :status => 404, :text => 'Documentation page not found'
    end
  end

  ...
  
end

ProjectsController

Action	Path	Model action	Views
index	/projects/	Projects.load_all	html js rss cctray
show	/projects/:id	Projects.find(:id)	html => redirect /builds/:project_id rss => projects/show_rss
code	/projects/:id/code/:path	Projects.find(:id)	html => redirect /builds/:project_id
build	/projects/:id/build	@project.request_build	js => redirect /builds/:project_id

Projects Routes : routes.rb

ActionController::Routing::Routes.draw do |map|
  ...
  map.resources :projects
  ...
  map.code 'projects/code/:project/*path', :controller => 'projects', :action => 'code'
  ...
  map.connect 'XmlStatusReport.aspx', :controller => 'projects', :action => 'index', :format => 'cctray'
  ...
end

BuildsController

Action	Path	Model action	Views
show	/builds/:project	@project.last_build	html
show	/builds/:project/:build	@project.find_build(:build)	html
drop_down	/builds/older/:project	Projects.find(:id)	html => drop_down.rhtml
artifact	builds/:project/:build/*path	@project.find_build(:build)	file => [build artifact]

Builds Routes : routes.rb

ActionController::Routing::Routes.draw do |map|
  ...
  map.builds_drop_down 'builds/older/:project', :controller => 'builds', :action => 'drop_down'
  map.project_without_builds 'builds/:project', :controller => 'builds', :action => 'show'
  map.build 'builds/:project/:build', :controller => 'builds', :action => 'show', :build => /[^\/]+/
  map.build_artifact 'builds/:project/:build/*path', :controller => 'builds', :action => 'artifact', :build => /[^\/]+/
  ...
end

The artifact action breaks the skinny controller rule, it may well be better to refactor this into a method on the builds model eg. Build::get_artifact(path) however this is complicated by the desire to handle directories differently.

Fat artifact method : builds_controller.rb

class BuildsController < ApplicationController
  ...
  
  def artifact
    render :text => 'Project not specified', :status => 404 and return unless params[:project]
    render :text => 'Build not specified', :status => 404 and return unless params[:build]
    render :text => 'Path not specified', :status => 404 and return unless params[:path]

    @project = Projects.find(params[:project])
    render :text => "Project #{params[:project].inspect} not found", :status => 404 and return unless @project
    @build = @project.find_build(params[:build])
    render :text => "Build #{params[:build].inspect} not found", :status => 404 and return unless @build

    path = File.join(@build.artifacts_directory, params[:path])

    if File.directory? path
      if File.exists?(path + '/index.html')
        redirect_to :path => File.join(params[:path], 'index.html')
      else
        # TODO: generate an index from directory contents
        render :text => "this should be an index of #{params[:path]}"
      end
    elsif File.exists? path
      send_file(path, :type => get_mime_type(path), :disposition => 'inline', :stream => false)
    else
      render_not_found
    end
  end
  
  ...
end

Views

The views are very straightforward idomatic rails views.

They use templates, and partials and helpers effectively to modularise the views and limit the amount of embedded ruby
RJS is used to provide asynchronous actions, which creates a better user experience
Builder is used to generate tidy and simple xml views

Partials used for modularisation : templates/default.rhtml

<html>

...

<body>

  <%= render :partial => '/layouts/header' %>

  <div id="content">
    <%= yield %>
  </div>

  <%= render :partial => '/layouts/footer' %>

</body>
</html>

Helpers used for cleaner erb : projects/_project.rhtml

<div id="project_<%= project.name %>" class="project build_<%= project.last_complete_build_status %>">
<table class="project_pane">
<tr>

  <td class="project_summary">

    <div>
      <span class="project_name">
        <%= link_to project.name, :action => 'show', :id => project.name %>
      </span>
    </div>

    <div class="recent_builds">

      <% recent_builds = project.last_five_builds %>

      <span class="lead">
        <%= recent_builds.first ? 'build ' : '<span class="build_status">Never built</span>' %>
        <br/>
        <%= link_to image_tag('rss_light.gif'), :action => 'show', :id => project.name, :format => 'rss' %>
      </span>

      <% unless recent_builds.empty? %>
        <div class="builds_list">
          <% recent_builds.each do |build| %>
            <% if build == recent_builds.first %>
                <div class="build_summary"><%= link_to_build_with_elapsed_time(project, build) %></div>
            <% else %>
                <div class="build_summary"><%= link_to_build(project, build) %></div>
            <% end %>
          <% end %>
        </div>
      <% end %>
    </div>
  </td>
  
  ...
</div>

Asynchronous requests via form remote tag : projects/_project.rhtml

<td class="builder_control">
    <div class="buttons">
      <% form_remote_tag(
             :url => { :action => 'build', :id => project },
             :before => "$('build_#{project.to_param}').disabled = true; " +
                        "$('build_#{project.to_param}').className = 'build_button_disabled'; " +
                        "Element.update('build_#{project.to_param}', 'Wait...')"
           ) do %>
        <button 
            <% if Configuration.disable_build_now -%>
              onclick="alert('Build Now button is disabled on this site.'); return false;"
            <% end -%>
              id="build_<%= project.to_param %>" type="submit" class="build_button" value="txt" alt="Build Now">
          <% if project.builder_state_and_activity == 'builder_down' %>
            Start Builder
          <% else %>
            Build Now
          <% end %>
        </button>
      <% end %>
    </div>
    <%= display_builder_state(project.builder_state_and_activity) %>
  </td>

RJS Response : projects/index_js.rjs

page.replace_html('projects', :partial => 'project', :collection => @projects) unless @projects.empty?
page.replace_html('projects', :partial => 'no_projects') if @projects.empty?

Builder used for xml output : projects/index_cctray.rxml

# List of projects and their state in XML - to be consumed by CruiseControl.NET tray.

xml.Projects do
  @projects.each do |project|
    xml.Project(
    :name => project.name,
    :category => "",
    :activity => map_to_cctray_activity(project.builder_state_and_activity),
    :lastBuildStatus => map_to_cctray_project_status(project.last_complete_build_status),
    :lastBuildLabel => (project.last_complete_build ? project.last_complete_build.label : 'Unknown'),
    :lastBuildTime => (project.last_complete_build ? format_time(project.last_complete_build.time, :round_trip_local) : '1970-01-01T00:00:00.000000-00:00'),
    :nextBuildTime => '1970-01-01T00:00:00.000000-00:00',
    :webUrl => url_for(:only_path => false, :controller => 'projects', :action => 'show', :id => project))
  end
end

Scripts and Daemon

The cruise command is a simple shell script that delegates to one of the server, add_project or builder scripts depending on the command line arguments.

server is just a simple wrapper around the rails builtin server script. It starts the dashboard rails app with mongrel.
add project creates a new project with options passed on the command line
builder builder runs a projects scheduler loop. There can be many builders running at once.

cruise executable script : cruise

#!/usr/bin/env ruby

load "#{File.dirname(__FILE__)}/lib/cruise_control/version.rb"

def start
  load "./script/server"
end

def add
  load "./script/add_project"
end

def builder
  load "./script/builder"
end

...

def method_for_command(command)
  case command
  when 'start'                            then :start
  when 'add'                              then :add
  when 'build', 'builder'                 then :builder
  when 'version', '-v', '--version'      then :version
  when 'help', '-h', '--help', '/?', '-?' then :help
  else nil
  end
end

command = ARGV.shift
if method_for_command(command)
  require 'fileutils'
  FileUtils.cd(File.dirname(__FILE__)) do 
    self.send(method_for_command(command))
  end
...
end

add_project script : scripts/add_project

...

source_control = Subversion.new(scm_options)
project = Project.new(project_name, source_control)
projects = Projects.load_all
projects << project

builder script : scripts/builder

...

project = load_project(project_path)

# this will create builder.pid file in project's CC directory and grab an exclusive lock on it, or else
# blow up saying that something else is already locking it
ProjectBlocker.block(project)

write_to_log_and_console "Builder for project '#{project.name}' started"
puts "Logging to: #{File.expand_path(CRUISE_OPTIONS[:log_file_name])}"

while (true) do
  catch(:reload_project) do
    project.scheduler.run
  end
  project = load_project(project_path)
  # this will cause the next call to scheduler to run the build immediately
  project.request_build rescue nil
end

The Model

The meat and gravy of the application is in the model (as it should be). There are three domain objects represented in the model - Project, Build and SCM (source control manager). The Project has many responsibilities including configuration, persistence and the build system. The project class delegates many of these reponsibilities to helper classes including BuilderStarter, BuilderStatus, PollingScheduler, ProjectConfigTracker, and BuildSerializer. Even with this refactoring the Project class is still over 500+ lines, further refactoring using delegates or mixins would definitely help the code readability.

Associations

Cruise control doesn't use Activerecord for persistence, instead it uses the filesystem. A project is stored as a directory in the [cruise data]/projects directory, the project has a configuration file, a working copy of the source code, and a directory for each of the builds.

There are finder methods for Projects and Builds- these query the filesystem in order to locate the desired resource.

Projects

find

Project

build builds create_build find_build last_build last_builds last_complete_build last_complete_build_status last_five_builds next_build previous_build

Filesystem used to store associations : project.rb

class Project
  ...
  
  def builds
    raise "Project #{name.inspect} has no path" unless path

    the_builds = Dir["#{path}/build-*"].collect do |build_dir|
      build_directory = File.basename(build_dir)
      build_label = build_directory.split("-")[1]
      Build.new(self, build_label)
    end
    order_by_label(the_builds)
  end
  
  ...
end

Initialization

Projects

load_all load_project

Project

configure read instantiate_plugins load_and_remember load_config load_timestamp

On initialization the project loads the central configuration file and the local configuration file. The configuration files are written in ruby and are expected to use Project.configure to set configuration attributes on the project. There is a clever trick here in setting a class variable to the current project on initialisation, this provides a means of accessing the current project in the configuration file

Initialisation method stores project reference : project.rb

class Project
  ...
  
  def self.read(dir, load_config = true)
    @project_in_the_works = Project.new(File.basename(dir))
    begin
      @project_in_the_works.load_config if load_config
      return @project_in_the_works
    ensure
      @project_in_the_works = nil
    end
  end
  
  def self.configure
    raise 'No project is currently being created' unless @project_in_the_works
    yield @project_in_the_works
  end
  
  ...
end

Example project configuration file : cruise_config.rb

Project.configure do |project|
  
  project.email_notifier.emails = ['email1@your.site', 'email2@your.site']
  project.email_notifier.from = 'john@doe.com'

  project.rake_task = 'custom'

  project.build_command = 'build_my_app.sh'

  project.scheduler.polling_interval = 5.minutes

end

The Build System

Project

build_command build_command= build_if_necessary build_if_requested build_necessary? build_requested? build_requested_flag_file build_without_serialization builder_error_message builder_state_and_activity error_message log_changeset request_build save_timestamp scheduler scheduler=

Build

run

BuilderStarter

begin_builder path_to_cruise run_builders_at_startup= start_builders

PollingScheduler

run

BuildSerializer

serialize serialize timeout wait

The controller can start a build by calling request_build on a project. This method delegates starting the build to the BuilderStarter class. The build is started asynchronously by creating a new process which executes the builder script. The build request is stored in a flag file so that the build process is aware of the request.

Asynchronous build request : build_starter.rb

class BuilderStarter
  ...  
  
  def self.begin_builder(project_name)
    cruise_executable =
        if Platform.interpreter =~ /jruby/
          Platform.interpreter + ' ' + path_to_cruise
        elsif Platform.family == 'mswin32'
          "ruby #{path_to_cruise}"
        else
          path_to_cruise
        end

    verbose_option = $VERBOSE_MODE ? " --trace" : ""
    command = "#{cruise_executable} build #{project_name}#{verbose_option}"

    Platform.create_child_process(project_name, command)
  end
end

The builder script runs the project's scheduler loop, which calls build_if_necessary or build_if_requested to start a build. The build is started with the build method, which in turn delegates to build_without_serialization (serialization is performed by a BuildSerializer if required).

Scheduler Loop : polling_scheduler.rb

class PollingScheduler
  ...

  def run
    while (true) do
     begin
        @project.build_if_necessary or check_build_request_until_next_polling
        clean_last_build_loop_error
        throw :reload_project if @project.config_modified?
      rescue => e
        log_error(e) unless (same_error_as_before(e) and last_logged_less_than_an_hour_ago)
        sleep(Configuration.sleep_after_build_loop_error)
      end
    end
  end
  
  def check_build_request_until_next_polling
    time_to_go = Time.now + polling_interval
    while Time.now < time_to_go
      @project.build_if_requested
      sleep build_request_checking_interval
    end
  end

  ...  
end

A Build object is created which controls the build and provides access to and persistence for results. The run method creates the appropriate environment and runs the project build command or rake task.

Build's run method : build.rb

class Build
  ...
    
  def run
    build_log = artifact 'build.log'
    File.open(artifact('cruise_config.rb'), 'w') {|f| f << @project.config_file_content }

    begin
      raise ConfigError.new(@project.error_message) unless @project.config_valid?
      in_clean_environment_on_local_copy do
        execute self.command, :stdout => build_log, :stderr => build_log
      end
      build_status.succeed!(seconds_since(@start))
    rescue => e
      if File.exists?(project.local_checkout + "/trunk")
        msg = <<EOF

WARNING:
Directory #{project.local_checkout}/trunk exists.
Maybe that's your APP_ROOT directory.
Try to remove this project, then re-add it with correct APP_ROOT, e.g.

rm -rf #{project.path}
./cruise add #{project.name} svn://my.svn.com/#{project.name}/trunk
EOF
        File.open(build_log, 'a'){|f| f << msg }
      end

      File.open(build_log, 'a'){|f| f << e.message }
      CruiseControl::Log.verbose? ? CruiseControl::Log.debug(e) : CruiseControl::Log.info(e.message)
      if e.is_a?(CommandLine::ExecutionError) # i.e., the build returned a non-zero status code
        fail!
      else
        fail!(e.message)
      end
    end
  end
  
  ...
end

Triggers

Project

build_necessary? triggered_by triggered_by=

ChangeInSourceControlTrigger

build_necessary?

SuccessfulBuildTrigger

build_necessary?

build_necessary? delegates to the projects triggers. Triggers are simple objects which respond to build_necessary?. Cruise control ships with two triggers ChangeInSourceControlTrigger and SuccessfulBuildTrigger.

build_necessary? method : project.rb

class Project
  ...
  
  def build_necessary?(reasons)
    if builds.empty?
      reasons << "This is the first build"
      true
    else 
      @triggers.any? {|t| t.build_necessary?(reasons) }
    end
  end
  
  ...
end

ChangeInSourceControlTrigger : change_in_source_control_trigger.rb

class ChangeInSourceControlTrigger
  def initialize(triggered_project)
    @triggered_project = triggered_project
  end

  def build_necessary?(reasons)
    p = @triggered_project
    p.notify :polling_source_control
    
    if !p.source_control.up_to_date?(reasons)
      p.notify :new_revisions_detected, reasons.flatten.find_all{|reason| reason.is_a? Revision}
      return true
    else
      p.notify :no_new_revisions_detected
      return false
    end
  end
end

Plugins

Project

plugin add_plugin method_missing notify plugins respond_to?

BuildReaper

build_finished

EmailNotifier

build_finished build_fixed

BuilderStatus

build_finished build_initiated build_loop_failed build_requested polling_source_control queued sleeping timed_out

ProjectLogger

build_finished build_loop_failed build_started new_revisions_detected no_new_revisions_detected polling_source_control sleeping

Each project has a set of plugins which are notified of events in the build sequence. A plugin can respond to these events by defining a method of the same name as the event. The set of possible events is dynamic and events are despatched from several places including the project build methods, and the triggers. Some of the events include

configuration_modified
build_initiated
build_started
build_finished
build_broken
build_fixed
build_requested
build_loop_failed
sleeping

notify method : project.rb

class Project
  ...
  
  def notify(event, *event_parameters)
    errors = []
    results = @plugins.collect do |plugin| 
      begin
        plugin.send(event, *event_parameters) if plugin.respond_to?(event)
      rescue => plugin_error
        CruiseControl::Log.error(plugin_error)
        if (event_parameters.first and event_parameters.first.respond_to? :artifacts_directory)
          plugin_errors_log = File.join(event_parameters.first.artifacts_directory, 'plugin_errors.log')
          begin
            File.open(plugin_errors_log, 'a') do |f|
              f << "#{plugin_error.message} at #{plugin_error.backtrace.first}"
            end
          rescue => e
            CruiseControl::Log.error(e)
          end
        end
        errors << "#{plugin.class}: #{plugin_error.message}"
      end
    end
    
    if errors.empty?
      return results.compact
    else
      if errors.size == 1
        error_message = "Error in plugin #{errors.first}"
      else
        error_message = "Errors in plugins:\n" + errors.map { |e| "  #{e}" }.join("\n")
      end
      raise error_message
    end
  end
  
  ...
end

E-mail notifier plugin : email_notifier.rb

class EmailNotifier
  attr_accessor :emails
  attr_writer :from
  
  def initialize(project = nil)
    @emails = []
  end

  def from
    @from || Configuration.email_from
  end

  def build_finished(build)
    return if @emails.empty? or not build.failed?
    email :deliver_build_report, build, "#{build.project.name} build #{build.label} failed", "The build failed."
  end

  def build_fixed(build, previous_build)
    return if @emails.empty?
    email :deliver_build_report, build, "#{build.project.name} build #{build.label} fixed", "The build has been fixed."
  end
  
  private
  
  def email(template, build, *args)
    BuildMailer.send(template, build, @emails, from, *args)
    CruiseControl::Log.event("Sent e-mail to #{@emails.size == 1 ? "1 person" : "#{@emails.size} people"}", :debug)
  rescue => e
    settings = ActionMailer::Base.smtp_settings.map { |k,v| "  #{k.inspect} = #{v.inspect}" }.join("\n")
    CruiseControl::Log.event("Error sending e-mail - current server settings are :\n#{settings}", :error)
    raise
  end

end

Project.plugin :email_notifier

Source Control

Project

source_control update_project_to_revision

Subversion

check_externals checkout clean_checkout latest_revision up_to_date? update

The build system interacts with the source control using three methods latest_revision(project), revisions_since(project, revision_number) and update(project, revision = nil). The add project script also requires a checkout(revision = nil) method.

Cruise Control currently ships with only a Subversion source control manager, but git and mercurial support are in the works and are available from the projects git repository.

update_to_latest_revision method : project.rb

class Project
  ...
  def update_project_to_revision(build, revision)
    if do_clean_checkout?
      File.open(build.artifact('source_control.log'), 'w') do |f| 
        start = Time.now
        f << "checking out build #{build.label}, this could take a while...\n"
        @source_control.clean_checkout(revision, f)
        f << "\ntook #{Time.now - start} seconds"
      end
    else
      @source_control.update(revision)
    end
  end
  
  ...
end

Subversion client : subversion.rb

class Subversion
end

require 'builder_error'
require 'subversion/changeset_log_parser'
require 'subversion/info_parser'
require 'subversion/log_parser'
require 'subversion/propget_parser'
require 'subversion/update_parser'

class Subversion
  include CommandLine

  attr_accessor :url, :path, :username, :password, :check_externals

  def initialize(options = {})
    @url = options.delete(:url)
    @path = options.delete(:path) || "."
    @username = options.delete(:username)
    @password = options.delete(:password)
    @interactive = options.delete(:interactive)
    @error_log = options.delete(:error_log)
    @check_externals = options.has_key?(:check_externals) ? options.delete(:check_externals) : true
    
    raise "don't know how to handle '#{options.keys.first}'" if options.length > 0
  end
  
  def error_log
    @error_log ? @error_log : File.join(@path, "..", "svn.err")
  end
  
  def clean_checkout(revision = nil, stdout = $stdout)
    FileUtils.rm_rf(path)
    checkout(revision, stdout)
  end

  def checkout(revision = nil, stdout = $stdout)
    @url or raise 'URL not specified'

    options = [@url, path]
    options << "--username" << @username if @username
    options << "--password" << @password if @password
    options << "--revision" << revision_number(revision) if revision

    # need to read from command output, because otherwise tests break
    svn('co', options) do |io| 
      begin
        while line = io.gets
          stdout.puts line
        end
      rescue EOFError
      end
    end
  end

  def last_locally_known_revision
    return Revision.new(0) unless File.exist?(path)
    Revision.new(info.revision)
  end

  def latest_revision
    svn_output = log "HEAD", "1", ['--limit', '1']
    Subversion::LogParser.new.parse(svn_output).first
  end
  
  def up_to_date?(reasons = [], revision_number = last_locally_known_revision.number)
    result = true
    
    latest_revision = self.latest_revision()
    if latest_revision > Revision.new(revision_number)
      reasons << "New revision #{latest_revision.number} detected"
      reasons << revisions_since(revision_number)
      result = false
    end
    
    if @check_externals
      externals.each do |ext_path, ext_url|
        ext_logger = ExternalReasons.new(ext_path, reasons)
        ext_svn = Subversion.new(:path => File.join(self.path, ext_path), 
                                 :url => ext_url, 
                                 :check_externals => false)
        result = false unless ext_svn.up_to_date?(ext_logger)
      end
    end
    
    return result
  end

  def update(revision = nil)
    revision_number = revision ? revision_number(revision) : 'HEAD'
    svn_output = svn('update', ["--revision", revision_number])
    Subversion::UpdateParser.new.parse(svn_output)
  end

  def externals
    return {} unless File.exist?(path)
    
    svn_output = svn('propget', ['-R', 'svn:externals'])
    Subversion::PropgetParser.new.parse(svn_output)
  end
  
  private
  
  def revisions_since(revision_number)
    svn_output = log('HEAD', revision_number)
    log_parser = Subversion::LogParser.new
    revisions = log_parser.parse(svn_output)
    revisions.reject {|revision| revision.number == revision_number} # cut out the revision that was asked for
  end

  def log(from, to, arguments = [])
    svn('log', arguments + ["--revision", "#{from}:#{to}", '--verbose', '--xml', url], :execute_locally => url.blank?)
  end
  
  def info
    svn_output = svn('info', ["--xml"])    
    Subversion::InfoParser.new.parse(svn_output)
  end

  def svn(operation, arguments, options = {}, &block)
    command = ["svn"]
    command << "--non-interactive" unless @interactive
    command << operation
    command += arguments.compact
    command
    
    execute_in_local_copy(command, options, &block)
  end

  def revision_number(revision)
    revision.respond_to?(:number) ? revision.number : revision.to_i
  end

  def execute_in_local_copy(command, options, &block)
    if block_given?
      execute(command, &block)
    else
      error_log = File.expand_path(self.error_log)
      if options[:execute_locally] != false
        Dir.chdir(path) do
          execute_with_error_log(command, error_log)
        end
      else
        execute_with_error_log(command, error_log)
      end
    end
  end
  
  def execute_with_error_log(command, error_log)
    FileUtils.rm_f(error_log)
    FileUtils.touch(error_log)
    execute(command, :stderr => error_log) do |io| 
      result = io.readlines 
      begin 
        error_message = File.open(error_log){|f|f.read}.strip.split("\n")[1] || ""
      rescue
        error_message = ""
      ensure
        FileUtils.rm_f(error_log)
      end
      raise BuilderError.new(error_message, "svn_error") unless error_message.empty?
      return result
    end
  end
  
  Info = Struct.new :revision, :last_changed_revision, :last_changed_author
  
  class ExternalReasons < Struct.new :external, :reasons
    def <<(reason)
      if reason.is_a? String
        reasons << "#{reason} in external '#{external}'"
      else
        reasons << reason
      end
      self
    end
  end
end