Synthesized testing is about accurately simulating object interactions and verifying that each end point of every interaction has been tested to work. The end result of a code base tested employing this strategy forms a specification of the application’s ecosystem in terms of object communication.

Danilo has been recently contributing some excellent work around visual representations of the above. The code is being developed on the Synthesis experimental branch on github.

Consider the Synthesis test_project example.

class DataBrander
  BRAND = "METAL"

  def initialize(storage)
    @storage = storage
  end

  def save_branded(data)
    @storage.save "#{BRAND} - #{data}"
  end

  def dont_do_this
    @storage.ouch!
  end
end

class Storage
  def initialize(filename)
    @filename = filename
  end

  def save(val)
    File.open(@filename, 'w') {|f| f < val}
  end

  def ouch!
    raise Problem
  end
end

class Problem < Exception;end

Below are the complete specs for the above implementation.

describe DataBrander do
  it "should save branded to storage" do
    storage = Storage.new("")
    storage.should_receive(:save).with("METAL - rock")
    DataBrander.new(storage).save_branded("rock")
  end

  it "should delegate problem" do
    storage = Storage.new("")
    storage.should_receive(:ouch!).and_raise(Problem.new)
    proc {DataBrander.new(storage).dont_do_this}.should raise_error(Problem)
  end
end

describe Storage do
  it "should save to file" do
    begin
      Storage.new("test.txt").save("rock")
      File.read("test.txt").should == "rock"
    ensure
      FileUtils.rm_f("test.txt")
    end
  end

  it "should raise problem on ouch!" do
    proc { Storage.new("").ouch! }.should raise_error(Problem)
  end
end

A Synthesis run using the DOT formatter produces:

Removing the "should save to file" spec will cause the Synthesis task to fail.

Below is how a real (relatively small) project looks like.

I find the ability to inspect our application modeling through such a representation a very appealing added benefit to the confidence in our system Synthesis provides us with. The DOT formatter will become part of the Synthesis gem as soon as we iron out the few remaining glitches.

Bazaar is a distributed version control system written in Python, similar to Git. Bazaar places particular focus on usability, it is easy and natural to use, especially for ones visiting or migrating from the world of Git.

One of Bazaar’s striking features is the ability to publish branches with sftp, provided there is a web server available. RubyForge project accounts come with support for both, so publishing a Bazaar branch is as easy as:

bzr push --create-prefix sftp://you@rubyforge.org/var/www/gforge-projects/your-project/bzr

Developers can create their copy of the branch by:

bzr branch http://your-project.rubyforge.org/bzr

Ruby’s Object#send method offers an elegant alternative for invoking methods based on a command translating a symbol to a function dispatch. I was looking for similar functionality in Erlang and here’s what I came up with.

First, let’s see how we can achieve eval functionality in Erlang, i.e. evaluate strings as Erlang code at runtime.

-module (meta).
-export ([eval/2]).

eval(Code, Args) ->
  {ok, Scanned, _} = erl_scan:string(Code),
  {ok, Parsed} = erl_parse:parse_exprs(Scanned),
  Bindings = lists:foldl(fun ({Key, Val}, BindingsAccumulator) ->
    erl_eval:add_binding(Key, Val, BindingsAccumulator)
  end, erl_eval:new_bindings(), Args),
  {value, Result, _} = erl_eval:exprs(Parsed, Bindings),
  Result.

erl_scan is Erlang’s token scanner module. The string function tokenizes a list of characters. erl_parse is the Erlang parser module and the parse_exprs function parses a list of tokens, each Token representing an expression. It returns a list of the abstract forms of the parsed expressions, ready to be used with erl_eval, the Erlang meta interpreter. An arbitrary list of bindings can be provided alongside the parsed expressions to erl_eval:exprs.

With the meta:eval function in place, we can evaluate arbitrary strings of code at runtime.

Eshell V5.6  (abort with ^G)
1> c(meta).
{ok,meta}
2> meta:eval("20 + 30.", []).
50
3> meta:eval("A + B.", [{'A', 15}, {'B', 60}]).
75

We can build on meta:eval to achieve Ruby-like dynamic dispatches.

send(MethodName, Args) ->
  ArgNames = lists:foldl(fun ({K, _}, Acc) -> lists:append([K], Acc) end, [], Args),
  Code = atom_to_list(MethodName) ++ "(" ++ atom_join(ArgNames, $,) ++ ").",
  eval(Code, Args).

The send function takes two arguments, an atom which is the name of the function to be dispatched and a list of tuples with key/value pairs for the arguments to be passed to the function. atom_join joins a list of atoms into one string using the supplied separator.

atom_join([], _Sep) -> [];
atom_join(Items, Sep) -> lists:flatten(atom_join1(Items, Sep, [])).
atom_join1([Head | []], _Sep, Acc) -> [atom_to_list(Head) | Acc];
atom_join1([Head | Tail], Sep, Acc) -> atom_join1(Tail, Sep, [Sep, atom_to_list(Head) | Acc]).

Let’s add a couple of test functions to showcase what has been achieved.

hello() -> "hello, world".
hello(Who) -> "hello, " ++ Who.

Back to Eshell…

Eshell V5.6  (abort with ^G)
1> c(meta).
{ok,meta}
2> meta:send('meta:hello', []).
"hello, world"
3> meta:send('meta:hello', [{'Name', "rock"}]).
"hello, rock"

A big thank you to Chris Barrett who has been taking the time to port Synthesis to Java.

JSynthesis is registered as a GoogleCode project and it will surely be integral to my toolkit next time I work on a Java project.

With testing occupying a major part of our development process, we have often attempted to quantify test code quality. Like many things, it is worth considering how test code ultimately manifests itself in terms of added value. This is why Stuart and I recently tend to conclude that, stripped from technically granular details, test code must fundamentally contribute in building confidence that the system under test is complete, a proof that what we’ve built is and will continue working as intended.

A working system fulfilling its business objectives can be considered complete enough, but, if not easily extensible and maintainable, will not grant itself to the conclusion of being as good as it can be. Advancements in software development methodologies that assist in delivering working software that is easy to extend and maintain - higher level abstractions, modeling and design - have been driven by the need to reduce technical debt. Technical debt can be viewed as the cost of change.

Test code is code, too. As code bases grow more elaborate, test code also suffers from technical debt, demanding methods to eliminate the factors that hinder its maintainability and extensibility. Present procedures geared towards extensible and maintainable test code are habitually counter-proportional to the amount of confidence they achieve.

The different categories on the scale are not mutually exclusive, in fact they are commonly combined as members of a suite that exercises the system in various degrees of instrumentation. Walking the scale from left (empty) to right (full), we move from tests that are generally easier to write, understand, run and maintain but at the same time are less representative of the real system with all its components integrated.

Dependency neutral tests with all of the tested component’s dependencies stubbed are disconnected, vaguely describe how the component interacts with its environment and offer minimal proof that the component will work as specified once a member of the application ecosystem.

The fundamental difference between interaction based dependency neutral tests and their stubbed counterparts is the accurate interaction specification of collaborating components through the use of mock objects instead of stubs. Here, we concentrate on specifying the contract of communication between two components. Although much closer to how the actual system operates, these tests are still disconnected. Despite the accurate specification of the interaction, we have don’t have complete proof that the pieces fit. In particular, interaction based dependency neutral tests do not offer proof that the mocked collaborators have been tested to work.

It becomes apparent that the major flaw of interaction based dependency neutral tests is their disconnect from their peers.

As we move towards the “full” side of the confidence scale, tests tend to become larger and more complicated. Wired tests draw a picture much closer to that of the system in its entire form but suffer from poor defect localization (test failures are not always directly related to the intent of the specific test) and disrespect encapsulation (setup code often exposes the behavior of components irrelevant to the context of the current test). The dependency wired tests’ contribution to technical debt is much more significant.

Understanding the importance of confidence in our system and aiming to reduce technical debt, Synthesized Testing suggests a solution that attempts to rectify the disconnect of lightweight, interaction based dependency neutral tests and reduce the need of overarching, prone to technical debt dependency wired tests.

When using RubyGems from the command line, I almost always type sudo gem i synthesis as opposed to sudo gem install rails, the emphasis targeted at using “i” instead of “install”, of course. The gem executable happily understands what command it is being asked to execute when provided with the first few letters of the command, as long as those letters are not ambiguous, i.e. don’t clash with the names of other commands. So even though sudo gem u foo complains that Ambiguous command u matches [uninstall, unpack, update], sudo gem uni foo will uninstall the specified gem.

Here’s how this is implemented in RubyGems.

def find_command(cmd_name)
  possibilities = find_command_possibilities(cmd_name)
  if possibilities.size > 1
    raise "Ambiguous command #{cmd_name} matches [#{possibilities.join(', ')}]"
  end
  if possibilities.size < 1
    raise "Unknown command #{cmd_name}"
  end

  self[possibilities.first]
end

def find_command_possibilities(cmd_name)
  len = cmd_name.length
  self.command_names.select { |n| cmd_name == n[0,len] }
end

In the same vein, although not strictly a command abbreviation, Danilo pointed out git understands abbreviated revision hashes, so it’s possible to use something like git diff d0a..HEAD even with the hash’s complete representation being d0aa7dd4aa9a95090df1e0b9d0f426d5a5bd56ae.

Less typing is almost always a good option to have. The easy to implement Unambiguous command abbreviation trick adds a subtle usability improvement to command line interfaces and holds a nice treat to the utility’s power users.

Jabber and its underlying protocol XMPP are typically associated with instant messaging applications, although the breadth and flexibility of the technology allows for implementations that can span further from traditional online chatting.

ejabberd is a fault tolerant and clusterable Jabber/XMPP server written in Erlang and presents an interesting option as a simple, lightweight and scalable message transport for distributed applications.

EventMachine is a simple and fast library for lightweight concurrency in Ruby. Its use mainly involves, but is not limited to, spawning lightweight processes whose execution can be programatically scheduled, easy and fast socket abstractions and an implementation of the Deferrable pattern as introduced by the Twisted event-driven Python networking engine.

When a Ruby class includes the EventMachine::Deferrable module, it is provided with the ability to accept arbitrary callbacks and errbacks that will get executed when its deferred status changes, in particular when it is set to either :succeeded or :failed. Let’s look at a deferrable Worker class which performs a potentially long running operation.

class Worker
  include EM::Deferrable

  def heavy_lifting
    30.times do |i|
      puts "Lifted #{i}"
      sleep 0.1
    end
    set_deferred_status :succeeded
  end
end

Inside an EventMachine loop, we can add callbacks to a Worker instance and dispatch the expensive operation to a separate thread, or an evented process. The program’s execution will continue, with any callbacks attached to Worker executed once its deferred status is set.

EM.run do
  worker = Worker.new
  worker.callback {p "done!"}
  Thread.new {worker.heavy_lifting; EM.stop}
  puts "resuming remaining program operations"
end

Now, let’s look at combining Worker with Jabber to trigger long running jobs. For Jabber server duties, I am using ejabberd on an old laptop running Debian, but there’s no reason why a mass online Jabber service like Google Talk could not be used for playing around with the example. Also, I’m using the xmpp4r-simple Ruby library, which is a wrapper around xmpp4r.

jabber = Jabber::Simple.new("bot@thrash", "password")
at_exit{jabber.status(:away, "jabot down")}

EM.run do
  EM::PeriodicTimer.new(1) do
    jabber.received_messages do |message|
      case message.body
      when "exit" : EM.stop
      when "lift" :
        EM.spawn do
          worker = Worker.new
          worker.callback {jabber.deliver(message.from, "Done lifting")}
          worker.heavy_lifting
        end.notify
        jabber.deliver(message.from, "Scheduled heavy job...")
      else jabber.deliver(message.from, "Dunno how to #{message.body}")
      end
    end
  end
end

Inside an EventMachine loop, we check for new messages every second. The program understands two commands, exit and lift. The first quits the EventMachine loop and ultimately terminates the program’s execution. When lift is received, we instantiate a new Worker inside a spawned process and add a callback so that the Worker will notify the command issuer when the job has completed. Worth noting is the use of notify to schedule the spawned process. notify returns immediately making work dispatch non-blocking - upon issuing a lift command twice, a “Scheduled heavy job…” message will be sent to the job issuer twice before the first job completes.

I use Adium to send commands to the program - an interesting way of remote controlling or interacting with applications. Of course, the real interest lies in using the setup under discussion for inter-app communication. With multicast options, presence discovery, node status updates and more, there is lot to explore in terms of distributed application development, if simple and lightweight are two keywords to be found on the highest ranks of your list.

The common pattern for mocking class instances in Ruby involves setting an expectation on the new method of the class to return an object of the specified type. Consider the following User and Account classes.

class User
  attr_reader :name, :email

  def initialize(name, email)
    @name, @email = name, email
  end

  def contact
    send_email(@email)
  end
end

class Account
  def self.create(name, email)
    user = User.new(name, email)
    user.contact
  end
end

The test for the Account#create method would look like this:

require "rubygems"
require "mocha"
require "test/unit"

class AccountTest < Test::Unit::TestCase
  def test_emails_user_on_account_creation
    user = User.new("", "")
    user.expects(:contact)
    User.expects(:new).with("Kirk", "kirk@metallica.com").returns(user)
    Account.create("Kirk", "kirk@metallica.com")
  end
end

As an alternative, and this is the version I see most often used, a duck typed mock can be used in the place of the “real” User object.

class AccountTest < Test::Unit::TestCase
  def test_emails_user_on_account_creation
    user = mock
    user.expects(:contact)
    User.expects(:new).with("Kirk", "kirk@metallica.com").returns(user)
    Account.create("Kirk", "kirk@metallica.com")
  end
end

Both implementations have some potentially undesirable issues worth taking into consideration. None of the two is DRY, since in both cases we need to set a bespoke expectation for the object’s instantiation. In the first example, we are actually instantiating a “real” User. This compromises the dependency neutrality of the test, coupling it with User’s initialization implementation. The second example, despite its relative convenience, doesn’t fully respect the contract between Account and User. The interaction specification is inaccurate, because, as far as the application code under test is concerned, User#new doesn’t return an object of type Mocha::Mock, it returns an instance of User.

The following code can be useful in an effort to DRY up code as the one under discussion, whilst driving such interaction specifications closer to the actual contracts they are meant to express.

class Object
  def self.mock_instance(*args)
    class_eval do
      alias original_initialize initialize
      def initialize()end
    end

    instance = new
    expects(:new).with(*args).returns(instance)

    class_eval do
      alias initialize original_initialize
      undef original_initialize
    end

    return instance
  end
end

Here is how AccountTest can be written with the above Mocha extension in place.

class AccountTest < Test::Unit::TestCase
  def test_emails_user_on_account_creation
    User.mock_instance("Kirk", "kirk@metallica.com").expects(:contact)
    Account.create("Kirk", "kirk@metallica.com")
  end
end

If RSpec is your flavor, the same can be achieved with minor modifications in the Spec::Mocks::Methods module.

module Spec::Mocks::Methods
  def mock_instance(*args)
    class_eval do
      alias original_initialize initialize
      def initialize()end
    end

    instance = new
    should_receive(:new).with(*args).and_return(instance)

    class_eval do
      alias initialize original_initialize
      undef original_initialize
    end

    return instance
  end
end

With the above we can use mock_instance in specs as such.

describe Account do
  it "should email user on account creation" do
    User.mock_instance("Kirk", "kirk@metallica.com").should_receive(:contact)
    Account.create("Kirk", "kirk@metallica.com")
  end
end

During projects I’ve worked on which involved coding JavaScript, I’ve had positive experiences enjoying the event driven nature of the language and some bad, especially after code bases grew larger with events firing left and right, producing side effects which were difficult to manage or track down. During the bad days, a centralized way of managing events was often brought up in conversations with colleagues as a means of managing the issue.

I’ve long been a fan of the event driven style of programming as a particularly useful alternative promoting loose coupling of system components. Below is the code for a Ruby module which, when included, enables classes to act as event dispatchers. Other classes can subscribe to the events and be notified when an event occurs.

module EventDispatcher
  module ClassMethods
    attr_reader :listeners

    def subscribe(event, &callback)
      @listeners ||= {}
      (@listeners[event] || @listeners[event] = []) << callback
    end

    private

    def clear_listeners!
      @listeners = {}
    end
  end

  def self.included(receiver)
    receiver.extend(ClassMethods)
  end

  def notify(event, *args)
    self.class.listeners[event].each {|callback| callback[*args]}
  end
end

Let’s consider an example application where the resource management department of an organization hires recruits and upon admission notifies interested services to take relevant action.

class RM
  include EventDispatcher

  def hire(name)
    notify(:new_recruit, name)
  end
end

What is especially interesting here is how the RM class doesn’t need to know anything about any interested listeners. In order to test RM, it suffices to ensure that a notification is sent upon calling the hire method.

def test_rm_notifies_on_new_hire
  service = mock
  service.expects(:new_recruit).with("name")
  RM.subscribe(:new_recruit) {|name| service.new_recruit(name)}
  RM.new.hire("name")
end

Typically, listeners would directly register their interest to the event by subscribing to it. Let’s imagine a welcoming letter is issued to new recruits the moment they join the organization.

class WelcomeLetterService
  RM.subscribe(:new_recruit) {|name| greet(name)}

  def self.greet(name)
    "Welcome, #{name}!"
  end
end

The first thing to notice is WelcomeLetterService’s direct coupling to RM. Additionally, a code base heavily employing this strategy might suffer ill effects similar to the ones described in the JavaScript inspired first paragraph of this article. Allowing a centralized option for event registration and management could serve as possible remedy to the issue.

module EventRegistry
  def register_event_listeners
    RM.subscribe(:new_recruit) {|name| WelcomeLetterService.greet(name)}
  end

  extend self
end

The EventRegistry module acts as a centralized mechanism for declaring which listeners subscribe to which events. On top of that, the WelcomeLetterService is now completely oblivious to the RM class.

class WelcomeLetterService
  def self.greet(name)
    "Welcome, #{name}!"
  end
end

Due to the level of decoupling achieved, testing these two components becomes particularly easy.

def test_welcome_letter_service_greets_by_name
  assert_equal(WelcomeLetterService.greet("Name"), "Welcome, Name!")
end

def test_event_subscription_wiring
  WelcomeLetterService.expects(:greet).with("name")
  RM.expects(:subscribe).with(:new_recruit).yields("name")
  EventRegistry.register_event_listeners
end

All this ties well with the philosophy behind Synthesized Testing, where a coherent collection of lightweight tests becomes a major factor of confidence that the system under test is complete, reducing the need for complex overarching tests.

Sinatra doesn’t yet seem to come with built in support for Erubis. If you want the relative speed bump of Erubis over standard ERB for your Sinatra application, the code below should do the trick.

module Sinatra::Erb
  def erb(content, options={})
    begin
      require 'erubis'
      @@erb_class = Erubis::Eruby
    rescue LoadError
      require "erb"
      @@erb_class = ::ERB
    end
    render(:erb, content, options)
  end

  private

  def render_erb(content, options = {})
    @@erb_class.new(content).result(binding)
  end
end

Application code will not require any modifications, Sinatra will use Erubis if the gem is found in the load path.