Isolated subclass specs in Ruby

2012-07-01

Unit tests should isolate their subject, but tests of subclass functionality usually test the class as a whole. One reason for this is that in Ruby there is no way to dynamically change the superclass, but more importantly, it’s because you don’t want to modify the thing you’re testing. Sometimes however, isolating a subclass from its parent makes sense.

Schemabot, our tool for applying database changes in a replication friendly way, is based on the Sequel gem and its migration DSL. We added a preview function by subclassing the MySQL adapter and having it ignore any SQL that would change the database.

A simplified version of our implementation (allowing just SELECT and DESCRIBE queries) looks like this:

module Sequel
  module MySQLPreview
    class Database < Sequel::MySQL::Database

      def execute(sql, opts={})
        if sql.match(/^SELECT/) || sql.match(/^DESCRIBE/)
          puts "RUNNING: #{sql}"
          super(sql, opts)
        else
          puts "PREVIEW: #{sql}"
        end
      end

    end
  end
end

Satisfied with the assumption that all queries go through #execute and the fact that Sequel itself is already well tested, we wanted to add a set of specs that would simply validate that our #execute method passed queries on to the superclass or ignored them as appropriate.

Set an expectation on #execute or on #super?

Our #execute calls the #execute in the superclass, and that action is what we want to set an expection on. RSpec lets us set message expectations on a class (ClassName.should_receive), on an instance (ClassName.new.should_receive), or all instances (ClassName.any_instance.should_receive), but the #execute we care about is an instance method on the superclass. It’s not a class method, and it’s not a method on any regular instance.

Instead of checking calls to the superclass’s #execute, it would be nice to set expectation on the call to super. Although Ruby language features are often implemented as methods (require, method_missing, etc.) super is actually a keyword. It is possible to create a method named super, but mocking such a method would be no help, as it would remain completely separate from the super keyword, and require a different, more explicit syntax to invoke.

RSpec’s David Chelimsky has commented on this issue:

Since we don’t get into the class hierarchy in rspec’s mock/stub framework, there is no facility for managing this.

So, it’s up to us to intercept the call if we want to verify that the subclass is behaving correctly.

Sneak a module into the inheritance chain

Somehow we need to get some in between the subclass and its superclass. Including a module in a class will not modify its superclass (which can’t be changed), but the module will be inserted into the class’s list of ancestors, and be able to intercept super before the actual superclass is reached.

To demonstrate this, let’s start with a simple parent class and subclass, both implementing #execute methods:

class ParentClass
  def execute
    puts "ParentClass#execute"
  end
end

class SubClass < ParentClass
  def execute
    puts "SubClass#execute"
    super
  end
end

The output of SubClass.new.execute will be:

SubClass#execute
ParentClass#execute

Including a module changes the ancestors list:

module SneakyModule
  def execute
    puts "SneakyModule#execute"
    super
  end
end

SubClass.ancestors # => [SubClass, ParentClass, Object, Kernel, BasicObject]
SubClass.instance_eval { include SneakyModule }
SubClass.ancestors # => [SubClass, SneakyModule, ParentClass, Object, Kernel, BasicObject]

And SubClass.new.execute shows the sequence of execution to be:

SubClass#execute
SneakyModule#execute
ParentClass#execute

This technique allows us to programatically verify the behaviour of the subclass’s #execute method by having the sneaky module in the middle record and report on method calls it intercepts. Here’s an example implementation:

module FakeMySQLAdapter
  class << self
    attr_accessor :last_execute
  end
  def execute(*args)
    FakeMySQLAdapter.last_execute = args
  end
end

With this done, we can write an example to verify that super is being invoked appropriately:

before(:all) { subject.class.instance_eval { include FakeMySQLAdapter } }
before(:each) { FakeMySQLAdapter.last_execute = nil }

it "should pass on SELECT statements" do
  subject.execute("SELECT * FROM foo;")
  FakeMySQLAdapter.last_execute.should == ["SELECT * FROM foo;", {}]
end

You can see a full implemenation of this on GitHub.

This is a solution. It exercises the logic of our subclass’s #execute and verifies its query handling. It’s not a great solution. One problem is that this code modifies the subclass being tested (by including the module), and another is that the subclass stays modified. It’s not possible to uninclude the module. If we had other tests that relied on the subclass, running the subclass’s specs first might break others.

There is a cleaner way, as we’ll see next.

Extract a module with the method override

Rather than overriding #execute in the subclass directly, we can override it by including a module that implements #execute. This structure allows us to include the logic of interest in a dummy class to test it in isolation.

First, we refactor our preview adapter to the following:

module Sequel
  module PreviewMySQL

    module MethodOverrides
      def execute(sql, opts={})
        if sql.match(/^SELECT/) || sql.match(/^DESCRIBE/)
          puts "RUNNING: #{sql}"
          super(sql, opts)
        else
          puts "PREVIEW: #{sql}"
        end
      end
    end

    class Database < Sequel::MySQL::Database
      include MethodOverrides
    end

  end
end

Before we test the module, let’s write an assertion that Database is actually using the module:

describe Sequel::PreviewMySQL::Database do
  it "should include the override methods" do
    subject.class.ancestors.should include(Sequel::PreviewMySQL::MethodOverrides)
  end
end

After that, we can test the module. We start by creating a dummy superclass for the module to be included in. We’ll use method_missing to record all method calls passed up to the superclass by its subclasses.

By using Class.new to create anonymous subclasses that include the module being tested, we can ensure that each example in our spec runs with a fresh copy of the subject.

describe Sequel::PreviewMySQL::MethodOverrides do

  class FakeSuperClass
    attr_reader :last_super_call
    def method_missing(method, *args)
      @last_super_call = [method, args]
    end
  end

  subject do
    Class.new(FakeSuperClass).instance_eval do
      include Sequel::PreviewMySQL::MethodOverrides
    end.new
  end

  #...
end

This solution is easier to read than the one in the previous post, and it avoids the problems of modifying the subject. Although we are now testing a module rather than the actual subclass we’ll be using it in, our assertion that the subclass should include the module of override methods completes the loop and will automatically warn us if we make changes to our hookup code.

With this in place, our individual examples are clean and simple, as follows:

it "should allow SELECTs" do
  sql = "SELECT * FROM sometable"
  subject.execute(sql)
  subject.last_super_call.should == [:execute, [sql, {}]]
end

You can see a full implemenation of this on GitHub.

This is a clean solution, and one that I’d have been satisfied with if I’d come up with it first. However, the extra time on this got me thinking… it really shouldn’t be that hard to stub out irrelevant and troublemaking parts of the full class.

Test the full class and stub problematic parts

Testing the full class really shouldn’t be hard and RSpec should point out any code that gets in the way. I gave it another go.

First, I reverted to the original implementation of the subclass:

module Sequel
  module MySQLPreview
    class Database < Sequel::MySQL::Database

      def execute(sql, opts={})
        if sql.match(/^SELECT/) || sql.match(/^DESCRIBE/)
          puts "RUNNING: #{sql}"
          super(sql, opts)
        else
          puts "PREVIEW: #{sql}"
        end
      end

    end
  end
end

And started with a simple spec:

describe Sequel::MySQLPreview::Database do
  it "should allow SELECTs" do
    sql = "SELECT * FROM sometable"
    subject.should_receive(:puts).with(/^RUNNING:/)
    subject.execute(sql)
  end
end

The assertion here is on #puts rather than the superclass’s #execute method, but we’ll add more comprehensive assertions later.

That example raised an error with a rather brief backtrace:

1) Sequel::MySQLPreview::Database should allow SELECTs
   Failure/Error: super(sql, opts)
   Sequel::DatabaseConnectionError:
     Mysql::Error: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
   # <internal:prelude>:10:in `synchronize'
   # ./spec/whole_spec.rb:11:in `execute'
   # ./spec/whole_spec.rb:30:in `block (2 levels) in <top (required)>'

RSpec tries to show you just the parts of the full backtrace that are relevant to the code you’re testing, but you can tell it to be more verbose by running it with the --backtrace option. That got me this:

1) Sequel::MySQLPreview::Database should allow SELECTs
   Failure/Error: super(sql, opts)
   Sequel::DatabaseConnectionError:
     Mysql::Error: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/adapters/mysql.rb:110:in `real_connect'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/adapters/mysql.rb:110:in `connect'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/database/misc.rb:48:in `block in initialize'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool.rb:100:in `call'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool.rb:100:in `make_new'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:144:in `make_new'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:130:in `available'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:120:in `block in acquire'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:162:in `block in sync'
   # <internal:prelude>:10:in `synchronize'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:162:in `sync'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:119:in `acquire'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:91:in `hold'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/database/connecting.rb:229:in `synchronize'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `execute'
   # ./spec/whole_spec.rb:11:in `execute'
   # ./spec/whole_spec.rb:30:in `block (2 levels) in <top (required)>'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:87:in `instance_eval'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:87:in `block in run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:195:in `with_around_each_hooks'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:84:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:353:in `block in run_examples'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:349:in `map'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:349:in `run_examples'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:335:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:28:in `block (2 levels) in run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:28:in `map'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:28:in `block in run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/reporter.rb:34:in `report'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:25:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/runner.rb:69:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/runner.rb:10:in `block in autorun'

I could see that a #synchronize method was responsible for acquiring a connection to the database. This was the method I needed to stub to to isolate my test from an actual database instance.

Following through the execution path showed that the superclass’s #execute would eventually call #_execute to finish its work. Setting an expectation on this method would allow me to test whether or not the SQL query was in fact being passed up to the superclass for execution.

Incorporating these two points brought me to the final form of my specs:

describe Sequel::MySQLPreview::Database do

  let(:connection) { stub("connection") }
  before { subject.stub(:synchronize).and_yield(connection) }

  it "should allow SELECTs" do
    sql = "SELECT * FROM sometable"
    subject.should_receive(:puts).with(/^RUNNING:/)
    subject.should_receive(:_execute).with(connection, sql, anything)
    subject.execute(sql)
  end

end

The two assertions here cover both the direct actions of the subclass, as well as its correct interaction with the superclass.

So, it turns out that (with --backtrace) this was quite straightforward. You can see the full code on GitHub

In conclusion

If I’d gotten this full class spec working first, I probably would have stopped there. However, even though this tests the class as a whole - the normal way - it is not necessarily the best approach.

The downside here is that it requires stubbing the class under test (even though it’s a part of the class we’re not specifically interested in testing). The other thing is that it couples itself more tightly to the internal implementation of the superclass. It requires the #execute, #synchronize and #_execute methods to behave in certain ways. It also places on the reader the burden of understanding the significance of the #_execute expectation.

The assumptions made by the isolated module spec are simpler: that all SQL goes via the #execute method and that it can be stopped there without any problems. It’s a smaller, cleaner set of assumptions, and there’s no stubbing of the subject. I’d say that extracting and testing a module in isolation is the best approach here.