Enforcing spec coverage with CruiseControl, RCov, and RSpec

June 1, 2007

Now that Alex has gotten CruiseControl.rb setup setup at work, I wanted to use it to enforce good test coverage - and I wanted the output to look good doing it. Turns out this isn't that hard, but requires a few things:

Make sure RCov is installed on whatever machine CruiseControl is installed on (and RSpec, unless it's in your rails/vendor directory) At this point all you need is a custom cruise task (CruiseControl will run rake cruise instead of the default task if a cruise task exists). Mine looks like this:

lib/tasks/cruise.rake:

rspec_base = File.expand_path("#{RAILS_ROOT}/vendor/plugins/rspec/lib")
$LOAD_PATH.unshift(rspec_base) if File.exist?(rspec_base)
require 'spec/rake/spectask'
require 'spec/rake/verify_rcov'
RCov::VerifyTask.new(:verify_rcov) { |t| t.threshold = 100.0 }

desc "Task for cruise Control"
task :cruise do
  RAILS_ENV = ENV['RAILS_ENV'] = 'test' # Without this, it will drop your production database.
  CruiseControl::invoke_rake_task 'db:reset'
  CruiseControl::invoke_rake_task 'cruise_coverage'
  CruiseControl::invoke_rake_task 'verify_rcov' 
end

desc "Run specs and rcov"
Spec::Rake::SpecTask.new(:cruise_coverage) do |t|
  t.spec_opts = ['--options', "#{RAILS_ROOT}/spec/spec_rcov.opts"]
  t.spec_files = FileList['spec/**/*_spec.rb']
  t.rcov = true
  t.rcov_opts = ['--exclude', 'spec,/usr/lib/ruby', '--rails', '--text-report']
end

This will invoke RSpec telling it to use RCov with a text report. If RSpec fails (broken tests) your build will fail. If it passes it will call the verify_rcov task which will parse the RCov output and fail unless the coverage is equal to or greater the percentage you set in the RCov::VerifyTask. If you do not set both ENV['RAILS_ENV'] and RAILS_ENV to test, CruiseControl will run in production mode and drop your production database. CruiseControl is awesome like that.

Manually running the specs I like the colored progress output, but for viewing in CruiseControl the colors get mangled, and it's nice to see the specdoc format - so I made a seperate .opts file for RSpec in spec/spec_rcov.opts. If you don't care, change the spec_rcov.opts above to spec.opts and ignore this part. The spec_rcov.opts I used is:

--format
specdoc
--loadby
mtime
--reverse

And that's it! You'll get a readable output, and nobody will be able to commit testless code or broken tests without breaking the build. The one with RSpec is in rspec/rake_tasks/verify_rcov.rake - Take a look at that, as it might suite your needs!

Update: updated to use the verify_rcov that comes with RSpec 1.02+. Thanks Chad!

Posted in General
Tagged as rcov, rspec, ruby, and rails

Comments

1 ColinD October 25, 2007

Thanks for this, rake db:reset is not in my standard list of rake tasks, so I've opted for rake db:test:clone.

2 Jerrett November 23, 2007

db:reset will be in rails 2

3 Brian December 19, 2007

Is there a way for the rcov output to be visible on the cruisecontrol site? This setup didn't work for me. Going to give it another try. Said it couldn't find the coverage dir, so I'll create one. site doesn't have access to that dir on cruisecontrol. I thought there was an artifact directory for that purpose.

4 Brian December 19, 2007

This is what I get when I run this scrip. Any ideas why this might happen?
I do have the coverage directory. Index.html isn't there, its supposed to be generated.

/var/www/cruise/projects/feedback/work root$ ruby1.8 -e require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run rake aborted! No such file or directory - coverage/index.html (See full trace by running task with --trace) (in /var/www/cruise/projects/feedback/work) [CruiseControl] Invoking Rake task "cruise" [CruiseControl] Invoking Rake task "db:test:clone" [CruiseControl] Invoking Rake task "cruise_coverage" [CruiseControl] Invoking Rake task "verify_rcov" dir : /var/www/cruise/projects/feedback/work command : ruby1.8 -e "require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run" executed command : echo /var/www/cruise/projects/feedback/work root$ ruby1.8 -e "require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run" >> /var/www/cruise/projects/feedback/build-91/build.log && ruby1.8 -e "require 'rubygems' rescue nil; require 'rake'; load '/var/www/cruise/tasks/cc_build.rake'; ARGV << '--nosearch' << 'cc:build'; Rake.application.run" >> /var/www/cruise/projects/feedback/build-91/build.log 2>&1 exitstatus: 1 STDERR TAIL START STDERR TAIL END

5 Brian December 20, 2007

This is working. Key tip - don't check in the coverage directory. put that in your ignore list. I also added to ignore more files in the exclude.
The target coverage seems to be a little odd. In the % coverage totals it shows 28.4% but then for the verify it says coverage must be at least 90% but was 44.2%.
Also it looks terrible in IE, but great in Firefox...

6 Gaius December 23, 2007

I've found one good way to output the coverage information to the CruiseControl directory (so it shows up under "artifacts"):

1. create {RAILS_ROOT}/lib/build_settings.rb

2. in that file, put
<pre><code>
module BuildSettings
attr_reader :coverage_dir
def coverage_dir
@coverage_dir ||= ( ENV['CC_BUILD_ARTIFACTS'] || 'coverage' )
end
end
</code></pre>

3. In your rake file, do: <code>require "#{RAILS_ROOT}/lib/build_settings"</code> at the top

4. In each namespace in your rake file (including the base one), do: <code>include BuildSettings</code>

5. use <code>t.rcov_dir = coverage_dir</code> inside your SpecTask

7 Ben Mishkin February 6, 2009

Quick and dirty to display rcov alongside cruisecontrol.rb -

1. modify your cruise rake task from 'spec' to 'spec:rcov'

2. symlink your project's coverage directory inside cruisecontrol's public directory

ln -s /path_to_.cruise/projects/your_proj/work/coverage /path_to_cruisecontrol.rb/public/rcov

Rcov will be viewable at
http://your_cc/rcov/
and updated on every cc build.

8 PeterMontee July 3, 2009

I congratulate, a brilliant idea

Add comment

Blog Archives