This week was a fun week of code spelunking.
Turns out, ActionController::Session::MemCacheStore runs through all the servers and checks to see if they are alive when the session store is first setup from your environment.rb. This causes it to connect to the Memcache server(s) and if you use a web server like Litespeed or Passenger that fork from a parent process, you end up with multiple processes sharing the same socket.
This is bad because then your child processes trip over each other in their eagerness to get data from Memcache. This can result in all kinds of strange behavior from errors to having incorrect data returned. The Passenger documentation does a good job of example of showing an explanation of the problem (and a solution for those of you using Passenger).
The easiest way to solve this is to ensure your parent process disconnects from Memcache before the fork occurs. In Litespeed this means editing RailsRunner.rb - specifically adding a call to reset to RailsRunner.rb, right before the existing call to ActiveRecord::Base.clear_active_connections! (which is there for the same reason). To do this we have to create a Memcache client and tell Rails to use it, otherwise there is no clean way to access the instance it will create.
In environment.rb:
# you might have to require 'memcache' SESSION_CACHE = MemCache.new( 'myserver.com:11211', :namespace => 'myapp_session' ) config.action_controller.session_store = :mem_cache_store config.action_controller.session = { :cache => SESSION_CACHE, :key => '_my_app_session', :expires => 1.hour}
Then in Litespeeds RailsRunner.rb add SESSION_CACHE.reset right after where it does it's ActiveRecord::Base.clear_active_connections!
This should do the trick. Monkey patching the MemCacheStore to reset after doing the alive? check (or not doing the check) should also fix it.