Get the RSS
Browse the Archive
Saved by Replicas
There has been a few notes about Redis presharding lately from places such as Craigslist. The one thing these have in common is that you “planned” to go big from the start. However, what if you didn’t? What happens when a little feature becomes a raging success and your Redis servers are getting hammered? Well if you’re using the Ruby Redis client here’s how the built in replicas feature can save your ass.
To start let’s setup the scenario. You implement a feature for your game/web site that uses Redis sets. You launched with two Redis nodes. Your YAML file looks something like this:
production:
hosts:
- redis01:6379
- redis02:6379
This goes live and it’s a success. Everything is fine. Then it becomes a bigger success and you start to see some timeouts and slowdowns. Crap. You know you need to grow the cluster but how without losing existing data. You could create a client that moves data from one node to another while serving up the data but that’s not good for sets. Crap. Then you notice this method in Redis::HashRing
# Adds a `node` to the hash ring (including a number of replicas).
def add_node(node)
@nodes « node
@replicas.times do |i|
key = Zlib.crc32(“#{node}:#{i}”)
@ring[key] = node
@sorted_keys « key
end
@sorted_keys.sort!
end
Sweet gods, @replicas defaults to 160 so there are 320 keys in the ring. The damn client presharded for you. Yea. Wait how do I get new nodes into existing keys. Crap. Well here you go.
Modify the Redis::Client class to accept a name and a replicas_start attributes.
vendor/gems/redis-1.0.7/lib/redis/client.rb
@@ -155,10 +155,15 @@ class Redis
@sock = nil
@pubsub = false
+ @name = options[:name]
+ @replicas_start = options[:replicas_start] || 0
log(self)
end
+ def replicas_start
+ @replicas_start
+ end
+
def to_s
+ if @name
+ “Redis Client connected to #{@name} against DB #{@db}”
Modify Redis::Distributed to pass along the name and replicas_start attributes to the client. Also, modify it to pass the replicas_per_server to Redis::HashRing
vendor/gems/redis-1.0.7/lib/redis/distributed.rb
@@ -13,8 +13,15 @@ class Redis
if opts[:hosts].is_a?(Hash)
opts[:hosts].each_pair do |k,v|
+ if v.is_a?(String)
host, port = v.split(‘:’)
hosts « Client.new(:host => host, :port => port, :db => db, :timeout => timeout, :name => k)
+ elsif v.is_a?(Array)
+ v.each do |client|
+ host, port, replicas_start = client.split(‘:’)
+ hosts « Client.new(:host => host, :port => port, :db => db, :timeout => timeout, :name => k, :replicas_start => replicas_start.to_i)
+ end
+ end
end
else
opts[:hosts].each do |h|
@@ -23,8 +30,12 @@ class Redis
end
end
+ if opts[:points_per_server]
+ @ring = HashRing.new(hosts, opts[:points_per_server])
+ else
@ring = HashRing.new hosts
end
+ end
def inspect
to_s
Modify Redis::HashRing to use the replicas_start attribute of the client.
vendor/gems/redis-1.0.7/lib/redis/hash_ring.rb
@@ -23,8 +23,9 @@ class Redis
# Adds a `node` to the hash ring (including a number of replicas).
def add_node(node)
@nodes « node
+ start = node.respond_to?(:replicas_start) ? node.replicas_start : 0
@replicas.times do |i|
- key = Zlib.crc32(“#{node}:#{i}”)
+ key = Zlib.crc32(“#{node}:#{start + i}”)
@ring[key] = node
@sorted_keys « key
end
@@ -33,8 +34,9 @@ class Redis
def remove_node(node)
@nodes.reject!{|n| n.to_s == node.to_s}
+ start = node.respond_to?(:replicas_start) ? node.replicas_start : 0
@replicas.times do |i|
- key = Zlib.crc32(“#{node}:#{i}”)
+ key = Zlib.crc32(“#{node}:#{start + i}”)
@ring.delete(key)
@sorted_keys.reject! {|k| k == key}
end
Lastly modify the YAML file to pass in the correct options.
production:
points_per_server: 80
hosts:
redis01:6379:
- redis01:6379:0
- redis03:6379:80
redis02:6379:
- redis02:6379:0
- redis04:6379:80
Now replicate the data to the new servers and deploy the code. Bingo. Now get some sleep.
Ashley Martens
Server Engineer
