gamemakers @ ngmoco:)

Saved by Replicas

Wed, 02 Mar 2011 10:28:00 -0800

There has been a few notes about Redis presharding lately from places such as Craigslist. The one thing these have in common is that you “planned” to go big from the start. However, what if you didn’t? What happens when a little feature becomes a raging success and your Redis servers are getting hammered? Well if you’re using the Ruby Redis client here’s how the built in replicas feature can save your ass.

To start let’s setup the scenario. You implement a feature for your game/web site that uses Redis sets. You launched with two Redis nodes. Your YAML file looks something like this:

production:
    hosts:
      - redis01:6379
      - redis02:6379

This goes live and it’s a success. Everything is fine. Then it becomes a bigger success and you start to see some timeouts and slowdowns. Crap. You know you need to grow the cluster but how without losing existing data. You could create a client that moves data from one node to another while serving up the data but that’s not good for sets. Crap. Then you notice this method in Redis::HashRing

    # Adds a `node` to the hash ring (including a number of replicas).
    def add_node(node)
      @nodes « node
      @replicas.times do |i|
        key = Zlib.crc32(“#{node}:#{i}”)
        @ring[key] = node
        @sorted_keys « key
      end
      @sorted_keys.sort!
    end

Sweet gods, @replicas defaults to 160 so there are 320 keys in the ring. The damn client presharded for you. Yea. Wait how do I get new nodes into existing keys. Crap. Well here you go.

Modify the Redis::Client class to accept a name and a replicas_start attributes.

vendor/gems/redis-1.0.7/lib/redis/client.rb
@@ -155,10 +155,15 @@ class Redis
       @sock = nil
       @pubsub = false
+       @name = options[:name]
+      @replicas_start = options[:replicas_start] || 0

       log(self)
     end

+    def replicas_start
+      @replicas_start
+    end
+
     def to_s
+       if @name
+         “Redis Client connected to #{@name} against DB #{@db}”

Modify Redis::Distributed to pass along the name and replicas_start attributes to the client. Also, modify it to pass the replicas_per_server to Redis::HashRing

vendor/gems/redis-1.0.7/lib/redis/distributed.rb

@@ -13,8 +13,15 @@ class Redis

       if opts[:hosts].is_a?(Hash)
         opts[:hosts].each_pair do |k,v|
+          if v.is_a?(String)
           host, port = v.split(‘:’)
           hosts « Client.new(:host => host, :port => port, :db => db, :timeout => timeout, :name => k)
+          elsif v.is_a?(Array)
+            v.each do |client|
+              host, port, replicas_start = client.split(‘:’)
+              hosts « Client.new(:host => host, :port => port, :db => db, :timeout => timeout, :name => k, :replicas_start => replicas_start.to_i)
+            end
+          end
         end
       else
         opts[:hosts].each do |h|
@@ -23,8 +30,12 @@ class Redis
         end
       end

+      if opts[:points_per_server]
+        @ring = HashRing.new(hosts, opts[:points_per_server])
+      else
       @ring = HashRing.new hosts
     end
+    end

     def inspect
       to_s

Modify Redis::HashRing to use the replicas_start attribute of the client.

vendor/gems/redis-1.0.7/lib/redis/hash_ring.rb
@@ -23,8 +23,9 @@ class Redis
     # Adds a `node` to the hash ring (including a number of replicas).
     def add_node(node)
       @nodes « node
+      start = node.respond_to?(:replicas_start) ? node.replicas_start : 0
       @replicas.times do |i|
-        key = Zlib.crc32(“#{node}:#{i}”)
+        key = Zlib.crc32(“#{node}:#{start + i}”)
         @ring[key] = node
         @sorted_keys « key
       end
@@ -33,8 +34,9 @@ class Redis

     def remove_node(node)
       @nodes.reject!{|n| n.to_s == node.to_s}
+      start = node.respond_to?(:replicas_start) ? node.replicas_start : 0
       @replicas.times do |i|
-        key = Zlib.crc32(“#{node}:#{i}”)
+        key = Zlib.crc32(“#{node}:#{start + i}”)
         @ring.delete(key)
         @sorted_keys.reject! {|k| k == key}
       end

Lastly modify the YAML file to pass in the correct options.

production:
    points_per_server: 80
    hosts:
      redis01:6379:
        - redis01:6379:0
        - redis03:6379:80
      redis02:6379:
        - redis02:6379:0
        - redis04:6379:80

Now replicate the data to the new servers and deploy the code. Bingo. Now get some sleep.

Ashley Martens
Server Engineer

Using Cassandra from Erlang

Mon, 16 Aug 2010 17:15:04 -0700

I spent some time trying to access Cassandra from Erlang and ran in to a bit of difficulty. To start, Cassandra uses Thrift for it’s RPC and wire protocol. Using Thrift from Erlang is very poorly documented. I was eventually able to get it to work, with the help of some nice folks in the chat room. In the interest of returning the favor, I’ve put together a little sample application and posted it on github.

Being already familiar with using Thrift in Erlang, the real sticky point for me was figuring out how to assemble the rat’s nest of custom record types into working Cassandra requests. Look at the code and README file for details, but I’ll repeat the main roadblocks I hit here.

Use framed mode for Cassandra 0.7 or greater
Do not use framed mode for Cassandra 0.6 or below
The map type in the Cassandra IDL expects an Erlang dict type

-dave

Loosely related issues

Wed, 05 May 2010 11:48:00 -0700

As most of our users know we have been experiencing server issues with We Rule and Plus+ lately. The server teams have been pouring over logs and graphs for the past weeks trying to figure where our issues are and implementing fixes to those issues.

One of the issues we recently “dealt” with was a problem on our NFS server, it was swapping and had really high load. This machine had become unstable but was critical to sharing certain static files between our web servers. We are currently in the process of replacing this machine but right now we need to keep it running, so we decided to stop all non-production critical access to the NFS shares on this machine. As soon as one of our analytic processes stopped writing to the share the machine became much more stable. Great.

A few hours later we found that Plus+ and We Rule were responding really well. In fact our response times were great and had been holding level since we solved the NFS issue. Holy $#!@ that’s awesome but why?

Well it turns out that nginx was serving up static files directly from NFS and when NFS was having problems nginx workers would start blocking as they waited to read the file. This resulted in requests being queued in nginx which, when the block cleared, would send a large batch to our application processes. This would flood the database with requests which would block as it worked through. All this resulted in choppy response times.

Long story short. Serving web files directly from NFS is a bad idea. Let’s not do that anymore.

Long live fast Plus+ response times.

- Ashley

Play on the iPad: the Magic Circle and a marketplace

Fri, 02 Apr 2010 12:23:00 -0700

Play on the iPad: the Magic Circle and a marketplace:

The iPad looks like an amazing gaming device. Does Apple’s App Store promote or hinder innovation? What role does a gatekeeper play in making good games for large audiences? Our friends at O’Reilly asked us to consider these questions for their Radar website. Radar tracks technology trends, and ngmoco Producer Justin Hall takes a break from Touch Pets Dogs to explore Play on the iPad.

When Memcached Doesn't Scale

Wed, 24 Mar 2010 10:42:00 -0700

Memcached has been one of the greatest tools in the battle to scale web applications, especially in RoR. From write-through query caching to action caching to custom caching…. it’s been the bees knees. When memcached doesn’t work, because you need persistence for example, an alternative presents itself, Redis.

At some point, however, even the speed of memcached cannot save you from coding issues, which were fine with a limited number of processes but cripple your application when the throughput increases. One of our experiences with this was I18n_db. There were two problems with this plugin that we ran into, a per-request check and a database firestorm.

First, there was a check of the last updated_at integer during each request. This should be a super fast call to memcached which would return an integer. No big deal, until you make 1000 request per second for this value. At this point you have introduced a lock to your distributed, stateless code. You also get a really bad/neat side effect (and the way we [Scott] found the problem) of maxing out the gigabit ethernet controller (900Mbps). This causes other queries to this memcached server to also be slow, which cascades throughout your application causing strange slow queries.

Second, if this key was deleted the translations were reloaded by each process from the database. Good idea until you have 800 concurrent requests for the same data from the database. The database becomes completely locked as every connection tries to get this data and update memcached with it.

Solutions:

1) It’s okay for the translations to be a bit stale, so only check for an update once a minute or five.

2) Update the translations in memcached when they change in the database and create a background job to refresh the cached translations once a day or so. Do not allow either of those memcached keys to be removed from memory at all costs.

-Ashley

ngmoco:) 2010 Summer Co-op and Internship Program

Tue, 16 Mar 2010 11:30:00 -0700

Submission Deadline: April 11th

ngmoco:) Summer 2010 Co-Op and Internship Program

The game industry has changed. We used to work on games for two to three years, stick them in a boxes and ship them to retailers around the world, hope for a decent Metacritic score and move onto the next set of titles. We don’t do that anymore.

In this new era of gamemaking, the rise of digital distribution platforms, social networks, pervasive and fast network access, and powerful new mobile devices has forever transformed the landscape. Now, we are perpetually connected with our players, we are tuning and iterating our game experiences on a daily and hourly basis, and we dialogue with our users and our fans constantly.

At ngmoco:) we’re committed to building next generation games for this new ecosystem and user. Our goal is to create innovative, socially relevant, compelling mobile social game experiences that promote daily active usage.

We’re designing, creating and launching these games now. From the the Plus+ Network, to the free-to-play Eliminate, Epic Pet Wars and Touch Pets Dogs series, to the highly anticipated We Rule and GodFinger, ngmoco:) is leading by example.

If you’re interested in helping us define this new era of gamemaking, join us.

We are looking for the following intern and co-op candidates:

· Engineers

o Brilliant, passionate, driven future leaders of web & mobile software technology who can solve any technical problem, at any scale, in any language…our problems are solved usually in…

o Ruby on Rails

o On Servers (C, C++)

o On Client devices (Cocoa, Objective C, Open GL, UIKit)

· Associates

o Aspiring gamemakers & marketers with clear visions, communication skills and leadership ability

o An incredible personal passion for the intersection of games & business - opinions and insights into consumers & game design that are progressive but pragmatic

· Designers

o Aspiring gameplay & game mechanic visionaries

o Personal passion for the intersection of human interaction with core gameplay compulsions and an obsessive thirst to invent, iterate and understand core gameplay compulsion loops

Requirements:

· Must be within 2 years of receiving a bachelor’s degree or pursuing a masters degree or equivalent

· Knowledge of software development cycle

· Ability to thrive in fast-paced environment

· Strong analytical skills

· Ability to be flexible to quickly changing business needs, new technologies, and game requirements.

· Exposure to game design and development experience (academic, employment, or personal)

If this is you please send a cover letter, your CV, and two letters of reference/recommendation to internship@ngmoco.com. This is a paid internship program.

Positions are available in the summer from the end of May to late August. The dates are flexible depending on each applicant’s’ university schedule.

Please send your application no later than April 11th. We will notify accepted students no later April 23rd.

###

Headquartered in San Francisco, ngmoco was founded in 2008 by games industry veterans committed to the new mobile landscape opened up by Apple’s iPhone. ngmoco :) creates and publishes games for the iPhone made in collaboration with the best and brightest gamemakers in the world. Founded by Neil Young, Bob Stevenson, Alan Yu and Joe Keene, the company’s investors include Kleiner Perkins Caufield & Byers, Norwest Venture Partners & Maples Investments. Follow us on Twitter at www.twitter.com/ngmoco and at www.ngmoco.com.

ngmoco:) & GDC 2010

Mon, 01 Feb 2010 16:38:12 -0800

Apologies for not being as active on our gamemakers blog recently. With Game Developers Conference 2010 coming up, we’ll try to rectify that and post more frequently. Look for posts from various ngmofos before and after GDC as we share our experiences at the event.

As we’ve mentioned before, many ngmofos are active participants in the creative community and GDC in particular. ngmoco:) is honored to be presenting four lectures at this year’s conference.

Neil Young, our fearless leader, will be presenting: Things to Unlearn Moving From Traditional Development to the New Digital World

https://www.cmpevents.com/GD10/a.asp?option=C&V=11&SessID=10642

Clive Downie, VP of Marketing, riffs on Neil’s theme with a session focused on product marketing: Marketing in a Digitally Distributed World: What to Do When The Old Bag of Tricks Won’t Work Anymore

https://www.cmpevents.com/GD10/a.asp?option=C&V=11&SessID=10801

On the technical side, we’ll be presenting two lectures at the inaugural iPhone Games Summit. The first one by Steve Detwiler and James Marr of our Creative R&D Team examines Building the Server Software for Eliminate.

https://www.cmpevents.com/GD10/a.asp?option=C&V=11&SessID=10547

Later on in the day, producer Matt Roberts and our partner Andrew Stern at Stumptown Game Machine present a case study on the design of Touch Pets in a talked titled: New Dogs, New Tricks: Breeding Social Networking and Virtual Pets.

https://www.cmpevents.com/GD10/a.asp?option=C&V=11&SessID=10546

In addition, we’ll also be showing off some new developments in Plus+ Network (more details to come).

See everyone there!

-Alan

Hey all! David Cairns here! I just wanted to mention that Amanda Wixted (of Zynga) and I are doing...

Sun, 27 Sep 2009 01:04:01 -0700

Hey all! David Cairns here! I just wanted to mention that Amanda Wixted (of Zynga) and I are doing a talk about iPhone Game Programming at the 360|iDev Conference in Denver next Monday. If you’re attending, please come check out our session, or at least come over and say hello at some point.

If you’re not attending, WHY AREN’T YOU ATTENDING?

— David

Let’s talk about names, upgrades, leveling, matchmaking… and Energy

Wed, 09 Sep 2009 15:25:54 -0700

Last time I wrote about the game, I was reflecting on how the recent addition of armor sets really felt like a game-changer. Since that time, we went much deeper on player customization and drove it into the creative center of the game. It’s all about competing to earn credits and using those credits to purchase and upgrade new gear like weapons and armor. It really feels fresh and provides a compelling bonus reward cycle to the deathmatch gameplay at the heart of the experience.

If you haven’t seen the latest trailer, go check it out on the newly revamped Eliminate website as it attempts to describe the entire ecosystem of the game in under 2 minutes. See, as you compete in global matches, you’ll also earn credits. The better your performance in a match, the more credits you earn. You’ll earn something even if you’re in last place, but great performance will always pay back the highest earnings. As you use these credits to buy new weapons or armor… or to upgrade your gear along multiple upgradable dimensions, you’ll gain an advantage over your enemies if they’re not upgrading too. And that’s where our matchmaker algorithms really become critical.

As you start performing better and winning more matches – by pure skill or with the help of upgrades or any combination – the matchmaker will continually match you up against better performing players. This keeps the individual matches balanced overall even as one player chooses to upgrade and another chooses not too. You’ll always find yourself matched against other players of similar potency. Ultimately, the player at the top of the global leader board will possess both skill and a healthy assortment of upgrades. Part of the fun is gearing up your character to reflect the style of play you enjoy the most. Are you all about offense? Maybe evasive moves? Or do you love blasting enemies with rockets above all else? The in-game armory and upgrade options give you the tools to play the way you like.

If you use up all of your energy, you can continue practicing on the global servers, but no credits will be earned until your energy levels are sufficient. You can recharge your energy at any time by using Power Cells. Your app download will include some Power Cells and you can purchase additional Power Cells from iTunes or acquire them through other methods to be revealed later. Or you can wait for your energy to recharge automatically.

Why bother with Energy? We use energy for a lot of things: to regulate and balance the game content, to help stack-rank the queue of players waiting for their next match (current default wait time is 5 seconds), and of course to pay for the global servers and infrastructure we host for lobbies, matchmaking, and gameplay. It’s a pretty huge undertaking and is not feasible without a scalable revenue stream tied to it.

If you really enjoy the game and want to engage deeply, you can do exactly that. We expect some portion of players will be willing to pay a fair price for something they really enjoy. And by doing so, those deep players can accelerate their progress in the game by earning more credits and leveling up faster than someone who is not engaging as deeply. That’s okay. The deep players are helping support the game for everyone, making online practice mode possible. And the deep players are given priority in the event the servers are slammed by tons of simultaneous players. It seems like a pretty fair solution—to offer the core online competitive FPS experience with credit-earning and upgrades at a fair price to everyone who wants to give it a try—while also offering deeper engagement with additional benefits to those players willing to invest more into the game. You can still progress and level up even if you’re not buying extra power cells, but eventually you’ll run out of energy and will need to replenish before earning more credits and leveling farther.

You may be wondering if players willing to throw piles of money at the game will simply rocket to the top of the leader boards overnight. The answer is NO. Most items and upgrades are level bound, meaning you have to earn the prerequisite level through combat experience in order to qualify for the armor or upgrade or whatever. So just because you pay more doesn’t mean you don’t have to play like everyone else. That’s the whole point of the game – to play and win! You’ll level up faster when you’re earning credits, so energy is a factor, but you definitely have to play and earn those credits.

We recently wrapped up a pretty big session of hands-on play tests with real fps players who are not game developers. The whole process was a great source of insightful feedback. We had 65 gamers blasting the shit out of each other, earning credits, buying new gear, using energy… the whole nine yards. The complete cycle felt very natural. We’re definitely fine-tuning some things as a result of those playtests and the game is already better as a result. The reason I bring this up is that it gave me a lot of confidence that players do understand this new ecosystem and the game will ultimately be better as a result. Eliminate’s release is not going to be a fire-and-forget launch. We’re in this for keeps. We want people to bring it anytime, anywhere. And extend their FPS lifestyle beyond the desktop or livingroom – into wherever their lives take them.

-chris

Develop: Alan Yu talks ngmoco, App Store and the rise of microstudios

Wed, 15 Jul 2009 11:55:52 -0700

ngmoco co-founder Alan Yu is currently on assignment at the Develop Conference in Brighton: He’s participating in a panel about the future of the iPhone and partaking in assorted tomfoolery. But Yu still has time to look back over the past year and answer a bevy of questions from Develop’s Michael French.

In this conference-side chat with an ngmofo, Yu talks about the founding of ngmoco, publishing games for the App Store, Plus+ and how the iPhone has changed game making.

From Develop:

Can you remind us what the story is behind the founding of Ngmoco?

The company was founded by four of us – Neil, Bob Stevenson, myself and Joe Keen. In short: we love the iPhone.

Like a lot of people, I waited in line on day one to get mine. Neil tells the story when he does talks about the wait that day, and I was right there with him. It was hot, irritating, but it felt like it was going to be big. Even then, before all the software came, we knew iPhone was going to be huge – just interacting with it was different. Then, when the SDK was out we realised what a great device it was to make games for. We left our jobs, formed a company in July 2008 and got it incorporated in August.

The great thing about it for many of us, especially those in the company that game from much bigger organisations, is that it’s a refreshing change from the two or three year production cycle needed for most other games. And it’s such a difference from managing a studio of around 400 people working on one or two products; we released eight games – a mix of free and paid-for apps – in the space of just seven months. A year later that number is 11.

Creativity, the iPhone has reinvigorated me – and Neil and a lot of other people.

Why did Ngmoco have to be an independent company? Why couldn’t you just do this within the EA business?

The market moves so fast that we felt we needed entrepreneurial focus – and it was a risk, a risk we were willing to take. EA is a great company, I love it and the people and I learned so much while I was there. But you can move faster when you are independent and not a part of those larger organisations.

And if you look at even the trends in the game making space for iPhone, a great democratisation has taken place. It’s $99 for the SDK, you need a Mac to program on… and that’s it, you’re off. That’s reflected in the types of games we’re seeing in that space and the kinds of developers making them. Instead of 80 person teams working on a game that they hope will be a hit, you have two to three man teams – or in the case of Rolando, one guy – working on a release schedule that covers just a few months.

But Ngmoco describes itself of a publisher of these smaller games – how does that work?

We do a range of things. We have no internal development, but there are concepts we come up with internally – such as Topple, Maze Finger, Dr Awesome and Word Fu – which we then find developers externally to move into production under our direction. Then there are games we publish and go and acquire such as Rolando and Star Defence – these concepts come to us from other developers. We also do marketing and other platform development stuff in the form of our Plus+ platform.

What do independents get out of working with you rather than just submitting the game themselves?

I think it’s a misnomer that you can self-publish on the App Store. You can self distribute – but it’s like someone gave you the key to GAME or Virgin Megastore. That’s great, because you can go in and put whatever you want on the shelf, but it’s bad because 25,000 other people can come in and do the same thing.

So any competent developer will have financing and marketing to get a product in front of as many people as possible. And marketing in this space is really different, it’s not about getting a game into stores or buying TV ads; it’s word of mouth and more socially driven.

In working with us developers get capital, financing and marketing – plus we are game makers ourselves. Neil, Bob, myself and the rest of the team have worked on games that have sold in the millions and millions of units. So the power is creative management and feedback to developers. [Rolando studio] Hand Circus plus Ngmoco is better than Hand Circus or Ngmoco on its own; there are real benefits for all of us by working together.

A lot of people have positive things to say about the iPhone, but in what ways do you personally see how the format has changed the games industry?

One refreshing area is talent. I’ve been in the game industry a long, long time, and I used to run GDC and then worked at EA. I realised that through all that time I was really talking to the same people – great friends, but no new blood. But here on iPhone with a $99 devkit and a Mac Mini, so many more people can move into this space. For me it’s been reinvigorating and awesome to speak to people like [Rolando creator] Simon Oliver, and people from outside games who are now coming in by making an iPhone game. Of course, that has meant a lot of people, and a bigger cast of characters – which allows for more orthogonal thinking, it’s changing the echo chamber that is the games industry. So it is very exciting to be able to not only speak with new talent all the time but help them realise their dreams.

Another great thing is the rise of the microstudio. You can now get a game out there with just a few people. That’s one of the major changes we are seeing. There are no great bands that are more than a couple of guys – and when you have that kind of rock star-style development you aren’t talking about managing a huge budget or all the other trappings you’d have in a larger organisation, and it’s a lot easier to get up and running.

Will the encroachment of EA and Gameloft into iPhone upset the kind of model you have established?

I think there’s lots of room for all kinds of companies in this space. Our advantage is that we are focused entirely on the iPhone, that’s all we do. I think comprehensively understanding the features of the hardware is a great benefit – especially when it came to understanding the great features in 3.0 and what that enables us to add to games.

But the other advantage we have is understanding the user space and knowing how people use their iPhone. It’s a benefit to us to solely focus on this device and not be distracted by other things the way bigger games publishers might.

Plus there is our publishing platform, which gives us an advantage over other companies.

Tell me about the Ngmoco platform.

Basically, inside our games is a very powerful analytics package. We call it our eyes and ears. In that I can tell how many Maze Finger mazes you have completed, where you might get stuck in Topple – it was level four, by the way – where people touch the screen in Rolando.

Or even the average play session, which is around 22 minutes – that’s something which, as an aside, shows that this really isn’t just a casual platform, that’s a core gamer-style stat.

So our platform talks back to us, gives us a relationship with the consumer and means we can live tune the software seamlessly for the user without an update. That stuff is invisible to the user.

Our system also includes in-game ads which we don’t sell – we just use to cross-promote our games – plus an achievement system, friends list, social referalls, push notifications, asynchronous multiplayer, live multiplayer and in-game currency.

All those features get battle tested in our games – once we’ve written them for Star Defence, for instance, it gets fed back in to the platform, so it is trusted and proven.

The thinking is that these new independent developers coming to this space don’t want to write all that stuff from scratch. That’s another answer to that question about the value we as a publisher can offer to developers.

There’s still a discovery issue around iPhone games in terms of telling users about them – and as you say the marketing requirements are different – does that mean games need to be designed differently to ‘traditional’ games?

I think so. iPhone is moving very close to the social gaming area and pushing the boundaries on what we typically expect of a game. One game we are working on is Touch Pets, our pet simulator – you can have a pet and even send them on play dates with other players. That takes the game beyond just the device and builds a social canvass for players to paint on. In all, consumer expectations have been fundamentally changed by the new functions the phone allows and the games have to match that.

Party with ngmoco at WWDC

Fri, 05 Jun 2009 18:44:59 -0700

Are you going to be in San Francisco for WWDC? We’re hosting a little get together on Monday, June 8 at the W’s XYZ bar (3^rd St and Howard St) from 6pm to 8pm.

Swing by to say hi to the ngmoco team, Simon Oliver of Hand Circus and Erik T’Sas of Rough Cookie. We’ll also have our summer line-up of games—Star Defense, Touch Pets, Rolando 2, and our FPS—on hand.

Space will be limited. If you plan on stopping by, please RSVP to lcaplan at ngmoco dot com and arrive prepared to show your WWDC badge at the door.

We look forward to seeing you there.

—The ngmoco Team

Star Defense: The Tuning Challenge

Fri, 29 May 2009 12:35:00 -0700

Tuning defines Star Defense—possibly even more so than its 3D play-scapes. It is an important and essential part of game design. The tuning of a game’s individual components is what makes it a unique and compelling experience. Changing just one element would transform it into a completely different game.

For example, if the game allowed you to pause it while you placed towers or if there were a few more seconds between wave launches, Star Defense would lose its fast pace. If we altered the strength of the towers or how much it cost to upgrade them, it would influence how you played the game. If we handed out more credits for each enemy unit killed, it would change how you managed tower purchases and upgrades.

The tuning decisions we made in Star Defense transformed what is otherwise a turn-based genre into something that feels and plays a bit more like a real-time strategy game. Because the action never stops—and you can actually choose to advance waves—you’re forced to constantly survey the action. It changes the way you have to think about defending.

But defining the kind of game Star Defense would become was only half of our tuning challenge. We needed to craft a difficulty curve that would encourage players to explore new tactics and learn all of the game’s systems. If it were too hard, players would grow too frustrated to continue playing. With too little challenge, they would soon grow bored.

The past few weeks have been spent nailing this balance.

The core Star Defense experience is made up of seven planets that can be played on three difficulties. On average there are 60 waves per planet (although you can opt to play against an endless stream of S’rath). We also slowly introduce new S’rath unit types as the player progresses through the game. For each of the 21 levels to present a unique challenge, we ultimately had to create at least 21 distinct difficulty curves and 420 individual waves.

The first task was to understand the different variables we could use to create the dynamic, hand-tuned waves we were gunning for. We had already learned that changing hit points alone was not enough.

After a few experiments, we realized that we should not rely on changes in speed because we defined some of our enemy archetypes based on how quickly they moved. We could not rely on incremental shifts in unit spacing because it was too difficult for players to identify these changes until the S’rath had slipped through their defenses and were pounding on the shield walls.

We were left to play around with hit points, the number of S’rath found in each wave, and obvious changes in spacing.

With this knowledge, we settled in to start locking down levels by hand. It took us an entire week to tune three planets across one difficulty level. Then all that work was effectively undone when we decided to alter some S’rath behaviors and reorder the planets.

We needed a way to procedurally generate the waves in Star Defense—or we’d end up spending two months attempting to tune levels that could ultimately be undone if we changed any other part of the game.

In the end, we used procedurally generated waves as a rough framework, which then allowed us to jump in and strategically fine tune individual waves. Those hand-crafted waves ensured that each planet felt like a unique character with its own difficulty pacing and surprises that would catch players off guard.

The tuning, combined with tower characteristics, wave timing and the unique paths along 3D planets, has created a level progression that will drive players to master Star Defense as they internalize enemy behaviors, towers’ strengths, and the ultimate placement of such weapons systems.

Most of all, our tuning has created a unique tower defense experience. Let us know what you think once the game lands in the App Store.

—Allen Ma

Stanford University and Apple were kind enough to invite me to...

Fri, 22 May 2009 19:15:00 -0700

Stanford University and Apple were kind enough to invite me to speak at today’s iPhone App Development class. (Here’s a link to the presentation, which is now up on iTunes). I gave students an overview of iPhone OpenGL optimization techniques and showed a demo of a fireworks app as well as my new game LiveFire.

Here is the full source code for the Fireworks demo I showed today:

http://skyfell.org/downloads/Fireworks.zip

This is basically the effect that I used in Dropship and MazeFinger. If you are new to OpenGL on iPhone, or OpenGL in general, perhaps you may find it useful in some way. Please don’t just recompile and sell this… Add something cool to it. I’d love to see:

More types of fireworks
Rocket trails
Smoke plumes
Music
Crowd cheering sounds

If you send any cool changes you make to me, I might update this blog with your new features!

Tim Omernick
ngmoco:) Creative R&D

IDENTITY & IN-APP COMMERCE IN FPS Our deathmatch anywhere,...

Mon, 18 May 2009 08:19:49 -0700

IDENTITY & IN-APP COMMERCE IN FPS

Our deathmatch anywhere, anytime, first-person shooter has progressed considerably since we first showcased the concept at Apple’s OS3 announcement event. We’re fast-approaching gameplay Alpha with all the weapons in and playable, a handful of really fun maps, an identity system, ranking and matchmaking from our global servers, a great new look and a whole lot more.

One of the OS3 features I blogged about previously was in-app commerce and its role in this and other games. There’s been quite a bit of thought on that subject as well as some decisions and this seemed like a good time to share where we are with it.

First I’ll talk about identity in our FPS game, because its related to our approach to in-app commerce.

Because our FPS is a dedicated online-only game (over 3G or wifi), every player will have a unique login (account) with an associated identity inside (and outside) the game. Your in-game identity starts with your account name, and is further defined by your cumulative performance stats (kills, exp…) and Rank. Your stats and Rank, when compared with other players’ stats and rank, determine a relative skill level used by our matchmaker to ensure players are matched up with other players of similar skill whenever possible.

These stats are presented in-game as your “Combat Card”, a stat summary UI browse-able in the pre and post-game lobby as well as in the armory (more on that in a bit). So if you’re playing with someone really good, you can easily tap their name or avatar to open their combat card and study their stats or check out your own card to keep tabs on your performance. Each rank has a unique insignia as well, so you can quickly identify higher ranked players on sight during combat or in a lobby.

Your Name, Combat Card and Rank Insignia are the bases of your identity, but we’re taking it farther than stats sharing. You can also choose your own armor set, which is reflected in the appearance of your in-game avatar, in the lobby and on your combat card. When we started experimenting with custom avatar visuals, it really stood out as the most powerful way to enhance your in-game identity. We’re planning to have certain armor materials that are only available to players of a certain rank, so if you see a guy with black armor, you better watch your back.

And this is where in-app commerce started to find a comfortable fit with our game design. You’ll unlock new ways to personalize your appearance (mainly material color changes) as you play the game and rank up, but you’ll also be able to visit the in-game armory and purchase cool new armor sets via in-app commerce. These purchasable armor let players choose an appearance that better suits the identity they want to project (a heavy armored dude, amphibious suit, stealth, etc). The armor sets have different silhouettes and material schemes than base armor. We’re working with some of the artists form the original Halo™ team and the results have been very encouraging.

We’re also in the process of experimenting with gameplay-affecting attributes for purchasable armors. This feature is in the early stages of testing for balance and is something we may or may not ship with, but the design idea is to differentiate the armors sets by speed, protection, physics and other properties that align with the concept of a given armor. So a heavily armored combat suit would take more damage than base armor, but would also come with slower movement. The intent is to give each a bonus and a trade-off, rather than a complete upgrade, to keep things in balance with the base armor set. Another idea we’re discussing is giving each armor different modifiers to power-ups (either instead of or in addition to base stat modification), so a heavy battle suit might enjoy longer bonus from damage power-ups and shorter bonus from a jet-pack powerup.

The armory feature lets you change your armor whenever you want between games, so you can play as an armored guy all the time or switch between a variety of armor sets at will.

So that’s the latest on our FPS design as it relates to in-app commerce. We have more to say in the coming days – about weapons and gameplay…. and the name! We had a working title when we first announced the game, but now that we’re so much further along and the features have evolved from cool ideas to great gameplay, we’re revisiting the name. The quality of the game is obviously the most important thing, but we want to get the name right too – something players can identify with and we can take forward with this and future FPS games.

Stay tuned for that.

Cheers and good hunting,

-chris & the ngmoco fps team

Twitter Games

Fri, 15 May 2009 10:50:06 -0700

Our Twitter channels filled with Star Defense chatter last week after we sent journalists a preview build of the game. We were pleasantly surprised by the interest their flurry of challenge tweets generated. People started inquiring about the release of Star Defense on forums and on Twitter, and traffic to the game’s website nearly tripled.

By now, we all know the power of social networks. News, memes, trends, and drunken utterances spread like wildfire across Twitter and Facebook. By integrating our games with these platforms, we’ve created an easy way to become a part of that conversation. We’ve added virality and social interaction to a game that is otherwise a self-contained, single player experience.

This not only adds value to the player, who can now challenge their friends and foes to high score contests at the press of a button, but it also has the potential to increase play sessions, extend a game’s natural life cycle, and even generate new sales.

When we first dreamed up the idea of linking Twitter and Facebook to our games, it was really nothing more than a “Wouldn’t it be cool if…?” We were planning to update WordFu and Topple 2 with asynchronous multiplayer via email exchanges - just like playing chess by mail, only substituting the neat shorthand Rhd4 for a really long URL packed with the necessary challenge data. Then someone suggested we try broadcasting challenges over Twitter and Facebook.

We had a Twitter prototype working in two days.

The Challenge
The biggest barrier to using these social networks to virtually toss gauntlets to the floor is the services’ self-imposed character limits. A typical challenge URL for WordFu, which has to convey high score, challenge time, and available letters, looks like this:

http://wordfu.ngmoco.com/challenge.php?q=Bg
EAAFyaAQAtAAAAAAAAAAAAAAAAAEEMmUPC4EoYwj5L
8wAAAAABAAAAAAAAAMK0AABCDKCeQvfWw8KYBQIAAA
AAAgAAAEK0AAAAAAAAQxYIZsLZv_JCA2eBAAAAAAMA
AADCtAAAAAAAAENNErnCUazrwqGy9gAAAAAEAAAAAA
AAAEK0AABDdOv0QmKmz8Iy86gAAAAAAgAAAEK0AAAA
AAAAQwP17cEwF2VDBdDfAAAAAAMAAADCtAAAAAAAAE
KuaQVC0p4eQon1IQAAAAADAAAAwrQAAAAAAABDjXGN
wvq-skMD724AAAAAAQAAAAAAAADCtAAAQ5AbvMIkLH
1Coh89AAAAAAIAAAAXAAAABgAAAGF3aXNlcgAAAAAA

A string of 421 characters works fine in an email because you can hide the address itself in html, but it is way too large for the wee 140-character tweets. Star Defense challenges—which only have to transmit the number of waves survived—are a bit more manageable at 92 characters. But that still leaves little room to provide context for a Tweet or Facebook status update.

Our solution: Use Tiny URL to compress the challenge strings. The Tiny URL API allows us to shorten URL’s in an application’s background, ensuring that players never have to exit the game to issue challenges.

With the URLs taken care of, it was just a matter of hooking our challenge broadcasts into the Twitter and Facebook APIs so the messages were properly delivered to their respective channels.

But getting the message to these services is only half the battle. Both Facebook and Twitter require developers to represent them in a particular way and to use specific terminology when describing their services. Following their guidelines will avoid copyright headaches down the line.

Additionally, people will be clicking these links from all manner of devices – not just iPhones. The last thing you want is to be inhospitable to a potential player who has clicked on a link from his computer. We resolved this problem by sending every user to a splash page that detects what device they’re using. If you’re on the iPhone, the page will open with links that launch the game. If you’re not on your mobile, it will send you to a Web page filled with game details.

The Impact
Right now, we haven’t collected enough data to determine how well our Twitter and Facebook challenges are resonating with players. The social networks were integrated into WordFu (on April 21) and Topple 2 (May 7) after the games had already launched. Neither has been integrated with the social networks long enough to make any grand conclusions.

With that in mind, here’s what we’ve learned so far:

When the Topple 2 1.2 update hit the App Store, the number of session starts spiked upwards—just as they did with the release of 1.1. Not enough time has passed to determine whether the challenges will prolong this effect.

The number of unique players has increased since the integration, but it’s difficult to discern what role the addition of Twitter and Facebook challenges played in that.

Half of all Topple 2 challenges are sent through Facebook. Email ranks second. Twitter is used the least.

Twitter challenges account for 75 percent of all challenge backs. Challenge backs are when a player accepts a challenge, beats it, then posts their winning score back to the challenger. Although it has the smallest user base, Twitter is the most active when it comes to challenge dialog.

Star Defense will be the first of our games to launch with Twitter and Facebook challenges built in from the get-go. It will be the first true test of how social network integration might affect gamers’ play patterns.

Our hope is that it will keep people engaging with Star Defense in new, interesting ways – and really give players a cool reason to play it more over time. We’d love to hear your thoughts.

- Elio, Allen, and Stephanie

I hate flying. More specifically, I hate landing. It’s scary to plummet down from the...

Sun, 10 May 2009 20:35:59 -0700

I hate flying.

More specifically, I hate landing. It’s scary to plummet down from the safety of your 36k-foot cruising altitude and slam back down to the ground. The ground is hard and real and immovable.

To put it another way - nobody ever died from bad flying, but bad landings are another story.

Finishing a game can be kind of like landing that jumbo jet. There’s a lot of safety up there in the clouds, in the realm of goals, ideas and intentions. Landing those ambitions is a different kind of challenge.

And just like in real life, it’s sometimes easier to circle the airport for just… one … more … loop…

Just like landing a plane, when you are closing out a game you start to really feel the physics of flight more acutely. All those design ideas that propelled you for months? They now push you back against your chair in an acceleration of bug fixes. That marketing strategy you thought was going to guarantee a safe touchdown? Tradewinds have pushed you off-course – time to re-calibrate and adjust trajectory to focus on that sweet spot of game appeal your potential players need to understand.

Touching down a hit game means flawless coordination and trust on a team – so many tiny parts have to function perfectly for that jet to kiss the runway with just the right amount of speed and timing. Tiny decisions across a network of dozens of people combine in sync to prevent disaster.

OK – so launching a game isn’t as dangerous or heroic as being a pilot, but hey: you get the point. In an App Store with 40k+ choices, the difference between a good landing and a bad one can mean the difference between arrival and survival.

Skies are busy at ngmoco these days.

-Matt

Localized iPhone Fonts

Tue, 14 Apr 2009 18:10:55 -0700

Typically during production, adding support for non-ASCII character sets is not exactly a high priority for developers (even though it might be for publishers). But unlike most consoles, the iPhone delivers a nearly complete solution out of the box.

And I’m not referring to UIFont. While its results are solid, UIFont is not only slow to load but also carries a heavy performance cost when used in an OpenGL context.

The key is to use the same system that underlies UIFont, which is the Carbon-based CGFont. CGFont provides the same results as its ObjC counterpart, but also gives us a way to minimize the runtime cost. This can be done by creating glyph textures by using a CGBitmapContext as an intermediary between CGFont and OpenGL.

There’s a catch, however: we need glyph indices in order to use this function, and CGFont doesn’t give us any way to convert characters into their corresponding glyph indices. One way to get around this is to use the open source font library FreeType. FreeType has a flexible license, and can be built for the iPhone fairly easily.

The iPhone provides one more thing that we can take advantage of: localized system fonts. These can be found in /System/Library/Fonts/Cache/, and can save valuable time and disk space, as Japanese fonts alone can easily weigh into tens of megabytes. At the same time, there’s also a risk involved: Apple may move or withdraw access to these fonts at any time, so if you use them, be prepared to provide an alternate solution at short notice.

- Andrew

GDC 09: How connected players are changing our game

Thu, 02 Apr 2009 19:39:00 -0700

I have been to several GDCs; however, this was the first year I attended the Mobile sessions of the conference. Being new to ngmoco:) I wanted an immersion into the world of Mobile. What became clear is that platform / device-based walls in the industry are quickly dissolving due to the convergence enabled by connectivity.

Flash quiz: do the following descriptions relate to Console, PC, Handheld or Mobile games? Social, Viral, Downloadable, Multiplayer, Live, Virtual World, Leaderboards, Free to Play, In-game Currency, Micro-transactions, Community…

In addition to building games for a device by taking advantage of unique hardware features, my observation is that game makers are also converging on common features to appeal to the connected player.

And I’m not the only one… One interesting session that I attended was led by Kristian Segerstrale of Playfish. I love Pet Society, btw. The topic: Five Lessons from Social Games that Matter to the Rest of the Games Industry. One assertion is that games will become services as digital distribution becomes prevalent. As games migrate from product to service, the business of developing and publishing games will change, regardless of platform or device. Here are a few examples of the changes that Kristian spoke about and what I know from my experience in the industry:

— Revenue curves aren’t what they used to be. The revenue stream morphs from having a spike with launch sell-in and a tail of catalog sales to a gradual buildup due to a lower barrier to entry (free or low initial price) to attract a greater volume of players from which a certain percentage will convert to paying customers over a longer time period. The once straight forward revenue model of units sold times sales price becomes obsolete.

— Secondly, the cost of development gets extended to fund ongoing content creation, live support and community management, which astonishingly could shift between 50%-80% of development effort post launch. What once was an upfront investment / cost management focus is now a longer view of operating margin.

— Marketing is not excluded from these fundamental business model shifts. Offering connected experiences grants game makers the holy grail of direct access to their customers. With online registration and play time, mass marketing converts to something more personal — targeted messages to the individual as we are now armed with specifics of age, gender, location, game affinity, who their friends are as well as play and spend patterns. Customer acquisition, retention and monetization all become numbers driven challenges. Data mining and metrics become critical business management tools to make real-time decisions that support customer demands and predict behavior patterns. Correlating numbers and statistics becomes a superpower to be envied (and with great power comes great responsibility).

— Lastly, as a business person I won’t get into game design; however, another important consideration is that games as a service offered to connected players will be multi-platform / device agnostic which adds a 3D view to the changes impacting our industry, how it is bringing us together and how connected players are changing our game.

- Christina

GDC 09: When was your "Golden Age" of games?

Tue, 31 Mar 2009 18:27:52 -0700

I must say that my first GDC was a great experience. As ngmoco’s sole intern and this year being my first in the game industry, I felt it was my responsibility to soak in as much information as I could. I attended a lot of great lectures about Production, but Mythic Entertainment’s Paul Barnett gave a great speech on the last decade of game design and some great insights on how we, as developers, can strive to make better games in the future.

I was actually fortunate enough to be able to meet Paul a couple of years ago during my undergraduate studies. He was an inspiration for me and confirmed my aspirations to pursue a career in the game industry once I had graduated. I was able to spend some time with him aside from his lecture as well, where we were able to catch up and talk more about his ideas that he presented in his lecture.

In his lecture, Paul’s main point was that we are finally in a great position for the game industry to really thrive. Independent developers now have more outlets to express their creativity and make great games. He explains this through a few key points:

Paul first pointed to the App Store, Xbox Live Arcade, and now Wii and DS Ware. Everyone now has the potential to make great games, even right from home. He went on to explain that we are not limited to the “console gods”, where only big name developers have the true ability to release games. While many of these games by independent developers may not reach a large audience or be popular on a large scale, many of these games inspire us to think in new and innovative ways.

Paul also talked about our “Golden Age” of games; this is a 10-year period (more or less) when we started obsessing about games. While Final Fantasy VII may be one of the most influential games in my “Golden Age” (I’m still relatively young), it may very well be Spore for my 7-year old cousin, who may have just entered his golden age. I could show my cousin Final Fantasy and he may dismiss it right away. He may not understand why the graphics are “not as cool” or why there isn’t any voice acting.

When talking to people about your golden age, Paul pointed out that your history is only really relevant to you, and it’s just a curiosity to others. He went on to explain how our culture and upbringing defines how we think creatively. With the current rise of independent developers, I believe we really see people’s creative genius coming to life with games like Braid and World of Goo. It doesn’t always take a large team and a lot of time to create a great, unique gaming experience.

Finally, Paul said that things are new and exciting when YOU discover them. While my cousin may have tried Final Fantasy and thought it was old or boring, that doesn’t change the fact that it was one of the greatest gaming experiences I’ve ever had. We must use our discoveries and our own golden age to think creatively about how our next generation of games will be, regardless of how others feel about our experiences. Whether it is the most simple or most complex game, we can always discover new and exciting things that we can take with us when making games in the future.

-Brandon the Intern :)

Sailing the App Store

Tue, 31 Mar 2009 18:24:00 -0700

30,000 apps as of today entered the App Store since its launch, up from the 25,000 counted on March 6th. The weekly average of new app releases per month continues to grow and now more than doubles those of Fall 2008.

Of the new apps released since the 25,000 app milestone (March 6), nearly 40 broke into either the Top 100 Paid or Free for the first time, indicating a sub 1% “hit” rate. Of the apps in the Top 100 Free/Paid at that time, over 85 dropped out.

Competition like this convinces some to believe that the App Store is a market where players have come to accept existing structural conditions as permanent and compete on the basis of price in pursuit of temporary monopolies.

Not to be contrarian, but that view may be short-sighted. So how can developers set sail for bluer waters?

1.    Abandon the notion that the market’s median price is a resistance level.
2.    Focus on value innovation in ideation, game design and live tuning.
3.    Articulate the value intrinsic to an app through strategic marketing.

Pricing Rethought

The average and median prices of the Top 100 Paid are ~$2.60 and $1.99, respectively. The $0.99 price is most prevalent across the App Store, but in the Top 100 appears to have given ground to higher price points since the introduction of iTunes 8.1.

Median prices may continue to hover at current levels so long as no structural change is made to the App Store, and yet even if, there remains an economic rationale to release apps differentiated by their content and priced above these levels.

Having a $100 caviar delicacy on a menu, David Edery explained at GDC, is important. It may be the seldom-picked item, but can help facilitate sales of relatively more inexpensive items. Hence, price disparity on a menu matters.

One application of this on the App Store is a title’s release of multiple SKUs at different price points. Let’s look at SGN’s Agency Wars. What’s noteworthy is that the popularity of Agency Wars 20 more than doubles that of Agency Wars Lite. What’s also interesting is that $9.99 is the proper midpoint of an unequal distribution of prices. Both the $19.99 and $9.99 SKUs have made downloads for themselves, and more importantly debuted Agency Wars in the Top 100 Paid (vs. Free).

Relative pricing, as this example suggests, has a potential to expand the Top 100 mix beyond the micro-priced majority. Liberated from the mentality that the market’s median price is a ceiling by which to make cost/value tradeoffs, developers can create more meaningful value for their customers and themselves.

Game Design Innovation

Today’s 30,000 app milestone suggests that the App Store offers a plethora of variety in concept and quality.

If ~25% of the Top 100 Paid apps has a 4.0 rating or higher, and ratings trend toward a 3.0 in a relatively short span of time (days, possibly weeks), is there room for quality improvement across the App Store? Are gamers truly getting the experience they care for most… Fun? Identifying their needs and likes/dislikes is where value innovation in ideation, game design and live tuning begins.

Chris Swain and Dan Arey shared insights at GDC about the science behind the art of game design. Aside from outlining common metrics (e.g. speed of level progression, ease of gesture-based controls), they encouraged seeing data patterns as fugues à la Gödel, Escher, Bach and applying the psychology of daily well-being to game design. Game design should be examined for the needs it strives to address: Autonomy, Competence and Relatedness (which supports the former two).

Relatedness represents a particularly exciting opportunity for iPhone gaming. Location-aware games with push notification and microtransactions have a potential to make connections with others uniquely meaningful.

Strategic Marketing

Ahead of the release of iPhone OS 3.0, it is worth revisiting the history of the Top 100 to glean how apps made their mark.

Smule’s Ocarina and Bolt Creative’s Pocket God serve as excellent examples of how value can be created in new ways and marketed effectively. Ocarina clocked in a consecutive 20 days and Pocket God a total 10 days at the Top 100 Paid’s No. 1… Nice.

Smule introduced an interactive sonic app and created a new form of Relatedness with Ocarina, where players’ creative expressions could be shared across the world. Its screenshots made use of the app look easy (no musical training required), and its music video captured the harmony of an ensemble. The YouTube video contest, “This Contest Blows”, created viral marketing well aligned with the app’s theme and helped extend its life in the Top 25 over the winter holiday.

Bolt Creative marketed Pocket God with an open commitment: “We aim to update every week for the next few updates”. This episodic microgame offers its audience a feeling of Competence and an ongoing reason to build on that and have something to look forward to, like a return on time invested. Marketing the precise dates of these content updates is one strategy that sets Pocket God apart from iFitness (mind you, iFitness caters to a specific market). iFitness also has a pattern of software updates, but Bolt Creative’s marketing makes its commitment to value more explicit.

Ready for iPhone OS 3.0?

In sum, these takeaways can help retune our compasses before sailing into the iPhone OS 3.0 era. As developers and publishers, shifting our focus away from today’s common pricing tactics and toward value innovation supported by strategic marketing has the potential to unlock new demand and provide greater value to App Store customers.

It’s time to use game design metrics like true number ninjas and get ready to embrace iPhone OS 3.0. With social gaming and microtransactions in the form of level packs and virtual items, games will become more than the first impression and need to upsell their services. That in itself represents a new market opportunity for content differentiation.

This new wave of innovation will in turn give more importance to the strategic value of marketing. For the same reason the App Store’s review system was updated with iTunes 8.1, information has value. Price tags, game descriptions, media assets and other product information are intended to make the market more efficient.

There is room for more creativity in game design and marketing. Our community has an interest to take the longer term perspective, where benefits can accrue to both buyers and sellers. Jibe ho.

Many thanks to the aforementioned GDC speakers for their insights and time:

David Edery (http://www.edery.org/), Chris Swain (http://cinema.usc.edu/faculty/swain-christopher.htm) and Dan Arey (http://cinema.usc.edu/faculty/dan-arey.htm).

-Marie