About

All articles, tagged with “scalability”

Following In My Father’s Footsteps

Tintin Hair

From 2 years before I was born, until just before I started working on Second Life at Linden Lab, my Dad worked at an innovative technology company with a large consumer photography business: Kodak. From January next year I’ll be working at an innovative technology company with a large consumer photography business: Facebook.

Looking at the march of technology from the perspectives of these companies is amazing. I had a summer job building my first web application for Kodak while I was studying Computer Science in Nottingham and remember one of the researchers there joking that film needed to last a long time as a roll would often have pictures of christmas trees at either end with a summer holiday in the middle. Photography was so expensive that people would only take a few dozen pictures a year. Now we happily take a dozen pictures of our lunch, wouldn’t consider buying a telephone without a built in multi-megapixel camera and people upload hundreds of millions of images to Facebook every day.

While the cost of creating photos has fallen to almost zero, their value hasn’t. Some of my most enjoyable moments recently have been looking at and commenting on the latest pictures of my brand new nephew, Charlie, on Facebook and so it goes for Facebook’s other billion users. Photos that are now so cheap to create that Kodak has filed for chapter 11 protection become social objects that are so valuable that Facebook can host the photos for free and still make a good business from advertising around the conversation.

Working for Facebook might seem like a strange move after a decade working on 3D environments, but virtual worlds like Second Life and EVE are also social spaces, just with virtual nightclubs or space battles as the social objects. While 3D environments allow more immersion than Facebook, the price is a much higher barrier to entry. Although a few people from my family tried Second Life while I worked at Linden Lab: most of my family use Facebook already. My brother could create a gallery of pictures of my nephew in Second Life and we could meet there to talk about them, but then most of my nephew’s other aunts and uncles wouldn’t be able to join us. Ubiquity trumps immersion. Virtual worlds like Second Life still need their iPod moment if they’re going to cross the chasm from niche technology used by gamers, early adopters and academics to become a mainstream communication technology. Even though Second Life is free to use and paid for by the publishers of the 3D content, it’s still too hard to navigate for most people to use almost a decade after its launch.

Facebook is already used by a billion people to keep in touch, while still evolving and developing at an incredible pace. I’m going to help new uncles connect with new nephews around the world while working on new technologies, which I think is going to make Facebook a fun and rewarding place to work.

After that, who knows? My Dad’s working on some pretty amazing stuff these days: if I keep following in his footsteps and change keeps accelerating, the next thing is science fiction now, just as Second Life and Facebook were in 1975.

@scalecamp

On Friday I jumped on the train to London to attend the first scalecampuk, an unconference about scalability, at the Guardian offices.

The sessions were all very interesting and mostly very relevant. I learned new things about XSS and CSRF and Django’s defences against them from Simon Willison, new things about messaging, pubsub, queues and data stores (process 1 message at a time, use message hospitals, send URLs to unavailable data that can be polled for with JavaScript and that just check memcache entries) lots about Varnish ) and it’s use at Wikia from Artur Bergman (Wikia runs off 18 apaches and 8 varnishes with 60GB of RAM and SSDs to serve 25 million pages and 950Mbps at peak, Varnish is generally better than squid ), but you need a modern kernel).

Lots of the talks were about moving storage, caching and queuing out of the application and just writing a small piece of business logic to tie them together. Against this background Alex Evans’ talk about the back end for Media Molecule’s Little Big Planet stood out like a sore thumb. Having not enjoyed using a Java web stack, Alex has just rewritten the whole of the back end as proprietary technology as a single binary in order to know the code from end to end. While it may be true that having custom physics or rendering middleware might distinguish Little Big Planet from other games, I can’t believe that custom technology to serve HTTP requests is going to be a competitive advantage. I hope Alex’s good ideas become Redis contributions rather than a maintenance nightmare and barrier to agility.

The lightning talks were also very good. Simon’s “ScaleFail” talk about the Guardian MP expenses app was hilarious (lesson: do load testing) and Gareth’s talk about Dumbo (a Python Hadoop client) was very useful.

At times it felt like the talks were suited to an ops audience, but “Dev’s should know about this!” was a regular refrain. Don’t worry: I listened and learned a lot. Thanks to everyone who made it a great day.