I wrote a project to learn d3js and to find video games that aren't always mentioned on "Best of" lists. I started with the Super Nintendo because it is the greatest console of all time. To do this I made a simple spider to crawl over GameFAQs and collect review information. This is what I ended up with.

I submitted the results to r/dataisbeautiful and got some great feedback. Some popular requests asked for more modern systems and filtering by sub-genres. For me the height of the action was watching Google Analytics realtime stream. At one point I had over 100 active visitors on the site, and I was running all around the house screaming with excitement. I pounded F5 all day to read any new comments and watch those upvotes!

Mobile

After it was all over I looked through the demographics and noticed a HUGE concern. Mobile browsers hitting the site accounted for roughly 25% of all traffic. I never thought about mobile. I quickly loaded up the site in my iphone and tried it out. Nothing. The controls don't work and the pages look weird. I wouldn't look at this site twice. This HAD to be fixed.

Adding touch support

First priority, get that interactivity working. This was a lot easier than I thought. Drop in touch punch for jQuery, have d3 listen for touchstart, touchmove, and touchend. Boom. Done.

Get more datasets

Next get more data for those PS3 gamers. Using the scrapper I had setup for the SNES this was no problem. I spent more time worrying that GameFAQs might not like to see my spider crawling over their site all day. So I spaced out every system collection by 12 hours.

Submit to r/Games

Time for round 2. Armed with more graphs and better mobile support I felt I had a good chance getting the love of a large gaming audience. Unfortunately things didn't go as well as I had hoped. This sub gets a lot more submissions and my post ended up taking roughly 6 hours to get out of the new branch and onto the front page of the sub. Compared to the 1 hour it took for my first submission.

Where is PC?

Ok, it finally hits the front page of r/games. Now everyone wants to know. How come I got all this info, but I didn't include the most loved platform on the internet. To be honest, it's because I'm scared.

The PC has a long lifespan encompassing TONS of games. Scraping all of those titles is something I don't want to do without thought. Why am I so scared then? Because scraping is in a little bit of a legal grayzone. I wouldn't do this if I thought it was illegal, but if GameFAQs can prove I am harming their servers with my spider I'm going to see trouble. Realistically I am nowhere near causing any impact to a site which gets hundreds of thousands of daily page views, but it's good to be careful.

In the end

This was a LOT of fun. I learned d3js, felt like a badass collecting data, got 18k unique page visits, and got some sweet sweet karma. I'm not sure what my next project will be, but mobile is definitely not going to be an after thought.