We Survived an Influx of Roughly 187,000 Reddit Users Caused By an Old Post on Secret Coca-Cola Millionaires

As tends to happen once every year or two, the unexpected “make the front page of Reddit” event occurred last night, in the midst of the site renovations; an inopportune time but I’ll take it.

Aaron and I were watching Jonathan Strange and Mr Norrell on iTunes at around 2 a.m. when I saw the MySQL server straining under an incredible amount of requests.  It was the old post about the secret Coca-Cola millionaires in Quincy, Florida.  Something like 90 requests a second were slamming the data center up in Michigan.  Traffic source: Reddit.  Short investigation later, turns out it became the number one post of the “Today I Learned” subreddit and made the overall front page of the site.

I got in touch with the webhosting engineers, one of whom began to work her magic.  The team that put time in on the project today upgraded some PHP things and switched me over to an alternative setup they explained that should allow exponentially more connections whenever it happens again.  Concurrently – and I have no idea how or why but for which I am extremely grateful – the co-founder of Cloudflare, which we use as the CDN, sent me a tweet telling me to set a page cache for the specific page that was receiving the huge traffic influx, which I was able to do so it made no MySQL requests at all.  Within a minute or two, the live visitors had doubled.  Shortly thereafter, tripled.  It was crazy.  I was watching 1,400+ people from around the world in the middle of the night at one point browsing the site (alas, I missed the 7-9 a.m. spike when it increased by several fold because I finally had to catch a few hours of sleep … I’ll need to check the stats to see how many people were on actively reading at one time).

The good news is that this diversion resulted in permanent improvements on the site I probably wouldn’t have gotten to otherwise, especially as I work to rationalize the javascript and css so it results in a mere fraction of the server requests going forward.  The regular community is going to enjoy much faster load times than we otherwise would have; hopefully, fully visible by Autumn.

The behavior patterns were odd, though as these folks were not like you all:

1. They all swooped in, sustained for nearly twelve hours, spending a couple of minutes on the page that was getting the attention, then left.  Practically nobody looked at additional pages or clicked around the blog itself.  That is not how I read the Internet at all.
2. Almost no one left a public comment despite 937 comments (at this moment) on the Reddit page itself discussing the post.
3. Several people in the discussion thread on Reddit were making some derivation of the inquiry: “What’s the next Coca-Cola?”  I broke my usual rule and logged in to the official joshuakennon Reddit user, which is not the account I normally use, to leave a response to one of them, which you can read here.

Look at how beautifully Cloudflare handled the whole thing … of the 303.63+ Gigabytes in bandwidth served over the past twenty-four hours, it took care of 95.4% of it, leaving only 4.6% for the server itself to handle.  When the site went down a few times for a minute or two, the “always on” feature served a static, non-live version of whatever each visitor was reading so the user experience wasn’t interrupted terribly despite 187,000 or so individual people showing up.

For all of the trouble (and my exhaustion today given I barely closed my eyes before having to get up to go to work), we blew the previous single-day advertising record out of the water as those page views turned into tasty tasty profits.  The Reddit folks like to click ads.  I feel as if I should use it to buy more Coke stock.  It would be appropriate, if not entirely rational.

I have to say, I found myself feeling … uneasy … about the whole thing.  It was as if our little (or not so little, anymore, as most of you regulars are lurkers) community was overrun.  I didn’t like it.  It could have been, too, having the most personal post I’ve ever written on the sidebar and front page, along with another post that had a picture of my nephew (update: I have subsequently removed many family pictures as the site has grown while trying to salvage the majority of the old posts).  It made me feel uncomfortably exposed.  We’re back to normal.  They’re mostly gone (though any of you who stuck around permanently, welcome!).  All is well.  They got their “Today I Learned”, I got their money.  It was a fair exchange.