Startup Stories: The Time I Almost Ruined Christmas
Back in late 2000, I was running Network Operations for Volera, which was a nascent joint venture spinout created by Novell, Nortel, and Accenture. Novell had acquired JustOn, a file-sharing and cloud applications company I was a co-founder of.
(More on that side of the story in another post….as my inspiration for sharing this one is Cyber Monday and the upcoming consumer-driven holidays.)
Volera was an exciting, and fast-moving Content Distribution Network (CDN) company. We had industry-leading tech and a unique approach vis a vis the other CDNs at the time. We made websites a lot faster, helped them scale without adding webserver hardware and had a global network of our own super-fast systems up and running. Naturally, the best way to show a large website operator how much value you can deliver is to demo their own site for them, right? Here’s your website…and here’s your website on Volera. Boom!
First, a little background on what was going on at this time. A persistent problem with the internet back in the very early 2000s was web visitor capacity. This was pre-cloud, pre-AWS, pre-DigitalOcean and pre-pretty-much-everything but bare-metal in a rack if you needed performance. For every 150–300 additional website visitors you needed to support, your website or web app likely needed another server. This meant an expensive piece of hardware, loaded up with memory and taking up power and rack space in your datacenter. Each connected client (browser) tied up a chunk of the web server, and thus it's very finite memory. The issues compounded back then with how web sessions were handled and how slow clients were to connect and obtain data. A great example of this, which some of you will remember, was the problem of getting “Crunched” — the dubious honor of getting covered by TechCrunch only to immediately see your website go down due to the traffic load.
Volera operated a global network of extremely fast reverse-proxy servers. We were seeing huge improvements in website speed and reductions in server load when our CDN sat in front of a website that had a lot of static content, video, or large objects. A/B testing a website without our CDN in front and with was the best sales tool we had. The benchmark differences were substantial and we were a best-in-class caching appliance. When it came to being a best-in-class hosted CDN, well, we were still learning…it was all very new.
Back in 2000, there was no CyberMonday. eComm was new. Amazon was just beginning to sell other items in addition to books. eToys was the hot .com in the eCommerce space and their website was getting slow around the holidays. How did we know this? Because of our most recent Nerf gun purchase for the office (turns out this is still a thing 20 years later, good for Nerf!).
We realized eToys was a great sales lead for Volera so naturally, we started running some tests to benchmark the gains our CDN could provide. Well, in our haste and excitement we made a few unforced errors in our approach to testing….
The largest one was that rather than run a totally contained test with one cache, we provisioned their domain on the test side of our production system. The minute we started testing, we inadvertently triggered our global caching network to pre-fill and synchronize every page and object on the site that allowed caching. To the network and security team at eToys, what this looked like was basically a DDOS attack. High-speed servers on the internet backbone pounding their site from all over the world. Not good. While this was going on, we were running tests to ensure we hadn’t broken their shopping cart by using caching. How? By ordering more Nerf guns of course! Unfortunately, we totally missed that our entire network of caches were all doing the same thing at the same time. We had pretty good monitoring, operational process, and data across our platform for a young company but this test was hasty and we skipped steps. Too much startup hustle, not enough planning. A major SNAFU.
The all-star team at eToys was, rightfully, very pissed and concerned. They did the right thing and blocked our IPs from continuing to pull content from their servers, and managed to get ahold of us. I was in my 20s at the time and responsible for the technical operation of the CDN network. I remember being pretty freaked out and realizing that this aggressive move to show them how great we were could end up with a lawsuit, me out of a job, or both! I think the term for my mental state at the time was “shi**ing twinkies”, but I knew I had to take responsibility for the issue. A few days after the incident I remember, vividly, the call with the eToys team. I’m pretty sure I was probably shaking and sweating while pacing around my desk. After a rough 10 minutes of listening (er, getting chewed-out), it was my turn to speak.
Rather than be defensive and try to justify our actions or make excuses, I just agreed with them. I flat-out acknowledged how stupid and poorly executed our work was in that moment. I took responsibility and told them that if there was material economic loss we’d address it. I explained where we had made our mistake and why it had happened, transparently. I recall a long moment of silence on the line, that extra beat where you are pretty sure the other side has muted you before the eToys team came back and said something I wasn’t expecting. I don’t remember the exact words or who from the team said it to me, but the gist was “You are the first tech company we’ve dealt with who has been honest about a screw-up. We are still beyond pissed and may pursue restitution, but thank you for taking responsibility, and no, we won’t be doing business with you.”
In the end eToys went on to have a great holiday season (only to declare bankruptcy in Feb of 2011). I kept my job and Volera went on to grow nicely until the crash of 2002 arrived and the bottom of the entire industry dropped out. It was game over for us too.
I learned two big lessons from this episode that have stayed with me since then. The first lesson was around integrity and doing the right thing when it matters most, even if that means its not the best thing for you personally. I knew I was a steward of the company’s brand and reputation in the market and the hard work and capital everyone else had put in was on the line too. It was pretty obvious we weren’t going to win that business, but the reputational damage could have been 10x worse for us had we handled that differently.
The second lesson is around the value of transparency and authenticity in business. We humans are not perfect, and yet it is our incredible minds that can turn ideas into reality. Sometimes remembering that we are human, even in a tough business context, can be powerful. When we have an absence of information or don’t trust the source, our brains go to dark places. Fear, uncertainty, and doubt dominate our minds and our ability to “assume good intent” can wane quickly. We lose the ability to be rational. I found that being transparent and open about mistakes humanized the situation and reduced the negative energy. It's easier to be mad at a thing, at a company, or a policy than directly at a person who is doing the best they can. This kind of transparency is not always possible in business for legal or risk-management reasons, but I believe there is more room for this than we realize!
The eToys episode is one of the many vivid memories I have from the early days of my career because of how big it was for me personally. I screwed up and hurt another business. I could have lost my job, cost my company money (oh I did that too, will share that story also), but in the end learned some great things. The digital world looks very different today than it did in 2000 too. Happy Cyber Monday! (belated, since it took me an extra day to edit this!)
