Have you heard Weston Jossey explain his company's hybrid cloud strategy before? If you haven't, get comfortable and enjoy the recording of our most recent OpenStack Podcast. In it, Wes explains why his company chose to deploy a private cloud, what they use it for, and how they use it in conjunction with AWS to achieve maximum efficiency.  Seriously-if you're looking for tips about how to do hybrid well, this is the podcast for you. Wes makes it easy to understand both the upside and downside of private clouds in general and OpenStack-based private clouds in particular. Plus, he's extraordinarily honest about what OpenStack is good for and what it's not so good for. Among other things, in this interview, Wes discusses:

• Tapjoy's big data architecture
• How they handle 5-10 billion transactions per day
• The TCO of their OpenStack-based cloud
• How he finds great hires in a competitive market
• How Tapjoy automates deployment

Have a show idea? Tweet Jeff and Niki at @openstackpod

See past episodes, subscribe, or view the upcoming schedule on the OSPod website.

To see the full transcript of this interview, click "Read more" below.

Niki Acosta: Hello, Hello, Hello! It's Niki Acosta from Cisco filling in for Jeff Dickey from Redapt, and I'm so excited about today's guest, Weston Jossey! Introduce yourself.
Weston Jossey: Hey, Niki! Thanks for having me on. As you said, I'm Wes; I'm Director of Engineering over at Tapjoy. Man, I guess, Niki, you and I have known each other for what, almost coming up on a year...
Niki Acosta: Yeah!
Weston Jossey: ... since I started working with Metacloud and doing a bunch of stuff on OpenStack.
Niki Acosta: Yes, and we're really excited! We've received a ton of feedback from our listeners. Thank you to our loyal listeners, who said, "Man, it would be really great to hear from more users." So, we reached out to Wes. Wes has spoken at an OpenStack, two OpenStack Summits, right?
Weston Jossey: Yep. Well, no, one OpenStack Summit and then recently spoke at Inter- or not Interop, spoke at-
Niki Acosta: Cisco Live.
Weston Jossey: Cisco live in San Diego, yeah.
Niki Acosta: What was that like to speak at the ... That was the Paris Summit, right?
Weston Jossey: That was the Paris Summit. Yeah, it was terrifying, is probably one way to define that. We flew out to Paris on a Sunday night and I spoke on a Tuesday night and I was still majorly jet-lagged by the time I had to give the presentation on Tuesday morning. It was great; it was a gigantic amphitheater. Where were we Niki? It was ...
Niki Acosta: The Palais de Congrès or something.
Weston Jossey: Palais de Congrès, yeah. So when they walk you in and you have to do your rehearsal the day before, you're going through the same entrances where all of the theater people normally walk through, with gigantic posters everywhere of like all the different shows they've put on, all the different artists that they've actually had there, the gigantic concerts. Then you ride up this elevator into the back of the amphitheater and then you walk out on stage and it's, what, 2, 4, 6,000 seats? I have no idea how many seats are in that place, but it's just these gigantic lights beaming down on you and it's completely empty. They're like, okay, now, do what you're gonna do tomorrow, but for an empty room. You're just absolutely terrified!
Then yeah, gave that talk in front of thousands and thousands of people and hopefully it resonated the story of how Tapjoy came to be involved in OpenStack and our journey to get to where we are.
Niki Acosta: You geeked out meeting Tim Bell back there, didn't you?
Weston Jossey: Yeah, so you had Tim Bell on, what, a couple weeks ago when you were up in Vancouver, is that right?
Niki Acosta: Yes.
Weston Jossey: Yeah, and if anybody hasn't had the opportunity to meet Tim Bell yet, one, probably one of the nicest people in tech, right? If we had a "Nicest Person in Tech Award," Tim Bell would at least be nominated, if not be the winner. He gives tons of great talks about the story of CERN and how they started using OpenStack and how they've integrated into their technology. But more than anything it was just great sitting back there and talking to somebody who makes you stare at him and wonder at how he and his team manage to do the things that they do at CERN, and the good that he does for the entire world obviously with the research that they're doing there. It's pretty spectacular in getting to be involved in it!
Yeah, I sat backstage and was a little bit nervous and hadn't actually met him before, but knew who he was, and kind of fumbled my first conversation with him. Then sat backstage after he gave his talk and we chatted for a little bit. Just a really great guy, it was good to see him in Paris. I'm hoping he'll be in Tokyo in the fall. I know he was up in Vancouver. I don't know if you guys fly him, or if the OpenStack Foundation flies him everywhere, but hopefully they should.
Niki Acosta: Yeah, funny thing about Tim, I was talking to him at the last summit in Vancouver, and some of the guys on his team were having some beers. He had a few, but I'm like talking to him about his family and the fact that his wife knows who Niki is, because she follows him on Twitter and I tweet on him every once in a while. I was asking him about life at CERN and his wife is actually a particle physicist at CERN. He was telling me about how his kids go to school with the kids that are basically the other UN kids, and the area that they live, and how everyone hates them, because they kick everyone's ass in the science fair. It's not even a close game!
Weston Jossey: Daddy let me run a quick experiment on the particle accelerator, won me the national prize for the science fair. Yeah, those are the perks of having a father who has 20,000 computers at his disposal, right?
Niki Acosta: Totally! Then he was talking about how right now they're doing an upgrade at CERN, and I don't even remember ... I think he told me, it was probably in Celsius, but right now the CERN Hadron Collider is like the coldest place on Earth right now, because they have to-
Weston Jossey: Right, it takes a while to cool it down, and so they have to do it over the course of weeks and weeks and weeks.
Niki Acosta: So cool! Anyway, fascinating stuff.
Weston Jossey: I don't know; I lived in Boston this past winter and I'm pretty sure Boston was the coldest place on Earth for a period of time, for about four weeks. At least the most snowy place on Earth it felt like.
Niki Acosta: I know; my cousin sent me a picture of her house and she had carved the walkway, and the snow was past her head.
Weston Jossey: Yeah, I'm still recovering. It's a beautiful day outside; it's probably about 78 degrees, sunny, gorgeous, and I still whenever I close my eyes, I just see mounds of snow just above my head, and I see my dog trying to run through it and just disappearing and then popping up and then disappearing again, and then popping up.
Niki Acosta: Where are you from Weston? You're not used to snow.
Weston Jossey: No, that's the thing; I'm used to snow. I was born in California actually. I was born in Los Angeles, but then grew up in Ohio and spent my entire childhood there. We would get snow ... I mean, we would get snow! But like a big winter would be, "Oh, man, we got a foot of snow this year," like over the course of the entire year. It was not normal to get 6, 7, 8 feet of snow like we got this year, and I've lived in Boston now since 2008. I moved here after college, had the pleasure of going to Northeastern University for a year, got into their PhD program, summarily dropped out of their PhD program to go into industry. So I've lived through a couple of Boston winters, but nothing compared to the experience of last year.
I live right next to the Tapjoy offices where I work, and we're about two blocks away, so I can walk into work. I basically have no excuse to ever not get into work, other than today where I'm doing a podcast, because if I did it at the office I'd have people behind me teasing me and taunting me. We had a coworker who lives down in Quincy which is down in South Boston, and he, I am not exaggerating ... He could not get into the office for an entire month, could not get into the office for an entire month! It was absolutely insane, so yeah, you should be lucky and be happy to be in Texas Niki.
Niki Acosta: Because it's like 80% humidity and like 95 today or something?
Weston Jossey: That's what air conditioning is for. It's fun.
Niki Acosta: Yeah, right.
Weston Jossey: You got central air.
Niki Acosta: Totally! I know; I'm wearing a sweater inside, because I'm actually kind of cold in my own house.
Weston Jossey: Well, I'm wearing my OpenStack sweater.
Niki Acosta: Yeah! Oh, my God!
Weston Jossey: I figured I'd do some branding.
Niki Acosta: No, I was so distracted, that I forgot to ask you the question we always start off with. You told some of it, but how did you get started in tech?
Weston Jossey: Oh, that's an interesting question. I don't know; we've got, what, another 50 minutes, so I'll do the next 40 ... no. I've always been a bit of a nerd, as I'm sure most of the people who are joining in on this podcast. I got into computers probably when I was about 8 or 9 years old. I remember getting our first Gateway computer playing Myst in my basement. I was always interested in computers, but my parents didn't really want me to spend all that much time on it. They kept telling me to go outside and go do other things, but I always had showed a lot of interest in it, so in high school my high school girlfriend, her father was a software architect. He actually was one of the first people to ever graduate with a computer science master's from Duke University.
Niki Acosta: Wow!
Weston Jossey: I foreshadowed, I guess it would have been my junior year of high school, I went with him to his work and walked around with him and got to see what his job was like. I remember actually thinking back then, wow, he just did a ton of meetings! That doesn't seem all that interesting ... And because he was a software architect, and software architects sit in a lot of meetings and talk to a lot of people and don't necessarily get to write a lot of code themselves. But really had ... It kind of made an imprint on me. I stayed interested in it, but really didn't do much around software and tech until I got to college, and I actually went to a really small, private liberal arts college in upstate New York, actually where his daughter was going to school. That was part of why I went to New York.
So they had a small computer science program. There were five people in each of the graduating classes, but I actually went in originally for physics, did a semester of it, loved physics, but also took my first computer science class and loved computer science! One of those ... Most kids by the end of their sophomore year of college are still trying to figure out what they want to do in their career and what they want to study, I knew ... Niki, are you still trying to figure out what you want to do, Niki?
Niki Acosta: Oh, totally! I don't know what I want to be when I grow up.
Weston Jossey: Yeah, you'll figure it out. Yeah, I ended up taking a bunch of computer science courses. As I said, it was a very, very small department. I would have classes of ten ... I had one class of one where I was the only person in the class. I had a fantastic advisor by the name of Tom [O'Connell 00:10:10] who's still at the department today, who really mentored me. More than anything, because it was a liberal arts college, it was all about problem-solving, and so you're always trying to learn how to solve problems, less so than you're learning a trade skill. When I was interviewing for a bunch of jobs out of college, I wasn't necessarily doing that great in terms of getting the placements that I wanted. Secretly I just wanted to work at ESPN, but they didn't let me.
So I ended up being able to get into a PhD program at Northeastern, as I said, because I thought it'd be interesting to teach ... Did it for a year, loved it, but just turned out to not be for me, and then kind of jumped into industry. Then I've had a completely different story over the last, I guess now, what, six, seven years of how I got here, but that's sort of my tale of how I got into actual technology.
Niki Acosta: So in a nutshell the last six or seven years, I know you were acquired into Tapjoy?
Weston Jossey: Yeah, that's right. I worked for a company called Viximo; we took social games that were on Facebook, because don't forget, about five years ago social games were really big on Facebook ... Farmville, and we would take those games, work with the publishers and actually port them to other social networks around the world. Because believe it or not, five years ago, there were also other social networks around the world other than Facebook. We would take games and put them onto Spanish social networks like Tuenti, we would take them onto German social networks like StudiVZ, and so a lot of what I was doing was around trying to read API documents in German. That was exciting, trying to get all that stuff to work.
It turns out good business, not a great business, just weren't quite making enough money to justify all the venture capital we had taken. So a sister company of ours, Tapjoy, through North Bridge Partners who is our VC firm, basically said that they thought there was some overlap in skills, ended up making a bit of an acqui-hire, and myself, I think along with nine or ten other people joined Tapjoy. Basically it's a company that's all about helping app developers make money. That's really what it all comes down to. We got acquired by them and then I started my journey from basically just being a regular software engineer to starting to dive into systems and diving into operations. So I basically knew nothing when I started at Tapjoy, when it came to actually how to run systems. At Viximo we had maybe 100 servers, somewhere in there, all running inside of AWS, but it wasn't really something that I was actively involved in. We had two other guys who were fantastic at doing that kind of stuff, so it was all off in the netherworld for me.
But when we joined Tapjoy, they were understaffed on that front. There was one full-time guy running 500 plus servers for their entire infrastructure, and we were on fire all the time. Everything was on fire! We had tons of downtime every quarter, 20 to 30 hours a quarter. It was just constantly like we were just trying to bail water out, and so I'm a fixer. I don't know if that's maybe the best description I have for me, and so when I see a problem I just want to fix it. Myself and this other guy by the name of Eric [Abbott 00:13:34] were just like, "All right, we're just gonna jump in and we're gonna just start working on this stuff." For the next year we basically started the process of overhauling everything we could under the sun, upgrading their databases, getting them onto the latest technologies, making sure that everything was properly automated, really trying to take Tapjoy and evolve it, and so we had a bunch of people working on it, a bunch of great stuff happened. Then started transitioning into what eventually ended up becoming our journey to OpenStack.
Sorry, I'm just rambling, so I'm gonna ...
Niki Acosta: No, that's great. This is great! You're making my job super easy today.
Weston Jossey: Yeah, yeah. Yeah.
Niki Acosta: We don't even need Jeff today.
Weston Jossey: I'm sorry?
Niki Acosta: We don't even need Jeff today.
Weston Jossey: Yeah, well it would have been nice. Ironically, one of the two people that I worked with at Tapjoy who was part of the systems team when I first joined was a guy by the name of Jeff Dickey. I was like, "Oh, man, does he have a podcast with Niki? Is this like a thing that he does now?" A different Jeff Dickey apparently.
Anyways, we got to a point in Tapjoy where things were stable, things were working, but it was still not terribly efficient, is probably the best way that I can put it. We ended up bringing in somebody who had been at Tapjoy early on at the beginning. He understood our systems well, so by the request of one of our board members who was going into every single board member, or board meeting, pounding on the table saying we're spending too much money, we're spending too much money on AWS, we gotta figure stuff out, started to explore a bit of the OpenStack community, and explore what it would possibly look like if we were to build it ourselves. Being a cloud company, being a company that basically was totally invested inside of AWS, totally invested in the concept of elastic resources and being able to spin up VMs and spin down VMs and the core concepts that come with operating inside of a cloud, we didn't feel like it was prudent to go down the route of just running just racks and racks and racks of bare-metal.
So what we decided was that we were gonna basically start to explore a bunch of different partners in the OpenStack space and price it out and try and understand what it was gonna cost if we wanted to use AWS, use one of the bare-metal providers that are out there now, use an OpenStack provider themselves, or try and build it ourselves and put all that together. It ended up being the last two years of my life, basically this journey of how we ended up getting here. So we basically started this whole process in April and actually at the Cisco Live Conference that you and I were at, I had that diagram threw up on the board, and really I wish it was Paul Santinelli, board member, yells at board and says, "Need to spend less money on AWS!" That's probably like the beginning line. Then a couple months go by and we basically start to narrow in our focus and decide, okay, it looks like this is gonna work, it looks like we can actually do this, what kind of ROI can we get.
We worked with a ton of different partners and we looked at the big companies like a Dell, like a HP, we looked at smaller resellers, and what we basically settled on was we wanted to build it ourselves in terms of we wanted to build- design the gear that we wanted that fit our use case. Then we desperately needed help, and that's basically where my relationship with you actually started, because we got to a point where we were gonna end up having to hire four or five OpenStack engineers to basically try and help us build, deploy, manage, upgrade, do all this work to be on OpenStack, and we knew we wanted to be on OpenStack. But then we actually got introduced to Metacloud, which is now ... I'm sorry; [inaudible 00:17:45]. Go ahead.
Niki Acosta: Cisco OpenStack Private Cloud.
Weston Jossey: Yeah, you guys need an abbreviation. I just can't do it. I'm sorry.
Niki Acosta: I'm sorry, Wes. No, I will pass that video feedback to the folks that make those decisions.
Weston Jossey: It's a bit wordy. But at the time Metacloud was still an independent company, had just raised a good amount of VC funding, was nice up into the right ... Or I guess I was up into the left, I'm not sure which direction it's gonna look like ... Up into the right trajectory and man, the first time they came in, it was magic! You guys have such an incredible team! It's really remarkable! I don't know how often you actually get to see them nowadays now that you're still down in Texas?
Niki Acosta: I troll them online.
Weston Jossey: Yeah?
Niki Acosta: Yeah.
Weston Jossey: Yeah, so they came, they pitched us on this idea of why having a managed, private infrastructure with Metacloud was such a great idea, totally sold it, totally bought it!
Niki Acosta: Did they take you through that the TCO/ROI tool? Did they do that with you?
Weston Jossey: They didn't have to.
Niki Acosta: Oh!
Weston Jossey: We had already done it ourselves. Yeah, so we had already done basically the modeling, so for us we had basically gone through and decided we wanted to price in everything. We wanted head count included, we wanted gear included, we wanted just general maintenance included, warranties, everything, everything was gonna get priced into the ROI. Because Amazon, that's part of what Amazon is providing to you, right? They're providing cloud engineers in effect behind this ... You don't see it, but they are. They're running all that infrastructure. We priced it all in and long story short, it ended up depending on the metrics that you were using, saving us over 10X what we would have paid if we had decided to stick all of our stuff back into the public cloud. Yeah, and it's been a glorious event ever since!
Niki Acosta: Gah-lee! I wish every customer was like you! You make my job so easy Weston Jossey!
You actually have a lot of the data behind this too, which I think is really cool, because you can actually show ... It was the GigaOm article that came out that I think may have had a screenshot of that data, an older version, or what have you.
Weston Jossey: Yep.
Niki Acosta: Talk to us about what you're actually do- because you guys sort of have a hybrid infrastructure. Tell us about all the components in your hybrid infrastructure and what you're running on each other's environments.
Weston Jossey: Yeah, because we were born in the cloud and we're so heavily invested inside of AWS, we knew that we wanted to have ourselves living in AWS and inside of OpenStack. We didn't want to have to make a choice, because we didn't feel like we had to make a choice. I think there's a lot of the conversations that you hear in the OpenStack community, it's like how do we win, how do we beat AWS, how do we basically become the dominant solution. I don't know if that's necessarily the way that I would even have to think about it in that I think that OpenStack and AWS are very complementary systems, exceptionally complementary systems really! What we decided we were gonna do is when we were picking out our colo provider, who ended up being Equinix, which is also a short plug for Equinix, a fantastic company, great to work with.
We decided that we wanted to plant ourselves down right next to AWS in Virginia. We wanted to be as close as we could be geographically to their facilities for a multitude of reasons; one, it makes it easy to get a direct connect between the two sites, and then also because it's just better to be geographically in the same area in terms of latency. We wanted to have a low latency connection between our two systems. What we've basically done is we've built this kind of bridge and this gap between our two systems, and we have inside of AWS we have our core production API systems, so when you're interacting with Tapjoy, you're interacting usually with servers that are sitting inside of AWS. But when you're interacting with our data science department, when we're making an optimization or we're making a decision about what to do with one of those requests, that's actually getting funneled all the way down into our OpenStack environment.
A huge number of the requests that are flowing through our production system right now actually hit both AWS and OpenStack all of within milliseconds of one another to basically make us better. You were asking what we run in there, and the reason why we have to hit our OpenStack cloud is that we're basically running all of our Big Data solutions inside of OpenStack. So those sort of solutions for us are Hadoop, HBase, MemSQL, and then a multitude of other myriad of things that you do when you're engineers and you build out stuff all the time. We have those servers running in these two different areas, and then we've written a bunch of technology to try and gap that stuff. We recently open-sourced a project called Dynamic, we've written some stuff called circuitbreakers that we use internally, which is all about intra-datacenter communication to make sure that if the link is ever broken, we have clear fallback plans, kind of a waterfall, as you will, if things don't work correctly.
We've basically just built everything around this concept of duality of OpenStack and AWS.
Niki Acosta: Are you in a situation where you own your base and write your spike, or are you just kind of keeping those environments separate by function?
Weston Jossey: I would say we're closer to owning by function at this point. I think we could easily go down the road where it is by spike, and I think that that's the most cost-effective solution out there. So for a particularly large company, and I would even put Tapjoy in that category, it is cost-effective to basically think about your infrastructure as a moving walkway. At the beginning of the moving walkway is something like an AWS or a big, public cloud where you're paying for elasticity, you're paying for flexibility, and then as you keep moving down that walkway, everything becomes more predictable. You understand what you already have, you understand what you need, and that's when you think about transitioning it into your own private infrastructure and transitioning it into an OpenStack.
That's the beauty of it, because then everything kind of gets transitioned, and then all you're ever using the public cloud for is new stuff and spike, which makes it exceptionally cost-effective, and it basically uses the two solutions for what they're best at.
Niki Acosta: How many transactions are you guys doing per day at this point?
Weston Jossey: Oh, gosh.
Niki Acosta: Or requests?
Weston Jossey: Yeah, requests, there's a couple of numbers. External to internal, I want to say we're coming up on 3 billion a day, something like that, external to internal. Internal, like if you think about micro services, there's a lot of chatter inside of the network. If we took all of our services and probably combined them together, I mean, we're probably talking about, I don't know, 5 to 10 billion requests per day, or 5 to 10 billion transactions that we're basically internally trying to handle and do calculations on. We were consuming 30 to 50 terabytes a day of data that we basically consume, process, run through Hadoop, analyze, store. We have well over a petabyte worth of storage that we use for basically near-term storage, that's our near-term storage is I think close to 1-1/2 petabytes.
Then if you talk about long-term storage, we actually are pretty heavy users now of Amazon S3 and Glacier just because the pricing's pretty good, and so that extends out that lifespan even further. Tapjoy is actually a really kind of cool company in that we've basically held onto every transaction, everything that's flowing through our system for the last five years. We have every single record of it. So I can actually replay a bunch of our production traffic going back five years ago. It's pretty-
Niki Acosta: You probably have a file on Niki Acosta. You're the folks responsible for showing me videos for like cleaning products and stuff that fit my demographic, right?
Weston Jossey: Your persona, yeah, it would be technically your persona. Yeah, someone who cleans I think would be your persona. I don't know what that would actually categorize into our system. Yeah, I mean, we have a great data science department that does some really good stuff around making sure that the ads that we're showing we're showing at, we like to say at the right moment and with the right context. So it's more than just I want to show Niki an ad. The majority of our ads are opt-in, so we're actually a rewarded ad network, and so users choose to interact with us, which is kind of a different thing than you're used to than when you go onto ESPN.com and something pops up in your face. We tend to do our advertising model is through opt-in, so people are saying, "I want to get something in this game, so show me a video." Or, "I want to get something in this game, so I'm gonna go fill out a survey or I'm gonna go check out an app." There's a bunch of different things that we allow users to do.
So if we saw that you were requesting a video, we want to show you a video that's relevant to you, because we want to make sure that you're getting in front of the right brand that matters to you. So that's what we need, this massive amount of infrastructure that we purchased and put inside of our OpenStack deployment, is to make sure that we can make those decisions and we can make them in quite literally milliseconds. We need to make those decisions [inaudible 00:27:18].
Niki Acosta: Is that why some of the apps that I download for my son ask when your birthday is?
Weston Jossey: That's COPPA compliance that you're getting.
Niki Acosta: Oh, okay.
Weston Jossey: That's because-
Niki Acosta: He's always running up to me going, "Mommy, I need this app!" It's because it was one of those [inaudible 00:27:31] videos.
Weston Jossey: That's because, and I'm not a lawyers\, so I may butcher it, but I'm sure our lawyers are very good in making sure that we're COPPA compliant. But basically if our users are under a certain age, you're not allowed to collect information about them and use it for advertising. That's why they will ask you for that stuff is to make sure they're respecting your son's privacy.
Niki Acosta: Oh! Well, in that case, I want to be 15 forever. Vallard is actually one of our Cisco team members, he's our micro-services geek. I think he got kind of excited. Val asks, "Can you tell us more about your Big Data architecture? Do you run instance per hypervisor, why OpenStack instead of just a standard Hadoop cluster? Thanks."
Weston Jossey: Yeah, it's a really good question. Yeah, when we went through and designed our infrastructure we basically decided that we were gonna look at one hypervisor per node and then we were going to have one Hadoop instance per physical node, so everything devolved down to what would that physical piece of infrastructure look like. We kind of were a little bit creative, and this was two years ago remember when we were doing all this work, and the Haswell chips were just about ready to get done inside of Intel, and so we were looking at the super low-ell chips, because Haswell was doing a really good job of getting the power really far down. We specifically bought, I'm pretty sure they're the 1265LV3 series, if anybody wants to look them up. They were the low-power Haswell 1265's, and the whole purpose of that was we wanted to draw the power down as low as we possibly could, because we wanted to cram as much as we possibly could into a single rack and we wanted to go basically as dense as we could outside of going purely down the open compute route.
So on every single node we basically said, okay, we got four cores running on this node, we're gonna max out the RAM, so no those particular instances it's about 32 gigs or RAM. Then we basically had slots for four physical hard drives, so we put 4 TB hard drives, and that became a physical Hadoop node, because we were looking at basically having a one-to-one spindle to core ratio, so every core on there wanted to have a single hard drive against it, make sure that we had sufficient IOPS to basically do our workloads. In our previous environment we were starting to see that we were running into some IOPS issues and so we wanted to make sure that we improved that. At the time when we were making all these purchases and I guess technically today, the SSDs just hadn't come quite down enough in price yet where we could really justify the cost. It was close, it was really close. I think if you looked at the three-year TOS ... Not TOS ...
Niki Acosta: TCO, ROI?
Weston Jossey: ROI, some three-letter acronym, yeah. Now it's probably cost-effective, but two years ago it wasn't quite there, so that was basically how we landed at that. Then the question of why not just Hadoop running on bare-metal; a lot of it came down to flexibility. We don't just have Hadoop running on these servers, and so we wanted to make sure that we had as much flexibility as we could around making it a virtualized environment which people were already comfortable with, and making sure that the spin-up and spin-down was easy, th