Bidtellect’s Information Technology team recently took on a huge endeavor to revamp and upgrade Bidtellect’s server environment. The undertaking took 32 days of nonstop working – “Work eat sleep repeat,” said Jason Taylor, Director of Information Technology – with some sleepless nights and 40 hour sprints in which the team would take turns napping. CTO Mike Conway explained that the impressive endeavor was akin to changing four tires on a car while speeding 80mph on the highway – and not crashing.
Bidtellect partnered with Cisco and R2 Unified Technologies to deploy wide-reaching upgrades including a modular Network and Server Fabric Platform with extreme throughput, compute, and storage density all while maintaining efficiency. The significant investment and upgrade includes large scale additions in computational capacity and data storage. Bidtellect can now listen to 3.5x the number of queries and process 3x the number of auctions per second, laying the foundation for the current growth trajectory.
According to JT, the team essentially made the capacity available to increase supply and demand and capture revenue, without increasing our monthly costs or EBITDA negatively. If Bidtellect were a shopping mall, we expanded the mall to make room for more shops and more shoppers to buy!
Check out our Official Press Release here: Bidtellect’s Answer for Growing Advertiser Demand is Major Infrastructure Environment Overhaul
Check out JT’s breakdown below:
What did we do?
Replaced and upgraded a majority of the environment!
- All UCS Errthang via Cisco Fabric Interconnects – Extremely fast networking and centralized server management
- New Native Blades – Faster CPU’s, increase to 96GB of RAM, ~ 10-15% capacity increase
- Tracking – Migrated DC3 tracking to repurposed VM blades, 24C/48Tx 4
- Kafka, Aerospike – Replaced, some performance increase
- Namenodes – Upgraded CPU/RAM, all SSD for improved indexing speed
- Datanodes – 3x storage increase, 2x RAM and CPU performance increase
- Firewalls – Upgraded 3x cryptography throughput, redundancy in DC3
- Core Network – New ASR’s and Nexus in DC3 to support more throughput and port density
- Load Balancers – 3-3.5x raw QPS capacity increase, 5-6x SSL offload capacity increase
- VM Platform – Increase VM platform capacity, more cores, more RAM
- Modular scalability
Why did we do it?
- Replace aging and end of life equipment.
- We were at capacity limitations of the platform with no path for growth.
- Able to deliver more revenue without negatively affecting profitability. In Fact this helps profitability.
- First step of a truly scalable Infrastructure environment that will allow us to continue increasing our Demand and Supply diversity along with capturing more revenue.
Who was involved?
- Helgi – Never stopped working
- JT – Never stopped complaining
- Chris – Never stopped eating
- Mike – Never stopped asking if it was done yet
- Rob – Always ready and willing
- Luke – The secret weapon
- Roei – Silent but deadly
- Karl – Morale support
How much did it cost?
- Little to no increase in MRC costs by replacing Winmark and seeking our own financing
- Opex NRC cost for temp SE, travel, lodging, car, food and misc datacenter needs
- Why is this important?
- These costs directly affect profitability, EBITDA, which directly affects EBITDA and Bonus
- Allows us to capture more revenue without increasing Infrastructure Costs
- A little piece of my soul
- Our power usage is equal to approximately 12-15 single family homes in the peak of a FL summer, yet our floor space is less than 350 sq ft combined.
- We could hypothetically build approximately 1,796 home computers if we divided up the compute resources in Ashburn and San Jose.
- Helgi and I went to Target on Easter Sunday and didn’t understand why they were closed. We also didn’t realize it was Sunday, let alone Easter Sunday. The concept of “time of day” or “day of the week” was already lost at this point. But we were tired of doing laundry and were just going to buy new clothes.
- In the last hotel we stayed at, Helgi and I had to fix the hotels internet so that we could work. We got on the phone with their ISP’s support and worked through the issue. The hotel still didn’t give us free breakfast.
- The Load Balancers we deploy in DC3 are the same devices that are used as an internet gateway for entire country of Jamaica.
Exploring what may be coming next…
- MORE QPS & MORE $$$
- More Native and Tracking resources
- Upgrade AD-IX port and enable public peering
- Additional uplinks to service providers and enable Multipath BGP(capacity level redundancy)
- Load Balancing the Load Balancers…..wait what?
- Infrastructure Patch Management, a step towards patch automation
- Windows 2016/2019? .Net 5.0? (test in EU first)
- EU going physical, in a good way
- APAC footprint
- Dedicated Stage Environment
- Deploy an ACI environment (10x scale plan)
Stay tuned for an official Press Release!