Stirring a little controversy by comparing my First Principles (things I know to be true) related to building high performing, scalable systems with current conventional wisdom.
Among most software architects today, current conventional wisdom states your architecture must follow a microservice software pattern, use containers like Docker, capture data in an event streaming platform like Kafka, persist data in a NoSQL database, be managed by something like Kubernetes or Mesosphere and run on Linux if you want to have a high performance, scalable system.
What is something I know to be true from the dot com era?
Back in the day, I was fortunate enough to be part of a team that built an energy trading platform that allowed multiple counterparties to buy and sell financial instruments (think NASDAQ). This platform was built on 32-bit Windows Server 2000 with some kind of Intel Pentium CPU. Classic Active Server Pages would send and receive XML fragments over HTTPS to and from brokers from every major energy company + NYMEX and the Intercontinental Exchange (ICE) following a RESTful pattern. Incoming data was queued in MSMQ and free-threaded objects pooled inside Microsoft Transaction Server (MTS) moved that data in and out of 32-bit SQL Server 2000. The databases were clustered and Windows Server got 1 GB of RAM whereas SQL Server got most of the remaining 3 GB of RAM. The Internet Information Servers running ASP 3.0 used the built-in network load balancing service (NLB). Our counterparties spent tens of millions of dollars on hundreds of load-balanced UNIX servers from Sun Microsystems. We used Microsoft. Our system didn’t break a sweat. When we needed to update the software, we just took the servers out of the cluster one by one to make the update. No biggie. No containers needed.
So what’s the takeaway from all this? As someone who has played a prominent role in the Internet of Things space over the years, I struggle to find modern systems or public cloud instances that have to deal with the transactional and analytical load that I witnessed with the system I helped build 20 years ago. In spite of all the promise and talk, I’ve yet to see any IoT system that deals with a fraction of the load the world’s financial systems handle effortlessly with their so-called, antiquated architectures and relational databases.
Why were we able to do so much more with so much less back then?
Remember folks, we’re just sending electrical current through a gate to establish a high or low voltage at a particular point in the circuit. The farther away you abstract yourself from this, the more resources your system will require and the slower the system gets. Don’t be impressed by architectural diagrams with hundreds of lines, boxes, arrows and triangles going every which way. Complexity kills. New programming languages, frameworks, and architectural patterns come along all the time. Use you best judgement and fall back to your own first principles before deciding to jump on the next bandwagon.