Kafka and the state of the software industry

By following the latest Kafka Summit in London (https://kafka-summit.org/events/kafka-summit-london-2018/) I found analogy between software industry and long distance race where racers have different opinion about the length of the race. In 80-ties big players like IBM thought it will be a sprint that can be easily won. Small players of that time like Apple understood that they need more endurance and that race will be at least 5000 meters. No one anticipated it will be longer than that. As they passed 5000 meter line, they all realised more realistic goal should be set. So 10 000 meter limit was set. As they were approaching this goal, dispersion of runners was evident. Some ware leading in the front some are lagging behind.

Finally the 10 000 limit was reached, only to discover that there is long way of uncharted territory in front of them. Nowadays no one is certain anymore is that a full marathon to be run, ultra marathon or something else.

What was evident after passing 10 000 meter limit is that only those runners with endurance and innovative running techniques are capable of leading the run. Another evident fact was grouping of runners. That was in it self logical and good outcome of such a long run, as it is easier to run in the group that is pushing you and occasionally provide you with the wind shield, than to run alone. However as the distance of the run become longer and longer the dispersion of the groups become wider, up to the point that they where not able to see each other.

This is the state of software industry today. We have companies that still rely on technology that is more than thirty years old, company that are using more “recent” technology, and companies that are embracing the change and adopting new technologies on their projects. What scares me the most is when I see companies investing time and money in developing and implementing technologies they consider “recent” or “widely used” from their viewpoint, which will put them on 10 000 meter mark while in reality other companies are several kilometres ahead. I see this gap widening each day as I follow new technology and in the same time working on the project where hundred thousand messages per day, processed with traditional messaging system and relational databases, seams to be huge problem to process. Demand to process this volume of messages requires huge hardware and bare metal processing power. In the same time I am talking to colleagues from my company to whom I helped to set architecture based on Kafka and Kafka streams message processing technology, and colleagues from other companies. They do not have problem with processing billions of messages per day by using the same hardware capacity as traditional message bus solutions processing at least ten times less messages per day. So what is the catch. Catch is that successful solutions for message processing all have common denominator. They all use Kafka and Kafka streams as a event handling asynchronous messaging and message processing system.

It appears that difference in performance, with the lover operational cost, is so huge compared to the traditional enterprise service bus systems that they practically even do not see each other. It appears that development managers of these products can not even grasp or comprehend the fact that technology like Kafka is such huge leap away that they consider it: “childish”, “not serious enough”, “not reliable” despite the fact that most of the major players like: Linked In, Google, Apple, Workday, are using it. They are investing money and time in projects implementing traditional message bus systems to integrate their existing portfolio of software products, just to be on the safe ground. They want to have message reliability and security, blatantly unaware that some of the product they are integrating (like Workday for example) are already using Kafka for message processing hawing the same message reliability and security they are requiring. It is even funny to see their thinking in leading the race until they pas the curve just to realise that another company is already several kilometres in front of them, just due to the fact that they have embraced new technology on time. They where not scared to jump into the Cloud and to use Open Source. They took the risk, took the energy booster and they are leaders now. Old kings think that they are still big, but their kingdom is falling apart.

I am a Paraglider pilot and I know that in order to fly I have to run, but in one moment the fast ground will vanish under my feat. I know that I have to be prepared for that as this is the only way to go from running with the paraglider that is creating drag, to flying with the paraglider that is now your support. I think that same analogy applies to industry or business in general. Especially software business. So we can see that some companies like Amazon, Microsoft, Apple, Google, Linked In, Facebook or Workday, are already flying while others are still running. You can just imagine gap being created between flyers and runners. And that is because flyers are not scared to embrace change and new while other are, despite what they say. If you challenge them they will reveal their through face, and find all possible excuses why they are not prepared to embrace change: “Ahh you know, we are in banking business, and we have to take care of our customer data …” Well Kafka takes care of your customer data. To the matter of fact it takes care even better than traditional Cobol or database systems as it logs every change in data record by design. So what is really the case: “I want to be secured that my data is kept safe and non compromised” which Kafka does anyway, or “I am afraid of new thing and change as I have to learn and adopt something new, and eventually change my company mentality”?

So what is your answer? Kafka?