If you choose Kinesis, you get:
- Bad documentation
- Limited support when it comes to languages/SDKs
- Bad latency (Kinesis isn't really an asynchronous broker, it's an inefficient polling beast disguised as an asynchronous stream)
- Limited throughput (a shard can ingest a maximum of 1 MB/s)
- A heavily limited quota (you can get a maximum of 10 MB or 10,000 records per read, and you can only do a maximum of 5 reads per seconds, or retrieve 10,000 items every 5 seconds). Any quota extension requires an expensive product upgrade through an annoying "contact our customer support" process
- You can create a maximum of 50 data streams - any extension requires, again, the "contact Amazon support" bullsh*t
- An environment that is hard to replicate and test locally - you need to use their closed localstack Docker image, which breaks very frequently
- Additional configuration required to connect to the correct cloud service (region, environment, default topics that should be available etc.)
- Pay Amazon to provide you this shit
If you choose Kafka installed on your premises, you get:
- Top-level documentation and community support
- Support for virtually any programming language and tons of SDKs and libraries
- A true asynchronous broker, with a latency that is as good as the latency of your infrastructure
- A throughput as good as the throughput of your infrastructure/server
- No quota or restrictions whatsoever, unless you want to set them explicitly in your configuration
- You can create as many topics as you like. You don't need to spend time on phone with Amazon's customer service if you want to have more than 50 topics
- An environment that can be easily tested and replicated locally. Just pull and run a Kafka image, and you're good to go. No configuration about regions, clusters, shards and default topics is required, unless you need it or want it
- You don't have to pay anybody for anything, except the costs of running your own broker on your own servers. Everything is free and open-source
My company has realized the mistake it made by relying on some severely limited enterprise shit instead of readily available open-source solutions, and it's now in the process of migrating its data stream infrastructure to Kafka.
A lot of time has been wasted both by the infra and development teams in the meantime. A lot of business value has been held behind by a bad data stream product. A lot of developers got frustrated by all these unneeded constraints. A lot of time will be required to migrate the whole codebase to a new data infrastructure - we're talking of several months. All because the management was like "let's use this off-the-shelf AWS solution instead of running our data stream infrastructure using 'amateur' open-source products".
Amazon is literally producing copycats of open-source products that are stable and free for anyone to use. But those copycats are much worse, much more limited, and you have to pay money for them. Why on earth would anybody pay for this crap?
Are senior managers so dumb that if you sprinkle a bit of marketing and enterprise powder on top of some crap they're ready to pay for it (and even pay A LOT) rather than use something that is better and free?
Even the "it scales better" argument makes no sense. Amazon provides you the worst of the two worlds: you have to think of your whole application as working in an opaque cloud even when you're just running on your local machine, but real throughput is severely limited, the strict quotas mean that you won't be able to easily go from 100K to 10M records per day (let alone per hour) within a day, and every infrastructure upgrade to ensure that it can handle more traffic requires a human contacting another human at Amazon and requesting an upgrade.
This is the kind of shit that is holding back technological innovation today.
"Are senior managers so dumb..."
In my experience, most definitely yes. There is a really ingrained attitude that expensive corporate solutions are more "professional" and thus "better". And when you consider what people mean by "professional", it seems to boil down to a cultural phenomenon more than anything else, i.e. "walking the walk and talking the talk". And maybe also an egotistical element to it like owning a Rolex, or driving an "executive" style car.
@BB well at least an expensive car or a Rolex are something that everybody considers valuable and that can be shown off. Not so sure about the value of showing off expensive but mediocre Amazon or Microsoft solutions 😁
Ha! Careful about defining "everybody" – I don't have any use for either, though I see what you mean. For the software, I suppose it's the same warm and fuzzy feeling of superiority they bring to the egotistical types that counts 😛
A platform about automation, open-source, software development, data science, science and tech.