It’s not done yet.
Is Generative AI the ‘Emperor’s New Clothes’ of Tech?
Impressive, but not fully dressed yet.
Mark Piper, Principal Explorer
1 August, 2024
One of the hardest challenges for independent researchers is when their opinions don't appear to align with that of the market. You’ve done the research - the leg work. You’ve gazed behind the headlines, recoiled slightly, and kept gazing. You’ve read the analyst call transcripts. You’ve even kicked the tires of the various products and done the maths.
And nothing about what you are looking at makes sense.
Regardless, the market is bullish and moving ahead at full steam procuring services while ignoring the elephant in the room. It’s a strange experience, and can definitely leave you feeling like you’re either missing something, or you’re just standing there awkwardly in a world full of people excitedly talking about the Emperor's New Generative AI.
One such example of this was from June, in our second major report The Great Augmentation, we wrote the following in the aptly named section Where angels fear to tread:
“The problem with this cycle is that marketing largely sells ideas right now. Most, if not all AI solutions, are in early phases and under significant active development. The features are incomplete. The resulting output is unreliable. The legal precedents have not been set. The marketing is so aggressive that it’s hard to understand what is functional and available today as opposed to a future feature release.
It’s hard to imagine a time where the market may have been so full of incomplete or “minimal viable products” as it is now. This can be taken as yet another indication on how transformer architecture, AI-based computing, is here to stay. The vendors are banking that they will finish the job.
When procuring LLM capabilities for your business, it is important to acknowledge that they are all in roughly the same early stages of development. AI vendors are currently competing for licensing commitments for when the full capabilities of their systems will be realised in the future, not for today.”
This is regardless of whatever that product is. If it has suddenly gained ‘Artificial Intelligence’ in the last 12 - 24 months - it’s likely not done yet.
Our research found that if you are suddenly in the market for generative AI capabilities to augment your workplace, or boost productivity, we believe that there’s a good chance you would be buying not only incomplete, but wildly incomplete capabilities - sold to you with doctored and flashy examples from some marketing video and 2-page sales collateral.
So our ultimate recommendation from this research was (in bold) the following:
It is for this reason that we strongly urge those looking at adoption of LLMs within their workplace to do so with a mid-long term (2-5 years) planning view.
This wasn't a recommendation we made lightly. At the risk of saying what no one seemingly wants to hear right now, while we truly believe in the potential of generative AI, we also believe in keeping our collective risk and economics heads screwed on when it comes to what decisions organisations make today. Very little of the services we looked at could be considered functional, let alone complete in any way at the time of our research.
So it was with some vindication that over the last few months we have started to see a market correction and a slow down of the non-justified and forceful procurement of broken dreams.
First, we had our hearts filled with joy at witnessing the huge support for Nikhil Suresh’s masterpiece essay ‘I Will Fucking Piledrive You If You Mention AI Again’. In it, he not only calls out the unbelievable arguments being presented for generative AI procurement, successfully cutting through the AI hype, but he also calls for a practical and grounded approach to how we tackle this in the future.
I’m not sure how many executives read his work, but I would suspect far more than would ever publicly admit it (and if you haven’t, you should).
This was quickly followed by more realisation that in order to achieve the dreams, we still have massive compute challenges to overcome in order to deliver on the AI dreams.
Despite all the technical reasons why we’re failing to scale (such as H100 cluster inter-connects across regions), the most staggering figure to help understand where we are at was Google’s emissions target data. For the FY 2023 year they reported 14.3 Million tCO2. That may not seem much, but it’s a 48% increase on the 2019 number. A increase they directly contribute to AI computation requirements.
At a time in which compute demands are only going to increase, it’s safe to assume that not only Google, but every major provider are going to be doubling or tripling their efforts to understand how they can provide the right level of compute for ambiguous AI and not completely destroy power grids - and the planet - at the same time. Hell, Microsoft are reportedly considering that mini-nuclear reactors in your neighbourhood data centre might alleviate the pain.
Let’s pause on this for a moment - Microsoft (and likely others) are seriously considering Small Modular Reactors to ensure AI can scale and deliver on promises. Whether you are for, or against nuclear power doesn’t matter - it is an astonishing ‘you are here’ marker on a map of the future.
Then, this week we saw the market analysts scratch their heads, smash fingers on their various calculators and ponder if the $53 Billion (USD, $80.1 Billion AUD for those following along in the southern hemisphere) Microsoft has spent in AI costs is going to get a matching the return for all this stuff?
The analysts did ask some soft ball questions and the answers were … muddled. “It’ll pay off" they argued, but no-one really knows when.
It must have been deja vu for the analysts, because they’ve been asking the same about OpenAI’s rumoured $5 Billion loss for the year. A pattern is emerging.
It’s clear that the vast majority of AI announcements of the last 12 months were jumping the gun. The vendors who made them have been scrambling to catch up and provide the functionality that is currently being sold.
You don’t need to look far to validate the situation. If you compare the launch videos of co-pilot or gemini for the workplace, then grab a licence for either service, you will find a myriad of functionality still in development, missing, unreliable or just generally broken. This isn’t to slight the engineering teams working on these solutions, what has been achieved to date is genuinely impressive - a look into the window of the future as it could be - it just isn’t ready today.
Take for example, Microsoft’s co-pilot vision. In the initial announcement it espoused the benefits of LLM technology meeting spreadsheets. And this makes sense - given the open secret that spreadsheets are the core of most major enterprises, being able to improve or enhance the spreadsheet experience is a massive win for everyone who has ever tried to decipher the 42 line formula calculating costs.
Getting LLMs to work with spreadsheets has turned into a hard computer science problem, regardless of how nice an idea it is. To illustrate this, Microsoft threw some of their top minds at on the problem and have invented an entirely new and novel way to integrate LLMs with spreadsheets. I’m not entirely sure what is required to take this from research to production, but I would argue that we are still 12 - 24 months away from realising the benefits of LLMs in Excel in any meaningful way.
If you speak to early adopters, they all state the same no matter the technology stack:
“It’s not done yet”
There is one particularly infuriating argument for selling incomplete products and that is they are just following ‘the trend’. It started with Tesla, they argue, and their self-driving car feature. It was (and is) nowhere near complete! If Tesla can charge $10k for something that isn’t done yet, so can we!
The problem here is twofold. First, as buyers, Tesla customers understood that the feature was incomplete. They understood that the feature was in development. It was clearly stated, no room to misinterpret.
Secondly, as a buyer, you still have a car while you wait. You can drive it. You can insure it. You can crash it. There is no doubt at all that the car exists and you can use it.
For software, it’s a very different situation. You have no tangible asset, and the software doesn't work as intended yet. You might not even be able to insure against misuse.
Building meaningful LLMs is hard. Integrating them is hard. Providing compute at scale is hard and hard problems take time to resolve. So why are they still selling you licences at a premium price today?
We want to believe in the dream. We want you to believe in the dream.
But we also want the reality to be believable.
As we enter into the ramp-up of a new financial year, over halfway through the calendar year, I strongly believe that waiting a little longer before placing large, cost heavy bets on generative AI is the right thing to do. It is entirely possible that we are wrong this time, but I believe we will see a strong market correction over the next 12 - 24 months as history proves that in this tough economic climate, caution is absolutely justified.
This wait won’t be the death of AI, or the AI-driven future. No business will be left behind because they were slower to adopt bleeding-edge technology. As we can see, there’s already been billions invested in the next-generation software and platforms and there is no way that investment will be laid to waste. The market simply won’t let it.
Taking a little time won’t be the end of your business or the future AI might provide. In fact, this careful approach may well position your business to adopt AI more effectively when the time is right, ensuring long-term success rather than rushing into a product that doesn’t (yet) do what it says on the tin.
