How LLMs are Reshaping Strategy
LLMs can disrupt data work - just not in the way we think
We're almost two years into the LLM hype cycle, and LLMs should be massively and positively disrupting how we data professionals work. The catch is that it isn't the change we were expecting.
It is a strong claim, so let me take you through my learning journey before closing in on how we should think about LLMs in data. (and yes, I still insist on calling LLMs LLMs and not AI - hopefully, that gives the data science readers some confidence I've not caught AI fever)
I left my CDO role almost two years ago with a clear plan. I'd spent six years working on a data transformation that was signed off as a success, and I wanted a career break before setting up a business to replicate the success as a service.
Three months into that plan, ChatGPT took off. Eight months in, I was at a data leaders event and was due to lead a discussion on getting value from data when I panicked. In the first half of the event, everyone talked in detail about the in-house LLMs they were building and how everything had changed. I started questioning if all my data strategy experience was suddenly void.
Being the most honest I've probably ever been in an article, I took a tactical break outside to quickly consider whether I should change my plan for the discussion. Fresh air did the trick it always does, and I concluded it was still relevant to discuss the value and ROI of data work.
It was early days then, but whilst there were many ideas for ROI, there were no proven use cases. Incidentally, just four weeks ago, I heard of a Gartner panel discussion that was lost for words when someone asked for documented, high-ROI use cases of LLMs. They didn't have any.
So why is someone known for preaching about value also pro-LLM disruption despite the lack of high ROI use cases for LLMs?
We're looking for value in the wrong place. LLMs have much greater potential in data product strategy than in data products themselves.
My first LLM strategy breakthrough came from trying to recreate some analysis I'd done in JLR. To make the case for more investment in data, we mapped out all data products that could materially add value to the organisation, function by function. We started research-led by mapping the value streams and how data could help optimise them. We then iterated our ideas with each function before proudly presenting a catalogue of data products to the board that could improve the bottom line by between £1.2 and £1.4 billion. Saying that figure with a straight face to the board remains a career highlight.
One of my first LLM experiments was to get an LLM to automate drafting a data product catalogue for any organisation. Getting an effective prompt took about two days, compared to months of teamwork and research before.
One user of the prompt said that in minutes, it produced seven of the same ten recommendations as a leading consultancy after a three-month engagement. It works surprisingly well.
This was the start of a series of experiments to see how effective LLMs could be at strategy work. We've since developed many strategy and management prompts, ranging from producing emotive and powerful vision and mission statements to stakeholder communications plans.
As an example, here is a prompt to map data domains and then prioritise them based on competitive advantage. Drop me a message (benny@datent.com) if you want more context on how to use this as a strategy tool and I can share a webinar recording on getting competitive advantage from data where this prompt was discussed.
So what's going on here? How is it that an LLM might be able to help with strategy and why is that of any value?
To generalise, LLMs are great at two things - spotting and following patterns and generating content.
Strategists, like artists, may think they perform some creative magic, but really, all we do is spot and follow patterns, like LLMs. Therefore, strategy work is now easily automatable with all the same caveats we make for the automation of creative content - LLMs aren't as good as master craftsmen.
Starting with the simplest example, the most basic strategy patterns are little more than "Client A got value from data by improving stock management. Client B has a lot of money tied up in stock, so let's type up the case study and see if we can sell data-driven stock management to them". If you've seen as many consulting decks as I have, you'll have seen that presentation a lot.
Until now, countless strategists have done the more advanced version of strategy pattern matching as a career step. After spending years solving problems (either in consulting, industry or both), become an independent consultant and create frameworks for the strategy patterns you have spotted so that others can learn from your experience. I am guilty of this and, in all likelihood, this is the reason most people have subscribed to this substack.
Now, we have a slightly more advanced version of strategic pattern spotting. If you can condense your strategy experience into a framework, you can also codify it into a prompt that enables an LLM to draft a strategy or provide strategic advice. Is it perfect? No, but it doesn't need to be. It is at least as good as the first strategy draft or initial advice from a junior consultant, and you can then iterate it and improve it the same way you would with a draft consulting report.
We've now created many prompts for frequent strategy and transformation challenges and a range of ways to give the LLM the data it needs for a 'well-informed' answer. I'm even thinking of building an automated consultant and calling it McCain.
In summary, there are three parts to effective use of LLMs:
The prompt.
The context data (for a well-informed answer).
The follow-ups (to refine the output).
So far, we've looked at prompts as an evolution of frameworks. The best context data I've found to attach to these prompts are other strategy documents (from strategy presentations to published accounts) and meeting transcripts, the new 'oil' in data.
I first realised the power of meeting transcripts when, with a one-hour prompt 'conversation' with an LLM, I turned nine hours of transcripts and feedback from the pilot of our Accelerator into a restructured vastly improved course curriculum. The LLM didn't just add structure to an initial MVP product; it mapped content to pain points and even highlighted gaps.
I lost the conversation, but when I asked for critical feedback, part of the reply was, "Your content and attendees' feedback talk about the personal challenges of leading a data transformation, but there is no advice in the course on how individuals can manage their leadership journey."
That insight was brilliant. We now finish the course with a session on the data leadership journey. It helps finish the course by getting everyone confident about setting action plans that take them out of their comfort zones. Attendees love this final session, and honestly, I don't think I would ever have thought of including it.
With transcripts from meetings (where possible) or monologue meeting summaries and LLMs, everyone now has access to a strategy coach.
I could go on at length about strategy or management use cases, and an equally length article could be written on use cases in:
Data Governance: From helping identify potential data quality issues to assisting in creating data governance policies and procedures.
Data Viz: From critiquing dashboard designs to generating ideas for data visualisations.
Data Engineering and Pipeline Management: From assisting in creating ETL jobs to generating synthetic data for testing and development.
So what? Where's the value? These are all fair questions.
Let's go full circle and start with the patterns.
The first surprising pattern is that, yet again, we don't seem to be good at predicting where AI will succeed and where it will struggle.
Five years ago, many people were convinced that AI was close to solving driving but that we were a long way from AI being able to write convincing essays. Today, driverless cars still seem a long way from taking over, and every teacher is struggling with LLM-generated essays.
Likewise, at the beginning of the current AI hype cycle, lots of people set out to use LLMs to automate the analyst, as analysts, like drivers, apparently do 'relatively simple' work that could be automated. However, the analyst role is neither simple nor easy to automate. By contrast, consulting, management and coaching are relatively easy to automate.
Final, so what? Consultants excel at helping organisations identify and focus on their priorities, i.e., the areas in which they are most likely to get value. And, this is also the problem that most data teams struggle with, as they often focus on maximising outputs over outcomes.
If we all start using LLMs to be more value-focused with strategic prompts and LLM coaching, the value implications are huge. Productivity could fly through the roof, not because we've increased throughput through automation but because we've used LLMs to be more strategic and value-focused.
The takeaway? Start using LLMs to guide your strategy and to focus your efforts on value creation.
Want a simple starting point? Record the transcript from a sprint review or planning session. Run it through an LLM start with a simple prompt like:
"You are an experienced product manager. We're trying to focus more on value as a data team. To what extent did our sprint review (transcript attached) demonstrate a focus on value and delivering measurable outcomes? Be both critical and ensure your advice is actionable i.e. tell me how could we improve"
I'd like to hear your thoughts in the comments.
And now for a bold offer. Last year, I snuck into a Substack post that, if you were quick, you could get a free place on a pilot of a data transformation Accelerator. The accelerator has gone on to do amazing things, and a bunch of fast readers got a free place.
Now, we want to extend the Accelerator with a more lightweight (mostly e-learning) version. If you are in North or South America you are in a timezone I can't currently support (our Accelerator sessions are in the morning UK time). So I'm looking for a pilot group of people in these time zones to attend a one-off hybrid version of the Accelerator with a crazy 80% discount. You'll get the live sessions as recordings, but there will be a weekly group coaching session at a sensible time, and your feedback will help shape a product we're launching next year.
If you'd like to know more, contact me at benny@datent.com. And be quick, in a few days this paragraph and offer will disappear...
Want a reason to subscribe - every now and then I through freebies like this into the articles :-)