Using ChatGPT For News Coverage: An Experiment With Budget 2023
ChatGPT has made the news. But is it any good for news-related work? I test it out during one of the biggest news events of the year in Singapore.
Budget season is the busiest time of the year for newsrooms in Singapore. The opening day of the roughly month-long affair is typically the most intense, as that’s when the Finance Minister delivers a lengthy speech packed with new policy announcements that journalists and editors have to digest and publish within a very tight deadline.
In other words, it’s a good opportunity to see if ChatGPT, a large language model which has taken the world by storm, is suitable for news-related work. Despite its popularity since debutting in late 2022, questions persist as to whether the AI tool can be relied upon for “real” or “important” work. Well, let’s find out.
For this experiment, I used a paid version of ChatGPT, aka “ChatGPT Plus”. It is supposedly “optimized for speed” and more stable, unlike the regular public version of ChatGPT which can be inaccessible during peak hours.
I used this official public version of the 2023 Budget speech by Singapore Finance Minister Lawrence Wong for my ChatGPT inputs. The 13,837-word speech was delivered in Parliament on February 14 2023.
Test #1: Summarization
ChatGPT handily beats most open source machine learning models in summarizing news articles, letters, commentaries and the like. But what about nitty-gritty policy details? No sweat, it seems.
The AI model easily handled requests to distill and summarize large numbers of paragraphs into bullet points. But what I found impressive was its ability to further re-organize the summaries into a reader-friendly format with thematic sub-headings. All it took was a simple prompt asking ChatGPT to convert the summarized text into a format that be used in an infographic:
Earlier iterations of ChatGPT, such as GPT-3, are highly prone to mistakes in such “text-to-text” tasks involving multiple numerical values. For instance, GPT-3 has a tendency of making up a statistic or dollar amount in its output that’s not present in the original text it was given.
But in my dozen or so attempts at summarizing various parts of the Budget speech, ChatGPT mostly avoided making such mistakes. I only encountered errors in two separate summaries where ChatGPT cooked up a figure of $3.5 billion as the cost for a raft of policies.
Like its predecessor, ChatGPT continues to be tripped up by ambiguous references to time and dates, such as “today”. In the Budget speech, for instance, Mr Wong announced that all eligible Singaporean children “born from today onwards” would be given a higher cash gift of S$3,000. Since it was not told that the speech took place on February 14, ChatGPT assumed that the policy would kick in from 2023 and wrote it as such in its summary.
By now it won’t surprise anyone that ChatGPT can make mistakes. But you could say the same for journalists working under a tight deadline.
What’s amply clear in my tests since late 2022 is that ChatGPT makes far fewer mistakes than its predecessors in tasks like summarization, an improvement that’s largely gone unappreciated, in my view.
Test #2: Listicle & Changing On The Fly
Most AI writing tools in the market currently operate on a “once and done, take it or leave it” basis, meaning you either accept the suggested output from the model or reject it completely and try again with a different text input. You can’t improve or fine tune the model’s answer by instructing it to change the focus, tone, or length of its initial answer.
ChatGPT, however, is able to “remember” its “conversation” with a user (up to a point) and modify its output accordingly. In the simple example below, I asked ChatGPT to generate a listicle of the top 10 things to know from Singapore’s fiscal position and economic outlook, after feeding the model the relevant paragraphs from three different sections of the speech.
ChatGPT’s first answer (see screen-cap on the left) was serviceable. As a regular reader, I would rather read this than a lengthy news article.
But I changed my mind and felt that the original listicle was too skimpy. So I entered a new prompt within the same chat and asked ChatGPT to redo the listicle, but elaborate on each point while keeping to a maximum of three sentences each.
The new listicle is on the right. You can make further modifications, such as by re-ordering the sequence of the items or dropping those that you think aren’t essential.
The ability to instruct and refine ChatGPT’s answers on the fly is one of the biggest breakthroughs for AI in recent years— and one that makes it appealing and practical for use in newsrooms where decisions can change fast.
One common fear of automated or AI-generated content among newsroom users is that they might end up publishing the same cookie-cutter content as their rivals. ChatGPT’s ability to improve on its answers based on unique user input reduces the potential for such embarrassment.
Test #3: Drafting News Articles & Re-Angling
ChatGPT is capable of mimicking the recognizable style behind journalistic writing — but only up to a point.
In the example below, I first gave ChatGPT the relevant paragraphs on the newly announced efforts to support families and parents, and instructed the model to write a news article without specifying which policy to focus on. Then I asked ChaGPT to shorten the draft, and re-angle the story to focus on the “baby bonus”:
Both drafts would be considered passable to casual readers, but not newsroom editors. For one, ChatGPT doesn’t know when to add relevant direct quotes from the newsmaker and instead rephrases the original speech in the same manner, paragraph after paragraph.
Both drafts also dropped the honorifics for the newsmaker, and adopted American spelling throughout. The paragraphs are “chunky” and hard to read.
In the second draft on the right, ChatGPT made a factual error in stating that the cash gift would be increased for eligible Singaporean babies born from 2023. What Mr Wong said, in his speech on February 14, was that the increase would apply to babies “born from today onwards”. This is a common problem with GPT models going back to its earliest incarnations.
But let’s be clear: these are not catastrophic mistakes. Anyone who has worked with rookies or harried journalists have encountered worse. Every newsroom has an established system of checks prior to publication, and there’s no reason why it shouldn’t be applied — perhaps more rigorously so — to AI generated content.
Meanwhile, other editors will quibble with the tone and writing style of the ChatGPT drafts. Unfortunately, these stylistic issues can’t be adequately addressed until OpenAI allows for a process called “fine tuning”, where you can create a bespoke version of the GPT model using your own custom data.
Overall, I am very impressed with ChatGPT’s ability to re-angle and re-organize story drafts on the fly. Manual “do-overs” like this can be painful affairs in newsrooms, taking hours in some extreme cases. This is one area where I can see ChatGPT saving editors and journalists a lot of grief.
I also tested ChatGPT on a bunch of other tasks, such as translation and a sentiment analysis of Mr Wong’s speech. I even asked ChatGPT to generate commentaries arguing for and against a particular policy change in this year’s Budget speech.
The possibilities are seemingly endless, so I’ll leave some of these examples for another time.
Can’t Spell Fail Without AI?
“Hold on,” you might say. “What about all these news reports and social media posts I’ve seen about ChatGPT’s mistakes and this somewhat spectacular example of Microsoft’s version of ChatGPT going off the rails?”
I haven’t replicated all these mistakes, but I have certainly come across my fair share of them while experimenting with ChatGPT in recent months. I’ve also managed to recreate a viral “prompt hack” that could get ChatGPT to respond as a snarky smart aleck known as DAN (short for “Do Anything Now”).
But the bottomline for me is that these issues don’t detract from the very real utility I get from using ChatGPT for a wide range of newsroom-related writing and editing tasks. Much of your comfort level with the tool will depend on how much you want to buy into the hype and fear surrounding AI.
I view ChatGPT through a very narrow use case — that of a powerful writing-editing assistant. I’m not interested in having casual conversations with the AI model, or using it as an oracle, stock picker or search engine. And I only use ChatGPT for writing-editing tasks involving information that I can easily verify.
In short, I use ChatGPT on a strictly “low trust, always verify” basis — an approach I take with all AI/machine learning products. If you want to blindly trust ChatGPT or use it for unusual purposes, then you should be prepared for unexpected and potentially bad outcomes.
This will no doubt create some discomfort among newsroom users, who are more accustomed with software and tools that behave in predictable ways. Unfortunately, ChatGPT is a new class of technology that is powerful and yet not fully predictable by design. Even its creators are still trying to figure out the model’s full capabilities and limitations.
But as long as you are clear about the writing-editing tasks you are using ChatGPT for and stick to a set of well-worn newsroom rules, the risks can be managed, in my view.
And it’ll certainly take a bit of practice and experimentation before you gain enough experience to use ChatGPT well.
While not a perfect analogy, I would say using ChatGPT is much like driving a car. In the hands of a new or reckless driver, it could cause chaos on the roads and endanger others. But if you pass a test, drive sensibly and obey the rules of the road, then it saves you time and gets you to the destination in comfort.
You can’t fully eliminate the risks of driving on the roads and accidents will happen from time to time. But the risks can be managed and millions have successfully done so around the world.
What’s true for driving will apply to AI in the months and years to come.
As always, if you spot mistakes in this or any of my earlier posts, ping me at:
- Twitter: Chua Chin Hon
- LinkedIn: www.linkedin.com/in/chuachinhon
PS: This article was not written by ChatGPT, though I should perhaps try to use it for my next Medium post.
