Customer Case, Workflow, Events

How the Associated Press (AP) uses AI to slash editing time

Charlotte Coppejans
October 4, 2021

The Associated Press processes around 20,000 hours of original footage on a yearly basis, both live and non-live content. To reduce the time taken to edit, and to ensure such content is available across the organisation and to its customers in the shortest possible time frame, they wanted to achieve the next level of automation with ‘automatic shot listing’. Limecraft, Vidrovr and Trint joined forces for this innovation challenge and built a solution that was brought to production this year. “All news companies spend an enormous amount of time on manual work. Given where AI is today, we felt like we could use that to tackle our problem”, says Sandy McIntyre, Vice President News at Associated Press.

 

Discover in this video how the AP applied AI to deliver automatic descriptions of live and non-live content.

The Associated Press wanted to know if AI could help them reduce the amount of time they spend on manual tasks. While those tasks are important, they are not part of the creative storytelling or the journalistic process. That is why AP wanted to remove the “grunt” work from the workflow: that led them to AI.

Limecraft delivered a pipeline using Vidrovr for scene description, facial recognition and gesture detection, as well as utilizing Trint for audio transcription, together delivering a single coherent and frame-accurate description of each individual shot. Moreover, Limecraft cracked the challenge to execute such processing on expanding files. Doing so, the AP has effectively emancipated several hundreds of hours of manual labour, and significantly reduced their turn around times.

There were two specific drivers to use this technology.

“The first driver is the large volume of material we have to deal with, which is increasing exponentially. The Associated Press transmits to hundreds of broadcasters and digital publishers around the world, processing about 20,000 hours of live content,” explains Sandy McIntyre.

“That is up to 150,000 edited items every year or hundreds of thousands of minutes of content –  which must be manually transcribed and shot-listed. The scale is vast and therefore the potential to save time is huge. We hoped that if we could solve a problem for ourselves, we’re also solving a problem for everybody who is a subscriber or customer of AP. That would be 700 broadcasters and probably twice that number of digital publishers.”

Another driver for AP was the human element: if you can automate the basic things, you give back time to your people to be more creative and add value to the business. Sandy McIntyre: “You have to think about this in an era of fake news. Fact checking and getting it right has never been more important. So, if you think about most news organisations, it is about being first with the news, and about being right and accurate. This allows both speed and accuracy to improve. And of course, more creativity in the editing process.”

How does Limecraft help maintain the quality of processing content?

Maarten Verwaest: “In our capacity as go‑between company, bridging core technology and end-users like The Associated Press, it’s our task and our mission to tune and push technology to a level where it becomes acceptable to journalists and end‑users. There is nothing more frustrating than correcting artificial intelligence over and over because it’s not learning and is then systematically recognising the wrong person or the wrong word.”

“This is where technology has evolved a lot in the last 12-24 months and it is also where the need for training becomes absolutely paramount. A good engine is a necessary, but insufficient, requirement. The more important success factors are training, training, and more training.

“Lastly, we recommend that those willing to deploy artificial intelligence look for a proper proofreading interface, allowing an ‘expert in the middle’ workflow. Artificial intelligence will make mistakes from time to time, and that creates a need for efficient correction and quality control. That must be fitted in as seamlessly as possible in your whole process.

“Based on these three conditions – the best engine, proper training, and a proper user interface – you will significantly increase your chances of success and quality.”

How did the Associated Press introduce AI to the work process?

Sandy McIntyre: “The Associated Press did two things. The first thing is that we realised that throwing all our content into the AI pot and trying to get AI to recognise it, wasn’t sustainable. The technology is not that smart yet. We realised that most of our content is governmental, political and diplomatic. For example, it might be US president Joe Biden getting off a plane, getting in a car or getting on a stage, making a speech, having a bilateral meeting, greeting the crowd, running a press conference and so on.”

“All these actions and behaviours you could apply to any world leader, any foreign minister, any celebrity. We knew that we could teach AI to recognise those kinds of actions, recognise speeches, and that we could possibly take 20 – 25% of the news content flowing through the ecosystem every day and gain understanding of that. So, we’ve taught the machine to recognise the top 300 names that appear most frequently on screens all over the world, and the actions that those folks might take within such political or diplomatic domains.”

“We’ve taught the machine to recognise the top 300 names that appear most frequently on screens all over the world, and the actions that those folks might take within such political or diplomatic domains.”

“The second is that we did this offline. It’s been a fast, iterative process which didn’t get hung up early on about how it integrates with legacy systems. Our approach was; let’s get something in a beta phase in the sandpit that works, and then let’s figure out how we integrate it into both the technical workflow and the editorial workflow. Let’s have the people who’ve done that work in the sandpit be the ones who are leading the conversation about integration. By doing it that way, we very quickly got away from a them-versus-us situation. Also, it allows you to take a very hard line. Is this going to work? How much help will it be? Is it worth persisting with?

“AI is as clever, sophisticated, and experienced as your assistant producer. But when you bring the learning loop to life, whereby knowledge and experience of the real world is captured in 12 months’ time, the machine will become way more experienced, more helpful, and more like a senior member of the editorial team.”

How did the Associated Press win over the users?

Sandy McIntyre: “With the introduction of all new technologies, there is an element of fear; fear that people might lose their jobs, fear of change, fear that users won’t get used to the new ways. So, to win over the users we had to be very open and honest with them about what we were trying to achieve: to liberate quality time.

“To win over the users we had to be very open and honest with them about what we were trying to achieve: to liberate quality time.”

“We also had to recognise that real time video is now such a big part of the news ecosystem and a growing part of it, and that that isn’t going away. Therefore, liberating that time is doubly important because the volume is going up. The speed at which we are judged in terms of success, preferably in live and real time, is the main reason we had to make these changes. So, we brought together a coalition of the willing.”

How can you increase the chance of users having a positive mindset towards AI?

Maarten Verwaest: “Quality of the output of artificial intelligence is a critical success factor for acceptance. Critical because it’s not proportional: zero accuracy obviously means zero acceptance and 100% accuracy means 100% acceptance, but, in between, it’s not a linear curve. Automatic speech recognition for professionals is an example. If the word error rate is 4 or 5%, it will cost you more time to correct the machine transcription than to draft it from a clean sheet. The word error rate should be pushed below 2%, and that’s a real challenge from an engineering point of view.

How does the AP deal with naysayers and doubters?

Sandy McIntyre: “We deliberately asked naysayers and doubters to come to the table because their input was super important. We knew that over time, if we could get this right, they’d probably become our biggest and most evangelical supporters. But, crucially, it had to be people who were doing this work, so that they could know the difference between how it was before and how it will be in the future. Also, so they could get to what Maarten was talking about, about understanding the tipping point between where it is faster to do it all manually versus when AI wins the race. So, what we have ended up with is a shift to how I would describe AI in terms of today: it’s the best assistant producer, it’s the best intern you ever had.”

Where is the future of AI for the Associated Press?

Sandy McIntyre: “The short-term future is cementing all the different components (like face recognition, speech recognition): up until now, the components were individual slices; now we need to put them together.”

“Next, we need to figure out how or if AI can help us with that knowledge, and take on the accidental bias of any content. An organisation like AP prides itself on fair, accurate, impartial news. Therefore, to measure our impartiality, to measure that fairness, to ensure that our reporting both reflects fairness and the world in which we live in terms of the people we choose to interview, there will definitely be an AI component to understanding if you got the tone right, or if you were biased in favour of one side or the other, one gender or the other or one ethnic group.“

“It’s my hope that other customers, like local news desks, will also start providing feedback to the data model so that we establish a momentum of co-creation.”

Maarten Verwaest: “Specifically, what we want to set up with the Associated Press is the integrated feedback loop from journalists into the brain, continuously updating the data model going forward. We hope this data, this corpus of data, will be usable by third parties in the future. It’s my hope that other customers, like local news desks, will also start providing feedback to the data model so that we establish a momentum of co-creation, rather than to leave it up to the big technology companies to consolidate those data models for their own profit. It is my hope that the technology evolves in that direction.”

To learn more about the use of AI in media and content-oriented automation, subscribe to our monthly newsletter. (Scroll to the bottom of this page)