Scott Man ran into a problem: too many f-bombs.

The writer-director spent production on “Fall,” his vertigo thriller about rock climbers stuck atop a remote TV tower, encouraging the two leads to enjoy their dialogue. This improvisation landed 35 “f-cks” in the movie, putting it firmly in the R-rated territory.

But when Lionsgate signed on to distribute Fall, the studio wanted a PG-13 adaptation. Film sterilization means abrasion All but one of obscenities.

“How do you solve that?” Mann recalled from the glass-lined conference room of his Santa Monica office in October, two months after the film’s debut. A prop eagle he had captured from the set sat in the lobby.

Reshoots are, after all, expensive and time consuming. Mann filmed “Fall” on top of a mountain, he explains, and struggled the whole time with not only COVID but also hurricanes and thunderstorms. A colony of fire ants had taken up residence inside the film’s main set, a hundred-foot metal pipe, at some point; When the crew woke them up, the swarm enveloped the group “like a cloud”.

“Fall was probably the hardest movie I ever made,” said Mann. Can he avoid a repeat?

He realized the solution might be a project he was developing alongside the film: artificially intelligent software that could edit shots of actors’ faces well after principal photography wrapped, seamlessly changing their facial expressions and mouth movements to match newly recorded dialogue.

It’s a deceptively simple use of a technique that experts say is poised to transform nearly every dimension of Hollywood, from business dynamics and financial models to how audiences think about what’s real or fake.

AI will do for motion pictures what Photoshop did for still images, said Robert Wahl, an associate professor of computer science at Concordia University Wisconsin who has written about CGI ethics, in an email. “We can no longer fully trust what we see.”

Software solution to questionable duplicates

It took a particularly soul-wrenching collaboration with Robert De Niro to propel Mann into the world of software.

De Niro appeared in Mann’s 2015 crime thriller “Heist,” and the two put a lot of time and thought into the famous actor’s performance. But when it came time to adapt the film for foreign releases, Mann said, he wasn’t satisfied.

When films are shown abroad, dialogue is often re-recorded in other languages. This process, called “dubbing,” makes the film universally accessible, but can also result in a jarring sight of the actor’s mouth flapping out of sync with the words he is supposed to be saying. A typical solution would be to rewrite the dialogue so it better fit the pre-existing visuals – but for the sake of clarity, these changes sacrifice the creative team’s original vision.

Of the dubbing, Mann said: “All the things I worked on in nuance with Robert De Niro have now changed.” “I was kinda devastated.”

A follow-up film, Final Score, deepened those frustrations. Mann tried scanning the heads of the cast members so he could better synchronize their speech, but the process proved expensive and the end result looked bizarre.

It wasn’t until the search for more new solutions that visual effects enthusiasts found 2018 Academic paper Identify a possible solution: neural networks, or computer programs imitation of the structure of the brain, which sought to transfer the facial expressions of one actor to the face of another.

Impressed, Mann reached out to the paper’s authors and began collaborating with some of them on a rudimentary “fobbing” tool—visual, not audio, dubbing. And the subsequent addition of Nick Lynes, a friend of a friend with a background in online gaming, has given the team a foothold in the technology sector as well.

Together, the emissaries of three very different worlds—cinema, science, and the software industry—build Flawless, an AI filmmaking venture with offices in both Santa Monica and London.

In very general terms, the company’s technology can identify patterns in actors’ vocalizations (or the sounds they make) and visions (or what they sound like when they make those sounds), and then—when presented with newly recorded voices—update the onscreen visuals to match. Last year, Time magazine It is considered One of the best inventions of 2021 is the “fix for movie dubbing”.

The scramble to remove dozens of f-bombs from the Fall movie, however, begged a question with potentially much broader ramifications: Rather than simply changing the language the characters speak, could Flawless alter the content of what they said?

“We went to a recording studio in … Burbank with the actresses and said, ‘Okay, here are the new lines,’” said Mann, who lives in Los Angeles. “Then they plugged the new voice into vubbing software, which adjusted the stars’ on-screen facial movements accordingly.” .

He said, “We put the footage down, the MPAA re-reviewed it and gave it a PG-13, and that’s what went into the movie theaters.”

Sitting in his Santa Monica conference room several weeks after the film’s release, surrounded by posters of “Blade Runner” and “2001: A Space Odyssey,” Mann laid out the results with a scene in which one of Fall’s protagonists bemoans.

“Now we’re stuck in this awful, stupid tower in the middle of nowhere!” Virginia Gardner yelled at Grace Caroline Curie as the two huddled atop a precariously high platform.

Virginia Gardner and Grace Caroline Corey in “Autumn.”

(Lionsgate)

Advertisement

A moment later, Mann repeated the scene. This time, however, Gardner’s dialogue is noticeably harsher: “Now we’re stuck in this stupid f-cking tower in the middle of nowhere.”

The first version was what came out in August to more 1500 American theaters. But the latter—which contains dialogue fit for a sailor—was what Mann actually filmed on a fire ant-infested mountaintop. If you didn’t know that the neural network reconstructed the actors’ faces, you likely had no idea their cleaned-up dialogue was a late addition.

Mann said, “You can’t tell what’s real and what’s not, and that’s the whole thing.”

Synthetics ethics

When it comes to filmmaking, that realism has obvious benefits. Nobody wants to spend money on something that looks like it came out of MS Paint.

But the advent of software that can seamlessly change what someone said has major implications for a media environment already awash in misinformation. The Flawless Core product is, essentially, just a more legit version of the “Deepfakeor CGI that mimics a person’s face and voice.

It’s not hard to imagine a troll who, instead of using these tools to cut vitriol from a movie, would make a viral video of Joe Biden declaring war on Russia. Pornography made with the digital likeness of a person also has it become a problem.

Flawless is not the only company working in this field. Papercup, which generates synthetic human voices for use in dubbing and voiceovers, aims to “make any video watchable in any language,” CEO Jesse Shemen told The Times.

CTO Hannu Pass said the visual effects use digital domain machine learning to render actors in situations where they can’t appear themselves, such as scenes that require a stunt double.

As these companies and others increasingly automate the entertainment industry, ethical questions abound.

Hollywood is already counting in its new ability to digitally recreate dead actors, as with Voiced by Anthony Bourdain In the documentary “Roadrunner” or Peter Cushing And Carrie Fisher In the recent Star Wars series. Holographic revival Being a late celebrity is also possible now.

The digitally altered dialogue “threatens to compromise the consent of the participants in the original,” said Scott Stroud, director of the University of Texas at Austin Media Ethics Program. “What the actors thought they approved of is not literally what they were created to do.”

The technology could open the door to changing movies long after they debut, said Denver Drozario, a Howard University marketing professor who has studied reanimations of dead actors.

“Let’s say… in a movie a guy drinks a can of Pepsi, and 20 years from now you get a sponsorship from Coca-Cola,” Drozario said. Would you change the Pepsi can to Coca-Cola? At what point can things change? At what stage can things be purchased? “

Mann said the advantages of his technique abound, from breaking down barriers of language and eliciting sympathy across borders to sparing actors the headache of reshoots. In his view, scenarios such as D’Rozario’s hypothetical sponsorship of Coca-Cola represent new revenue streams.

Mann added that Flawless was proactive in building a product that helps rather than replaces authentic human performance.

“There is a way to use techniques the same way he does [visual effects] The industry has already created, and it’s like: Do it safely, do it right, do it legally, with the consent of everyone involved,” he said.

The manager went on to say that the company has already involved “all the major syndicates” on how to make and use this technology in a reasonable way.

SAG-AFTRA representatives stressed that AI filmmaking technology can help or hurt actors, depending on how it is used.

“Technologies that do nothing more than digitally improve the work of our members may only require the ability to provide informed consent, and possibly additional compensation,” Jeffrey Bennett, SAG-AFTRA General Counsel, said in an email. “At the other end of the spectrum are technologies that may replace traditional performances or that take the performances of our members and create an entirely new one; for those we maintain they are a compulsory bargaining chip.”

It’s a train that has already left the station, for better or worse.

Fall is currently streaming, and Mann said other films his company has worked on will be released this Christmas — though he can’t yet publicly name them.

If you watched a movie during the holidays, perhaps AI helped you create it.

Will you be able to say? Will it make a difference?