OpenAI's video generation tool Sora took the AI ​​community by surprise in February. With fluid, realistic video that looks miles ahead of competitors. But the carefully staged debut left out a lot of details — details that a filmmaker filled in through early access to create a short using Sora.

Shy Kids is a Toronto-based digital production team selected by OpenAI as one of the few. To make short films Primarily for OpenAI promotional purposes, although they were given considerable creative freedom. In creating an “airhead”. In one Interview with visual effects news outlet fxguidepost-production artist Patrick Cederberg described “actually using Sora” as part of his work.

Perhaps the most important point for most people is this: while a post featuring OpenAI's shorts might lead readers to believe that they sprung more or less entirely from Sora, the reality is that these were professional productions. , which was complete with strong storyboarding, editing, color correction. and post-work such as rotoscoping and VFX. Just like Apple says “shot on iPhone” but doesn't show studio setups, professional lighting, and color work after the fact, Sora's post only talks about what it tells people to do. gives, not how they actually did it.

Cederberg's interview is interesting and fairly non-technical, so if you're at all interested, Go to fxguide and read it. But here are some interesting nuggets about Sora's usage that tell us that as impressive as it is, this model might be less than we thought.

Control is still the thing that is most desired and also the most exciting at this point. … The closest we could get was just to be very descriptive in our notation. Defining the wardrobe for the characters, as well as the balloon type, was our approach to consistency since there is no feature yet for full control over shot-to-shot / generation-to-generation consistency.

In other words, matters that are simple in traditional filmmaking, such as choosing the color of a character's clothing, require extensive work and scrutiny in a generative system, as each shot is created independently of the others. This can obviously change, but at the moment it's definitely a lot of work.

Sora outputs also had to be looked at for unwanted elements: Cederberg explained how the model would routinely generate a face on a balloon for the main character's head, or a string hanging from the front. They had to be removed in post, another time-consuming process, if they couldn't get a signal to remove them.

Precise timing and movement of the characters or the camera isn't really possible: “There's a little bit of temporal control over where these different movements happen in the actual race, but it's not precise … it's like shooting in the dark. ” said. Cedarburg.

For example, timing a wave-like gesture is a very approximate, suggestion-based process, unlike manual animation. And a shot like an upward pan on the character's body can reflect what the filmmaker wants – so in this case the team rendered a shot in portrait orientation and cropped pan in post. The produced clips were also often in slow motion for some reason.

An example of a shot as it came out of Sora and how it ended up in the short. Image credit: Shy kids

In fact, the use of everyday filmmaking language, such as “panning right” or “tracking shot” were generally inconsistent, Cederberg said, which the team found quite surprising.

“The researchers, before they approached the artists to play with the instrument, weren't really thinking like filmmakers,” he said.

As a result, the team ended up using hundreds of generations, 10 to 20 seconds each, and only a handful. Cederberg estimated the ratio at 300:1—but of course we'd all be surprised at the ratio of a typical shoot.

The team actually Made a little behind the scenes video Explaining some of the problems they ran into, if you want to know. As with AI-related content, The comments are quite critical of the entire effort. – although not as rude as that We recently saw an AI-powered ad.

The last interesting wrinkle concerns copyright: if you ask Sora to give you a “Star Wars” clip, she'll refuse. And if you try to get around it with “marauding guy with a laser sword on a retro-futuristic spaceship”, it will refuse too, as it recognizes through some mechanism that you What are you trying to do? He also refused to do an “Aronofsky typeshot” or a “Hitchcock zoom.”

On the one hand, this makes perfect sense. But that begs the question: If Sora knows what these are, does that mean the model was trained on that material, better able to recognize that it's infringing? OpenAI, which keeps its training data cards close to the vest – to the point of absurdity, as CTO Meera Murthy's interview with Joanna Stern – will almost certainly never tell us.

As for Sora and its use in filmmaking, it's clearly a powerful and useful tool in its place, but it's not the place to “make movies out of whole cloth.” Yet. As another villain once famously said, “That comes later.”