Toward Intelligent Subtitling
Part 7: A New Level of SDH

Since its inception, subtitling for the deaf and hard of hearing has come a long way. From the early days of including only dialogue, to labeling speakers, plot-critical sounds and prosodic elements, and then to the evocative captions of today, we’ve witnessed a steady evolution in how we approach SDH.

And so has evolved the role of the captioner: from a technical operator who transcribes speech to text, to a sound annotator who enables media accessibility, and finally to a creative interpreter who makes that access not just functional and utilitarian — but artful, engaging and human.

It’s been an interesting trajectory, and after much research and thinking, I’m ready to share my vision for the next, fourth evolutionary stage of this role — holistic designer.

Below you will find my musings, inspired largely by Sean Zdenek, Mark Mangini and Randy Thom.

Sonic storytelling

When people talk about sound in movies, they usually refer to scores, soundtracks and hard effects, like gunshots, explosions, and jump-scare stingers. “Wow, that film’s sound was great!” — is what you’ll usually hear about seat-shaking booms and gripping orchestration rather than anything else. And while indeed much of audio design goes into keeping the audience at the edge of their seat, making them feel sentimental, or simply advancing the plot, upon closer look you will find that there’s a lot more to cinematic sound than most people know.

You see, sound is a storyteller. Just like picture, it guides your thoughts and feelings in a deliberate, coordinated way, as intended by the film’s creators. An invisible second narrator, it seeks to tell you about the characters, the world they live in, how they relate to each other, and how they change over time — all in service of one singular goal: to move you. To fascinate, to engage, to shift perspectives.

And how does it do that? By using a few smart methods which I’ll cover below.

1. Characterization

When well-crafted, a film’s sound can give us a glimpse into a character’s state of mind or role in the story. It can be a certain type of music played when we see them — perhaps mournful, whimsical or triumphant; the tone, register or style of their speech (e.g. stuttered for a shy teenager or seductive for a femme fatale); and also other acoustic details surrounding the character:

A rapid, amplified heartbeat can indicate stress or paranoia
A harsh scrape of a kitchen utensil can give away suppressed anger
Muffled jingles from fiddling with keys in one’s pocket can betray nervousness

To give you a concrete example, here is the sound team behind Ford v. Ferrari explaining their approach to creating Henry Ford’s footsteps:

Source: Vanity Fair YouTube channel

The audio mixing here aims to portray Henry Ford as a powerful, authoritative, larger-than-life figure — and so should your SDH sound descriptions, which must include the steps as loud, echoing, booming, resonant or even commanding.

Here’s another example from How to Train Your Dragon. Consider how much we can tell about these beasts just by listening to their diverse, painstakingly crafted vocalizations:

Frenzied, squabbly and pest-like at first, then a bit comical — “all bark and no bite”, and finally affectionate: not mindless monsters that must be slain but creatures that can be understood and befriended.

[excited chirping]

[aggressive squawks]

[tiny, squeaky roar]

[dizzy moan]

[inquisitive trill]

[soft, contented purring]

Describing all these vocalizations as simple [growls], [hisses] and [roars] would flatten out these reptiles’ personalities and make them less vibrant, less interesting and less understandable for DHH viewers.

An absence of sound can also be quite revealing. Take a look at this factory fight scene from Terminator 2 and see if you notice anything unusual about its acoustics:

Indeed — the sounds of human struggle are missing: no laboured grunting, panting, moaning or screaming. Through such unusual omission, we are reminded that these are two cold, unfeeling, emotionless cyborgs. Naturally, we must convey this for deaf viewers by adding [silent struggle] or something to that effect.

2. Worldbuilding

More often than not, a film’s sound will give you a sense of the world its characters inhabit: the different locations, environments, historical periods, etc. Think of The Lord of the Rings and its regional leitmotifs:

Source: StudioBinder YouTube channel

Each region has it own sonic signature that conveys its character and function in the movie. As a subtitler, you should respect these musical phrases in your SDH and describe them properly.

Another good example is Blade Runner 2049’s multi-layered audio track. Conjure the following in your mind:

Constant whir and hum of machinery, implying a world overrun by technology
Soulless ads and police announcements, projecting corporate and state control
Cityspeak in a mix of languages: a global yet fractured, alienating urban environment
Inescapable city noise suggesting immense overpopulation and a lack of personal space

A dark, dystopian world revealed through not only image but also sound.

How will you describe these sounds? What will you choose to keep and to omit? Whatever you decide, make sure to capture the essence of the film’s cinematic universe and all its components.

3. Plot device

Sound can also be integral to the viewer’s understanding of the story. One of my favorite examples comes from Blow Out. In one of the scenes, John Travolta’s character — a sound technician — hears something strange at a local park while trying to record effects for a movie. He can’t put his finger on what could be the source of this odd sound, and neither can we, the audience:

Shortly after, in that same park he witnesses a car accident in which a presidential candidate dies, and then, much later in the movie, we hear this whirring coming from a hitman’s strangulation wire:

And that’s when it clicks: the car crash was no accident — this guy was behind it. We instantly connect the two scenes based on one oddly specific sound. So, its SDH description should stand out yet remain vague, to make it intriguing in the first instance and connectable in the second — e.g. [high-pitched whirring].

But plot-device sounds don’t have to be as elaborate as the example above. It can be [tepid applause] to indicate a public speaker’s low popularity, [uncomfortable laughter] to portray a woman in trouble, or someone saying “I’m from New York” to a police officer [in a thick Boston accent] (yes — he’s lying). What matters is that you retain this information in your captions and don’t leave deaf viewers confused or ignorant of what’s going on.

4. Mood and emotion

We tend to think that it’s the music that does the heavy lifting for imbuing things with sentiment, and that is kind of true, but effects can play a big part as well. Think of the sound of closing a door — what emotive qualities can it convey? On the surface, not much, but if you give it some thought...

A door slam can indicate rage, conflict or irreversible decisions
A soft gentle click can be a sign of love, tenderness or care
A long heavy creak can bring suspense, dread or foreboding
A near silence can hint at someone trying to not make a sound, which creates tension
A pressurized hiss or mechanical sequence can give a sense of danger from whatever is locked inside.

Or imagine the sound of rain. What can it do? How can it affect the mood? Let Thomas Flight explain:

Source: Thomas Flight YouTube channel

It’s not just “rain” anymore — no, it’s [sizzling rain], [gentle pattering], [raging downpour], and so on. An important element of storytelling, it should ideally be accessible to DHH viewers.

Going back to music, it can indeed move us quite strongly: it can amuse, sadden, exhilarate, and do much more. I particularly like the epic, goosebump-inducing score accompanying Dune Part Two’s arena fight:

A note of caution, though: remember to not overdo it, trying to subtitle every sound you encounter. After all, people don’t want their eyes to be glued to the bottom of the screen, so skip audio cues that can be gathered from the visuals, that overlap with plot-pertinent dialogue, or that carry little narrative significance.

I watched this movie with captions enabled, and all they had to say for this part was [dramatic music playing]. That irked me a bit. What are deaf viewers supposed to imagine? Aren’t there better, more apt, more evocative words for this in the English dictionary? — [grandiose score erupts] or something like that? Of course, we should not over-indulge in fancy descriptions but rather try to step into DHH shoes and think in terms of what they need, but a little bit of flair to evoke the right emotion would seem not only acceptable but also welcome.

5. References

Think of the BRAAAM from Inception, or the DUH-dum Jaws theme, or the Wilhelm Scream. These and other famous sounds get added as allusions in movies, shows, games and even short YouTube videos all the time, usually for comedic or dramatic effect. For instance, here’s a clip from a sketch about forgetting reusable grocery bags. Can you catch the reference?

Source: Aunty Donna YouTube channel

Yes, exactly — both the sound and the picture refer to Denis Villeneuve’s Dune. Now, imagine you are deaf and all you see is [suspenseful music]. How do you interpret a sudden shift in the color palette and lighting? It will still make some sense as a moment of realization, but for those who’ve watched Dune, the connection will be lost. Do you keep the reference or not? If so, how? — The decision is yours.

6. Metaphors

I’ve already covered this function of sound in a previous article — the train screech in The Godfather conveying Michael Corleone’s internal struggle, the Looney Tunes TV sounds in 12 Monkeys mirroring Jeffrey’s craziness, and of course the Alka Seltzer drink in Taxi Driver expressing Travis Bickle’s sizzling anger.

So, let’s find another example. How about one of the most famous sonic metaphors in film history: the ceiling fan in Apocalypse Now’s opening scene — an acoustic embodiment of PTSD:

Mm... [ceiling fan thumping rhythmically]? [ceiling fan thumping like helicopter blades]? Something else?

This one sound gives us a portal into the character’s mind and hints at the film’s overall message about war and its effects. So, it must be preserved in your captions.

Alright, enough for now. There are many more functions of sound in movies, and a fair few examples to show, but I need to get to the point.

The Next Level

What I’ve described above, albeit illuminating to a Level 2 SDH creator, is not news to a creative interpreter. At Level 3, they know how to gather creative intent behind sounds and how to convey it in their descriptions, which sounds to keep and which not to, and what this or that captioning decision will mean for deaf viewers. They deliver not just access but an experience.

Yet, a Level 4 captioner — a holistic designer — goes even further. Instead of focusing on individual sounds or scenes, they aim to see the full picture:

What story is this film trying to tell?
In what way does it attempt to move the viewer?
How does its sound contribute to that?
How do I reproduce that contribution for a deaf audience?

The idea here is to deconstruct the movie narratively and to create an SDH strategy for it, which will inform your captioning choices and produce a cohesive, directed experience.

Genre

To achieve the movie’s intended effect, you first need to think about its general type and where it’s trying to take the audience. For instance, in a horror film, which aims to evoke fear and suspense, you might want to:

Contribute to the gripping, eerie atmosphere:

[low, ominous drone]

[indistinct murmurs]

[a child’s faint giggle]

Help to create tension and make the viewer guess what happens next:

[twig snaps in distance]

[floor creaks behind door]

[doorknob turns slowly]

Add to discomfort and shock:

[blood-curdling shriek]

[gasping, desperate breaths]

[head splatters wetly]

Conversely, in a comedy, the fun factor takes center stage. The guiding principle here is to make the show as humorous for DHH viewers as it is for everyone else. So, we can let loose and adopt a more playful approach:

Keep the joke

[prolonged, trumpety fart]

[fart ends on a high note]

Focus more on prosody:

[in squeaky voice] You will pay for this!

[sarcastically] Ha. Ha. Ha.

Use onomatopoeia:

[AWOOGA]

[loud, metallic BONK]

As far as action movies, which want to blow you away with all the effects, you should:

Create a sense of dynamism:

[tires screech urgently]

[flurry of sharp gunshots]

[tense, pulsing synth beat]

And convey the bombastic sounds:

[explosion rumbles the street]

[truck engine roars to life]

[epic rock anthem kicks in]

And so on: whether it be era-accurate descriptions for period drama; imaginative, futuristic ones for sci-fi; or poignant SDH for romance — you want to pick a particular angle for your captions and coordinate them for the best effect.

Theme

A film’s central idea, the topic it looks to explore, is its theme. It can be something simpler, like friendship, passion, coming of age, or good versus evil — or it can be more nuanced and layered, philosophical even.

Famous filmmakers quite often have a strong thematic preference. Christopher Nolan likes delving into the concept of time, Martin Scorsese has been known for religious motifs in his movies, and David Lynch never made a secret of his fascination with dreams.

So, what does that mean for SDH?

A clear example here is Raymond Chandler’s script for never-created movie Playback which sound designer Randy Thom wrote about in his blog. Randy asked himself this question: “If I could work on the film in close collaboration with its director, how would I go about it?” — and in a series of articles gave a brilliant analysis. Here are a few quotes that pertain to us:

“ Immediately I’m thinking about themes. The name of the movie is Playback. It’s about a re-playing of something [bad] already experienced… a repetition. I’m wondering how I can underscore with sound design the idea of repetition, especially a kind of repetition that is not necessarily pleasant. ”

“ I’m thinking mainly about rail clacks and other kinds of repetition inside this train passenger compartment. I’m wondering if objects the train passes close to could make a double whoosh. Maybe we would assume that for each object we see going by there might be another one we can’t see which would generate a second doppler. Or perhaps the director would consider seeing two electrical poles fly by the train windows in quick succession in some cases. ”

“ The sound of water droplets falling onto a metal surface on the balcony could be in roughly the same pattern as the train wheel clacks we heard earlier. ”

“ These rhythmic and tonal ideas I may propose are going to need to be coordinated with the composer. ”

Throughout the film, its score, music and effects would again and again play into the theme of repetition. So, we’d need to do the same in our captions by using a specific vocabulary with words like dual, double, twin, repeated, etc. — and by viewing each sound through this theme’s lens, seeing if there’s a connection, and if there is, subtitling accordingly.

Narrative arcs

A film’s sound design strategy doesn’t have to be monolithic; it can vary from one part of the story to another, sometimes quite drastically. Dream sequences with surreal, reverberating acoustics; outer space acts featuring the complete silence of the cosmos; war combat scenes with a barrage of zipping bullets and metallic ricochets — if combined in one movie, such different segments may require not only separate angles to your SDH but also some coordination between them to retain the original creative idea.

A good example of this comes from Cast Away. Early on in the filmmaking process, director Robert Zemeckis instructed Randy Thom to not include any sounds of birds, insects, frogs or other life for the entire island arc — to portray Tom Hanks’s character Chuck as utterly alone and isolated. They even avoided inserting a score or anything non-diegetic for some 40 minutes of runtime, until the escape part, for the same reason. In contrast, Chuck’s pre-island life appears noisy, even cacophonous. It’s bustling, fast-paced and time-driven — he can hardly carve out a moment for anything or anyone, including his girlfriend Kelly. Compare:

Hearing viewers can feel his seclusion subconsciously. So, to respect this directorial sonic vision in our SDH, we would need to:

Underscore the urgency and density of pre-island soundscape via prosody and descriptions:

[pager beeps incessantly]

[hurriedly] Let’s go! Let’s go!

[busy, overlapping conversations]

[jet engine whines loudly]

Create a sense of isolation on the island by subtitling frugally (to avoid excessive “chatter”) and using “solitary” phrasing where warranted:

[silence, broken only by waves]

[wind fades into stillness]

[thunder rumbling in distance]

Ensure contrast between the two arcs through adding slightly more subtitles than normal in the former and slightly fewer in the latter, thus contributing to a palpable change in the visual momentum.

It’s not going to be a 1:1 experience for the deaf — that isn’t the goal of SDH — but it’ll approximate the idea.

Character development

It’s quite standard for movies and shows to feature characters that evolve over the course of the story: Walter White’s moral descent, Prince Zuko’s redemption arc, Sansa Stark’s maturation — there’ve been countless protagonists, villains and sidekicks experiencing a gradual personality change.

And so changes the sound that accompanies these characters, which may be produced by themselves or their surroundings, or be a non-diegetic leitmotif. Randy Thom gave dozens of examples in his blog, but one I particularly like comes from Joker, in which Arthur Fleck’s laughter goes through four stages, along with his personality:

1. Subjugating (at the start). An inescapable source of shame and isolation — involuntary and painful.

2. Cathartic (after the train murders). A release of pent-up humiliation — the laughter is still pathological, but it’s growing more and more genuine and controlled.

3. Deliberate (at the show). Defiant, performative and mocking. The laughter is more of a choice now.

4. Liberated (at the end). Private and contented. Arthur now owns his laughter. He has become Joker.

This acoustic transformation must be correctly captured in your SDH; your descriptions need to evolve along with it — from [strained laughing fit with painful sob] to [quiet contented laughter mixed with cackle].

As far as leitmotifs, The Imperial March from Star Wars is a textbook example. Associated with Darth Vader, it slightly changes throughout the franchise to follow his character development:

Source: StudioBinder YouTube channel

Captioning evolving leitmotifs can be difficult, especially if accompanied by dense dialogue or onscreen text, but you need to detect them and understand what they’re trying to convey — and then map out how you will render those motifs in a cohesive, harmonious manner.

Allegory

There’s the plot and the story — and then, beneath the surface, there’s what the film is really about. Indeed, upon attentive viewing, you’ll discover that:

Avatar examines colonialism and exploitation of indigenous people
Dawn of the Dead comments on mindless American consumerism
Zootopia explores systemic racism and prejudice
SpongeBob SquarePants Movie is a retelling of Homer’s Odyssey.

Of course, not every film has a second layer — most don’t — but when there is one, SDH becomes rather tricky, as you need to preserve both the literal and the figurative readings. Two birds with one stone, so to speak.

When I was much younger, I watched A Bug’s Life, an animated movie about ants rising up against their bullies, grasshoppers. Recently I watched it again and it felt... different. Workers collecting resources for an “upper class”? Struggling to feed themselves and to comply with ever-increasing demands? Overthrowing their oppressors upon realizing that they outnumber them? — Seems to me like a communist manifesto of sorts — or, at the very least, an allegorical story about class struggle. And the often-militaristic sound design plays into this metaphor:

Heavy thumping of a transport helicopter, a fighter squadron droning over an orchestral battle score, raindrops bursting like artillery shells, loud warhorn hrooming, dogfight maneuvering — your job here is to make sure to keep these sounds in your SDH and to add warlike qualities to your descriptions in a careful, non-overt way that works both for deaf kids enjoying a fun movie and deaf adults connecting all the dots.

Engagement

A good portion of media content out there is meant for passive consumption, as a way to relax after a hard day at work and to replenish on endorphins. Occasionally, though, a movie will invite you to take a more active role, to engage with it. Think of the classic whodunit where you get to piece clues together along with the characters, or an unreliable narrator story that requires rearranging its parts in your mind to make sense of what’s going on, or a wholly interactive movie where you must make some decisions. In such cases, audio can play a crucial role, and if you don’t describe it holistically, with clear intent to engage, deaf audiences won’t be able to participate in an equitable manner — in solving the mystery, reconstructing the timeline, or choosing what to do next.

To demonstrate, let me present you with a curious case: The Guilty. Save for a few brief shots, almost the entire movie consists of one character — a 911 emergency dispatcher — talking on the phone, trying to help a woman who has been abducted. As a hearing viewer, you aren’t so much watching the movie as listening to it, intently, to interpret the speakers’ tone of voice, to catch faint but important background noises in each call, to analyze every pause and frightened breath — and get a clear picture of the unfolding story.

An exercise in active listening, The Guilty engages you completely. So, to make it work for DHH audiences, we must carefully inspect each conversation — the prosody, the pacing, the sounds — as well as the music accompanying it, and then produce subtitles that provide a well-choreographed, inclusive experience.

◯ ◯ ◯ ◯ ◯

The Caveat

Now, even though my ideas above might seem great on paper, they clash with the stark reality of our trade. With such tight deadlines and limited pay, how can we afford to watch each film or episode multiple times and analyse it deeply? There aren’t enough hours in a day! And furthermore, how can we possibly know what the authors really meant? Or what instructions the director gave to the sound team?

Well, the truth is, we can’t — and that’s why communication between the creators and the subtitlers is a must for Level 4 SDH — or, if not that, at the very least we would need a reference document outlining the movie’s sound design strategy, thematic scope, allegorical concept, recurring leitmotifs, evolving elements as well as other important details. Without that, Level 3 SDH will often remain the best that we can do.

Here’s hoping that things will change and we’ll be able to give our viewers fantastic, equitable captions!

Alright, this concludes the article. Thanks for making it all the way to the end! As always, if you have questions, thoughts, or remarks, feel free to share them in a comment below. And if you enjoyed my writing, make sure to subscribe to my blog.

Until next time!