What Is Multimodality? The Problem of Constrained Translation in Localisation

A game screen may carry speech, subtitles, background music, a counter and a warning icon all at once. The user reads all of it as a single whole. That is what makes translation difficult: the only thing you change is the text, yet the result shows across the whole. This is what is known as multimodality.

Contents

The concept describes meaning being produced through more than one channel at the same time. Image and writing, sound and pop-up window, speech and movement combine to form a single message. Almost every digital product works this way; an interface made up of text alone is the exception today.

This article takes up the definition of the concept, the channels that carry meaning, the debate around constrained translation and the distinction between internal and external constraints. We then show how that framework translates into everyday localisation practice.

What is multimodality?

In its theoretical definition, a mode is any resource used to produce meaning for the purposes of representation and communication. Writing is a mode; so are colour, font size, moving image, sound effect and the direction of the text. The user processes these resources together rather than one by one.

The warning this perspective brings to translation is plain: the text is only one of many components interacting inside the source product. Placing the verbal dimension ahead of the other meaning-making channels turns translation into an incomplete operation. What has to be preserved is the whole as it is seen on screen.

Meaning is also formed in the encounter itself. The meaning potential the medium carries, the social and cultural setting in which it is used and the knowledge the user brings to that moment all work together (Jewitt 2016). The same screen is read differently in an emergency and at leisure.

In practice this means that the same sentence is translated differently according to context. The continue on an insurance application screen and the continue in a game menu are the same word, yet they do not set up the same expectation. One foregrounds seriousness, the other fluency, and that difference shows in the register of the translation.

The four channels that carry meaning

In multimodal texts the user takes in information from different channels and turns it into a coherent whole. Pedersen (2011) groups these channels under four headings. The model was developed for games, but it offers a workable framework for any product being localised.

1. Writing that appears on screen, that is, the verbal visual channel.
2. Music and sound effects, that is, the non-verbal auditory channel.
3. Speech and voice-over, that is, the verbal auditory channel.
4. Movement and the sequence of events, that is, the non-verbal visual channel.

The four channels do not carry equal weight in every product. On a help page made up of text alone the first channel dominates. In a promotional video the third and fourth come to the fore. Budget and schedule should be planned according to that weighting.

Channels may support one another and may also contradict one another. If a warning sound plays while a positive message is on screen, the user hesitates. An eye that looks only at the writing during translation will miss that contradiction.

This is why the pre-delivery check in multimodal work has to be carried out on screen. A translation that looks flawless in the file may land in the wrong place inside the product or collide with another element.

Constrained translation and subordinate translation

Translation studies has long examined work of this kind under a heading of its own. For translations in which the text is obliged to conform to elements outside itself, the terms constrained translation (Titford 1982) and subordinate translation (Mayoral et al. 1988) are used. Localisation falls largely into this class.

Although the word constraint carries a negative ring, it is descriptive. The translator here is not a free writer, but works within a space framed by the width of the screen, the duration of the audio and the content of the image. That frame shapes both the process and the product.

The presence of a constraint also changes the criterion of quality. A rendering that carries the whole of the source sentence but does not fit on the button does not count as a success. A short rendering that keeps the core of the meaning and respects the limit is in most cases the more accurate solution.

For this reason per-word pricing on its own can be misleading in multimodal work. Fitting a short string within a limit can take longer than translating a long paragraph. The type of constraint should also be taken into account when estimating time.

Internal constraints and external constraints

Constraints fall into two groups. Internal constraints have to do with how the text agrees with visual or auditory elements. External constraints are limits imposed from outside, such as the size of the screen or synchronisation with speech. The table below compares the two with examples.

Type of constraint	Example situation	What the translation must do
Internal constraint	A screenshot on a help page	Match the button name in the image exactly with the name in the text
Internal constraint	A label presented together with an icon	Keep the text consistent with the action the icon suggests
External constraint	A narrow mobile screen	Produce a short rendering that respects the character limit
External constraint	The duration of a character’s speech	Write text that fits the duration without breaking the rhythm

The most frequently overlooked example of an internal constraint is the screenshot. The manual text is translated into Turkish, but the interface in the image stays in English. The user cannot find the button being described and the help page has precisely the opposite effect.

With external constraints the problem is usually one of measurement. If the limit is not stated in advance, the translator writes to their own estimate. Shortening strings that overflow after delivery takes longer than working to the limit from the start, and produces more errors.

Which practices did multimodality give rise to?

Some established habits in the industry are a direct result of this structure. Visual localisation tools that work on a what-you-see-is-what-you-get logic were developed to let the translator see the text in its place on the screen. The aim is to make the constraint visible at the moment of translation.

Stating a maximum character count for every line in game scripts arose from the same need. The text has to fit both the time and the space available. If the limit is written into the file, the translator gets the length right at the first attempt.

The note fields in file formats are part of this framework too. A short note explaining which action a segment relates to and which screen it appears on is the multimodal context set down in writing.

In some workflows visual material is shared directly instead of a note. A picture of the screen on which the segment appears, or a short screen recording, does the job faster than long explanations. Material that takes a few minutes to prepare can spare hours of correspondence during translation.

Finally, visual and audio assets need to be planned as a separate line of work. Subtitling, voice-over and the reproduction of the writing inside images do not run to the same schedule as text translation. You can find more detail on our services page.

How multimodality relates to accessibility

Meaning being carried through more than one channel becomes a problem when the user cannot reach one of those channels. A user with a visual impairment cannot use the visual channel, and a user with a hearing impairment cannot use the auditory one. In a multimodal product, accessibility means offering the equivalent of the missing channel through another channel.

This is why accessibility rules require a text equivalent for non-text content. Descriptions of images, transcripts of audio and text equivalents of video are all produced on this logic. In localisation these equivalents have to be carried into the target language as well, and this is exactly the part that is most often overlooked.

Conditions of access are not limited to permanent impairments. Playing a game with subtitles in a noisy environment, using it one-handed with an arm in plaster, or relying on spoken directions while driving all fall within the same framework. A variety of channels helps in these temporary situations too.

Multimodality, then, is not only a difficulty but also an opportunity. A product that can present the same information through more than one channel reaches a wider group of users. The localisation stage is a natural occasion to make use of it.

The Türkiye context

In Türkiye, digital products are promoted largely through short video and visual content. That shifts the centre of gravity of localisation from writing to audiovisual material. Projects that translate only the web copy and leave the images as they are come across to the target audience as unfinished.

The structure of Turkish makes external constraints heavier. Because the language is agglutinative, renderings grow longer, and since dropping suffixes usually damages the meaning, there is little room for shortening. Allowing extra width for Turkish at the design stage is more economical than repairs made afterwards.

Another local detail is that the text in images is not stored together with its layer information. When the source files cannot be found, the image cannot be reproduced and goes live in its English form. Archiving visual assets in editable form makes every future translation easier.

There is a similar gap on the voice-over side. Most domestic products arrive with a written interface only; spoken guidance, in-app narration and video content are added later. These added layers can end up out of step with the terminology in the interface and leave the user with the impression of two separate products.

What does this mean for you?

When you place a localisation order, describing the scope by word count alone falls short. You also need to state how much of the meaning in your product comes from channels other than writing. That information has a direct effect on both price and schedule.

1. Share screenshots and a short promotional video along with the text to be translated.
2. Prepare editable source files for the writing inside images.
3. State the character limits per string in the file.
4. Plan audio and subtitling work as a separate line item.
5. Carry out the post-delivery check in the working product, not in the file.

Preparation that meets these five points reduces revision rounds markedly. For a plan of work suited to your scope you can fill in the quote form, and you can look through articles on related subjects in our knowledge centre.

Frequently asked questions

What does multimodality mean in localisation?

It means that meaning is built not by text on its own but by text working together with image, sound, movement and interactive elements. The translator changes only one part of that whole, yet is responsible for all of it.

What does constrained translation mean?

It is the kind of translation in which the text is obliged to be consistent with elements outside itself. Factors such as screen width, audio duration or fit with an image narrow the translator’s options.

What is the difference between an internal and an external constraint?

Internal constraints have to do with how the text agrees with visual and auditory elements; for example, a screenshot on a help page matching its counterpart in the interface. External constraints are limits imposed from outside, such as screen size or synchronisation in time.

Do tools that translate what you see solve the problem?

They make it easier, but they do not solve it on their own. Seeing the text in place reduces errors of length and context; decisions such as fit with the audio and cultural appropriateness still call for human judgement.

References

• Jiménez-Crespo, M. A. (2024). Localization in Translation. Routledge, pp. 104–105.
• Pedersen, J. (2011). Subtitling Norms for Television. John Benjamins, p. 10.
• Jewitt, C. (2016). Introducing Multimodality. Routledge, p. 68.