Pragmatics is concerned with context-based interpretations of language use, where linguistic choices are seen to derive their meanings not only from the system of language itself (that is, grammar, discourse systems and lexis), but also from contextual factors, such as the background of the speakers and the nature of their social relationships and respective intent, and other factors in the situational and cultural context of the interaction.
It must also be noted that, in a multimodal text, each mode finds its most immediate context of reference in the other mode/s, and this influences greatly the usage of each system. Since information can be drawn from different sources, in multimodal texts the message communicated by a single code is incomplete without the remaining information, as a code can rely on the other to express what it has left unexpressed or to enhance its meaning: usage in context, and hence Pragmatics, becomes then a key factor in Multimodality, possibly even more so than in ‘monomodal’ texts.
In social semiotics, language is viewed as dependent on the context in which it occurs; that is, the social relations are context-based as Halliday puts it “language is a system of signs with social functions in which meaning is constructed” with “text in use” (Kress and Hodge, 1988). Social semiotics is the coinage of the British linguist Michael Halliday in his book entitled Language as a social semiotic in 1978 in which language functions within a socio-cultural context and the importance of culture in shaping the way how verbal language “interacts” with other systems of communication. Halliday found out that to understand other codes, the verbal code plays a major role in bringing them all together to terms. His linguistic theory has developed major key issues on which social semiotics relies in formulating its concepts. Kress and van Leeuwen (2006) emphasize the importance of the context in which visual communication takes place depending on the “uses and values” available in that society.
Language as Halliday proposed has a function within a society and is not separated from it. The relationship between language and society has a root in the concept of culture as it needs people to perform the act of interaction hence this is a crucial point to be considered. Language is studied within a social semiotics framework since culture is defined within the environment in which the interaction takes place and language is considered as one system of semiotic systems within a culture in which meanings are shaped in lieu of the terms that constitute that culture.
Kress and van Leeuwen build their framework on Halliday’s social semiotic approach which focuses on three metafunctions of language; the ideational, the interpersonal, and the textual in which they give a new terminology in describing those metafunctions. The ideational is described by Kress and van Leeuwen as ‘representational’ instead of ideational’; ‘interactive’ instead of interpersonal’; and ‘compositional’ instead of ‘textual’ (van Leeuwen and Jewitt, 2001).
Multimodal social semiotics is a theory which can account for social and cultural influences in texts as well as meanings made in multimodal visual narratives; multimodality and social semiotics together benefit in the understanding of forms of communication (Huang, 2009:2;Kress, 2010;16). In their book Reading Images: The Grammar of Visual Design (2006 [1996]), Gunther Kress and Theo van Leeuwen have proposed a new direction in the process of analyzing visual semiotics in the lights of social semiotics in which they are dealing with it as a “grammar”, that is, the interaction between the different modes as a whole compromises what is intended. They call their approach a ‘grammar’ to draw attention to “culturally produced regularity”.
