Author ORCID Identifier
https://orcid.org/0000-0001-8262-2476
Document Type
Conference Paper
Disciplines
1.2 COMPUTER AND INFORMATION SCIENCE, Computer Sciences
Abstract
Internet memes are characterised by the interspersing of text amongst visual elements. State-of-the-art multimodal meme classifiers do not account for the relative positions of these elements across the two modalities, despite the latent meaning associated with where text and visual elements are placed. Against two meme sentiment classification datasets, we systematically show performance gains from incorporating the spatial position of visual objects, faces, and text clusters extracted from memes. In addition, we also present facial embedding as an impactful enhancement to image representation in a multimodal meme classifier. Finally, we show that incorporating this spatial information allows our fully automated approaches to outperform their corresponding baselines that rely on additional human validation of OCR-extracted text.
DOI
https://doi.org/10.1007/978-3-031-26438-2_25
Recommended Citation
Hazman, Muzhaffar; McKeever, Susan; and Griffith, Josephine, "Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Face Embedding" (2023). Conference papers. 395.
https://arrow.tudublin.ie/scschcomcon/395
Funder
Science Foundation Ireland Centre for Research Training in Digitally-Enhanced Reality (d-real) under Grant No. 18/CRT/6224.
Creative Commons License
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.
Publication Details
https://link.springer.com/chapter/10.1007/978-3-031-26438-2_25#citeas
https://doi.org/10.1007/978-3-031-26438-2_25
Hazman, M., McKeever, S., Griffith, J. (2023). Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Face Embedding. In: Longo, L., O’Reilly, R. (eds) Artificial Intelligence and Cognitive Science. AICS 2022. Communications in Computer and Information Science, vol 1662. Springer, Cham.