B. Schenkenfelder, M Wimmer: Towards Multimodal Entity Linking for LLM-based Metamodel Generation, Proceedings of the 2026 2nd International Workshop on Low-Code Development and Software Architecture, LArc 2026, Rio de Janeiro, Brazil, April 12-18, 2026, pages 27-30, publisher ACM, 2026. Doi: 10.1145/3786179.3788325
Building Low-Code Development Platforms (LCDPs) tailored to specific domains and users is a time-consuming and manual task. Large Language Models (LLMs) are a powerful method for automating this process by generating modeling constructs from an organization’s heterogeneous artifacts containing domain knowledge, whether implicit or explicit. However, it is not necessarily clear where these artifacts overlap, how they complement each other, and which parts are relevant to the scope of a particular LCDP. This work presents an approach for Multimodal Entity Linking (MEL) of LLM input, i.e., combining, integrating, and augmenting data such as text (natural language), documents, structured data, audio files, images, videos, and code. Inspired by pilot studies of LLM-based metamodel generation from a variety of files containing domain knowledge related to a popular card game-which serves as a case study to simulate heterogeneous artifacts in industrial settings–the approach links entities at the sub-file level, e.g., connecting an object in an image to a paragraph in a document. When applied to the card game example, the MEL approach shows promising results in terms of the quality and scope of the generated metamodels, outperforming both unimodal input files and merging metamodels produced from unimodal inputs.