Block Diagram-to-Text: Understanding Block Diagram Images by Generating Natural Language Descriptors

초록

Block diagrams are very popular for representing a workflow or process of a model. Understanding block diagrams by generating summaries can be extremely useful in documentsummarization. It can also assist people in inferring key insights from block diagrams without requiring a lot of perceptual and cognitiveeffort. In this paper, we propose a novel taskof converting block diagram images into textby presenting a framework called "BloSum".This framework extracts the contextual meaning from the images in the form of triplets thathelp the language model in summary generation. We also introduce a new dataset for complex computerized block diagrams, explain thedataset preparation process, and later analyzeit. Additionally, to showcase the generalizationof the model, we test our method with publiclyavailable handwritten block diagram datasets.Our evaluation with different metrics demonstrates the effectiveness of our approach thatoutperforms other methods and techniques. © AACL-IJCNLP 2022.All rights reserved