Home >

news Help

Publication Information


Title
Japanese: 
English:Pyramid Coder: Hierarchical Code Generator for Compositional Visual Question Answering 
Author
Japanese: SHEN Ruoyue, 井上 中順, 篠田 浩一.  
English: Ruoyue Shen, Nakamasa Inoue, Koichi Shinoda.  
Language English 
Journal/Book name
Japanese: 
English:2024 IEEE International Conference on Image Processing (ICIP) 
Volume, Number, Page         pp. 430-436
Published date Sept. 27, 2024 
Publisher
Japanese: 
English:IEEE 
Conference name
Japanese: 
English:2024 IEEE International Conference on Image Processing (ICIP 2024) 
Conference site
Japanese: 
English:Abu Dhabi 
File
Official URL https://ieeexplore.ieee.org/document/10648180
 
DOI https://doi.org/10.1109/ICIP51287.2024.10648180
Abstract Visual question answering (VQA) is the task of providing accurate answers to natural language questions based on visual input. Programmatic VQA (PVQA) models have been gaining attention recently. These use large language models (LLMs) to formulate executable programs that address questions requiring complex visual reasoning. However, there are challenges in enabling LLMs to comprehend the usage of image processing modules and generate relevant code. To overcome these challenges, this paper introduces PyramidCoder, a novel prompting framework for PVQA models. PyramidCoder consists of three hierarchical levels, each serving a distinct purpose: query rephrasing, code generation, and answer aggregation. Notably, PyramidCoder utilizes a single frozen LLM and pre-defined prompts at each level, eliminating the need for additional training and ensuring flexibility across various LLM architectures. Compared to the state-of-the-art PVQA model, our approach improves accuracy by at least 0.5% on the GQA dataset, 1.4% on the VQAv2 dataset, and 2.9% on the NLVR2 dataset.

©2007 Institute of Science Tokyo All rights reserved.