GAM-SGR: Graph Structure Information-augmented Multimodal Large Language Models for Scene Graph Reasoning
-
摘要: 多模态大模型(multimodal large language models, MLLMs)在场景图生成(scene graph generation, SGG)中面临关系幻觉与任务针对性推理不足的问题. 为此, 提出基于图结构信息增强的多模态大模型场景图推理方法(graph structure information-augmented multimodal large language models for scene graph reasoning, GAM-SGR), 使MLLMs显式关注图结构信息. 核心是引入图标记(graph token), 深度耦合视觉、拓扑和语言信息. 对于开源MLLMs, 提出参数高效的联合微调方法, 增强其对图结构信息的注意力. 图结构信息由图神经网络提取, 改进的Transformer层用于视觉语言信息与图结构信息的融合推理. 对于闭源MLLMs, 由于无法调整其内部参数和结构来改变注意力, 通过指令提示将生成的场景图与图神经网络的关系推理结果交叉验证, 引导模型关注与关系有关的图像内容. 实验表明, GAM-SGR提升开源模型的场景图推理性能, 稳健抑制闭源模型的关系幻觉, 并表现出良好的泛化能力. 关系幻觉率指标评估进一步表明, 该方法在多种骨干模型上均显著降低幻觉率.Abstract: Multimodal large language models (MLLMs) face relational hallucinations and insufficient task-specific reasoning in scene graph generation (SGG). We propose graph structure information-augmented multimodal large language models for scene graph reasoning (GAM-SGR), which enables MLLMs to explicitly attend to graph structure information. Its core is the introduction of graph tokens to deeply couple visual, topological, and linguistic information. For open-source MLLMs, a parameter-efficient joint fine-tuning method enhances attention to graph structure information. Graph neural networks extract this information, and improved Transformer layers fuse it with visual-language information for reasoning. For closed-source MLLMs, whose internal parameters and structures cannot be modified to change attention, instruction prompts cross-verify generated scene graphs against relational reasoning results from graph neural networks, directing attention to relation-relevant image content. Experiments show that GAM-SGR improves scene graph reasoning performance for open-source models, robustly suppresses relational hallucinations in closed-source models, and generalizes well. Evaluation using the relation hallucination rate metric further shows significant reductions in hallucinations across multiple backbone models.
-
图 2 基于图结构信息的高效参数微调方法
Fig. 2 Parameter-efficient fine-tuning method based on graph structure information
表 1 在Visual Genome测试集上的对比实验(%)
Table 1 Comparative experiments on the Visual Genome test set (%)
方法 骨干模型 $ \mathrm{AP}_{50} $ PredCLS SGCLS SGDET $ \mathrm{R}@20 $ $ \mathrm{R}@50 $ $ \mathrm{mR}@20 $ $ \mathrm{mR}@50 $ $ \mathrm{R}@20 $ $ \mathrm{R}@50 $ $ \mathrm{mR}@20 $ $ \mathrm{mR}@50 $ $ \mathrm{R}@20 $ $ \mathrm{R}@50 $ $ \mathrm{mR}@20 $ $ \mathrm{mR}@50 $ MOTIFS[23] - 20.0 58.5 65.2 10.8 14.0 32.9 35.8 6.3 7.7 21.4 27.2 4.2 5.7 KERN[48] - 20.0 59.1 65.8 - 17.7 32.2 36.7 - 9.4 22.3 27.1 - 6.4 PCPL[49] - - - 50.8 - 35.2 - 27.6 - 18.6 - 14.6 - 9.5 GB-Net[50] - - - 66.6 - 19.3 - 38.0 - 9.6 - 26.4 - 6.1 RelDN[51] - - 66.9 68.4 - - 36.1 36.8 - - 21.1 28.3 - - VCTree-TDE[52] - 28.1 39.1 49.9 17.2 23.3 22.8 28.8 8.9 11.8 14.3 19.6 6.3 9.3 GPS-Net[53] - - 67.6 69.7 17.4 21.3 41.8 42.3 10.0 11.8 22.3 28.9 6.9 8.7 SVRP[54] - - - - - 24.3 - - - 12.5 - - - 10.5 VS$ ^3 $[55] - - - - - - 27.8 36.6 - - - - - - BGNN[56] - 29.0 - 59.2 - 30.4 - 37.4 - 14.3 23.3 31.0 7.5 10.7 BGT-Net[57] - 28.1 60.9 67.1 16.8 20.6 41.7 45.9 10.4 12.8 25.5 32.8 5.7 7.8 IMP[22] - - 58.5 65.2 - 9.8 31.7 34.6 - 5.8 14.6 20.7 - 3.8 CISC[58] - - 42.1 53.2 - - 23.3 27.8 - - 7.7 11.4 - - G-RCNN[24] - 24.8 - 54.2 - - - 29.6 - - - 11.4 - - FCSGG[59] - 28.5 33.4 41.0 4.9 6.3 19.0 23.5 3.7 3.7 16.1 21.3 2.7 3.6 SGTR[60] - 29.6 62.1 65.5 19.2 21.0 30.1 33.4 7.9 11.0 17.9 20.6 7.1 15.8 RelTR[61] - 26.4 63.1 64.2 20.0 21.2 29.0 36.6 7.7 11.4 21.2 27.5 6.8 10.8 LLaVA-SpaceSGG[62] LLaVA 27.0 62.9 64.0 20.7 21.1 30.1 37.8 8.5 11.3 22.5 29.3 7.6 13.0 SGTR+RAHP[31] Qwen2-VL 28.1 64.9 65.1 20.6 21.6 32.1 38.2 8.8 11.5 26.0 35.2 8.9 15.9 VS$ ^3 $+LLM4SGG[63] Qwen2-VL 27.8 64.8 68.1 21.5 22.2 33.6 39.8 8.4 11.0 24.0 27.1 6.9 10.2 PGSG[64] Qwen2-VL 28.2 65.2 68.9 22.0 22.5 35.1 40.3 8.5 11.7 25.3 28.8 7.9 13.3 SDSGG[65] Qwen2-VL 27.6 64.9 66.1 20.7 21.4 33.1 35.4 8.0 10.9 24.1 27.9 7.1 10.9 IndVisSGG[66] GPT-4o 27.9 65.1 67.3 21.4 21.9 34.9 37.9 8.5 11.0 22.0 28.3 7.5 11.9 GAM-SGR(本文方法) GPT-4o 30.1 67.9 70.1 21.1 22.0 43.1 43.8 10.5 13.3 26.1 34.9 8.1 15.5 GAM-SGR(本文方法) LLaVA 32.3 68.8 72.0 23.6 24.2 45.2 45.8 11.1 15.3 28.9 37.9 10.0 18.2 GAM-SGR(本文方法) Qwen2-VL 35.1 69.9 73.2 25.9 27.0 46.1 47.0 12.5 16.0 29.8 38.2 11.0 18.9 注: 粗体表示同列最大值; "-" 表示原文献未报告该项或未标注骨干模型. 本文方法的骨干模型单列给出. 
下载: 导出CSV
表 2 在Open Images V6测试集上的对比实验(%)
Table 2 Comparative experiments on the Open Images V6 test set (%)
方法 骨干模型 $ \mathrm{R}@50 $ $ \mathrm{wmAP}_{\mathrm{rel}} $ $ \mathrm{wmAP}_{\mathrm{phr}} $ RelDN - 73.08 32.16 33.39 VCTree - 75.34 33.21 34.31 GPS-Net - 74.81 32.85 33.98 BGNN - 74.98 33.51 34.15 G-RCNN - 74.51 33.15 34.21 SGTR - 59.91 36.98 38.73 RelTR - 71.66 34.19 37.46 GAM-SGR(本文方法) LLaVA 80.19 42.76 43.99 GAM-SGR(本文方法) GPT-4o 77.24 40.35 42.01
下载: 导出CSV
表 3 在Panoptic Scene Graph测试集上的对比实验(%)
Table 3 Comparative experiments on the Panoptic Scene Graph test set (%)
方法 骨干模型 新类别 $ \mathrm{R}@20 $ $ \mathrm{R}@50 $ $ \mathrm{mR}@50 $ LLaVA-SpaceSGG LLaVA 19.5 31.7 5.9 SGTR+RAHP Qwen2-VL 20.3 33.2 6.3 VS$ ^3 $+LLM4SGG Qwen2-VL 20.5 34.0 6.2 PGSG Qwen2-VL 18.9 30.1 5.1 SDSGG Qwen2-VL 19.6 27.1 4.2 IndVisSGG GPT-4o 21.4 33.9 6.0 GAM-SGR(本文方法) GPT-4o 22.0 33.7 5.7 GAM-SGR(本文方法) LLaVA 24.0 35.0 6.6 GAM-SGR(本文方法) Qwen2-VL 25.8 37.9 7.5
下载: 导出CSV
表 4 GAM-SGR在不同闭源骨干网络下的泛化性与稳定性评估(Visual Genome测试集, SGDET设置, %)
Table 4 Generalization and stability evaluation of GAM-SGR with different closed-source backbone networks (Visual Genome test set, SGDET setting, %)
骨干网络 方法 $ \mathrm{R}@50 $ $ \mathrm{mR}@50 $ RHR GPT-4V 原生模型 24.8 3.5 14.8 GPT-4V +GAM-SGR 34.0($ \uparrow9.2 $) 14.9($ \uparrow11.4 $) 7.2($ \downarrow7.6 $) Claude-3-Opus 原生模型 25.9 3.9 12.2 Claude-3-Opus +GAM-SGR 35.3($ \uparrow9.4 $) 16.1($ \uparrow12.2 $) 6.3($ \downarrow5.9 $) GPT-4o 原生模型 25.4 3.2 12.5 GPT-4o +GAM-SGR 34.9($ \uparrow9.5 $) 15.5($ \uparrow12.3 $) 6.1($ \downarrow6.4 $) 注: 原生模型为未搭载GAM-SGR的骨干网络; +GAM-SGR表示搭载本文框架. 括号内箭头和数值表示相对原生模型增加或减少的百分点.
下载: 导出CSV
表 5 消融实验(SGDET设置, %)
Table 5 Ablation experiments (SGDET setting, %)
类型 GNN PosEnc PEFT Bi-Dir $ \mathrm{R}@20 $ $ \mathrm{R}@50 $ $ \mathrm{mR}@20 $ $ \mathrm{mR}@50 $ 闭源 $ \checkmark $ $ \checkmark $ - - 26.1 34.9 8.1 15.5 闭源 $ \checkmark $ $ \times $ - - 22.7($ \downarrow3.4 $) 29.4($ \downarrow5.5 $) 6.5($ \downarrow1.6 $) 11.8($ \downarrow3.7 $) 闭源 $ \times $ $ \times $ - - 19.9($ \downarrow6.2 $) 25.4($ \downarrow9.5 $) 2.4($ \downarrow5.7 $) 3.2($ \downarrow12.3 $) 开源 - - $ \checkmark $ $ \checkmark $ 28.9 37.9 10.0 18.2 开源 - - $ \times $ $ \times $ 18.2($ \downarrow10.7 $) 24.9($ \downarrow13.0 $) 2.4($ \downarrow7.6 $) 3.1($ \downarrow15.1 $) 开源 - - $ \checkmark $ $ \times $ 23.5($ \downarrow5.4 $) 30.0($ \downarrow7.9 $) 6.6($ \downarrow3.4 $) 13.0($ \downarrow5.2 $) 注: GNN为闭源图神经网络验证模块; PosEnc为空间位置编码; PEFT为开源参数高效微调; Bi-Dir为双向图结构交互. 勾号、叉号分别表示保留和移除; "-" 表示不适用. 每组首行为完整模型; 括号内数值为相对该行下降的百分点.
下载: 导出CSV
表 6 场景图双向交互层选择配置的横向消融实验(Visual Genome测试集, SGDET设置)
Table 6 Comparative ablation experiments on scene graph bidirectional interaction layer configurations (Visual Genome test set, SGDET setting)
交互层配置 层数 $ \mathrm{R}@50 $(%) $ \mathrm{mR}@50 $(%) 时延(s) 2、4、6、8 4 26.3 7.9 0.21 24、26、28、30 4 30.1 8.8 0.22 1$ \sim $32 32 38.5 19.1 0.58 6、12、18、24(本文方法) 4 37.9 18.2 0.23
下载: 导出CSV
表 7 外部GNN组件注入不同比例预测噪声时GAM-SGR最终性能的鲁棒性测试
Table 7 Robustness test of the final performance of GAM-SGR under different prediction noise proportions injected into the external GNN component
噪声比例(%) 关系预测准确率(%) $ \mathrm{mR}@50 $(%) 变化(百分点) 0 68.5 18.9 0.0 10 61.2 15.4 −3.5 20 54.8 15.2 −3.7 30 47.3 14.6 −4.3 注: 变化以本表 0% 噪声时的18.9% 为基准计算.
下载: 导出CSV
表 8 在Visual Genome和Panoptic Scene Graph测试集上的RHR对比(%)
Table 8 Comparison of RHR on the Visual Genome and Panoptic Scene Graph test sets (%)
骨干模型 搭载前 搭载后 变化(百分点) LLaVA 17.4 6.8 −10.6 Qwen2-VL 15.3 6.5 −8.8 GPT-4o 12.5 6.1 −6.4 注: 变化为搭载后减去搭载前的RHR.
下载: 导出CSV
表 9 GAM-SGR在GPT-4o骨干网络下的推理效率与成本评估
Table 9 Inference efficiency and cost evaluation of GAM-SGR with the GPT-4o backbone network
方法 二次推理
比例(%)平均耗时
(s)标记消耗
($ 10^3 $)成本增加
(%)RHR
(%)原生GPT-4o 0 1.25 450 0 12.5 GAM-SGR 23.2 1.47 489 8.7 6.1
下载: 导出CSV
-
[1] Johnson J, Krishna R, Stark M, Li L J, Shamma D, Bernstein M, et al. Image retrieval using scene graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE, 2015. 3668–3678. [2] Shiny S, Abisheak A, Balaji P, Rohit K. Image captioning using scene graph generation. In: Proceedings of the International Conference on Wireless Communications Signal Processing and Networking. Chennai, India: IEEE, 2025. 1–5. [3] Nguyen K, Tripathi S, Du B, Guha T, Nguyen T Q. In defense of scene graphs for image captioning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021. 1407–1416. [4] Gao L, Wang B, Wang W. Image captioning with scene-graph based semantic concepts. In: Proceedings of the 10th International Conference on Machine Learning and Computing. Macau, China: ACM, 2018. 225–229. [5] de Oliveira Souza B C, Aasan M, Pedrini H, Rivera A R. SelfGraphVQA: A self-supervised graph neural network for scene-based question answering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. Paris, France: IEEE, 2023. 4640–4645. [6] Yoon S, Kang W Y, Jeon S, Lee S, Han C, Park J, et al. Image-to-image retrieval by learning similarity between scene graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35. Virtual Event: AAAI Press, 2021. 10718–10726. [7] Ashual O, Wolf L. Specifying object attributes and relations in interactive scene generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, Korea: IEEE, 2019. 4561–4569. [8] Johnson J, Gupta A, Fei-Fei L. Image generation from scene graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018. 1219–1228. [9] Wang Z, You H, Li L H, Zareian A, Park S, Liang Y, et al. SGEITL: Scene graph enhanced image-text learning for visual commonsense reasoning. In: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36. Virtual Event: AAAI Press, 2022. 5914–5922. [10] Liu H, Li C, Li Y, Lee Y J. Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2024. 26296–26306. [11] Li J, Li D, Savarese S, Hoi S. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research. Honolulu, USA: Proceedings of Machine Learning Research, 2023. 19730–19742. [12] Radford A, Kim J W, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. Virtual Event: Proceedings of Machine Learning Research, 2021. 8748–8763. [13] Li J, Selvaraju R, Gotmare A, Joty S, Xiong C, Hoi S C H. Align before fuse: Vision and language representation learning with momentum distillation. In: Advances in Neural Information Processing Systems, volume 34. Virtual Event: Curran Associates, 2021. 9694–9705. [14] Bai Z, Wang P, Xiao T, He T, Han Z, Zhang Z, et al. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv: 2404.18930, 2024. [15] Sun S, Lin Z, Wu X. Hallucinations of large multimodal models: Problem and countermeasures. Information Fusion, 2025, 118: 102970. [16] Han K, Wang Y, Guo J, Tang Y, Wu E. Vision GNN: An image is worth graph of nodes. In: Advances in Neural Information Processing Systems, volume 35. New Orleans, USA: Curran Associates, 2022. 8291–8303. [17] Lin X, Ding C, Zhan Y, Li Z, Tao D. HL-Net: Heterophily learning network for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE, 2022. 19476–19485. [18] Krishna R, Zhu Y, Groth O, Johnson J, Hata K, Kravitz J, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 2017, 123(1): 32−73 doi: 10.1007/s11263-016-0981-7 [19] Kuznetsova A, Rom H, Alldrin N, Uijlings J, Krasin I, Pont-Tuset J, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 2020, 128(7): 1956−1981 doi: 10.1007/s11263-020-01316-z [20] Yang J, Ang Y Z, Guo Z, Zhou K, Zhang W, Liu Z. Panoptic scene graph generation. In: Proceedings of the European Conference on Computer Vision. Tel Aviv, Israel: Springer, 2022. 178–196. [21] Lu C, Krishna R, Bernstein M, Fei-Fei L. Visual relationship detection with language priors. In: Proceedings of the European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 2016. 852–869. [22] Xu D, Zhu Y, Choy C B, Fei-Fei L. Scene graph generation by iterative message passing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017. 5410–5419. [23] Zellers R, Yatskar M, Thomson S, Choi Y. Neural motifs: Scene graph parsing with global context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018. 5831–5840. [24] Yang J, Lu J, Lee S, Batra D, Parikh D. Graph R-CNN for scene graph generation. In: Proceedings of the European Conference on Computer Vision. Munich, Germany: Springer, 2018. 670–685. [25] Zhang J, Kalantidis Y, Rohrbach M, Paluri M, Elgammal A, Elhoseiny M. Large-scale visual relationship understanding. In: Proceedings of the AAAI Conference on Artificial Intelligence, volume 33. Honolulu, USA: AAAI Press, 2019. 9185–9194. [26] Tang K, Zhang H, Wu B, Luo W, Liu W. Learning to compose dynamic tree structures for visual contexts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019. 6619–6628. [27] Jiang B, Zhuang Z, Shivakumar S S, Taylor C J. Enhancing scene graph generation with hierarchical relationships and commonsense knowledge. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Tucson, USA: IEEE, 2025. 8865–8876. [28] Wu S, Fei H, Chua T S. Universal scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE, 2025. 14158–14168. [29] Li R, Zhang S, Lin D, Chen K, He X. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2024. 28076–28086. [30] Chen Z, Wu J, Lei Z, Zhang Z, Chen C W. Expanding scene graph boundaries: Fully open-vocabulary scene graph generation via visual-concept alignment and retention. In: Proceedings of the European Conference on Computer Vision. Milan, Italy: Springer, 2024. 108–124. [31] Liu T, Li R, Wang C, He X. Relation-aware hierarchical prompt for open-vocabulary scene graph generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, volume 39. Philadelphia, USA: AAAI Press, 2025. 5576–5584. [32] Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, de Laroussilhe Q, Gesmundo A, et al. Parameter-efficient transfer learning for NLP. In: Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. Long Beach, USA: Proceedings of Machine Learning Research, 2019. 2790–2799. [33] Hu E J, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. LoRA: Low-rank adaptation of large language models. In: Proceedings of the International Conference on Learning Representations. Virtual Event, 2022. [34] Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. 3045–3059. [35] Ben Zaken E, Goldberg Y, Ravfogel S. BitFit: Simple parameter-efficient fine-tuning for Transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Dublin, Ireland: Association for Computational Linguistics, 2022. 1–9. [36] Zhou X, He J, Ke Y, Zhu G, Gutierrez Basulto V, Pan J. An empirical study on parameter-efficient fine-tuning for MultiModal large language models. In: Findings of the Association for Computational Linguistics. Bangkok, Thailand: Association for Computational Linguistics, 2024. 10057–10084. [37] Li M, Zhong J, Li C, Li L, Lin N, Sugiyama M. Vision-language model fine-tuning via simple parameter-efficient modification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Miami, USA: Association for Computational Linguistics, 2024. 14394–14410. [38] Prottasha N J, Chowdhury U R, Mohanto S, Nuzhat T, Sami A A, Ali M S, et al. PEFT A2Z: Parameter-efficient fine-tuning survey for large language and vision models. arXiv preprint arXiv: 2504.14117, 2025. [39] Ebrahimi S, Arik S O, Nama T, Pfister T. CROME: Cross-modal adapters for efficient multimodal LLM. arXiv preprint arXiv: 2408.06610, 2024. [40] Zheng K, Chen J, Yan Y, Zou X, Zhou H, Hu X. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. In: Findings of the Association for Computational Linguistics: ACL 2025. Vienna, Austria: Association for Computational Linguistics, 2025. 6193–6212. [41] Jiang C, Jia H, Dong M, Ye W, Xu H, Yan M, et al. Hal-Eval: A universal and fine-grained hallucination evaluation framework for large vision language models. In: Proceedings of the 32nd ACM International Conference on Multimedia. Melbourne, Australia: ACM, 2024. 525–534. [42] Gong B, Zheng Y, Kong F, Zhou J, Lu J. Attention at rest stays at rest: Breaking visual inertia to mitigate relation hallucinations. arXiv preprint arXiv: 2604.01989, 2026. [43] Liu C, Xu Z, Wei Q, Wu J, Zou J, Wang X E, et al. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models. arXiv preprint arXiv: 2505.21523, 2025. [44] Xu Z, Wang Z, Qian Z, Shi D, Tang F, Hu M, et al. Thinking in uncertainty: Mitigating hallucinations in MLRMs with latent entropy-aware decoding. arXiv preprint arXiv: 2603.13366, 2026. [45] Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y. Graph attention networks. In: Proceedings of the International Conference on Learning Representations. Vancouver, Canada, 2018. [46] Zhang S, Tong H, Xu J, Maciejewski R. Graph convolutional networks: a comprehensive review. Computational Social Networks, 2019, 6(1): 1−23 doi: 10.1186/s40649–019-0069-y [47] Hamilton W, Ying Z, Leskovec J. Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems, volume 30. Long Beach, USA: Curran Associates, 2017. 1024–1034. [48] Chen T, Yu W, Chen R, Lin L. Knowledge-embedded routing network for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019. 6163–6171. [49] Yan S, Shen C, Jin Z, Huang J, Jiang R, Chen Y, et al. PCPL: Predicate-correlation perception learning for unbiased scene graph generation. In: Proceedings of the 28th ACM International Conference on Multimedia. Seattle, USA: ACM, 2020. 265–273. [50] Zareian A, Karaman S, Chang S F. Bridging knowledge graphs to generate scene graphs. In: Proceedings of the European Conference on Computer Vision. Glasgow, UK: Springer, 2020. 606–623. [51] Zhang J, Shih K J, Elgammal A, Tao A, Catanzaro B. Graphical contrastive losses for scene graph parsing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019. 11535–11543. [52] Tang K, Niu Y, Huang J, Shi J, Zhang H. Unbiased scene graph generation from biased training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020. 3716–3725. [53] Lin X, Ding C, Zeng J, Tao D. GPS-Net: Graph property sensing network for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020. 3746–3753. [54] He T, Gao L, Song J, Li Y F. Towards open-vocabulary scene graph generation with prompt-based finetuning. In: Proceedings of the European Conference on Computer Vision. Tel Aviv, Israel: Springer, 2022. 56–73. [55] Zhang Y, Pan Y, Yao T, Huang R, Mei T, Chen C W. Learning to generate language-supervised and open-vocabulary scene graph using pre-trained visual-semantic space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023. 2915–2924. [56] Li R, Zhang S, Wan B, He X. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE, 2021. 11109–11119. [57] Dhingra N, Ritter F, Kunz A. BGT-Net: Bidirectional GRU Transformer network for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Nashville, USA: IEEE, 2021. 2150–2159. [58] Wang W, Wang R, Shan S, Chen X. Exploring context and visual pattern of relationship for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019. 8188–8197. [59] Liu H, Yan N, Mortazavi M, Bhanu B. Fully convolutional scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE, 2021. 11546–11556. [60] Li R, Zhang S, He X. SGTR: End-to-end scene graph generation with Transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE, 2022. 19486–19496. [61] Cong Y, Yang M Y, Rosenhahn B. RelTR: Relation Transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 11169−11183 doi: 10.1109/TPAMI.2023.3268066 [62] Xu M, Wu M, Zhao Y, Li J C L, Ou W. LLaVA-SpaceSGG: Visual instruct tuning for open-vocabulary scene graph generation with enhanced spatial relations. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Tucson, USA: IEEE, 2025. 6362–6372. [63] Kim K, Yoon K, Jeon J, In Y, Moon J, Kim D, et al. LLM4SGG: Large language models for weakly supervised scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2024. 28306–28316. [64] Li L, Ji W, Wu Y, Li M, Qin Y, Wei L, et al. Panoptic scene graph generation with semantics-prototype learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. Vancouver, Canada: AAAI Press, 2024. 3145–3153. [65] Chen G, Li J, Wang W. Scene graph generation with role-playing large language models. In: Advances in Neural Information Processing Systems, volume 37. Vancouver, Canada: Curran Associates, 2024. 132238–132266. [66] Wang Z, Yan Z, Li S, Liu J. IndVisSGG: VLM-based scene graph generation for industrial spatial intelligence. Advanced Engineering Informatics, 2025, 65: 103107 doi: 10.1016/j.aei.2024.103107 -
下载:
计量
- 文章访问数: 10
- HTML全文浏览量: 5
- 被引次数: 0
