《Entangled Transformer for Image Captioning》

目录

一、文献摘要介绍

二、网络框架介绍

三、实验分析

四、结论

本文有点长，请耐心阅读，定会有收货。如有不足，欢迎交流， 另附:论文下载地址

一、文献摘要介绍

In image captioning, the typical attention mechanisms are arduous to identify the equivalent visual signals especially when predicting highly abstract words. This phenomenon is known as the semantic gap between vision and language. This problem can be overcome by providing semantic attributes that are homologous to language. Thanks to the inherent recurrent nature and gated operating mechanism, Recurrent Neural Network (RNN) and its variants are the dominating architectures in image captioning. However, when designing elaborate attention mechanisms to integrate visual inputs and semantic attributes, RNN-like variants become unflflexible due to their complexities. In this paper, we investigate a Transformer-based sequence modeling framework, built only with attention layers and feedforward layers. To bridge the semantic gap, we introduce EnTangled Attention (ETA) that enables the Transformer to exploit semantic and visual information simultaneously. Furthermore, Gated Bilateral Controller (GBC) is proposed to guide the interactions between the multimodal information. We name our model as ETA-Transformer. Remarkably, ETA-Transformer achieves state-of-the-art performance on the MSCOCO image captioning dataset. The ablation studies validate the improvements of our proposed modules.

作者认为，在图像字幕中，典型的注意机制很难识别出等效的视觉信号，尤其是在预测高度抽象的单词时。这种现象被称为视觉和语义之间的鸿沟。这个问题可以通过提供与语言相对应的语义属性来解决。由于其固有的递归性质和门控操作机制，循环神经网络（RNN）及其变体是图像描述中的主要架构。但是，当精心设计注意力机制以集成视觉输入和语义属性时，类似的RNN变体由于其复杂性而变得不灵活。在本文中，我们研究了仅基于关注层和前馈层构建的基于Transformer 的序列建模框架，为了弥补语义上的鸿沟，我们引入了Entangled Attention(ETA)，使Transformer能够同时利用语义和视觉信息。此外，作者提出了门控双向控制器（GBC）来指导多模态信息之间的交互。实验表明，在数据集MSCOCO上达到了最新的性能。

二、网络框架介绍

作者提出的模型（如下图所示）包含三个部分：视觉子编码器(visual sub-encoder)，语义子编码器(semantic sub-encoder)和多模式解码器(multimodal decoder)。 生成过程分为三个步骤：（1）检测区域推荐和语义属性； &#

最后

以上就是大胆钻石为你收集整理的《Entangled Transformer for Image Captioning》的全部内容，希望文章能够帮你解决《Entangled Transformer for Image Captioning》所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。