two modalities are separated by the [SEP] token，the numbers in each attention module represent the Eq. number.
E h _h h is the mean， ∇ \nabla ∇A := ∂ y t ∂ A {∂y_t}\over∂A ∂A∂ytfor y t y_t yt which is the model’s output. ⊙ \odot ⊙ is the Hadamard product，remove the negative contributions before averaging.
aggregated self-attention matrix R q q ^{qq} qq，previous layers’ mixture of context is embodied by R q k ^{qk} qk.
感想
作者的实验在coco和ImageNet验证集上做的，不好follow
