Skip to content

zjunlp/Generative_KG_Construction_Papers

Repository files navigation

Awesome License: MIT

[:bell: News! :bell: ] We have released a new survey paper:"Generative Knowledge Graph Construction: A Review" based on this repository, with a perspective of existing Generative Knowledge Graph Construction! We are looking forward to any comments or discussions on this topic :)

🕵️ Introduction

Generative Knowledge Graph Construction (KGC) refers to those methods that leverage the sequence-to-sequence framework for building knowledge graphs, which is flexible and can be adapted to widespread tasks. In this study, we summarize the recent compelling progress in generative knowledge graph construction. We present the advantages and weaknesses of each paradigm in terms of different generation targets and provide theoretical insight and empirical analysis. Based on the review, we suggest promising research directions for the future. Our contributions are threefold: (1) We present a detailed, complete taxonomy for the generative KGC methods; (2) We provide a theoretical and empirical analysis of the generative KGC methods; (3) We propose several research directions that can be developed in the future. For more resources about knowledge graph construction, please check our tookit DeepKE.

👋 News!

  • We release EasyEdit, an easy-to-use framework to edit Large Language Models.
  • We release DeepKE-LLM to support knowledge extraction with KnowLM, ChatGLM, LLaMA-series, GPT-series etc.
  • We release a survey and paper-list for reasoning with language model prompting.
  • We release a prompt&KG paper-list at PromptKG.
  • Congratulations! Our work has been accepted by the EMNLP2022 main conference.
  • Due to the rise of generative extraction methods in the NLP community,we summarize recent progress in generative KGC and release our paper on arivx.
  • We release Eng/Cn slides at Silde file.

🚩Citation

If you find this survey useful for your research, please consider citing

@article{DBLP:journals/corr/abs-2210-12714,
  author    = {Hongbin Ye and
               Ningyu Zhang and
               Hui Chen and
               Huajun Chen},
  title     = {Generative Knowledge Graph Construction: {A} Review},
  journal   = {CoRR},
  volume    = {abs/2210.12714},
  year      = {2022},
  url       = {https://doi.org/10.48550/arXiv.2210.12714},
  doi       = {10.48550/arXiv.2210.12714},
  eprinttype = {arXiv},
  eprint    = {2210.12714},
  timestamp = {Fri, 28 Oct 2022 14:21:57 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2210-12714.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

⚓️ Preliminary on Knowledge Graph Construction

Knowledge Graph Construction mainly aims to extract structural information from unstructured texts, such as Named Entity Recognition (NER), Relation Extraction (RE), Event Extraction (EE), Entity Linking (EL), and Knowledge Graph Completion (KGC).

Generally, KGC can be regarded as structure prediction tasks, where a model is trained to approximate a target function $F(x) \rightarrow y$, where $x \in \mathcal{X}$ denotes the input data and $y \in \mathcal{Y}$ denotes the output structure sequence. For instance, given a sentence, "Steve Jobs and Steve Wozniak co-founded Apple in 1977.":

  • Named Entity Recognition aims to identify the types of entities, i.e., ‘Steve Job', ‘Steve Wozniak' $\Rightarrow$ PERSON, ‘Apple' $\Rightarrow$ ORG;

  • Relation Extraction aims to identify the relationship of the given entity pair $\langle$Steve Job, Apple$\rangle$ as founder;

  • Event Extraction aims to identify the event type as Business Start-Org where ‘co-founded' triggers the event and (Steve Jobs, Steve Wozniak) are participants in the event as AGENT and Apple as ORG respectively.

  • Entity Linking aims to link the mention Steve Job to Steven Jobs (Q19837) on Wikidata, and Apple to Apple (Q312) as well.

  • Knowledge Graph Completion aims to complete incomplete triples $\langle$Steve Job, create, ?$\rangle$ for blank entities Apple, NeXT Inc. and Pixar.

🏳‍🌈 A Taxonomy of Current Methods

In this Survey, we summarize recent progress in generative KGC. We propose to organize relevant work by the generation target of models and also present the axis of the task level.


1. Copy-based Sequence.

This paradigm refers to developing more robust models to copy the corresponding entity directly from the input sentence during the generation process. As shown in figure, the model copies the head entity from the input sentence and then the tail entity.


  • Directly copy entity

    • "Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism", ACL 2018

      • Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, Jun Zhao
      • [Paper]
    • "Learning the Extraction Order of Multiple Relational Facts in a Sentence with Reinforcement Learning", AAAI 2020

      • Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, Jun Zhao
      • [Paper]
    • "CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learning", EMNLP 2019

      • Daojian Zeng, Haoran Zhang, Qianying Liu
      • [Paper]
    • "Document-level Entity-based Extraction as Template Generation", EMNLP 2021

      • Kung-Hsiang Huang, Sam Tang, Nanyun Peng
      • [Paper]
  • Restricted target vocabulary

    • "A sequence-to-sequence approach for document-level relation extraction", BioNLP 2022
      • John Giorgi, Gary Bader, Bo Wang
      • [Paper]

2. Structure-linearized Sequence.

This paradigm refers to utilizing structural knowledge and label semantics, making it prone to handling a unified output format. As shown in figure, the output is a linearization of the extracted knowledge structure.


  • Per-token tag encoding

    • "Exploring Sequence-to-Sequence Learning in Aspect Term Extraction", ACL 2019

      • Dehong Ma, Sujian Li, Fangzhao Wu, Xing Xie, Houfeng Wang
      • [Paper]
    • "Neural Architectures for Nested NER through Linearization", ACL 2019

      • Jana Straková, Milan Straka, Jan Hajic
      • [Paper]
  • Faithful contrastive learning

    • "Contrastive Triple Extraction with Generative Transformer", AAAI 2021

      • Hongbin Ye, Ningyu Zhang, Shumin Deng, Mosha Chen, Chuanqi Tan, Fei Huang, Huajun Chen
      • [Paper]
    • "Contrastive Information Extraction with Generative Transformer", IEEE ACM Trans. Audio Speech Lang. Process

      • Ningyu Zhang, Hongbin Ye, Shumin Deng, Chuanqi Tan, Mosha Chen, Songfang Huang, Fei Huang, Huajun Chen
      • [Paper]
    • "Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning", ACL 2022

      • Swarnadeep Saha, Prateek Yadav, Mohit Bansal
      • [Paper]
  • Prefix tree constraint decoding

    • "Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction", ACL 2021

      • Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, Shaoyi Chen
      • [Paper]
    • "GenIE: Generative Information Extraction", NAACL 2022

      • Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, Robert West
      • [Paper]
  • Triplet linearization

    • "REBEL: Relation Extraction By End-to-end Language generation", EMNLP 2021

      • Pere-Lluís Huguet Cabot, Roberto Navigli
      • [Paper]
    • "De-Bias for Generative Extraction in Unified NER Task", ACL 2022

      • Shuai Zhang, Yongliang Shen, Zeqi Tan, Yiquan Wu, Weiming Lu
      • [Paper]
  • Entity-aware hierarchical decoding

    • "From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer", WWW 2022
      • Xin Xie, Ningyu Zhang, Zhoubo Li, Shumin Deng, Hui Chen, Feiyu Xiong, Mosha Chen, Huajun Chen
      • [Paper]
  • Unified structure generation

    • "Unified Structure Generation for Universal Information Extraction", ACL 2022

      • Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, Hua Wu
      • [Paper]
    • "DeepStruct: Pretraining of Language Models for Structure Prediction", ACL 2022

      • Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, Dawn Song
      • [Paper]
    • "Text-to-Text Extraction and Verbalization of Biomedical Event Graphs", COLING 2022

      • Giacomo Frisoni, Gianluca Moro, Lorenzo Balzani
      • [Paper]
  • Reformulating triple prediction

    • "Intent Classification and Slot Filling for Privacy Policies", ACL 2021

      • Wasi Uddin Ahmad, Jianfeng Chi, Tu Le, Thomas Norton, Yuan Tian, Kai-Wei Chang
      • [Paper]
    • "HySPA: Hybrid Span Generation for Scalable Text-to-Graph Extraction", ACL 2021

      • Liliang Ren, Chenkai Sun, Heng Ji, Julia Hockenmaier
      • [Paper]
    • "SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning", EMNLP 2022

      • Yushi Bai, Xin Lv, Juanzi Li, Lei Hou, Yincen Qu, Zelin Dai, Feiyu Xiong
      • [Paper]
  • Query Verbalization

    • "Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linking", ACL 2022

      • Tuan Lai, Heng Ji, ChengXiang Zhai
      • [Paper]
    • "Sequence-to-Sequence Knowledge Graph Completion and Question Answering", ACL 2022

      • Apoorv Saxena, Adrian Kochsiek, Rainer Gemulla
      • [Paper]
    • "Knowledge Is Flat: A Seq2Seq Generative Framework for Various Knowledge Graph Completion", COLING 2022

      • Chen Chen, Yufei Wang, Bing Li, Kwok-Yan Lam
      • [Paper]

3. Label-based Sequence.

This paradigm refers to utilizing the extra markers to indicate specific entities or relationships. As shown in figure, the output sequence copies all words in the input sentence, as it helps to reduce ambiguity. In addition, this paradigm uses square brackets or other identifiers to specify the tagging sequence for the entity of interest. The relevant labels are separated by the separator "$|$" within enclosed brackets. Meanwhile, the labeled words are described with natural words so that the potential knowledge of the pre-trained model can be leveraged.


  • Augmented natural language

    • "Augmented Natural Language for Generative Sequence Labeling", EMNLP 2020

      • Ben Athiwaratkun, Cicero Nogueira dos Santos, Jason Krone, Bing Xiang
      • [Paper]
    • "Autoregressive Entity Retrieval ", ICLR 2021

      • Nicola De Cao, Gautier Izacard, Sebastian Riedel, Fabio Petroni
      • [Paper]
    • "Structured Prediction as Translation between Augmented Natural Languages ", ICLR 2021

      • Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, Stefano Soatto
      • [Paper]
    • "Autoregressive Structured Prediction with Language Models", EMNLP 2022

      • Tianyu Liu, Yuchen Jiang, Nicholas Monath, Ryan Cotterell, Mrinmaya Sachan
      • [Paper]

4. Indice-based Sequence.

This paradigm generates the indices of the words in the input text of interest directly, and encodes class labels as label indices. As the output is strictly restricted, it will not generate indices that corresponding entities do not exist in the input text, except for relation labels.


  • Pointer mechanism

    • "Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction Authors", AAAI 2020

    • "Don’t Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing", WWW 2020

      • Subendhu Rongali, Luca Soldaini, Emilio Monti, Wael Hamza
      • [Paper]
    • "A Unified Generative Framework for Various NER Subtasks", ACL 2021

      • Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, Xipeng Qiu
      • [Paper]
    • "A Unified Generative Framework for Aspect-based Sentiment Analysis", ACL 2021

      • Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, Zheng Zhang
      • [Paper]
  • Pointer selection

    • "GRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction", EACL 2021
      • Xinya Du, Alexander Rush, Claire Cardie
      • [Paper]

5. Blank-based Sequence.

This paradigm refers to utilizing templates to define the appropriate order and relationship for the generated spans. As shown in figure, the template refers to a text describing an event type, which adds blank argument role placeholders. The output sequences are sentences where the blank placeholders are replaced by specific event arguments.


  • Template filling as generation

    • "COMET: Commonsense Transformers for Automatic Knowledge Graph Construction", ACL 2019

      • Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, Yejin Choi
      • [Paper]
    • "Document-Level Event Argument Extraction by Conditional Generation", NAACL 2021

      • Sha Li, Heng Ji, Jiawei Han
      • [Paper]
    • "Template Filling with Generative Transformers", NAACL 2021

      • Xinya Du, Alexander Rush, Claire Cardie
      • [Paper]
    • "ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification", ACL 2022

      • Yucheng Zhou, Tao Shen, Xiubo Geng, Guodong Long, Daxin Jiang
      • [Paper]
  • Prompt semantic guidance

    • "DEGREE: A Data-Efficient Generation-Based Event Extraction Model", NAACL 2022

      • I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, Nanyun Peng
      • [Paper]
    • "Dynamic Prefix-Tuning for Generative Template-based Event Extraction", ACL 2022

      • Xiao Liu, Heyan Huang, Ge Shi, Bo Wang
      • [Paper]
    • "Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction", ACL 2022

      • Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, Jing Shao
      • [Paper]
  • Language-agnostic template

    • "Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction", ACL 2022
      • Kuan-Hao Huang, I-Hung Hsu, Prem Natarajan, Kai-Wei Chang, Nanyun Peng
      • [Paper]

🏆 A List of Survey Papers

Survey Paper Publish
🚩 Generative Knowledge Graph Construction: A Review EMNLP 2022
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications TNNLS 2022
Multi-Modal Knowledge Graph Construction and Application: A Survey Arxiv 2022
Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey Arxiv 2021

🕚 A Timeline of generative KGC.

The time for each paper is based on its first arXiv version (if exists) or estimated submission time.

Papers Method Conference Code
Code4Struct: Code Generation for Few-Shot Structured Prediction from Natural Language Structure-linearized arXiv 2022 CODE4STRUCT
Autoregressive Structured Prediction with Language Models Label-augmented EMNLP 2022 ASP
Text-to-Text Extraction and Verbalization of Biomedical Event Graphs Structure-linearized COLING 2022 BioT2E
Knowledge Is Flat: A Seq2Seq Generative Framework for Various Knowledge Graph Completion Structure-linearized COLING 2022 KG-S2S
A sequence-to-sequence approach for document-level relation extraction Copy-based BioNLP 2022 Seq2rel
Unified Structure Generation for Universal Information Extraction Structure-linearized ACL 2022 UIE
Sequence-to-Sequence Knowledge Graph Completion and Question Answering Structure-linearized ACL 2022 KGT5
Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction Blank-based ACL 2022 PAIE
De-Bias for Generative Extraction in Unified NER Task Structure-linearized ACL 2022 -
DeepStruct: Pretraining of Language Models for Structure Prediction Structure-linearized ACL 2022 DeepStruct
Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction Blank-based ACL 2022 X-GEAR
Dynamic Prefix-Tuning for Generative Template-based Event Extraction Blank-based ACL 2022 -
ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification Blank-based ACL 2022 -
Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning Structure-linearized ACL 2022 HuSe-Gen
Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linking Structure-linearized ACL 2022 EPGEL
From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer Structure-linearized WWW 2022 GenKGC
SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning Structure-linearized EMNLP 2022 -
REBEL: Relation Extraction By End-to-end Language generation Structure-linearized EMNLP 2021 REBEL
Document-level Entity-based Extraction as Template Generation Copy-based EMNLP 2021 TEMPGEN
DEGREE: A Data-Efficient Generation-Based Event Extraction Model Blank-based NAACL 2022 DEGREE
HySPA: Hybrid Span Generation for Scalable Text-to-Graph Extraction Structure-linearized ACL 2021 HySPA
Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction Structure-linearized ACL 2021 Text2Event
Template Filling with Generative Transformers Blank-based NAACL 2021 GTT
A Unified Generative Framework for Aspect-based Sentiment Analysis Indice-based ACL 2021 BARTABSA
A Unified Generative Framework for Various NER Subtasks Indice-based ACL 2021 BARTNER
GRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction Indice-based EACL2021 GRIT
Document-Level Event Argument Extraction by Conditional Generation Blank-based NAACL 2021 BART-Gen
Structured Prediction as Translation between Augmented Natural Languages Label-augmented ICLR 2021 TANL
Intent Classification and Slot Filling for Privacy Policies Structure-linearized ACL 2021 PolicyIE
Autoregressive Entity Retrieval Label-augmented ICLR 2021 GENRE
Augmented Natural Language for Generative Sequence Labeling Label-augmented EMNLP 2020 -
Contrastive Information Extraction With Generative Transformer Structure-linearized TASLP 2021 -
Don’t Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing Indice-based WWW 2022 -
CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learning Copy-based AAAI 2020 CopyMTL
Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction Indice-based AAAI 2020 PNDec
Learning the Extraction Order of Multiple Relational Facts in a Sentence with Reinforcement Learning Copy-based EMNLP 2019 -
Neural Architectures for Nested NER through Linearization Structure-linearized ACL 2019 -
Exploring Sequence-to-Sequence Learning in Aspect Term Extraction Structure-linearize ACL 2019 -
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction Blank-based ACL 2019 COMET
Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism Copy-based ACL 2018 -

🌟 TIPS

If you find this repository useful to your research or work, it is really appreciate to star this repository.