Improving Link Prediction Models through a Performance Enhancement Scheme: A Study on Semi-Supervised Learning and Model Soup

doi:10.19682/j.cnki.1005-8885.2024.1015

中国邮电高校学报(英文) ›› 2024, Vol. 31 ›› Issue (4): 43-53.doi: 10.19682/j.cnki.1005-8885.2024.1015

• Artificial Intelligence • 上一篇下一篇

Improving Link Prediction Models through a Performance Enhancement Scheme: A Study on Semi-Supervised Learning and Model Soup

亓东林¹,陈曙东²,杜蓉²,佟达¹,余泳¹

1. 中国科学院微电子研究所；中国科学院大学
2. 中国科学院微电子研究所

收稿日期:2023-06-26 修回日期:2024-03-27 出版日期:2024-08-31 发布日期:2024-08-31
通讯作者: 陈曙东 E-mail:chenshudong@ime.ac.cn

Improving Link Prediction Models through a Performance Enhancement Scheme: A Study on Semi-Supervised Learning and Model Soup

Received:2023-06-26 Revised:2024-03-27 Online:2024-08-31 Published:2024-08-31
Contact: shudong chen E-mail:chenshudong@ime.ac.cn

摘要/Abstract

摘要： 目前，大多数构建的知识图谱无论其规模如何，大多有不完备性问题。这种不完备性会对基于知识图谱的应用产生负面影响。作为知识图谱补充的重要方法，链接预测近年来已成为热门研究课题。本文提出了一种基于半监督学习和模型汤思想的链接预测模型性能增强方案，通过对模型架构进行微小改变，有效提高了几种主流链接预测模型的性能。这一创新方案主要包括两个部分：（1）使用半监督学习策略预测图中的潜在事实三元组，（2）创造性地结合半监督学习和模型汤，进一步提高最终模型的性能，而不增加显著的计算开销。我们通过实验证实了该方案在各种链接预测模型上的有效性，特别是在具有密集关系的数据集上。对于测试的模型中整体性能最佳的模型CompGCN，在经过增强方案后，在FB15K-237数据集上的Hits@1指标提高了14.7%，在WN18RR数据集上提高了7.8%。同时，我们观察到增强方案中的半监督学习策略对于多类链接预测模型有显著改进，并且模型汤的引入带来的性能改进与具体的测试模型有关，某些模型的性能得到了改善，而其他模型的性能基本保持不变。

关键词: 自然语言处理，知识图谱，链接预测，模型汤，半监督学习

Abstract: As a fact, most constructed knowledge graphs are far from complete no matter its size. This incompleteness will cause negative influence on the applications based on knowledge graphs. As an important method for knowledge graph complementation, link prediction has become a hot research topic in recent years. In this paper, a performance enhancement scheme for link prediction models based on the idea of semi-supervised learning and model soup is proposed, which effectively improves the model performance on several mainstream link prediction models with small changes to their architecture. This novel scheme consists of two main parts: (1) predicting potential fact triples in the graph with semi-supervised learning strategies, (2) creativily combining semi-supervised learning and model soup to further improve the final model performance without adding significant computational overhead. We experimentally validate the effectiveness of the scheme for a variety of link prediction models, especially on the dataset with dense relationships. In terms of CompGCN, the model with the best overall performance among the tested models improves its Hits@1 metric by 14.7% on the FB15K-237 dataset and 7.8% on the WN18RR dataset after using the enhancement scheme. Meanwhile, we observe that the semi-supervised learning strategy in the augmentation scheme has significant improvement for multi-class link prediction models, and the performance improvement brought by the introduction of the model soup is related to the specific tested models, because performance of some models are improved while others remained largely unaffected.

Key words: natural language processing, knowledge graph, link prediction, model soup, semi-supervised learning

参考文献

[1] JI S X, PAN S R, CAMBRIA E, et al. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2): 494-514.

[2] BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (SIGMOD’08), 2008, Jun 9-12, Vancouver, Canada. New York, NY, USA: ACM, 2008: 1247-1250.

[3] LEHMANN J, ISELE R, JAKOB M, et al. DBpedia—A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web, 2015, 6(2): 167-195.

[4] MAHDISOLTANI F, BIEGA J, SUCHANEK F M. YAGO3: A knowledge base from multilingual Wikipedias. Proceedings of the 7th Biennial Conference on Innovative Data Systems Research (CIDR’15), 2015, Jan 4-7, Asilomar, CA, USA. 2014: 12p.

[5] KROMPAß D, BAIER S, TRESP V. Type-constrained representation learning in knowledge graphs. Proceedings of the 14th International Semantic Web Conference (ISWC’15): Part 1, 2015, Oct 11-15, Bethlehem, PA, USA. Berlin, Germany: Springer, 2015: 640-655.

[6] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data. Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS'13): Vol 2, 2013, Dec 5-10, Lake Tahoe, NV, USA. Red Hook, NY, USA: Curran Associates Inc, 2013: 2787-2795

[7] TROUILLON T, WELBL J, RIEDEL S, et al. Complex embeddings for simple link prediction. Proceedings of the 33rd International Conference on Machine Learning (ICML'16), 2016, Jun, 20-22, New York, NY, USA. PMLR 48. The Journal of Machine Learning Research (JMLR), 2016: 2071-2080.

[8] YANG B S, YIH W T, HE X D, et al. Embedding entities and relations for learning and inference in knowledge bases. Proceedings of the 3rd International Conference on Learning Representations (ICLR’15), 2015, May 7-9, San Diego, CA, USA. 2015: 12p.

[9] NGUYEN D Q, NGUYEN T D, NGUYEN D Q, et al. A novel embedding model for knowledge base completion based on convolutional neural network. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’18): Vol 2 (Short Papers), 2018, Jun 1-6, New Orleans, LA, USA. Stroudsburg, PA, USA: Association for Computational Linguistics, 2018: 327-333.

[10] WORTSMAN M, ILHARCO G, GADRE S Y, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. Proceedings of the 39th International Conference on Machine Learning (ICML'22), 2022, Jul 17-23, Baltimore, MD, USA. PMLR 162. 2022: 23965-23998.

[11] ROSSI A, BARBOSA D, FIRMANI D, et al. Knowledge graph embedding for link prediction: A comparative analysis. ACM Transactions on Knowledge Discovery from Data, 2021, 15(2): Article 14.

[12] LI D, YI M, HE Y K. LP-BERT: Multi-task pre-training knowledge graph BERT for link prediction. arXiv Preprint, arXiv:2201.04843, 2022.

[13] WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes. Proceedings of the 28th AAAI Conference on Artificial Intelligence (AAAI'14), 2014, Jul 27-31, Quebec, Canada. Palo Alto, CA, USA: Association for the Advancement of Artificial Intelligence (AAAI), 2014: 1112-1119.

[14] LIN Y K, LIU Z Y, SUN M S, et al. Learning entity and relation embeddings for knowledge graph completion. Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI'15), 2015, Jan 25-30, Austin, TX, USA. Palo Alto, CA, USA: Association for the Advancement of Artificial Intelligence (AAAI), 2015: 2181-2187.

[15] JI G, HE S, XU L, et al. Knowledge graph embedding via dynamic mapping matrix. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing: Vol 1 (Long Papers), 2015, Jul 26-31, Beijing, China. Stroudsburg, PA, USA: Association for Computational Linguistics, 2015: 687-696.

[16] NGUYEN D Q, SIRTS K, QU L, et al. STransE: a novel embedding model of entities and relationships in knowledge bases. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’16), 2016, Jun 12-17, San Diego, CA, USA. Stroudsburg, PA, USA: Association for Computational Linguistics, 2016: 460-466.

[17] JI G L, LIU K, HE S Z, et al. Knowledge graph completion with adaptive sparse transfer matrix. Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI'16), 2016, Feb 12-17, Phoenix, AZ, USA. Palo Alto, CA, USA: Association for the Advancement of Artificial Intelligence (AAAI), 2016: 985-991.

[18] NGUYEN D Q, VU T, NGUYEN T D, et al. A capsule network-based embedding model for knowledge graph completion and search personalization. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’19): Vol 1 (Long and Short Papers), 2019, Jun 2-7, Minneapolis, MN, USA. Stroudsburg, PA, USA: Association for Computational Linguistics, 2019: 2180-2189.

[19] VASHISHTH S, SANYAL S, NITIN V, et al. Composition-based multi-relational graph convolutional networks. Proceedings of the 8th International Conference on Learning Representations (ICLR’20), 2020, Apr 26-30, Addis Ababa, Ethiopia. 2020: 15p.

[20] MAO X, WANG W T, XU H M, et al. Relational reflection entity alignment. Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM’20), 2020, Oct 19-23, Virtual Event, Ireland. New York, NY, USA: ACM, 2020: 1095-1104.

[21] ZENG Z Z, CHEN K J, ZHANG S B, et al. A link prediction approach using semi-supervised learning in dynamic networks. Proceedings of the 6th International Conference on Advanced Computational Intelligence (ICACI’13), 2013, Oct 19-21, Hangzhou, China. Piscataway, NJ, USA: IEEE, 2013: 276-280.

[22] ZHANG D L, SONG X Z, LIU Z Q, et al. DSSLP: A distributed framework for semi-supervised link prediction. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data’19), 2019, Dec 9-12, Los Angeles, CA, USA. Piscataway, NJ, USA: IEEE, 2019: 1557-1566.

[23] RAYMOND R, KASHIMA H. Fast and scalable algorithms for semi-supervised link prediction on static and dynamic graphs. Proceedings of the 2010 Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD’10): Part 3, 2010, Sept 20-24, Barcelona, Spain. LNCS 6323. Berlin, Germany: Springer, 2010: 131-147.

[24] HISANO R. Semi-supervised graph embedding approach to dynamic link prediction. Proceedings of the 2018 International Workshop on Complex Networks, 2018, Dec 10-11, Cambridge, MA, USA. Cham, Switzerland: Springer, 2018: 109-121.

[25] ZHANG T, ZHANG K, LI X, et al. Semi-supervised link prediction based on non-negative matrix factorization for temporal networks. Chaos, Solitons & Fractals, 2021, 145: Article 110769.

[26] MAO X, WANG W T, XU H M, et al. MRAEA: an efficient and robust entity alignment approach for cross-lingual knowledge graph. Proceedings of the 13th International Conference on Web Search and Data Mining (WSDM’20), 2020, Feb 3-7, Houston, TX, USA. New York, NY, USA: ACM, 2020: 420-428.

[27] NATHANI D, CHAUHAN J, SHARMA C, et al. Learning attention-based embeddings for relation prediction in knowledge graphs. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, Jul 28-Aug 2, Florence, Italy. Stroudsburg, PA, USA: Association for Computational Linguistics, 2019: 4710-4723.

[28] COSTABELLO L, BERNARDI A, JANIK A, et al. AmpliGraph: a Library for Representation Learning on Knowledge Graphs. https://doi.org/10.5281/zenodo.2595043.

Improving Link Prediction Models through a Performance Enhancement Scheme: A Study on Semi-Supervised Learning and Model Soup

Improving Link Prediction Models through a Performance Enhancement Scheme: A Study on Semi-Supervised Learning and Model Soup

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 1

编辑推荐

Metrics

本文评价