Improving Text Summarization Quality by Combining T5-Based Models and Convolutional Seq2Seq Models

Arif Ridho Lubis; Habibi Ramdani Safitri; Irvan Irvan; Muharman Lubis; Al-Khowarizmi Al-Khowarizmi

doi:10.37385/jaets.v5i1.2503

Authors

Arif Ridho Lubis Politeknik Negeri Medan
Habibi Ramdani Safitri Politeknik Negeri Medan
Irvan Irvan Universitas Muhammadiyah Sumatera Utara
Muharman Lubis School of Industrial Engineering Telkom University
Al-Khowarizmi Al-Khowarizmi Universitas Muhammadiyah Sumatera Utara

DOI:

https://doi.org/10.37385/jaets.v5i1.2503

Keywords:

Model T5, Seq2Seq Convolutional, ROUGE

Abstract

In the natural language processing field, there are several sub-fields that are very closely related to information retrieval, such as the automatic text summarization sub-field. obtained from the convolutional T5 and Seq2Seq models in summarizing text on hugging faces found features that can affect text summary such as upper- and lower-case letters which have an impact on changing the understanding of the text of the document. This study uses a combination of parameters such as layer dimensions, learning rate, batch size, and the use of Dropout to avoid model overfitting. The results can be seen by evaluating metrics using ROUGE. This study produces a value of ROUGE-1 on 4 documents that are tested which produces an average of 0.8 which is the optimal value, for ROUGE-2 on 4 documents that are tested which results in an average of 0.83 which is an optimal value while ROUGE-L on 4 documents conducted tests that produce an average of 0.8 which is the optimal value for the summary model.

Downloads

Download data is not yet available.

References

Abdel-Salam, S., & Rafea, A. (2022). Performance study on extractive text summarization using BERT models. Information, 13(2), 67.

Abdulateef, S., Khan, N. A., Chen, B., & Shang, X. (2020). Multidocument Arabic text summarization based on clustering and Word2Vec to reduce redundancy. Information, 11(2), 59.

Alambo, A., Banerjee, T., Thirunarayan, K., & Cajita, M. (2022). Improving the Factual Accuracy of Abstractive Clinical Text Summarization using Multi-Objective Optimization. 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 1615–1618.

Allahyari, M., Pouriyeh, S., Assefi, M., Safaei, S., Trippe, E. D., Gutierrez, J. B., & Kochut, K. (2017). Text summarization techniques: a brief survey. ArXiv Preprint ArXiv:1707.02268.

Batra, H., Jain, A., Bisht, G., Srivastava, K., Bharadwaj, M., Bajaj, D., & Bharti, U. (2021). CoVShorts: News Summarization Application Based on Deep NLP Transformers for SARS-CoV-2. 2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO), 1–6.

Chaves, A., Kesiku, C., & Garcia-Zapirain, B. (2022). Automatic Text Summarization of Biomedical Text Data: A Systematic Review. Information, 13(8), 393.

Cheng, J., Zhang, F., & Guo, X. (2020). A syntax-augmented and headline-aware neural text summarization method. IEEE Access, 8, 218360–218371.

Chouikhi, H., & Alsuhaibani, M. (2022). Deep Transformer Language Models for Arabic Text Summarization: A Comparison Study. Applied Sciences, 12(23), 11944.

Christian, H., Agus, M. P., & Suhartono, D. (2016). Single document automatic text summarization using term frequency-inverse document frequency (TF-IDF). ComTech: Computer, Mathematics and Engineering Applications, 7(4), 285–294.

Dhivyaa, C. R., Nithya, K., Janani, T., Kumar, K. S., & Prashanth, N. (2022). Transliteration based Generative Pre-trained Transformer 2 Model for Tamil Text Summarization. 2022 International Conference on Computer Communication and Informatics (ICCCI), 1–6.

Di Egidio, G., Ceschini, L., Morri, A., & Zanni, M. (2023). Room-and High-Temperature Fatigue Strength of the T5 and Rapid T6 Heat-Treated AlSi10Mg Alloy Produced by Laser-Based Powder Bed Fusion. Metals, 13(2), 263.

El-Kassas, W. S., Salama, C. R., Rafea, A. A., & Mohamed, H. K. (2021). Automatic text summarization: A comprehensive survey. Expert Systems with Applications, 165, 113679.

Farahani, M., Gharachorloo, M., & Manthouri, M. (2021). Leveraging ParsBERT and pretrained mT5 for Persian abstractive text summarization. 2021 26th International Computer Conference, Computer Society of Iran (CSICC), 1–6.

Fendji, J. L. E. K., Taira, D. M., Atemkeng, M., & Ali, A. M. (2021). WATS-SMS: A T5-Based French Wikipedia Abstractive Text Summarizer for SMS. Future Internet, 13(9), 238.

Gupta, A., Chugh, D., Anjum, & Katarya, R. (2022). Automated news summarization using transformers. In Sustainable Advanced Computing: Select Proceedings of ICSAC 2021 (pp. 249–259). Springer.

Jung, S.-Y., Lin, T.-H., Liao, C.-H., Yuan, S.-M., & Sun, C.-T. (2022). Intent-Controllable Citation Text Generation. Mathematics, 10(10), 1763.

La Quatra, M., & Cagliero, L. (2022). BART-IT: An Efficient Sequence-to-Sequence Model for Italian Text Summarization. Future Internet, 15(1), 15.

Lalitha, E., Ramani, K., Shahida, D., Deepak, E. V. S., Bindu, M. H., & Shaikshavali, D. (2023). Text Summarization of Medical Documents using Abstractive Techniques. 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), 939–943.

Lee, C.-H., Yang, H.-C., Chen, Y. J., & Chuang, Y.-L. (2021). Event Monitoring and Intelligence Gathering Using Twitter Based Real-Time Event Summarization and Pre-Trained Model Techniques. Applied Sciences, 11(22), 10596.

Liang, Z., Du, J., & Li, C. (2020). Abstractive social media text summarization using selective reinforced Seq2Seq attention model. Neurocomputing, 410, 432–440.

Lubis, A. R., & Nasution, M. K. M. (2023). TWITTER DATA ANALYSIS AND TEXT NORMALIZATION IN COLLECTING STANDARD WORD. Journal of Applied Engineering and Technological Science, 4(2), 855–863. https://doi.org/10.37385/jaets.v4i2.1991

Lubis, A. R., Nasution, M. K. M., Sitompul, O. S., & Zamzami, E. M. (2022a). The feature extraction for classifying words on social media with the Naïve Bayes algorithm. IAES International Journal of Artificial Intelligence (IJ-AI), 11(3), 1041–1048. https://doi.org/10.11591/ijai.v11.i3.pp1041-1048

Lubis, A. R., Nasution, M. K. M., Sitompul, O. S., & Zamzami, E. M. (2022b). Spelling Checking with Deep Learning Model in Analysis of Tweet Data for Word Classification Process. 2022 9th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), 343–348. https://doi.org/10.23919/EECSI56542.2022.9946476

Lubis, A. R., Prayudani, S., Lubis, M., & Nugroho, O. (2022). Sentiment Analysis on Online Learning during the Covid-19 Pandemic Based on Opinions on Twitter using KNN Method. 2022 1st International Conference on Information System and Information Technology, ICISIT 2022, 106–111. https://doi.org/10.1109/ICISIT54091.2022.9872926

Ramesh, G. S., Manyam, V., Mandula, V., Myana, P., Macha, S., & Reddy, S. (2022). Abstractive Text Summarization Using T5 Architecture. Proceedings of Second International Conference on Advances in Computer Engineering and Communication Systems: ICACECS 2021, 535–543.

Ranganathan, J., & Abuka, G. (2022). Text summarization using transformer model. 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS), 1–5.

Reddy, K. M., & Guha, R. (2023). Automatic Text Summarization For Conversational Chatbot. 2023 IEEE 8th International Conference for Convergence in Technology (I2CT), 1–7.

Shi, T., Keneshloo, Y., Ramakrishnan, N., & Reddy, C. K. (2021). Neural abstractive text summarization with sequence-to-sequence models. ACM Transactions on Data Science, 2(1), 1–37.

Song, C., Shao, T., Lin, K., Liu, D., Wang, S., & Chen, H. (2022). Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models. Applied Sciences, 12(21), 11117.

Vogel-Fernandez, A., Calleja, P., & Rico, M. (2022). esT5s: A Spanish Model for Text Summarization. In Towards a Knowledge-Aware AI (pp. 184–190). IOS Press.

Widyassari, A. P., Rustad, S., Shidik, G. F., Noersasongko, E., Syukur, A., & Affandy, A. (2022). Review of automatic text summarization techniques & methods. Journal of King Saud University-Computer and Information Sciences, 34(4), 1029–1046.

Xi, X., Pi, Z., & Zhou, G. (2020). Global encoding for long Chinese text summarization. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(6), 1–17.

Xu, S., Zhang, X., Wu, Y., & Wei, F. (2022). Sequence level contrastive learning for text summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10), 11556–11565.

Zhang, Y., Wang, Y., Liao, J., & Xiao, W. (2018). A hierarchical attention seq2seq model with copynet for text summarization. 2018 International Conference on Robots & Intelligent System (ICRIS), 316–320.

Zolotareva, E., Tashu, T. M., & Horváth, T. (2020). Abstractive Text Summarization using Transfer Learning. ITAT, 75–80.