Abstract: With the continuous development of artificial intelligence, the application of pre-trained language model has been widely used in various fields, especially in complex text generation tasks.