A Transformer-Based Model for Multi-Track Music Generation

Background

Similar to most sequence-to-sequence (Seq2Seq) models(Sutskever et al., 2014), Transformer uses an encoder-decoder structure. However, the previous model usually uses a recurrent neural network (such as LSTM) in the encoder and decoder. The disadvantage of this network structure is the problem of long-term dependence and the inability to calculate in parallel. In order to improve the efficiency of parallel computing and capture long-term dependencies, Transformer gave up the RNN cycle generation and used the self-attention model to build a fully connected network structure, thereby implementing an architecture based entirely on the feed-forward attention mechanism. In this context, OpenAI's GPT pre-training model came into being(Radford et al., 2018). GPT uses a generative method to train language models. GPT uses the decoder structure in Transformer and does not use a complete Transformer to build the network. With the wide application of GPT, OpenAI proposed GPT-2, which has a larger training data set and can do diverse tasks without supervision on the basis of GPT. The structure of the GPT-2 model is still the same as GPT, where the core idea is that unsupervised pre-training models can be used to do supervised tasks(Radford et al., 2019). Similar with the pre-training model, a weekly-supervised deep hashing method is proposed by using weekly-supervised information(Li et al., 2020). BERT (Bidirectional Encoder Representations from Transformers) also modifies the pre-training target on the basis of GPT, and uses a larger model and more data to pre-train to obtain the best results at present.