Information Technology for Foreign Languages Remote Learning with Adaptation to the User Based on Machine Learning

Abstract

Since the goal of the work is to improve the process of remote learning of foreign languages, it was chosen to create an application for translation based on the choice of the native language and the language being studied. An educational process using a set of telecommunication technologies aimed at enabling students to learn the basic amount of information they need without direct contact between students and teachers during the learning process (which can take place both synchronously and asynchronously), and can be both an independent form of education, as well as a supplement to another more traditional form of education (full-time, part-time, extramural or externship), if necessary, giving a person the opportunity to study a foreign language training course. So, on the basis of this concept, a translation application was developed, which accurately translates both ordinary language and phraseological units, slang expressions, etc. The model is used as the basis of training, so let's analyze the model according to the main indicators. The model was pre-trained on BookCorpus, a dataset consisting of 11,038 unpublished books and the English Wikipedia (excluding lists, tables and titles). The texts are written in lowercase and tokenized using WordPiece and a dictionary size of 30,000. With probability 0.5, sentence A and sentence B match two consecutive sentences in the original corpus, and in other cases it is another random sentence in the corpus. Note that a sentence here is a continuous stretch of text, usually longer than one sentence. The only limitation is that the result with two "sentences" has a total length of less than 512 tokens. The masking procedure details for each sentence are as follows: 15% of tokens are masked; in 80% of cases masked tokens are replaced by [MASK]; 10% of the time, masked tokens are replaced by a random token from the one they replace; in the remaining 10% of cases, masked markers remain unchanged. The model was trained on 4 Cloud TPUs in a Pod configuration (16 TPUs in total) for one million steps with a batch size of 256. The sequence length was limited to 128 markers for 90% of the steps and 512 for the remaining 10%. Adam optimizer is used with learning rate: β1=0.9, and β2=0.999, weight decay 0.01, learning rate warm-up for 10,000 steps and learning rate linear decrease after. After training the network, the mean squared error decreased from 34.2 to 3.3. Also, training the network made it possible to reduce overtraining and improve its ability to generalize to new data. In the trained network, the number of layers and neurons was increased, which allowed it to reproduce more complex dependencies in the input data. Training the network made it possible to improve its results on test data, increase its ability to generalize, optimize its structure and parameters, choose a more effective activation function, and reduce the risk of overtraining. © 2023 Copyright for this paper by its authors.