How does transfer learning work

Transfer learning: basics and areas of application

Transfer learning is a method from deep learning with which a pre-trained artificial neural network is used to solve new problems. For this purpose, the learning progress of the existing model is transferred. This results in advantages such as: faster creation, better model quality and less use of resources.

Transfer learning is used in particular in the field of image and word processing, as this is an extremely helpful procedure for very complex deep learning models. Above all, it also enables smaller companies to get started with the topic.

Let's jump right in:

  1. What is transfer learning?
  2. Application areas of transfer learning
  3. When is transfer learning used?

What is transfer learning?

Transfer learning is a Machine learning technologythat takes a model that has already been trained for one task and applies it to another task. In particular, unstructured data, such as image, video and audio data, make such a deep learning approach interesting.

The advantage of transfer learning is that you can skip parts of the very complex training and thus save time and resources, because complex deep learning models often take weeks to train.

The simplest example of transferred learning is the processing of image data for object recognition. The following graphic is intended to underline this using an image classification for object recognition.

Here a pre-trained model (e.g. ResNet) is used for transfer learning, which can already recognize around 1000 different objects. However, the pre-trained model is not trained for our special application (here using the example of dog breeds) and does not yet know dog breeds such as the Chihuahua. Since we want to recognize the different breeds of dogs with our deep learning model, we have to train the algorithm again.

Most of the work is already done with transfer learning. Now we only have to add one last layer (classification layer) to the deep learning algorithm and we can predict dog breeds correctly. A relatively easy task that would probably have taken weeks without the pre-trained model.

Application areas of transfer learning

Transfer learning is used particularly in the area of ​​image, video and text data. Let's take a closer look at the topic:

Transfer learning with image data

Transfer learning for image data is the best-known example, which is also used very often in practice. An example is the detection of objects in pictures or videos (which are also nothing more than many pictures).

This type of problem is common in deep learning and is a very active field of research. But not only in research, industry is also very active in this area of ​​deep learning. Well-known (pre-trained) models are:

The creation of these models can take several days or weeks, even on special hardware.

The models mentioned can be downloaded for free online and you can integrate them directly into your model, for example to create an object classification.

Transfer learning with text data

Another example of the use of transfer learning is the processing of text data with Natural Language Processing (NLP). For this purpose, the deep learning model is trained on very large text documents.

Similar to the processing of image data with deep learning, there are already pre-trained models that have learned, for example, connections in speech and text. Words and contexts can be used to extract important information from a text.

For this purpose, so-called “word embeddings” are used, which then map the relationships and meanings of individual words in a high-dimensional space (in a vector). For example, different words that have a similar meaning have a similar vector.

Here, too, there are large companies and research institutions that invest a lot of money in the development of these deep learning models for the interpretation of text data. Two well-known deep learning models that can be used for transfer learning of text data:

These models are available for download and can be used immediately in machine learning models.

When is transfer learning used?

Transfer learning is an abbreviation for training machine learning models and thus saves time and resources and can ensure better results. Transfer learning can be helpful in three cases when creating a machine learning model:

  1. Better start (Higher start)
  2. Faster improvement of the model (higher slope)
  3. Better model quality (higher asymptote)

In the best case, you can benefit from all 3 advantages. A precise analysis of whether an existing model already exists and whether it is applicable to the problem at hand can save you days of work.

Especially when not a lot of data is available, transfer learning can help to create really good models that would simply be difficult to achieve in practice (or only with an enormous amount of data).