1. Encoder-Decoders

Encoder-decoder architecture has been pioneered in 2014, primarily being the core technology behind Google Translator.

This architecture works as a multi-label classification, with an Encoder Vector as lower dimensional representation of the data, connecting the inputs and outputs.

When we use the same data as input and output, we have an Autoencoder. Autoencoders can be used for:

• Dimensionality reduction: Smaller dimension space representation of the inputs
• De-noising data: When trained with clean data, the model can filter out noise during reconstruction
• Anomaly detection: If the input has unseen information to the model, which means an anomaly, the output will be a poor reconstruction of the data.

To demonstrate the encoding-decoding process, we will create a very simple and basic Autoencoder for MNIST digits.

# Build our autoencoder
autoencoder = Sequential()
autoencoder.add(Dense(32, activation='relu', input_shape=(784,)))
autoencoder.add(Dense(784, activation='sigmoid'))

autoencoder.compile(optimizer='adam', loss='binary_crossentropy')

autoencoder.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 32)                25120     
                                                                 
 dense_1 (Dense)             (None, 784)               25872     
                                                                 
=================================================================
Total params: 50,992
Trainable params: 50,992
Non-trainable params: 0
_________________________________________________________________

Our Autoencoder has same number of nodes in input and output, corresponding to 784 pixels of each image. The hidden layer, which is the Encoder Vector, we applied 32 nodes.

# Build our encoder
encoder = Sequential()
encoder.add(autoencoder.layers[0])

encoded = encoder.predict(X_test_bin)

313/313 [==============================] - 1s 4ms/step

Our test set has 10 thousand digit images. After being encoded, each image is represented by 32 values, instead of 784 (28x28) pixels.

encoded.shape

(10000, 32)

# Build our decoder
decoder = Sequential()
decoder.add(autoencoder.layers[1])

decoded = decoder.predict(encoded)

313/313 [==============================] - 0s 1ms/step

Below we show examples of each digit, before and after encoding/decoding.

We also show 5 examples for each digit, showing their original and decoded form, as well as a representation of their encoded format. It is important to note that the encoded format does not have a real meaning, it is a row of 32 numerical values, but we present them here reshaped in a rectangular format, just for illustrative purpose.

1.1. De-noising application

We will pick 5 digits, artificially add some noise and apply the encoding/decoding to remove the noise.

compare_digits(X_test_bin.reshape(-1, 28, 28),
               X_test_noisy.reshape(-1, 28, 28),
               idxs, noisy=True)

encoded_noisy = encoder.predict(X_test_noisy)
decoded_noisy = decoder.predict(encoded_noisy)

compare_digits(X_test_noisy.reshape(-1, 28, 28),
               decoded_noisy.reshape(-1, 28, 28),
               range(5))

1/1 [==============================] - 0s 44ms/step
1/1 [==============================] - 0s 21ms/step

1.2. Anomaly detection

For anomaly detection we will apply same encoder-decoder networks, with an image which is not a digit and look the output.

image = data.horse()
image = resize(image, (28, 28))
image = image == 0

encoded_img = encoder.predict(image.reshape(1, 784))
decoded_img = decoder.predict(encoded_img)

1/1 [==============================] - 0s 30ms/step
1/1 [==============================] - 0s 25ms/step

2. Recurrent Neural Networks

Now let us take a step further, applying encoder-decoders. Recurrent Networks are built onto this architecture, but they expand its applications.

On our previous example, we built an autoencoder based on same data as input and output. Now imagine that, instead of encoding/decoding pixels from images, we encode/decode words from a corpus. With this concept we can grow into larger models with more complex relationships.

LSTM (Long Short Term Memory) structure is a Recurrent Network topology which uses Encoding/Decoding layers to build the model's input and output. Added to this, we have a recurrent structure, which linkages entire sentences, giving a sence of memory and importance to sentence order.

We won't dive deep into the LSTM structure, but it is important to note that is has a function responsible for weighting importance of previous data (picture below from DataCamp).

Since Recurrent Networks are able to capture sequence importance, it allows many different applications, such as:

• Speech to written text transformation;
• Language translation;
• Document summarization;
• Musical composition;
• Image captioning;
• Text generation/prediction;
• Time series modeling.

2.1. Text Generation

We will perform a simple example below, creating a simple model for word prediction, based on a text corpus.

Based on the text from the book Lord of the Rings: The Fellowship of the Ring, our LSTM model will be trained to predict a fourth word, based on previous three. Since our model is extremely simple, for each sentence we want to predict, we will actually show the top 5 words with highest probabilities predicted.

The entire book has 9475 different words, and this will be our vocabulary of possible words. It is important to note that we did not remove any stopwords, neither we applied lemmatization or stemming here.

vocab_size

9475

# We build our LSTM model
model = Sequential()
model.add(Embedding(input_dim = vocab_size, input_length = 3, output_dim= 128, ))
model.add(LSTM(32))
model.add(Dense(32, activation='relu'))
model.add(Dense(vocab_size, activation='softmax'))
model.summary()

model.compile(optimizer='adam', loss='categorical_crossentropy')

Model: "sequential_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding (Embedding)       (None, 3, 128)            1212800   
                                                                 
 lstm (LSTM)                 (None, 32)                20608     
                                                                 
 dense_2 (Dense)             (None, 32)                1056      
                                                                 
 dense_3 (Dense)             (None, 9475)              312675    
                                                                 
=================================================================
Total params: 1,547,139
Trainable params: 1,547,139
Non-trainable params: 0
_________________________________________________________________

In order to represent our vocabulary of 9475 words, we applied Word Embedding, into 128 nodes. This layer is capable of representing our entire vocabulary in similarity vectors between words.

In the output of the LSTM model, it applies a Decoding layer, returning to 9475 nodes, where each one corresponds to a word of our vocabulary.

After training our model, we try to use it to predict words with highest probability for some sentences.

predict_text('One Ring to')

1/1 [==============================] - 0s 438ms/step

['be', 'go', 'the', 'see', 'have']

predict_text('Fellowship of the')

1/1 [==============================] - 0s 28ms/step

['shire', 'road', 'ring', 'world', 'mountains']

predict_text('Men and Elves')

1/1 [==============================] - 0s 28ms/step

['and', 'said', 'but', 'were', 'had']

predict_text('Gandalf was a')

1/1 [==============================] - 0s 32ms/step

['long', 'while', 'great', 'little', 'few']

Althought not perfect, our model was capable of predicting words that could make sense in a sentence. For example the next word predicted for Men and Elves were mainly verb or connective, while the next word predicted for Gandalf was a were adjectives and the next word predicted for Fellowship of the were mainly nouns. The model was capable of extracting this structure, without ever learning grammar or word classes.

LSTM were revolutionary and brought many applications, but they also have several shortcomings:

• They are extremely slow to train;
• Their hidden state depends on the sequence order, so this topology doesn't allow for parallel computation;
• Big sequences provoke exploding or vanishing gradients problem.

3. Transformers Networks

So here enters the Transformers, introduced by Google Brain, in 2017. Transformers is a new Neural Network topology, that replaces the sequential computation by attention mechanism, in recurrent networks (picture from original paper Attention is all you need). It uses multi-head attention layers, which captures and weights sequence importance, without requiring the model to be trained in sequence. The left rectangle behaves as the Encoder while the right rectangle behaves as the decoder.

This topology is mainly used for Language Translation, but also has been widely used in top-notch medical researches like new drugs discovery and diseases research.

The Transformers presents:

• Lower computational complexity compared to previous RNNs topologies;
• Allows for parallel computation, since the recurrence feature is modelled by attention layers;
• The multi-head attention layers also solve the vanishing gradient problem, allowing longer sequences to be used.

Below we present some examples, by applying pre-trained Transformers networks.

3.1. Language Translation

One main task is to perform Language Translation. This, however, is not a simple task, since different languages may have different grammar and phrase structure, and the model must be able to learn how to properly translate.

We show here a pre-trained model, from the NLP group from the University of Helsinki, built on MariamMT framework, which uses an engine built on C++ by Microsoft and academics. It is the same engine used by Microsoft Translator.

We will use it to translate English sentences to Portuguese.

# Get the name of the model
model_name = 'Helsinki-NLP/opus-mt-tc-big-en-pt'

# Get the tokenizer
tokenizer = MarianTokenizer.from_pretrained(model_name)
# Instantiate the model
model = MarianMTModel.from_pretrained(model_name)

def format_batch_texts(language_code, batch_texts):
    # Add target language token, so the model may identify the language
    formated_bach = [">>{}<< {}".format(language_code, text) for text in batch_texts]
    return formated_bach

def perform_translation(batch_texts, model, tokenizer, language="pt"):
    # Prepare the text data into appropriate format for the model
    formated_batch_texts = format_batch_texts(language, batch_texts)
    
    # Generate translation using model
    translated = model.generate(**tokenizer(formated_batch_texts, return_tensors="pt", padding=True))

    # Convert the generated tokens indices back into text
    translated_texts = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
 
    return translated_texts

english_texts = ["Artificial intelligence (AI) is intelligence - perceiving, synthesizing, and inferring information - demonstrated by machines, as opposed to intelligence displayed by non-human animals and humans.",
                 "Example tasks in which this is done include speech recognition, computer vision, translation between (natural) languages, as well as other mappings of inputs."]

# Check the model translation from the original language (English) to Portuguese
translated_texts = perform_translation(english_texts, model, tokenizer)

print(f"English texts:\n{[text for text in english_texts]}")
print(f"\nTranslated Portuguese texts?\n{translated_texts}")

English texts:
['Artificial intelligence (AI) is intelligence - perceiving, synthesizing, and inferring information - demonstrated by machines, as opposed to intelligence displayed by non-human animals and humans.', 'Example tasks in which this is done include speech recognition, computer vision, translation between (natural) languages, as well as other mappings of inputs.']

Translated Portuguese texts?
['Inteligência artificial (IA) é inteligência - perceber, sintetizar e inferir informações - demonstradas por máquinas, em oposição à inteligência exibida por animais não humanos e humanos.', 'exemplo de tarefas em que isso é feito incluem reconhecimento de fala, visão computacional, tradução entre línguas (naturais), bem como outros mapeamentos de entradas.']

The texts to be translated were from Wikipedia, about Artificial Intelligence and the translated texts were very good, showing proper grammar:

English	Portuguese
Artificial intelligence (AI) is intelligence - perceiving, synthesizing, and inferring information - demonstrated by machines, as opposed to intelligence displayed by non-human animals and humans.	Inteligência artificial (IA) é inteligência - perceber, sintetizar e inferir informações - demonstradas por máquinas, em oposição à inteligência exibida por animais não humanos e humanos.
Example tasks in which this is done include speech recognition, computer vision, translation between (natural) languages, as well as other mappings of inputs.	exemplo de tarefas em que isso é feito incluem reconhecimento de fala, visão computacional, tradução entre línguas (naturais), bem como outros mapeamentos de entradas.

3.2. Question Answering

In our final example, we will apply another Transformer, trained to Question Answering. This model is capable of perform NER (Named Entity Recognition), in order to analyze the question and what should be the answer.

For this simple model, however, it is necessary to provide a corpus of text, which it will use as a context for the questions.

model_qa = "deepset/roberta-base-squad2"

task = 'question-answering'
QA_model = pipeline(task, model=model_qa, tokenizer=model_qa)

Our context text contains some paragraphs from Wikipedia, with information about Artificial Intelligence:

Artificial intelligence (AI) is intelligence - perceiving, synthesizing, and inferring information - demonstrated by machines, as opposed to intelligence displayed by non-human animals and humans. Example tasks in which this is done include speech recognition, computer vision, translation between (natural) languages, as well as other mappings of inputs.

AI applications include advanced web search engines (e.g., Google Search), recommendation systems (used by YouTube, Amazon and Netflix), understanding human speech (such as Siri and Alexa), self-driving cars (e.g., Waymo), automated decision-making and competing at the highest level in strategic game systems (such as chess and Go). As machines become increasingly capable, tasks considered to require "intelligence" are often removed from the definition of AI, a phenomenon known as the AI effect. For instance, optical character recognition is frequently excluded from things considered to be AI, having become a routine technology.

Artificial intelligence was founded as an academic discipline in 1956, and in the years since has experienced several waves of optimism, followed by disappointment and the loss of funding (known as an "AI winter"), followed by new approaches, success and renewed funding. AI research has tried and discarded many different approaches since its founding, including simulating the brain, modeling human problem solving, formal logic, large databases of knowledge and imitating animal behavior. In the first decades of the 21st century, highly mathematical-statistical machine learning has dominated the field, and this technique has proved highly successful, helping to solve many challenging problems throughout industry and academia.`

text = 'Artificial intelligence (AI) is intelligence - perceiving, synthesizing, and inferring information - demonstrated by machines, as opposed to intelligence displayed by non-human animals and humans. Example tasks in which this is done include speech recognition, computer vision, translation between (natural) languages, as well as other mappings of inputs. AI applications include advanced web search engines (e.g., Google Search), recommendation systems (used by YouTube, Amazon and Netflix), understanding human speech (such as Siri and Alexa), self-driving cars (e.g., Waymo), automated decision-making and competing at the highest level in strategic game systems (such as chess and Go). As machines become increasingly capable, tasks considered to require "intelligence" are often removed from the definition of AI, a phenomenon known as the AI effect. For instance, optical character recognition is frequently excluded from things considered to be AI, having become a routine technology. Artificial intelligence was founded as an academic discipline in 1956, and in the years since has experienced several waves of optimism, followed by disappointment and the loss of funding (known as an "AI winter"), followed by new approaches, success and renewed funding. AI research has tried and discarded many different approaches since its founding, including simulating the brain, modeling human problem solving, formal logic, large databases of knowledge and imitating animal behavior. In the first decades of the 21st century, highly mathematical-statistical machine learning has dominated the field, and this technique has proved highly successful, helping to solve many challenging problems throughout industry and academia.'

QA_input = {
          'question': 'What is Artificial Intelligence?',
          'context': text
          }

model_response = QA_model(QA_input)
print(f"Question: {QA_input['question']}")
print(f"Answer: {model_response['answer']}")

Question: What is Artificial Intelligence?
Answer: intelligence - perceiving, synthesizing, and inferring information

QA_input = {
          'question': 'When was Artificial Intelligence created?',
          'context': text
          }

model_response = QA_model(QA_input)
print(f"Question: {QA_input['question']}")
print(f"Answer: {model_response['answer']}")

Question: When was Artificial Intelligence created?
Answer: 1956

QA_input = {
          'question': 'What are the tools of Artificial Intelligence?',
          'context': text
          }

model_response = QA_model(QA_input)
print(f"Question: {QA_input['question']}")
print(f"Answer: {model_response['answer']}")

Question: What are the tools of Artificial Intelligence?
Answer: speech recognition, computer vision, translation between (natural) languages

QA_input = {
          'question': 'Which big companies use Artificial Intelligence?',
          'context': text
          }

model_response = QA_model(QA_input)
print(f"Question: {QA_input['question']}")
print(f"Answer: {model_response['answer']}")

Question: Which big companies use Artificial Intelligence?
Answer: YouTube, Amazon and Netflix

Overall the answers were very good. It was not capable of creating sentences for the answers, this model was trained to only highlight key words regarding the questions. It would require other modellings to behave more human-likely as ChatGPT.

4. Conclusion

Being more complex and also bigger neural networks, with implementation of Transformers, it became more and more necessary to apply Transfer Learning, and also development of more efficient Transfer Learning techniques.

Currently we have these two state-of-the-art Transfer Learning models:

• BERT (Bidirectional Encoder Representations from Transformers): was developed in 2018 by Google. It helps to solve the most common language tasks such as NER (Named Entity Recognition), sentiment analysis, question-answering, text-summarization;
• GPT3 (Generative Pre-Training-3): was developed in 2020 by OpenAI. It is a multi-layer transformer, mainly used to generate new text. It became worlwide famous with the launch of ChatGPT.

With computational resources evolving faster and faster, AI area is also evolving, with new and interesting structures and topologies. We still have a lot more to evolve, not only in NLP, but in every area involving Machine Learning.