Guide to Natural Language Processing and Sequential Models

Introduction to Natural Language Processing

First we shall get introduced to natural language processing and enumerate various NLP applications. Deep learning has made huge inroads into NLP applications and things which were thought to be too difficult has become easier and achievable. We shall understand the concepts behind these algorithms and architectures and implement them on several applications as well.

We shall start off with the NLP pipeline and various pre-processing steps or components within it and understand the shortcoming of traditional techniques such as bag of words, one hot encoding or TFIDF that is Term Frequency Inverse Document Frequency and how word embeddings such as Word2Vec, GloVe that is global vectors for word representation were able to address these limitations.

Since text data is considered sequential data, we would need a different architecture to handle it. So, for that we use sequential models such as RNNs that is Recurrent Neural Networks for that. Within RNNs there are different variants such as LSTM that is Long Short Term Memory, and GRU that is Gated Recurrent Unit. We shall understand the concept behind that as well.

Researchers have been working on improving the performance constantly and hence we are observing better models or advancements in existing models, which includes bidirectional RNNs, encoder decoder model or Seq2Seq models with teacher forcing, attention mechanism, transformer models, GPT that is Generative Pre-Training and finally, BERT that is Bidirectional Encoder Representations from Transformers.

We shall be discussing these architectures in detail. Besides understanding the concepts we shall apply and implement various applications such as sentiment or text classification, language modelling, machine translation etcetera. Alright then, let us get started with the introduction to NLP.

The Deluge of Text Data and NLP Complexity

We are deluged with text data on everyday basis. Emails, Blogs, News and the list goes on and with social media platforms such as Twitter, Facebook, Quora etcetera it only gets compounded. Text data is one of the unstructured data types, which has gained lot of importance over the recent years.

Just few negative sentiments about a product, app, movie, service etcetera on any of the social media platforms, could prove detrimental to the success of the product and also mark the reputation of the company within a short span of time. Almost every company has started to realize that it is mandatory to analyze text data as the social media platforms and chatbot applications have become the medium for the customers to reach out to customer care than the traditional media such as Telephone, Emails etcetera.

Now that, we get an idea of its importance let us first define Natural language processing. Natural language processing is a field in artificial intelligence, which enables computers to understand natural, that is, human language. The world would be much simpler if we did not have natural languages, but instead had unnatural languages like the programming languages, which have a predefined rules and no ambiguity at all. And not adhering to it would result in exceptions or errors, which could be easily handled.

Fortunately though, we have let human language as natural, else it will have stifled communication, innovation and expressiveness; however, this does come at a cost. Natural language is difficult to learn as various complexities such as Sarcasm, Slang, Different dialects, Emoticons, Flexible Rules etcetera have to be accounted for while processing.

Adding to the complexity is the common usage of short forms of words, especially by the next generation. For instance, Right as ‘ryt’, How are you as ‘hru’ etcetera have become so common, that even keyboard predictions sometimes suggest these short forms than the actual words. These short forms, with different spellings make it even more challenging as these are not in standard dictionaries.

And if you account these most common short forms as new words, the list goes big and increase the size of the vocabulary. The bigger issue is that things change fast and there is no particular pattern to it. Another typical scenario, especially in countries with multiple languages is mixing of languages. For instance, it is common to see languages such as Hindi and English also known as ‘hinglish’ which is good for convenience, but makes it complex for processing as there are no formal rules yet.

The challenge is listed above and many more make NLP really a hard problem to solve. Nonetheless, there are techniques which do help in solving these issues with decent performance, but when deep learning was applied for NLP problems, things improved dramatically and great strides in the performance was observed. Let us list and understand few of the NLP applications, where deep learning is extensively used now.

Key NLP Applications

Sentiment Analysis is like the Hello World application of natural language processing. It might be one of the most basic applications any NLP practitioner would start to work with; however, it is one of the most important applications for any company. Either it is a product or a service one.

As most of the users or customers provides opinions, express their feelings and share their emotions on digital or social media platforms through comments ratings etcetera, which is openly available. And, if any negative sentiment not caught by the company, before it goes viral could prove detrimental to their brand and might have financial implications as well.

For major companies such as Amazon, Flipkart etcetera which sells huge number of products on their platforms, it is imperative for them to capture these data and process them for quick response. As they receive massive number of feedbacks, comments, likes, ratings etcetera every minute, it is not feasible to address them manually, but rather have to automate the whole process. And for that, it is important to use complex deep learning models for better accuracy.

Another application is the Intent Analysis, with digital medium becoming a choice over customer call center or IVR that is Interactive Voice Response, for many companies the companies need more than just sentiment analysis, that is to identify the intent behind the comment or statement and classify them as either Complaint, Feedback, Query, Suggestions, Opinions etcetera, as it might need to be routed to the specific team as per the Service Level Agreement, that is SLA would be different for different intents

For instance, complaints would need lower resolution time comparatively to opinions. This emphasizes the need for the team to handle massive text data as well. Sometimes, just knowing the intent would not be enough because it does not give granular information for quick turnaround time.

For instance, a complaint could be about a product bought in a store in a particular location. So, in this case, unless we are not able to automatically extract the store name, the location of the store and the product in reference, we are not able to route the issue directly to the team without manual intervention, which is time consuming and could also impact the SLA.

However, the techniques like NER that is Named Entity Recognition, we are able to extract the entities and feed it to the automated ticketing system of any company. NER plays a vital role in automating processes. For instance, it is quite extensively used in extracting candidates’ information from the resume, financial attributes or events from news for trading etcetera.

Language Translation is another common NLP application and the translation efficiency has increased dramatically that is from mere word-to-word translation to translation of sentences with the context. Chatbots another common application, especially when it comes to automating lead generation, employee engagement, obtaining transactional information etcetera is getting more sophisticated day-by-day that is from bounded questions or responses to free text conversation in multiple languages as well.

Autocomplete or Auto responses is quite common nowadays and are integrated with the tools, applications etcetera which is dramatically reduced the timing time and also reduced the errors while writing; however, Auto responses are still more conservative which is understandably.

So, as it is difficult to get the context from previous messages or emails or from conversations through other medium, which could make it difficult to come up with a lengthy text, the way we would have written it and besides an incorrect sentence could lead to a negative publicity or worst case legal implications as well.

Digital medium has changed the way we consume news and information; unlike earlier, where we had just very few media, that is Newspaper, TV with few channels to get any information or news. But with various digital media, we are overloaded with so much news and information, which makes it difficult to be abreast with everything. Besides, with so much information, it has also reduced the time a person dedicates per article.

Text Summarization comes very handy in such scenarios; where the whole news is condensed into a short passage, which conveys the main crux of it. And again, with deep learning models we have moved from extracting just the important statements from the passage to rewriting the whole article in different words, capturing the context as well.

Well, this is definitely not an exhaustive list of NLP applications. There are many more applications such as automated text generation, question and answer systems, text to speech and vice versa systems. Alright, with this amazing list of applications, let us get an overview of various components of an NLP pipeline and understand the various techniques underlying these applications and also implement few of them.

The NLP Pipeline: Tokenization and Pre-processing

For NLP problems, generally we deal with Text data which cannot be consumed directly because to perform mathematical computation we require numbers. However, before we numericalize the text data, we typically perform tokenization and optionally apply pre-processing steps, such as stop words removal, stemming, lemmatization, etcetera. Text data that is documents generally are set of paragraphs and sentences, which in turn are set of words with punctuation and symbols.

The process of breaking down these documents into set of sentences and each sentence into words is called Tokenization. Typically, the delimiter used to segregate the words is a space character, especially for English language. The tokens could be individual words punctuations, or part of short forms of the words. For instance, don’t could be broken down into do and n apostrophe t.

Different packages such as Natural Language Toolkit that is NLTK, SpaCy etcetera might perform tokenization differently and hence the tokens generated from the same document could be different, but majority of the tokens would be similar. Any language would generally have a set of rules and if we have to write or speak correctly, we have to adhere to these rules and English language is no exception.

However, while adhering to these rules and communicating we might also end up using many words, which could increase the dimension and thereby increasing the computational complexity and processing time. To reduce the dimension and still not impact the understanding of the sentence, we could perform certain text pre-processing steps, such as stop words removal, stemming, lemmatization, etcetera.

Once we get the tokens after tokenization, we end up with lots of individual words some of which provide semantic information and some of them exists for syntactical reasons that is, right grammatical construction. For instance in English articles such as a, an, the prepositions such as, of, on etcetera joining words or conjunctions such as and, but etcetera provides grammatical structure, but without these words we might still be able to extract the information from the sentence, such words can be called Stop words.

So, to reduce the number of words or dimension we could create a custom list of stop words or avail it from several packages such as NLTK, SpaCy etcetera and remove them from the list of tokens. However, with modern techniques and availability of computational resources we could use all the words instead of discarding the stop words. Practitioners generally take a decision of whether to remove the stop words or not based on the type of application.

For instance, in topic modeling application, which is an unsupervised learning application in which we identify several topics within a large Scopus. We might want to focus on words which clearly distinguishes the topics and hence we could remove the stop words and just retain the important words pertaining to the topic.

Whereas, in sentiment analysis application even an exclamation sign or single quote which could indicate importance, sarcasm or something else could change the sentiment and hence we might want to retain everything to capture the minutes of the details. The idea of stemming and lemmatization technique is to use only the base form of the words.

In languages such as English a word could have multiple variants for grammatical purposes. The goal of both these techniques is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For instance, words such as cries, cry, cried, crying could be converted to the base form cry. And reduce the number of words from four to one without losing too much information as they most of the times conveys similar meaning or information.

Stemming is a crude process of severing the suffixes of the word without checking whether the word after chopping is a valid dictionary form of a word or not. Whereas, lemmatization is a more refined way of converting the words to the base form, as it checks the dictionary after the process to ensure the word is a proper vocabulary term.

For instance, for the above mentioned variants of word cry stemming might convert them to Cri and lemmatization might convert to word cry that is Cry. So, the trade-off is between speed and accuracy. Since stemming is a crude process it is fast, but it might not be accurate and since the process of lemmatization is more sophisticated than that of stemming it is relatively slow, but accurate.

Both these processes are used quite commonly even though they have their own limitations, but with availability of computational resources at lower cost, practitioners could rather skip these processes to increase the chance of improving the accuracy as keeping the word in the same form captures granular information.

Besides these pre-processing steps there are many more pre-processing steps, such as creating Bi-grams that is instead of considering New Delhi as two words, we consider it as single word. Then we have Tri-grams etcetera. Part of speech tagging, which is useful for name entity recognition application.

These additional pre-processing steps could be useful, but they also increase the computational complexity and processing time. Based on the context and the type of application it is up to the practitioner to decide whether to perform these additional steps or not.

Encoding and Word Embeddings

Now that, we have the tokens and have performed the pre-processing steps, we need to encode or numericalize the output tokens into numbers. There are various ways of Encoding the token; including techniques such as One-Hot Encoding, Count Vectorizer and TF-IDF Vectorizer which falls under the bag of words model.

In this model, we use each token as features and find a measure to capture the weight of the word in each document of the corpus, that is the whole set of documents. However, these techniques have some limitations, which is majorly addressed by using word embeddings. But before that, let us get a quick overview of these techniques.

While creating a Bag of Word model, we create a dictionary of all the unique words in the corpus and then we create a Document Term Matrix, where the term is token or the word represented as the columns of the matrix and document is represented as the rows of the matrix. In one hot encoding technique the cells of the matrix are filled with 1 or 0 based on the presence of the word in the document.

It is similar, to how we handle categorical variables in a structured data set, that is we create dummy variables for every unique value of the categorical column and fill the value of 1, against the column represented by the value. The issue with one hot encoding is that the weight of the word in a document is one, irrespective of the frequency of occurrence. So, if a document has a word repeated more than once, the weight of that word would still be one and would not be increased even though the count is high.

The Count Vectorizer, addresses the issue by filling the cells of the document term matrix with the frequency of the occurrence of the word. So, besides accounting for the presence of the word in the document, it also accounts for the frequency of the word in the document; however, the common words could have higher frequency and the magnitude of those words could overshadow the significance of infrequent, but important words.

The idea behind TF-IDF, that is Term Frequency Inverse Document Frequency is to give higher weightage to a term or token based on the frequency of the word in a document and to normalize its weightage based on the frequency of its occurrence across the corpus. So, common words which appear in all the documents would have lower weightage.

We observe that each technique tried to overcome one of the shortcomings of the previous one, but these vectorizers inherently have few limitations which hinders the performance of any model built on these vectorizer. When the dictionary or vocabulary, that is the unique number of words or tokens is very high which is quite common, the matrix becomes high dimensional.

And since, the documents generally have fewer words when compared to the number of words in the vocabulary the matrix would be sparse, that is not many cells in the matrix would be filled with values. For instance, if we have a vocabulary of 50,000 words and if one of the documents has just 5 words, then the whole row would just have 5 values and the remaining value would be null or NA that is not applicable values.

Most of the documents would just have few words and we would end up with a severe sparsity issue though, most of the numerical packages such as NumPy etcetera can store these sparse matrices efficiently, but the processing becomes computationally expensive and time consuming. In languages such as English, a word could be used in multiple contexts and could have different meanings.

For instance, the word Apple is a company and of fruit as well and to understand its usage we need to know the context as well. The issue with these vectorizers is that the words are treated independently and the values neither capture the context nor the semantic meaning of a word and nor the relationship between the words. So, even though words such as Banana and Pineapple are related as fruits, the relationship is not captured by these vectorizers.

Properties and Visualization of Word Embeddings

Word embedding is a technique in natural language processing (NLP) that converts words into dense numerical vectors, capturing their semantic meanings and contextual relationships. Unlike traditional methods that use sparse representations, word embeddings provide a more compact and informative representation of words. This approach enables NLP models to understand and interpret language more effectively, as it incorporates nuances of word meanings and their usage in different contexts. By leveraging word embeddings, models can perform complex tasks such as measuring word similarity, identifying relationships between words, and enhancing context-aware operations, leading to improved language understanding and application.

To address the issues, we mentioned earlier, the encoding or embedding matrix should be dense and the weights of the words should capture the context, semantic meaning and the relationship between the words. So, that means, word should be represented as numbers that express the underlying factors or features which accounts for the association between the words as well. For instance, the table shows 6 words and their embeddings, which addresses the issues.

The matrix is not sparse, but dense as all the cells are filled up. Let us understand the significance of the values of the matrix and how it addresses the other issues before we understand how are they calculated. We observe in the first row, that the magnitude of the words Car, Bike, Mercedes-Benz, Harley Davidson are similar and high and because the common attribute among these words is vehicle, we can consider it to be the latent factor captured by the first row.

For the second row, we observe that the value is high for only Mercedes-Benz and Harley Davidson and not for other words and hence we could consider the underlying factor as Luxury, as it is the most differentiating common attribute. Similarly, based on the values in the third row, we could infer the latent factor to be Fruit and for the fourth row, we could infer it as Company, because Orange and Mango are names of companies as well.

We could have factors such as Network that is whether it is a network company, a capital that is capital of a country etcetera and many more such factors. However, these 6 words might not have any relationship with these features, but there could be other words in the vocabulary which could be related through these factors. The columns of the table show the weights of the words across all the dimensions or axis.

So, if the words are similar the values would be similar as well, that is points would be close to each other in the dimensional space. For instance, from the table, one can infer that Car and Bike are similar because, the values are similar and based on the values the relationship looks stronger than a relationship between words like Car and Orange. This helps us to find the similarity between the words.

Generally, the words which are closely related such as the word Orange and Mango would have similar values across most of the factors and might vary slightly for few factors. For instance, Mango could be also considered as clothing store, which could have a higher representation than Orange for Luxury Factor. Based on the distance between the columnar value, we could infer the underlying association between the words and use it to find related words.

For instance, if we consider the distance measured using distance measures such as Euclidean, Manhattan etcetera between the words Car and Bike and between Mercedes-Benz and Harley Davidson, we find the values to be very similar, which indicates the association that is the luxury brand of the respective vehicle type between these four words is strong and can be represented as Car is to Bike is Mercedes-Benz to Harley Davidson.

In the above table, we have shown only 6 factors; that is the 6 rows in the table, but the number of factors is a hyper parameter that is, we would need to mention it and generally we assign values such as 50, 100, 300, 500 etcetera based on how many factors we would want to capture.

The more the better, but it also increases the computation. We mentioned that the rows of the table indicate the latent factors such as Vehicle, Luxury, Company, etcetera. However, these were shown just for our understanding. As in real world scenarios, it is not straightforward to explain these factors and describe them.

It is very unlikely we would have a single row distinctly capture only one factor, as the vocabulary that is the columns is huge and with just few hundred dimensions that is a rows, factors would get overlapped and hence making it difficult to identify the underlying factors very clearly.

Nevertheless, we will be able to capture the relationship between the words; that is we could state that the association between word like Car and Bike is stronger than the words Car and Orange or Car and Mango. So, if we have to plot these words in two dimension and if the distance between these word vectors represent the association.

It could have been as shown; where words like Car, Bike, Mercedes-Benz, Harley Davidson would be relatively near to each other and words such as Orange and Mango would be relatively far away from these words were near to each other. Now, we understand the values in the matrix and its significance, but the big question is how do we get those numbers? Let us see how to learn these word embeddings.

Learning Word Embeddings

Learning word embeddings involves using algorithms like Word2Vec and GloVe to train models that generate dense vector representations of words, capturing their semantic and contextual relationships. These embeddings are created by analyzing large corpora of text data, which allows the model to understand word meanings and their usage in various contexts. The resulting embeddings offer a rich, nuanced representation of words, significantly improving performance on diverse NLP tasks such as word similarity, context understanding, and language generation. By leveraging these embeddings, NLP models can achieve more accurate and meaningful interpretations of language.

For the word embeddings to capture several contexts, semantic meaning and the relationship between the words the training data used to learn the embeddings should have multiple instances of the word in those many contexts. And to get good embedding values we would need a huge corpus. However, to get a representation of a word using the documents in the corpus in an unsupervised setting was challenging, and hence this problem was converted to a supervised learning setting.

Using neural network for language modeling, that is using the words from the previous time sequences that is \(W_1, W_2, W_3 \dots W_t\) to predict the next word \(W_{t+1}\) was a simple way of learning word embeddings. Let us consider an example sentence as: She is a great tennis player.

Using neural network shown in the figure we could build the language model, where we would want to predict the word player using the previous words, “She is a great tennis”. For that we shall represent each of the 5 words using one hot encoding based on the vocabulary size. Assuming the vocabulary has 1000 words, the first word would be 1000 dimensional vector.

We shall do the same thing for other words as well and that would constituent the input layer. The size of the embedding would be defined by setting the number of neurons in the hidden layer of the neural network. For instance, if we want the embedding to be represented by 500 numbers, we would set 500 neurons in the hidden layer. This hidden layer would be connected to the output layer with neurons equivalent to the vocabulary size that is 1000 in this case.

And a softmax activation is applied on the output value to get the probability of each word. And the word with maximum value is considered the next predicted word. Since different sentences would have different length, that is some could be very long and some very short sentences. It is common to fix a window size that is context window at the input layer. Let us say that we fix the window size to 3, then in the above case the model would predict the word based on the last 3 words, that is “A great tennis”.

This ensures that the model is independent of the length of the sentence and would always consider last 3 words before the word to be predicted. We could vary the window size as it is a hyper parameter. In language modeling the goal is to predict the next word using the previous words, but while learning what embeddings we can set bi-directional contexts that is to use not only the words before the predicted word, but also the words after that.

Let us consider another example sentence. She is a great tennis player and has won many awards. We could set the context to be 3 words to the left of the word player, that is “a great tennis” and 3 words to the right that is “and has won”. This model which predicts the target word, which is in the middle is commonly used to learn word embeddings.

So, if we consider a simple model, which predicts the next word based on the previous word the model architecture would look as shown. In this model the input layer connects the one hot encoded input to the hidden layer and the weights which connect the hidden layer to the output layer are the word embeddings of the word. Word2Vec one of the popular word embeddings model, uses the neural network model with different context. Let us discuss that now.

Word2Vec Model

Word2Vec is an NLP algorithm that learns word embeddings by training a neural network on extensive text datasets. It employs either the skip-gram or Continuous Bag of Words (CBOW) methods to predict words based on their surrounding context. Through this process, Word2Vec generates dense vector representations of words that encapsulate their semantic relationships and contextual meanings. These embeddings are typically represented in just a few dozen dimensions, enabling efficient and effective handling of language tasks such as measuring word similarity and performing various language processing applications. This compact representation facilitates improved language understanding and application.

Word2Vec Model has two different variants. Continuous Bag-of-Words that is CBOW and Skip-Gram. Both of them use single layer neural network, but differ in input and output layer. In CBOW architecture the input layer is the context, that is the surrounding words that is both to the left and the right of the target word and is used to predict the target that is the middle word as shown here.

In Skip-Gram Architecture, the input layer is the middle word and is used to predict the context that is the words to the left and the right of the target word as shown here. It is named Skip-Gram because some of the terms or the words are skipped from the context. We mentioned that when we consider the context Windows Size: 3, we take three words to the left and three words to the right of the target word, but in Skip-Gram model besides mentioning the window size parameter, we also mention another parameter which defines how many words to pick at random from that window.

So, if the value is 3 then we select only two random words from the context. For instance, in the sentence mentioned earlier for the target word player with context Window Size: 3 and the number of Skips: 2, the context would be “a, great, tennis, and, has, won”, but we would just select two of these 6 terms randomly. So, it could be “player -> has” and “player -> tennis” or any other combination.

In the figure we see that the word embeddings are represented by the weights connecting between the hidden and output layer. If we have 500 neurons in the hidden layer and 1000 neurons that is if the vocabulary is 1000 in the output layer we have to learn around 0.5 million weights, which might not be too huge but generally in any practical scenario we deal with bigger vocabularies. And if we even consider 10000 words in our vocabulary, then we have to learn a whopping 5 million weights.

Besides that we know that for embeddings to capture several context we would need a pretty huge corpus. So, training these many weights for a huge corpus and applying softmax on 10000 weights is computationally very expensive and sometimes infeasible. Mikolov and others in their paper titled ‘Distributed Representation of Words and Phrases and their Compositionality’, discussed how this issue could be addressed using negative sampling technique.

Generally, in neural networks all the weights are updated, but in negative sampling only few of the weights are updated. For instance in the above sentence if a target word is player and the context word is tennis, while training without negative sampling the output would be 1 for tennis and 0 for all the other words of the vocabulary.

But a negative sampling we shall use an hyper parameter k random negative words would be chosen whose output value would be 0 and the weights for these k words and the positive word tennis would only be updated. The negative words are randomly sampled from the words outside the context and the weightage of a word was given based on the frequency proportion with a slight modification that is by raising it to the power of 3 by 4.

For instance if we consider k to be 3, then we could have the following positive and negative samples. That is “Tennis, Player -> 1” and the 3 random words would be “Tennis, hello”; “Tennis, piece” and “Tennis, few” and the values would be represented as 0.

So, now instead of updating all 5 million weights, we just have to update the 2000 weights that is 4 into 500. And this drastically reduce the computation time and resources needed to run the network model. The authors ran many experiments and suggests the value of k to be in the range of 5 to 20 for small training data sets. While for large data sets the k could be in the range of 2 to 5. Now, that we know how the embeddings are learnt let us understand its properties.

Word Embeddings Properties and Visualization

Word embeddings in NLP exhibit several important properties: capturing semantic relationships, enabling compositionality, managing subword information, maintaining compactness, and adapting to context. These properties allow embeddings to effectively represent word meanings, construct phrase and sentence representations, handle out-of-vocabulary words, and reduce dimensionality. By incorporating these aspects, word embeddings enhance various language processing tasks, leading to improved understanding and performance in NLP applications. We will explore these characteristics in more detail to understand their impact on language models and their effectiveness in handling complex linguistic tasks.

Based on the neural network models or Word2Vec architecture if two different words have similar meaning and generally used in similar context, that is if they are associated to each other through a common factor, then the model outputs similar results which means the red embeddings are also similar. For instance, let us consider the below two statements - She is a great tennis player. And another sentence as - She is a great badminton player.

So, in the model if you are trying to predict the word player and if we consider the previous words as the context, then we could expect the embeddings of the word tennis and badminton to be similar as they are related through a common factor sports. Similarly, from a large corpus when we capture multiple contexts using different sentences, we could expect the word embeddings of similar words to be similar.

As mentioned earlier, to understand the meaning of the embedding values we would need to see how the values are in relation with other words as it is difficult to infer just based on the values of any word embedding vector independently. To measure the relationship we generally use the Distance Metrics, such as Euclidean Distance, Cosine similarity etcetera. One of the most remarkable properties of word embeddings is that it can help capture analogies.

So, if we have two words which are similar based on a factor then we can get another pair of words with the same factor based on the distance metric. So, in the embedding table shown earlier if we consider the word Car and Bike and for simplistic reason if we take the distance by just taking the absolute value of the difference of the two vectors then the output be approximately 1.9 followed by five 0’s.

And similarly for Mercedes-Benz and Harley Davidson, we would approximately get values around 1.9 followed with five 0’s. So, using the relationship between Car and Bike, we can get an analogous pair of Mercedes-Benz and Harley Davidson, which are majorly a luxury brand company of Cars and Bikes respectively. Typically, the embeddings are represented in 50 or higher dimensions, which makes it difficult to visualize the embeddings.

However, we could use the dimension reduction technique and reduce the dimension to 2 or 3 and plot it to check, whether we can observe any relationship between the words.

Word embeddings can be visualized in a reduced-dimensional space to provide insights into word relationships. This technique enables the observation of clusters of semantically similar words and the exploration of their connections in a visually interpretable format. By projecting high-dimensional embeddings into a lower-dimensional space, patterns and relationships between words become more apparent, facilitating a clearer understanding of their semantic similarities and differences. Such visualizations help in analyzing and interpreting complex word associations and the overall structure of the word embeddings.

Just like Word2Vec, Pennington and others created GloVe that is, Global Vectors for Word Representation to obtain the Word Embeddings using a huge corpus. The model is trained on multiple data sets including Wikipedia, Twitter and Common Crawl on billions of tokens and the embeddings are represented in different dimension size ranging from 50 to 300.

Since these representations are learnt on vary data sets each word would have seen many different contexts and the values would have encapsulated them quite well. The authors, have made the data code, scripts and above all the embeddings available to everyone. Let us download the pre-trained word vectors and visualize them.

We shall use the “glove.6B.zip” file available in the following website and just consider the 50-dimension representation. If we read the file, each line shows the words and its corresponding 50-dimension numbers. For instance, the word ‘of’ is represented as shown. As mentioned earlier, it is difficult to understand these numbers independently, as we do not know the factors it represents.

Let us see when we visualize the embeddings, whether we can see any relationships. But, as mentioned earlier, since these word embeddings are represented in 50-dimension, we need to reduce the dimension so that we can plot it. We shall use the dimension reduction technique like t-SNE that is, t-Distributed Stochastic Neighbor embedding to reduce the dimensions to 2 and plot around 500 words on those 2-dimensions as shown.

At the top part of the plot, we can see the words such as June, March are together and the words such as Central, City, Area etcetera are close to each other. On the right hand side, the word such as University, School are together and on the bottom side we observe cluster of Numbers and Symbols and also see words like Israel and Peace very near. Not sure how many would agree to that, but mostly we can see the related or similar words are close and the distance between them is very less.

The remarkable thing is even though we have drastically reduced the dimensions from 50 to 2, we still see the embeddings are able to retain the similarity to quite a great extent. This shows the word embeddings captures the underlying representations quite well and implicitly learns the relationship between the words, even though we did not provide any information upfront about these words or its relationships.

We understand that the numbers in the embeddings have lots of information and value. Now, let us learn its internal representation and understand, how it is stored and how to access this matrix for analysis.

The Embedding Matrix

In NLP, the embedding matrix is a crucial component that maps words to their respective vector representations. This matrix enables models to access and leverage the learned word embeddings during both training and inference phases. The size of the embedding matrix is determined by two factors: the vocabulary size, which represents the number of unique words, and the dimensionality of the embeddings, which indicates the number of features in each vector. By organizing word vectors in this matrix, models can efficiently use these embeddings to perform various language processing tasks and improve their overall performance.

The Embeddings are represented in a matrix with the number of rows equal to the vocabulary size and the number of columns equal to the dimension of the embeddings we define. The matrix shows the words, in the vocabulary along the rows and values along the columns as shown. To understand how this is used, let us consider a vocabulary of 10,000 words and an input of just one word.

The dimension of the input word represented in One hot encoding would be 10,000 into 1. And, all the values would be 0 except the index or the location representing that word; where the value would be 1 and in this case the first value. So, when we multiply the input vector with the embedding matrix, we would get the embedding vector of the word of size 300 into 1 that is assuming we are representing the embedding in 300 dimensions.

When you multiply, all the values would be 0 except for the row index which has a value as 1 in the input matrix. So, in the case of word ‘a’ the output would be the first row of the embedding matrix. Similarly, if the word was ‘great’ which is at the 524th location or index of the input matrix, then all the multiplication term for other locations other than 524th location would be 0 and the output would just be the 524th row of the embedding matrix.

We mentioned above that we do matrix multiplication of the embedding matrix with the input vector and we do this for every cell of the input vector even though all of them are 0s except one. This way of multiplication is not efficient, as we know that we will always end up with lots of 0s and hence in practice we just implement a simple lookup of the rows of the embedding matrix.

For instance, for the word ‘a’, since the value is 1 in the 1st cell of the input vector, we just look up and fetch the first row of the embedding matrix to get the embedding vector for the word ‘a’. Similarly, for the 524th word ‘great’ we just look up and fetch the 524th row of the embedding matrix. This process is highly efficient as we are just searching and fetching instead of doing matrix multiplication.

And since, indices are matched between the input and embedding matrix the search is also fast. Keras’s Embedding Layer also just maps indices to the vectors. It is common to use this layer as the first layer after the input layer for many NLP related applications including Text Classification, Sentiment Analysis, Machine Translation, Name Entity Recognition, Text Summarization etcetera.

In the field of computer vision, pre trained weights of state of the art convolution neural network architectures are used for classification related tasks on similar or different data sets. Similarly, we can leverage the pre trained weights of the words that is, the GloVe model or Word2Vec etcetera in our model and use it for NLP related tasks on other data sets. Let us now use the embedding layer in a case study and also perform transfer learning using these pre-trained embeddings.

Reference Papers:

• Distributed Representations of Words and Phrases and their Compositionality : Mikolov et.al

• Efficient Estimation of Word Representations in Vector Space: T. Mikolov et.al

Analyzing Sequential Data with RNNs

Sequential data consists of information organized in a specific order, where the sequence is meaningful. This type of data includes time series, text, audio, DNA, and music. Analyzing sequential data often requires techniques such as time series analysis and sequence modeling, using machine learning models like Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs).

Until now, we have been discussing about data captured at a point in time that is cross-sectional data. The data in the earlier case studies were cross-sectional data and did not have a temporal dimension. However, most often we observed data has time dimension, which is popularly known as time series data or sequential data.

One of the most common type of sequential data is Univariate Time Series Data; such as daily average temperature of a city, monthly revenue of a company and so on which we try to forecast for the next period, which could be either a day, month or quarter etcetera using the historical data.

In the case of multivariate time series data, we would forecast the variable based on multiple variables observed over time. Generally for the univariate or multivariate time series data, we consider the data for a single entity. However, there are scenarios such as IoT that is Internet of Things kind of environment, where we would have univariate and multivariate time series data for multiple entities like sensors.

Data is collected for multiple sensors at multiple time periods. Such sequential data is also called panel or longitudinal data. Besides the aforementioned time series data, we also have unstructured data such as speech, text, videos, music etcetera which are also considered sequential data as they are sequence of symbol, letters, notes, images, words etcetera. They inherently have time or position relationship which makes these data meaningful when seen in sequence.

When we deal with applications with sequential data, we could either have input or output data as sequential or both input and output data as sequential data. For instance, in speech or voice recognition or identification the input data is audio sequence, but the output is not a sequence as it is just a name or a person identifier whom we have to recognize. When we perform sentiment analysis again the input data is sequence of characters or words, but the output is not a sequence and just a category such as positive, negative or neutral.

For music applications such as music creation it is the other way around. That is we might enter a single value and could generate a sequence of musical notes. Similarly, in image captioning case from a single image we would generate a sequence of words describing the object and the scene. Besides these cases we also have scenarios where we have an input as sequence and output as sequence as well, but the sequence length could be same or different though.

For instance, in language translation such as English to French, Spanish to Portuguese etcetera the input and output are sequence of characters or words, but of different size as generally we do not map words one to one across languages. Video files are nothing but high or low frequency frame or images taken over a period of time or in other words they are nothing but just sequence of images.

Applications such as video activity recognition or object tracking where we have to recognize the activity in every frame, we would end up with both input and output as sequence data either with same number of values when we have to identify per frame or different value when the series of frame would lead to a single value. This list of scenarios is just to illustrate that sequential data is quite common and these applications which we have listed here are by no means an exhaustive list.

To handle such type of data a special type of neural network called Recurrent Neural Network or in short RNN is needed, which accounts for temporal dimension. We shall discuss about these networks in detail, how to train them and issues such as vanishing and exploding gradients and how to overcome them using variants of RNN cells that is LSTM; Long Short Term Memory and Gated Recurrent Unit that is GRU cells. Besides that we shall discuss about bi-directional RNNs which uses past and future data and lastly about stacked RNNs which are useful for complex scenarios.

Multilayer Perceptrons (MLPs) are designed to process fixed-size inputs, treating each input as an independent data point without considering any sequential or time-based relationships. Due to this limitation, MLPs cannot capture patterns that depend on the order of the data, making them unsuitable for time series analysis. In contrast, Recurrent Neural Networks (RNNs) are specifically designed to handle sequential information through their recurrent connections, making them a more suitable choice for tasks involving time series data.

We mentioned that we shall be using recurrent neural network a special type of neural network and not the standard neural network architecture that is multi layer perceptron for analyzing time series data. Now, one of the primary reasons for not using MLP is the way it is set up. For instance, let us consider a single instance of time series data and represent it as \(x^{(1)}, x^{(2)}, x^{(3)}\) and so on, where 1, 2, 3 are the time periods as shown.

We can see that the input data is independent of each other, but we know that there is a time relationship between the data points. That is the second input data point is dependent on the first input data point, which needs to be accounted for. Since this order relationship cannot be leveraged by multi layer perceptron architecture, we lose the temporal information, which is critical. In other words, these networks cannot learn from sequential data.

Besides that in multi-layer perceptron network the input and output size are expected to be fixed for every data instance, but for sequential data generally that is not the case always and we could expect it to be of variable size. Recurrent neural networks addresses these shortcomings and are preferred for sequential data. Let us now dig deeper into recurrent neural network.

Core RNN Architecture and Training

A Recurrent Neural Network (RNN) is a type of neural network designed for processing sequential data. It features loops that allow information to be retained across time steps, making it effective at capturing temporal patterns. This capability makes RNNs particularly useful for applications such as time series forecasting, speech recognition, and natural language processing. More advanced variants, like Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks, have been developed to overcome the limitations of traditional RNNs, such as difficulty in learning long-term dependencies.

Recurrent neural networks can learn from the previous time steps and hence are adept in handling sequential data. They are able to do so, because of using recurrent neurons instead of a typical neuron. In MLP architecture we use a regular neuron, which takes in an input and produces an output based on just the input as shown. However, in recurrent neuron since it accounts for time, output at time \(t\) is dependent not only on the input at time \(t\), but also on the previous output that is at time \(t-1\).

The difference between the regular neuron and recurrent neuron is that the output is looped back as shown. Unlike regular neuron which has only one weight \(W_x\) to represent its contribution, a recurrent neuron has two weights \(W_x\) and \(W_y\) to represent the contribution of the input and previous output respectively. The weights determine the weightage of current input when compared to the pass data that is if the current input has more significance then \(W_x\) would be bigger than \(W_y\).

So, the output of a neuron at a time \(t\) i.e., \(y_t\) could be represented as shown, where \(W_x\): The weights associated to the input, \(W_y\): The weight Associated to the previous output and \(b\): The bias term and \(\Phi\): The activation term which is typically ReLU in our case.

The recurrent neuron loops back the output to the next input step and this connection keeps on recurring, hence, the name recurrent. And if you flatten and arrange the network along time axis, it would unroll as shown.

We can see that the output \(y_3\) is dependent on \(x_3\) and \(y_2\), which is dependent on \(x_2\) and \(y_1\) which in turn is dependent on \(x_1\) and \(y_0\), which is the function of \(x_0\). Since this is the first input, it has no previous output value. This chain of dependencies of each y term at time t with all the previous inputs \(x_0, x_1, x_2\) and so on enables it to relate to all timestamps from the beginning that is at \(t = 0\).

Until now we have been discussing about each layer having just one recurrent neuron, but generally each layer has multiple recurrent neurons as shown. In this layer each neuron is connected to the input and the previous output and the number of units within each layer is a hyper parameter. When we unroll the network through time, it would look like the figure shown.

Also for an input vector a single neuron would produce a scalar output, but a layer with multiple neurons would produce output vector that is the output dimension is dependent on the number of neurons. For instance, if the layer has 4 neurons and if the batch size is 100 then the dimension of output matrix y(t): 100/4. The total number of weights depends on the input vector dimension and number of neurons in the layer.

So, for the layer of 4 neurons mentioned earlier if you provide an input of two features then the size of \(W_x\): Weight matrix of the input \(2 \times 4\) and the size of \(W_y\): Weight matrix of the previous step’s output would be \(4 \times 4\). So, the total number of weights: 8 + 16 + 4 = 28. The additional 4 is for the bias term that is one for each neuron.

Though it might seem that when the network is unrolled different weights are used at different time steps, but internally the weights are shared. We discussed earlier that in RNNs output at time t is derived from the previous time steps that is t minus 1, t minus 2, t minus 3 etcetera. So, that means, the recurrent neurons or cells maintain some kind of state or has a memory, which gets added or updated by the recurrent connections, which allows it to learn features from the sequential data.

Unlike a regular cell which has just input and output, RNN cell would have input, output and state \(h(t)\) which preserves memory. As illustrated in the figure, the output of a layer is a function of current input and the current state which is dependent on the state accumulated over a previous time steps. But for the first cell the output is dependent only on the input as there is no previous state at the first time step.

Different Types of RNNs

Different types of Recurrent Neural Networks (RNNs) can be categorized based on input-output cardinality:

1. One-to-One (1:1): This is a standard feedforward neural network used for non-sequential data.
2. Many-to-One (N:1): This type processes multiple inputs to produce a single output, such as in sentiment analysis.
3. One-to-Many (1-N): This setup uses a single input to generate multiple outputs, such as in image captioning.
4. Many-to-Many (N-N): This configuration handles multiple inputs and produces multiple outputs, which is common in machine translation.
5. Many-to-Many (N-M): This flexible structure allows for varying sequence lengths in both inputs and outputs, useful in applications like video analysis.

We observe in the previous figure RNNs take a sequence as inputs and also produce a sequence as outputs that is output at every time step. However, the output sequence that is time steps need not be same as that of input sequence and could vary. So, based on the cardinality that is the number of sequential step we consider in the input and output, we can define different network architectures. One such network is the sequence to sequence network with one to one cardinality.

In this type of sequence network the relationship between the number of input time steps to that of the output that is the cardinality is one to one. That is one output value is produced for every input time step as shown. Typical applications for such network would be character or word prediction after every character or word respectively. So, when we enter the character “t”, it would predict the next character such as “h” or “o” based on what the network has learned.

Another application could be forecasting the sales of a product one time step on a daily basis that is assuming time interval of the data is daily. So, if you provide last five days of sales value it would output the last four days sales value and the sales value for the sixth day. Another type of network is a sequence to vector network with many to one cardinality.

In this type of sequence network the relationship between the number of input time steps to that of the output that is the cardinality is many to one. That is one output value at the last time step is produced for the whole input sequence as shown. Typical applications for such network would be sentiment analysis where we take in sequence of characters or words and predict the sentiment that is either positive, neutral or negative.

Another application could be predicting machine failure in an IoT setup that is Internet of Things kind of environment, where we capture multiple parameters of the machine over a period of time that is a panel or a longitudinal data and predict for the machine failure. Another type of network is the vector to sequence network with one to many cardinality. In this type of sequence network, the relationship between the number of input time steps to that of the output that is the cardinality is one to many.

That is for input value provided at the first time step a sequence of output is produced as shown. Typical applications for such network would be music generation, where we provide a single value like a null value or just the genre and the output would be a music that is sequence of nodes of any category or of that particular category respectively. Another application could be image captioning where we provide a single image and the network describes the image in sequence of words.

Another type of network is the encoder-decoder model which is a sequence to sequence network with many to many cardinality. Besides that it can handle variable input and output sequence length as well. This network is divided into two sub networks that is encoder and a decoder.

Encoder is a sequence to vector network which produces a fixed length vector. Now, this is provided as an input state to the decoder which is used to produce the output sequence. This architecture is commonly used in language translation like from English to French, Spanish to Portuguese etcetera.

Since the number of words to express something in English could be more or less than the number of words in French, this kind of network is required which can handle variable input and output sequences. Alright, now that we know the different types of RNN architectures, let us see how do you train RNNs.

Training RNNs and Backpropagation Through Time (BPTT)

To train an RNN using Backpropagation Through Time (BPTT):

1. Unroll the RNN: Treat each time step as a separate layer.
2. Forward Pass: Generate predictions.
3. Calculate Loss: Compare predictions with actual values.
4. Backpropagate Error: Propagate the error through time.
5. Update Parameters: Adjust using an optimization algorithm.
6. Repeat: Continue for multiple epochs.

To prevent vanishing gradients in long sequences, use techniques like gradient clipping or advanced RNN variants like LSTM and GRU.

To train the multi-layer perceptron we use the back propagation algorithm and since recurrent neural networks have similar architecture, but account for the time access we apply a variant of this algorithm called Back-Propagation Through Time in short BPTT. BPTT is just running back propagation on an unrolled network through time steps. It performs forward propagation from left to right. So, for the first time step the input \(X_0\) is multiplied with the weight \(W_X\).

The bias term is added as well and the activation function is applied on this value to produce \(Y_0\), which is then compared with the actual output to calculate the error \(E_0\). Similarly, for the next time step input \(X_1\) is multiplied with \(W_X\) and the previous output is multiplied with the \(W_Y\) and the activation function is applied on the sum of these two values and the bias term to produce the output \(Y_1\), which is used to calculate the loss for that time step.

This way the loss terms are calculated for every time step and the cumulative loss is the loss for the entire sequence. The next step is to perform back propagation in which the gradients of the error is calculated with respect to the weights \(W_X\) and \(W_Y\). In the figure since we have unrolled the network we see different time steps, but it does not imply that the weights are different at each time step, but rather it has the same weights that is the network is rolled up and then the weights are updated.

And just like we used cumulative loss, we used cumulative gradients, which is calculated by summing up the gradients at each time step. The number of RNN units depends on the number of previous time periods we consider. So, if we consider the current output is dependent on many previous times periods then the RNN gets very deep that is it would have many units.

I have too many units, the whole network has to be unrolled very far back in time which makes it computationally expensive and might also lead to an exploding and vanishing gradients issue. We mentioned about the vanishing and exploding gradients issue earlier in the context of multi-layer perceptron architecture, especially the case when the model has too many layers.

We face the similar issue in the recurrent neural networks when the input sequences are long. For instance, if the paragraph starts with “I had been to Karnataka, one of the southern states of India” and if you continue writing about Karnataka for around 500 words without mentioning the word Karnataka and if the following sentence “I was able to learn the local language _____”.

Now, to predict the word Kannada, the model should have learned the long term dependency on the word Karnataka, which was the fifth word of the paragraph. So, we see that there could be dependencies on previous 100 or even 500 time periods that also means our model needs to have those many units. Unfortunately basic RNNs are not good at handling long sequences.

We observe that in deep multi-layer perceptron networks if the gradients are small it is difficult to propagate the gradients obtained at last layers to change the weights of the earlier layers. As by the time the gradients reach the earlier layers the magnitude might become negligible that is almost which reached to zero. So, after a point there is almost no gradient, which means the weights do not change and the learning stops.

Similar vanishing gradient phenomenon is experienced in RNNs as well. So, it becomes difficult to retain the long term dependency like the word Karnataka in our previous sentence. Shallow RNNs in which dependency is only on previous few time periods, the model seems to perform well.

When it comes to deep recurrent neural networks the performance deteriorates and as the gradients are quite volatile and because of which the learning also varies across the units, thereby making the network unstable and inconsistent. Just like vanishing gradient issue we could also face exploding gradient issue in which the gradients explode.

So, with units the gradient values increase and might increase to a very large value that instead of converging the model diverges. However, this issue is easier to observe while computation as a debugger would show values such as NaNs that is Not a Number; an indication of a numerical overflow. To address this issue one of the solutions which was put forth was to use truncated BPTT.

One of the primary concerns for the BPTT for a long term sequence is that running it per parameter update is computationally expensive and hence running it for multiple epochs is also not feasible. One of the solutions to deal with this problem could be to break the sequences into subsequences, which makes it computationally feasible, but the temporal dependency is reduced to the subsequence length. Truncated BPTT kind of follows this method.

BPTT processes the sequence one time step at a time and every \(k_1\) time steps that is forward propagation steps it runs BPTT for \(k_2\) time steps that is back propagation steps. So, a parameter update can be cheap if \(k_2\) is small. However, \(k_2\) should be big enough to capture the temporal dependency, but should not be too big that it causes vanishing or exploding gradient issue. So, balancing is quite critical.

Though relatively the computation is cheaper, but limiting unrolling to lesser number of time steps limits the model’s performance especially for those data which has long term patterns. So, we needed a technique which could have long-term memory and which is computationally feasible as well.

Types of RNNs: Pre-Read

Here’s a brief overview of different types of Recurrent Neural Networks (RNNs):

• Long Short-Term Memory (LSTM): LSTMs are a type of RNN designed to remember information for long periods. They use special units called memory cells that can maintain information in memory for long durations. LSTMs are effective for tasks like time series prediction and natural language processing.

• Gated Recurrent Unit (GRU): GRUs are similar to LSTMs but with a simpler structure. They use gating mechanisms to control the flow of information, making them faster to train and sometimes more efficient for certain tasks. GRUs are often used in similar applications as LSTMs, such as speech recognition and machine translation.

• Character Prediction: This refers to RNNs used for predicting the next character in a sequence. These models are trained on text data and can generate text one character at a time, making them useful for tasks like text generation and autocompletion.

• Stacked RNNs: Stacked RNNs consist of multiple layers of RNNs stacked on top of each other. This architecture allows the model to learn more complex patterns by capturing different levels of abstraction. They are commonly used in tasks that require deep understanding, such as language modeling and sequence-to-sequence tasks.

• Bidirectional RNNs: These RNNs process sequences in both forward and backward directions. By having access to both past and future contexts, bidirectional RNNs can better understand the entire sequence. They are particularly useful in tasks like speech recognition and text classification, where context is important.

These various types of RNNs can be combined or adapted for specific use cases, depending on the requirements of the task at hand.

(Note: The original article includes a duplication of the above Pre-Read section, which has been preserved to maintain strict verbatim integrity).

Types of RNNs: Pre-Read Here’s a brief overview of different types of Recurrent Neural Networks (RNNs):

• Long Short-Term Memory (LSTM): LSTMs are a type of RNN designed to remember information for long periods. They use special units called memory cells that can maintain information in memory for long durations. LSTMs are effective for tasks like time series prediction and natural language processing.

• Gated Recurrent Unit (GRU): GRUs are similar to LSTMs but with a simpler structure. They use gating mechanisms to control the flow of information, making them faster to train and sometimes more efficient for certain tasks. GRUs are often used in similar applications as LSTMs, such as speech recognition and machine translation.

• Character Prediction: This refers to RNNs used for predicting the next character in a sequence. These models are trained on text data and can generate text one character at a time, making them useful for tasks like text generation and autocompletion.

• Stacked RNNs: Stacked RNNs consist of multiple layers of RNNs stacked on top of each other. This architecture allows the model to learn more complex patterns by capturing different levels of abstraction. They are commonly used in tasks that require deep understanding, such as language modeling and sequence-to-sequence tasks.

• Bidirectional RNNs: These RNNs process sequences in both forward and backward directions. By having access to both past and future contexts, bidirectional RNNs can better understand the entire sequence. They are particularly useful in tasks like speech recognition and text classification, where context is important.

These various types of RNNs can be combined or adapted for specific use cases, depending on the requirements of the task at hand.

Gated Recurrent Unit (GRU)

Cho and others in 2014 proposed the GRU cell, a new variant of LSTM cell. In their paper in which they proposed the RNN Encoder Decoder model as well. The architecture of GRU cell is slightly different from LSTM cell. Unlike in LSTM, which had two different states that is long-term and short-term state, GRU merges the two states and hence has just one state \(c(t)\).

The number of neural networks is also reduced to 3 and there is a slight change in functionality of some of them as well. The architecture of the GRU cell is as shown. Unlike in LSTM, where we have two separate neural networks to control the input gate and forget gate, in GRU only one neural network is used.

The output of the update gate neural network \(z(t)\) controls the forget gate and the input gate. This value is passed on to the input gate as is and the remaining that is \([1-z(t)]\) is passed to the forget gate. So, if the value is 1 the input gate is open, and the forget gate is closed and vice versa when the value is 0.

The output \(z(t)\) would be given as per the equation as shown where \(W_{xz}\) and \(W_{cz}\) are the weight matrices of the connection between the hidden layer of the update gate neural network and input \(x_t\) and state \(C_{t-1}\) respectively. And \(b_z\) is the bias term. Just like the LSTM which has the main neural network with n number of neurons and tanh activation function, GRU also has a similar main neural network, but the inputs are slightly different.

The input \(X_t\) is the same and the second input that is the previous state term \(C_{t-1}\) is controlled by the Reset gate output and is not fed directly. The Reset gate neural network with n number of neurons takes an input as \(X_t\) and the previous state \(C_{t-1}\) and sigmoid activation function is applied on the output to produce \(r(t)\).

The Reset gate output \(r(t)\) is as given here, where \(W_{xr}\) and \(W_{cr}\) are the weight matrices of the connection between the hidden layer of the reset gate neural network and input \(X_t\) and state \(C_{t-1}\) minus 1 respectively. And \(b_r\) is the bias term. As discussed above the output of the main neural network \(g(t)\) would be represented as shown here, where \(W_{xg}\) and \(W_{cg}\) are the weight matrices of the connection between the hidden layer of the main neural network and input \(X_t\) and state \(C_{t-1}\) controlled by the reset neural network output respectively. And \(b_g\) is the bias term.

Unlike the LSTM cell which has an output gate the GRU cell has no such gate. The cell produces output \(y(t)\) and the updated state \(C(t)\) and the values need not be the same. For instance, \(y_t\) can be obtained by applying the Softmax function as is showed in the unrolling LSTM cell case. Assuming the output and the updated state as same it can be represented as shown here.

We observe that the GRU cell is smaller and less complex than the LSTM cell, but in practice both of them are still used and there is no consensus on the preference of one over the other. Alright, now that we have enough understanding of RNNs let us build a character prediction model. To understand the concept we shall build the model by implementing the RNNs from scratch instead of using the Keras API, which we shall use later though.

Stacked and Bi-Directional RNNs

Alright, we have now understood the LSTM and GRU architecture and implemented character prediction model using RNNs, but sometimes with single layer of RNNs we cannot capture complex features and for that we might have to use complex models or deeper models. Generally in Deep learning models to capture complex features, we tend to go deeper by adding layers. We use a similar strategy for RNNs as well, that is we could stack RNN layers and make RNNs deeper.

An architecture of a simple unidirectional Deep RNNs is as shown. The figure shows a 3 layer Deep RNNs. The number in the subscript represents the time period and the number in the superscript so represents the layer number. We could change few things in the architecture and it create different variants of the network based on the application by adding multi-layer perceptron layers on top of the stacked RNNs and not connecting them horizontally.

And based on capturing both left and right context, we could use bi-directional layers instead of unidirectional RNN layers. For applications such as time series forecasting language modeling use case where we predict the next character or word while training a model, we use only the past sequence of words that is words before the predicted word. Even though we have the words after the predicted word during training, which could provide better context and improve the accuracy, but we cannot use those words as during inference we will not have those future sequence of words.

However, for tasks such as Name Entity Recognition in which, we recognize words and phrases as entity and classify them as location, names organization etcetera we would need both past and future sequence of words for proper recognition. For instance, consider a following two sentences. “Honey bees are flying insects and their stings are painful”. And another sentence, “Honey Singh is a popular Indian music composer and rapper”.

For us to recognize and name the entities as insects for “Honey bees” or person name for “Honey Singh”, we would not be able to do so, by looking at just the first word Honey. But rather would need future sequence of words to know the context and name the entity properly. Here while training we are able to consider both past and future sequence of words because while predicting also we have the complete sentence.

Similarly, to achieve good performance in machine translation, you would need both the past and the future sequence of words as well. And since the complete sentence is available both while training and predicting, we can leverage them. Let us see how to change the RNN architecture to account for both past and future sequence of words.

The limitation of a Standard RNN is that, it takes inputs only from one direction that is left to right and does not consider words in future time steps. To address this limitation, we use bi-directional RNNs. Bi-directional RNN has two LSTM GRU cells instead of one, and uses the same input sequence as the input for both.

However one of them processes the input sequence from the first to the last word, that is left to right direction and other one computes the input in reverse direction that is from last to first word. This allows bi-directional RNN to get the context from the past and the future, which results in overall better learning with higher accuracy.

The architecture of bi-directional RNN is as shown here. In the forward recurrent layer to compute the last state value \(h_{t+n}\) in the forward direction we start from \(h_{t-1}\) in the forward direction and compute sequentially \(h_t\) and \(h_{t+1}\) and so on in the forward direction till the last state using the inputs at corresponding time point.

The backward recurrent layer though name backward is part of the forward propagation. The difference is that, the last word or time step value is taken as first value and so, \(h_{t+n}\) in the backward direction is calculated first and similar to the forward layer the states are calculated sequentially. And finally, the first state \(h_{t-1}\) in the backward direction is calculated.

The state in forward layer and backward layer at a particular time step is considered to produce the output for the corresponding time step. For instance \(h_{t-1}\) in backward direction and \(h_{t-1}\) in the forward direction is used to obtain \(y_{t-l}\) and so on and so forth. This way we learn from the left and the right context. As mentioned earlier this is useful for several applications, such as name entity recognition, machine translation etcetera. In fact, machine translation models used bi-directional RNNs in encoder decoder model with attention mechanism. We shall see this architecture in detail and also understand the attention mechanism, but before that let us dive deeper into encoder decoder model and we shall use machine translation as the application context.

Encoder -Decoder Model: Pre-Read

The Encoder-Decoder architecture is an RNN framework designed for sequence-to-sequence tasks. In this setup, the Encoder processes an input sequence and produces a context vector, which encapsulates the information from the input. The Decoder then uses this context vector to generate an output sequence. This architecture is commonly applied in areas such as machine translation, text summarization, and speech recognition.

(Note: The original article includes a duplication of the above Pre-Read section, which has been preserved to maintain strict verbatim integrity).

Encoder -Decoder Model: Pre-Read The Encoder-Decoder architecture is an RNN framework designed for sequence-to-sequence tasks. In this setup, the Encoder processes an input sequence and produces a context vector, which encapsulates the information from the input. The Decoder then uses this context vector to generate an output sequence. This architecture is commonly applied in areas such as machine translation, text summarization, and speech recognition.

The encoder decoder model as the name suggests encodes the information and then decodes it. The concept is very similar to that of the auto encoders which encodes the input information in compressed format and the decoder decodes the same input using the compressed vector representation.

However, unlike the auto encoders in which we have same input and output in the encoder decoder model for machine translation we have separate input and output that is the encoder part of the model encodes the information of the text in the source language which is used by the decoder part of the architecture to decode it in the target language.

The encoder model is a RNN model typically an LSTM as it can learn long term dependency between input and output sequence and the input is the English sentence as shown. But unlike any typical sequence model we do not produce an output as we are not making any predictions, but rather just retaining the state at the last time step.

The state vector represents the information of the whole input English sentence. The decoder model which is also a RNN model typically LSTM as well is the second part of the architecture which decodes the encoded information stored in the state vector to other language. In the figure, we observe that the decoder model takes in the encoder’s final state as one input and produces an output at every time step in the target language.

However, we also know in RNN architecture besides the state we also need to provide input at every time step. So, the question is what should we provide as input. One solution is to use a starting token such as <Start> in angular brackets as the input for the first time step and for the next time steps we could use the predicted output of the previous time step as the input at each time step as shown in the figure.

Since the output sequence length could vary in machine translation how do we decide on the number of time steps in the decoder model? To address this we fix a certain number of steps generally by taking the average length of the documents and introduce a token <End> in the angular brackets to indicate the end of a sentence.

While training the model if the words are beyond the length of the window, it could be truncated and the last unit would have the ending token and if the words are less than the window length then after the ending token padding is performed. This strategy makes sense while training the model, but while predicting the issue is that we do not have the entire sentence up front.

So, that means, we would have to predict word by word. So, during inference we keep on predicting the next word until we encounter the ending token or we reach the end of the window length. We mentioned that while training in the decoder model we use the previous predated output as input for the next steps.

But the issue with this approach is that if the prediction is incorrect in the first time step itself then the future inputs would also be erroneous. This could have a cascading effect and prove detrimental to the performance of the model. A better and effective method is to use the translated language text itself as the input shifting it by one time step and providing the starting token at the first time step.

So, that means, we provide the first word of the translated language text as input for the second time step instead of the previous predicted output at each time step as shown. This ensures that even if the prediction is wrong correction happens at that time step itself. An incorrect translation does not get propagated to the next time steps.

This strategy is called Teacher Forcing as the correct word acts as a teacher and forces the model to correct immediately when the prediction is wrong. Alright, now that we understand the encoder decoder architecture and its usage for machine translation, let us now implement it using Keras API and execute it in Google Colab.

Machine Translation Implementation

Before we execute the code in Google Colab, upload Machine_Translation.ipynb in the Google Drive of your account. Once you have done that, open the file in Google Colab. After you are connected, first we shall change the Runtime. For that go to Runtime and click on Change runtime type and select the Hardware accelerator to GPU and SAVE it.

We shall be using English to Spanish translation data available at this site. Let us import the necessary packages including the Keras’s text pre-processing functions, utility functions, layers etcetera. We shall then configure some of the hyper parameters including BATCH_SIZE for training; Number of EPOCHS; LATENT_DIM, which represents the number of neurons within the RNN models.

NUM_SAMPLES is the number of samples used to train the model; MAX_SEQUENCE_LENGTH sets the maximum length of sentences in either of the languages; MAX_NUM_WORDS is the maximum number of words in the dictionary and EMBEDDING_DIM is the size of the embedding. Once you have configured these hyper parameters, let us load the data.

I have kept the files in Google Drive and I would recommend the same as the files could be big. Then we shall mount the Google Drive folder, using the following code. It will ask for authorization for the first time and once we key in the code, it will provide the access to the main folder as /content/drive.

The input data that is, spa-eng.zip file is unzipped locally to the virtual machine. And, along with that, we shall unzip the pre-trained embeddings for the English words as well. Once the files are uncompressed, we see the two files in the File section. spa.txt is the input data and glove.6B.50d.text has the pre-trained glove embedding.

Quick glance of the first 5 lines of the ‘spa.txt’ file shows that, each line has English and Spanish words; separated by a tab, which is followed by the attribution information. So, the first word is Go and Ve, in English and Spanish respectively. We should initialize the list input_texts, target_texts and target_texts_input, which will hold the data and which will be used for the RNN structure of the encoder and decoder model.

The following code processes the file to remove the attribution data and append the data into the corresponding list defined earlier. input_text list holds the English sentences fed to the encoder model. target_text holds the translated that is English sentences with the end token at the <end> to indicate the end of sentences. This list could be used as a target sentence while training.

target_text_input list holds the translated sentence with the starting token appended in the beginning of the sentence. This list is provided as input to the decoder, as we will be applying Teacher Forcing Method. For simplicities sake and to ensure that we can execute the model quickly, we have just taken 10000 samples out of the file.

After the data is processed, we shall Tokenize the text and convert text_to_sequences. For the sample data, we have around 2337 unique English tokens and the maximum length of the input sequence is just 5. Similarly, we shall tokenize and convert text to sequence, the target_text and target_text_inputs list as well.

However, while tokenizing we need to ensure the ‘filters’ do not remove the punctuation as we have starting an ending token with angular brackets and hence we pass on the filter parameter with an empty string value. For the sample data, we have around 6316 unique Spanish tokens and the max_length_target sequence is 9.

Since we have variable length sequences in the source and target sentences, we padded up with 0’s if the length is less than the configured length which is set by max_len_input variable and max_len_target variable respectively. As mentioned earlier, for the source sentence, we shall use the pre-trained embeddings. We have already uncompressed the glove pre-trained embeddings of vector size of 50 and we shall use it to represent the words in English.

We shall create an embedding matrix and assign the pre-trained embedding vector for the words which appear both in the input data and glove embeddings. If the input data has words which are not in the embeddings, then it would be assigned a vector of zeros. We shall instantiate an embedding_layer by passing the vocabulary size, embedding dimension, the pre-trained weights and the input_length of the sentence.

Since Keras does not support sparse categorical cross entropy for sequences we shall use the categorical cross entropy. But then, while creating the target, we would need to create the one hot encoded version of the decoder targets. Hence, we create decoder_targets_one_hot matrix of the dimension = len ( input data), max_len_target and num_words in the target language dictionary.

And then assign the value as 1, based on the corresponding index. Now, that we have the data prepared in the format, we require we can build the model. First, we shall build the Encoder model. For that, we shall use the LSTM model. It is pretty straightforward as we just have the Input layer with embedding_layer which is passed to the LSTM layer.

We have the return_state=True, as we need that state to pass on to the decoder model. The model would give us the encoder outputs, h and c are the two states of the LSTM models. We shall store the two states in the encode_states variable. Similarly, we can build the Decoder model as well; that is we have the Input layer with the embedding layer which is passed to the LSTM layer.

However, we are not using the pre-trained embeddings and in the LSTM model we shall set return_sequence=True so that we can get a value at every time step. Besides that, we shall also pass the encoder_state that is state from the encoder model as the initial state to the decoder model.

Finally, we add the dense layer with number of neurons equal to the vocabulary size of the target language and activation function as softmax. The model summary shows the model architecture, layers and number of parameters, which needs to be trained. Once the model is compiled and we fit the model to the training data and run it for few epochs.

The Plot Diagnostics, that is the loss and the accuracy plots indicate overfitting; which is understandably so, as we just have taken a small sample of data. Typically, once the model is built, we just have to call the predict function of the model and pass on the new data, but in this case, it is slightly a different scenario.

While training we have the translated sentence as input for the decoder, which is not the case while predicting. So, for the decoder, we have to predict the word one at a time and take that as the input for the next step. For prediction, first we need to create the encoder model as shown. From the encoder model we get the encoder_state which will be the initial decoder hidden state.

For the decoder we create an Input layer which accepts single word as input and apply the embedding layer. decoder_lstm is the LSTM model to which you pass the input and the initial state. decoder_output is the output of the model to which the dense layer is applied and the output is reassigned to decoder_outputs.

To the model API you pass the input decoder_inputs_single and decoder_state_inputs and we get the output in decoder_outputs and the states in decode_states. idx2word_eng and idx2word_trans dictionaries map back the index to the real words. And the function decode_sequence runs a for loop in which you predict the word in the translated language and continue it until we get the <end> token.

The decode_seq function runs a for loop in which we predict the word in the translated language and continue it until we get the <end> token or reach the maximum target length. For the states to start with, we use encoder hidden state obtained by the predict function of the encoder model on the English sentence. And, after that, the states is updated for every time step within the for loop.

Finally, to test the model, we call the decode_sequence function by passing some English text or sentences. While we input couple of sentences, we might not see great results. Well, this was anticipated as we saw that the model was overfitting and besides, we used a very small sample set. We could expect a better conversion once we use the complete data set. Nonetheless, the researchers figured out, that for short sentences this model was working out great; but for translating long sentences its performance was not great.

We shall discuss later about the changes that is attention mechanism, transformer models etcetera which improved the machine translation performance quite considerably. But before that, let us understand how the translation performance can be refined using beam search technique and also understand BLEU score that is Bilingual Evaluation Understudy which is one of the most common metrics used to evaluate machine translation systems.