Top 6 Interview Questions on NyströmFormer

This article was published as a part of the Data Science Blogathon.

 Source: totaljobs.com

Introduction

Transformers have become a powerful tool for different natural language processing tasks. The impressive performance of the transformer is mainly attributed to its self-attention mechanism. However, training big Transformers with long sequences is impossible in most cases due to their quadratic complexity. To workaround this limitation, Nystromformer was proposed.

In this article, I have compiled a list of six imperative questions on Nyströmformer that could help you become more familiar with the topic and help you succeed in your next interview!

Interview Questions on Nyströmformer

Question1: When compared to RNNs, which key element was responsible for impressive performance gain in transformers? Explain in detail.

Answer: The main element that made it possible for transformers to work excellently was self-attention. It computes a weighted average of feature representations with the weight proportional to a similarity score between pairs of representations.

In essence, self-attention encodes other tokens’ impact/influence/dependence on each specific token of a given input sequence.

Figure 1: Attention Mechanism (Source: Kaggle)

Formally, an input sequence (X) of n tokens is linearly projected using three matrices WQ,
WK, and WV to extract feature representations Q (query), K (key), and V (value), with the dimension of query equal to the dimension of key dk = dq. The outputs Q, K, and V are computed as follows:

So, self-attention is defined as:

where softmax is a row-wise softmax normalization function; hence, each element in the softmax matrix S is dependent on all other elements in the same row.

Question 2: What are the drawbacks imposed by self-attention?

Answer: Yes. Despite being beneficial, self-attention is an efficiency bottleneck because it has a memory and time complexity of O(n2 ), where n is the length of an input sequence. This results in high memory and computational requirements for training large Transformer-based models, which has limited its application to longer sequences.

For example, with a single V100, training a BERT-large model takes four months. Hence, training big Transformers with long sequences (like n = 2048) will be expensive due to the O(n2) complexity.

Question 3: What is Nyströmformer? What’s its key approach?

Question 4: What is Nyström Method?

Question 5: How was the Nyström method adapted to approximate self-attention?

Question 6: List some key achievements of Nyströmformer.

Conclusion

This article covers some of the most imperative interview-winning questions on Nyströmformer that could be asked in data science interviews. Using these interview questions as a guide, you can not only improve your understanding of the fundamental concepts but also formulate effective answers and present them to the interviewer.

To sum it up, in this article, we explored the following:

1. The main element that made it possible for transformers to work excellently was self-attention. It computes a weighted average of feature representations with the weight proportional to a similarity score between pairs of representations.
2. Despite self-attention being beneficial, it is an efficiency bottleneck because it has a memory and time complexity of O(n2 ), where n is the length of an input sequence.
3. Nyströmformer makes use of the approximation of the BERT-small and BERT-base self-attention mechanism by adapting the Nyström approximation in a way that the self-attention complexity is reduced to O(n) both in the sense of memory and time.
4. Nystrom’s method minimizes computational costs by producing low-rank approximations.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.