YouTip

In 2017, Google published the paper "Attention Is All You Need", introducing the Transformer architecture. No one expected this paper to completely change the AI field. Today's GPT, Claude, Gemini, Llama... almost all major large language models are essentially variants of the Transformer. Understanding Transformer means understanding 90% of modern LLMs. In this article, we will dive deep into every component of Transformer: self-attention, multi-head attention, positional encoding, feed-forward networks, layer normalization... not just explaining "what" but also "why". > This is one of the most technically in-depth modules. We will use code to demonstrate the core computations, ensuring you not only understand the formulas but can also implement them yourself. * * * ## Transformer Prehistory: Limitations of RNN Before Transformer, sequence tasks (like translation, text generation) primarily used RNN (Recurrent Neural Networks) and its variants LSTM and GRU. ### How RNN Works The core idea of RNN is "step-by-step processing": input sequences enter the network one word at a time, and each step's output contains information from all previous words. ## Example # ============================================ # Simplified RNN Forward Propagation Demo # ============================================ import math class SimpleRNN: """Simplified RNN implementation for demonstrating the principle""" def __init__ (self, input_size: int, hidden_size: int): """Initialize RNN parameters""" import random random.seed(42)# Set random seed for reproducibility # Input to hidden layer weights self.Wx=[[random.uniform(-0.1,0.1)for _ in range(hidden_size)] for _ in range(input_size)] # Hidden to hidden layer weights self.Wh=[[random.uniform(-0.1,0.1)for _ in range(hidden_size)] for _ in range(hidden_size)] # Bias self.b=[0.0 for _ in range(hidden_size)] def step(self, x: list, h_prev: list) ->list: """Single step RNN: input x and previous hidden state h_prev, output new hidden state""" hidden_size =len(h_prev) h_new =[0.0 for _ in range(hidden_size)] # h_new = tanh(Wx·x + Wh·h_prev + b) for i in range(hidden_size): # Compute Wx·x wx_sum =sum(x * self.Wxfor j in range(len(x))) # Compute Wh·h_prev wh_sum =sum(h_prev * self.Whfor j in range(hidden_size)) # Add bias, pass through tanh h_new=math.tanh(wx_sum + wh_sum + self.b) return h_new def forward(self, sequence: list) ->list: """Process the complete sequence""" hidden_size =len(self.b) h =[0.0 for _ in range(hidden_size)]# Initial hidden state hidden_states =[] for x in sequence: h =self.step(x, h) hidden_states.append(h) return hidden_states # Test: Demonstrate with simple vector sequence input_size =4 hidden_size =3 rnn = SimpleRNN(input_size, hidden_size) # Assume input sequence is 4 words, each word represented by 4-dimensional vector sequence =[ [1.0,0.0,0.0,0.0],# Word 1 [0.0,1.0,0.0,0.0],# Word 2 [0.0,0.0,1.0,0.0],# Word 3 [0.0,0.0,0.0,1.0],# Word 4 ] hidden_states = rnn.forward(sequence) print("Tutorial Simple RNN Demo") print("=" * 40) for i, h in enumerate(hidden_states): print(f"Step {i+1} hidden state: {[f'{v:.4f}' for v in h]}") # Output: # Tutorial Simple RNN Demo # ======================================== # Step 1 hidden state: ['0.0204', '-0.0434', '0.0556'] # Step 2 hidden state: ['-0.0061', '-0.0529', '0.0775'] # Step 3 hidden state: ['0.0641', '-0.0409', '0.0655'] # Step 4 hidden state: ['0.0143', '-0.0805', '0.0277'] RNN seems reasonable, but it has three fatal problems: ### Problem 1: Gradient Vanishing, Difficulty Remembering Long-distance Dependencies RNN has a "chain" structure, where gradients at each step need to backpropagate all the way to the first step. After multiple multiplications, gradients decay exponentially, becoming nearly 0. For example, in the sentence "I went to Paris in 2010... that was my favorite city" — "city" needs to refer to "Paris", but there are too many words in between, making it difficult for RNN to learn such long-distance dependencies. LSTM and GRU alleviated this problem but didn't completely solve it. ### Problem 2: Cannot Compute in Parallel, Slow Training RNN must wait for step t-1 to finish before computing step t. This means: * Even with 100 GPUs, can only compute one word at a time * The longer the sequence, the longer the training time * Difficult to scale to ultra-large datasets ### Problem 3: Early Position Information Easily "Overwritten" RNN's hidden state is updated step by step, and later information continuously overwrites earlier information. Important information at the beginning of a sentence may become "diluted" by the end. The emergence of Transformer solved these three problems at once. | Feature | RNN/LSTM | Transformer | | --- | --- | --- | | Computation Method | Serial, step by step | Parallel, all at once | | Long-distance Dependency | Weak, gradient vanishing | Strong, direct connection at any distance | | Position Information | Naturally ordered | Needs positional encoding | | Training Speed | Slow | Fast (parallelizable) | * * * ## Self-Attention Mechanism Self-attention is the core of Transformer. Its idea is simple: every word needs to "communicate" with all other words in the sentence to see who is important to itself, then weighted sum based on importance. First, let's look at the overall architecture diagram:  ### Intuitive Understanding of Q, K, V Self-attention uses three vectors to describe each word: * **Q (Query)** — "Who am I looking for?" — which word this word wants to attend to * **K (Key)** — "Who am I?" — the identity of this word * **V (Value)** — "What do I have?" — the actual content of this word The computation process is: 1. Each word uses Q to "match" with all words' K to get attention scores 2. Use Softmax to normalize the scores, summing to 1 3. Use the normalized scores to weighted sum all words' V The diagram makes it clearer:  ### Attention Score Computation Formula The complete formula is: Attention(Q, K, V) = softmax( Q·Kᵀ / √dₖ ) · V Let's implement it step by step in pure Python: ## Example # ============================================ # Pure Python Implementation of Self-Attention # ============================================ import math def softmax(x: list) ->list: """Compute Softmax: convert a set of numbers into a probability distribution, sum to 1""" # Subtract max value to prevent numerical overflow max_val =max(x) exp_x =[math.exp(v - max_val)for v in x] sum_exp =sum(exp_x) return[v / sum_exp for v in exp_x] def matrix_multiply(A: list, B: list) ->list: """Matrix multiplication: A is m×n, B is n×p, output is m×p""" m =len(A) n =len(B) p =len(B) result =[[0.0 for _ in range(p)]for _ in range(m)] for i in range(m): for j in range(p): for k in range(n): result += A