Skip to main content

LSTM Gates: The Traffic Cops

We know the LSTM has a VIP Highway (Cell State) for long-term memory, and it uses "traffic cops" (Gates) to protect it.

But how does a math equation act like a traffic cop?

The Magic of Sigmoid (The Valve)​

To understand gates, you need to understand the Sigmoid activation function. If you feed any number into a Sigmoid function, it squishes the answer to be strictly between 0 and 1.

Think of it like a valve on a water pipe:

  • If Sigmoid outputs 0, the valve is fully closed. Nothing gets through.
  • If Sigmoid outputs 1, the valve is fully open. Everything gets through.
  • If Sigmoid outputs 0.5, the valve is half-open. Only some things get through.

An LSTM uses 3 of these Sigmoid valves to control the flow of information!

1. The Forget Gate (The Trash Can)​

Before the LSTM learns anything new, it looks at its past to see if it should forget anything. Imagine you are reading a book. If the chapter ends and you start a new chapter in a new city, your brain "forgets" the old scenery to make room for the new one.

How it works: The AI looks at the new word and the short-term memory, and pushes them through a Sigmoid valve. It creates a vector of numbers between 0 and 1. It multiplies this with the VIP Highway. Any number multiplied by 0 gets erased forever!

2. The Input Gate (The Notebook)​

Now that we've taken out the trash, what new information should we write down? If you read the word "apple," the AI has to decide if "apple" is important enough to put on the VIP Highway.

How it works: Another Sigmoid valve decides which parts of the new word are important (0 to 1). A different function creates the actual new information. The AI multiplies them together and adds the result to the VIP Highway.

3. The Output Gate (The Spokesperson)​

Finally, the AI has an updated VIP Highway. But it doesn't want to just dump its entire long-term memory to the outside world! That would be overwhelming.

Instead, it needs to figure out what tiny piece of that memory is relevant right now.

How it works: A third Sigmoid valve looks at the new word and says, "Based on this word, what part of our long-term memory should we talk about?" It filters the VIP Highway and outputs the final short-term memory (Hidden State) for the next step!

By using these three simple valves (Forget, Input, Output), the LSTM became the undisputed king of AI language translation for nearly two decades!

Next Up: LSTMs are amazing, but they are heavy and slow. What if we could get the exact same performance, but with less math? Enter the GRU!