Mateus Fernandes and Axel Disasi struck in the 93rd and 96th minutes as West Ham levelled the score at 2-2 at London Stadium and forced extra time.
No more goals were scored, before Leeds won the penalty shootout 4-2.
In the shootout, West Ham debutant Finlay Herrick saved a penalty from Joel Piroe but Leeds eventually prevailed with Pascal Struijk scoring the winning penalty.
"At least I'm old enough that I was already born when there was the last semi-final for Leeds United in the FA Cup in the '80s," Leeds manager Daniel Farke said.
"It was a crazy game and it has cost lots of energy, but in the FA Cup quarter-final both teams are able to fight for quite another piece of history for the club.
"To reach, for the first time since a while, the semi-final, the game should be like this with ups and downs and setbacks.
"We could have made our life a bit easier after an excellent start to the game. But West Ham is also a Premier League side who is of course giving everything and trying everything to come back into the game."
The thousands of West Ham fans who had left early were trying, and failing, to get back in when Taty Castellanos thought he had put the Hammers ahead in the opening seconds of extra time after a bad error from Leeds goalkeeper Lucas Perri, only for VAR to rule Castellanos offside.
Then Jarrod Bowen crashed a shot against the crossbar, with Pablo offside when he rolled in the rebound.
The 20-year-old Herrick came on as a replacement for Alphonse Areola, who left the field to receive treatment with five minutes of extra time remaining.
Ao Tanaka and Dominic Calvert-Lewin's penalty had previously built a 2-0 lead for Leeds in a classic FA Cup game between two relegation-threatened teams in the Premier League.
Leeds will now play Chelsea in the semi-finals in a repeat of the 1970 FA Cup final, which Chelsea won after a replay.
Manchester City and second-tier Southampton meet in the other semi-final match with games to be played on April 25-26 at Wembley.
