Celta entered the return match at the Balaidos with a 2-1 lead from the first leg, and a 2-2 draw in the reverse was enough to send the Vigo side through and eliminate Los Blancos in the process.
Celta boss Eduardo Berizzo rung the changes from his team's 1-0 defeat to Real Sociedad at the weekend, with John Guidetti, Iago Aspas, Theo Bongonda, Gustavo Cabral and Nemanja Radoja among those in the starting XI, but Chilean midfielder Pablo Hernandez missed out through suspension.
As for Real Madrid, an interesting selection saw Casemiro start in central defence, while Kiko Casilla, Isco, Mateo Kovacic, Danilo and Marco Asensio were all handed starts as Zinedine Zidane also made alterations from the weekend.
Real Madrid went after Celta from the first whistle as they looked to level the tie overall, but the visitors had to wait until the sixth minute to register their first attempt on target - Isco bringing a decent save from Sergio Alvarez.
Celta, who entered the match off the back of a fine run of form, were also a threat, however, and Guidetti fired over Casilla's crossbar in the ninth minute during what was an end-to-end period of the quarter-final.
Alvarez had to make a stunning save to prevent Real Madrid from taking the lead in the 27th minute after a mixture of Cristiano Ronaldo and Cabral had sent the ball flying towards the top corner, and Ronaldo then incredibly hit the post from the rebound as Celta somehow survived a very dangerous moment.
The home side had a glorious chance of their own 10 minutes before the break when a poor pass from Casemiro presented Aspas with time and space inside the Real Madrid box, but the Spaniard decided to look for Guidetti rather than the shot and Sergio Ramos was on the cover to clear his lines.
Guidetti was next to come close for Celta after breaking into the Real Madrid box, but the Sweden international could not find a route past Casilla, moments after Alvarez had kept out a header from Ramos down the other end.
Celta did make the breakthrough in the 44th minute, however, when Danilo found the back of his own net after Casilla palmed Guidetti's shot straight into the path of the right-back, whose stride took the ball into his own goal.
Chances were few and far between in the early stages of the second period, but Real Madrid, as expected, were the team pressing and Alvarez had to keep out a low Toni Kroos free kick just before the hour mark.
Los Blancos were level on the night in the 62nd minute, however, when Ronaldo, who could have had three or four on the night, found a route past Alvarez with a stunning free kick from distance.
Ramos had a brilliant chance to level the tie at 3-3 in the 66th minute when the centre-back met a dangerous free kick from Kroos, but the Spaniard's header was just wide of the post as Celta struggled to gain a foothold in the match.
Ronaldo was next to come close for Los Blancos as he stood over another free kick, but this time the Portuguese's effort bounced just wide of the far post as it remained 1-1 on the night entering the final 20 minutes of normal time.
Alvaro Morata and Lucas Vazquez were both introduced for the away side as Zidane looked for a second goal, but Celta made sure of a spot in the final four when Daniel Wass threaded one into the bottom corner in the 85th minute.
Real Madrid equalised on the night in the 90th minute when Vazquez headed home at the far post, but the visitors could not find another in the four minutes of added time as Celta did just enough to advance 4-3 on aggregate.
Celta have joined Alaves and Atletico Madrid in the semi-finals, while the lineup will be confirmed following Thursday's quarter-final second leg between Barcelona and Real Sociedad at Camp Nou.