Dani Alves and Neymar gave PSG a two-goal lead but Presnel Kimpembe's own goal and Mexer's header either side of the interval levelled the match at 2-2 and led to extra time.
France star Mbappe saw red just before the end for an awful challenge and after a perfect 10 penalties from the designated takers, Ismaila Sarr put Rennes 6-5 up and Christopher Nkunku blazed over to hand Rennes the trophy.
Alves struck a magnificent opener in the 13th minute, meeting Neymar's corner directly to the edge of the box with a first-time volley into the left corner of the net.
Tomas Koubek denied Mbappe a quickfire second but it was provided in the 22nd minute by Neymar, who ran onto Angel Di Maria's through-ball and chipped the keeper.
M'Baye Niang fired against the right-hand post at the other end but Rennes did pull a goal back before half-time when Kimpembe turned Hamari Traore's cross past his own keeper.
Neymar was booked for a foul on Ramy Bensebaini and Koubek saved well from Mbappe and Alves to keep the half-time deficit to a single goal.
And after Alphonse Areola saved from Benjamin Bourigeaud and Colin Dagba made a goal-saving challenge on the same player, Rennes got a deserved equaliser when Clement Grenier's corner was met at the near post by an unstoppable header from defender Mexer.
Koubek twice denied Neymar, who was then just unable to connect with Mbappe's cross as the match went to extra time, which began with PSG introducing Edinson Cavani in place of Julian Draxler.
Neymar drew another sharp save from the impressive Koubek and Mbappe struck a post in the first half of extra time – but the latter received a straight red card with a minute to play after going in with his studs up on Damien Da Silva.
The shoot-out went to sudden death after Neymar scored PSG's fifth penalty, but Nkunku's terrible effort cleared the crossbar by a yard and left the underdogs celebrating.