The underdogs, who are ranked 15th in the HSBC Rugby Sevens Series, managed to claim a 14-12 victory over the third-best team in the world.
New Zealand were the first to concede as they went 7-0 down, but they managed to find their rhythm to put 12 points on the board.
Japan responded with a try to edge the scoreline, and it was a nervy finish as New Zealand tried to get back on top, but after the ball was pushed out of bounds, the full-time whistle blew.
To add salt to the wound for New Zealand, Sonny Bill Williams, who has won two World Cups with the All Blacks, limped off the pitch with an injury.
Meanwhile, Great Britain defeated Kenya 31-7 in their opening match of the tournament.