Andy Murray will face brother Jamie in the men's doubles quarter-final at the Cincinnati Masters.
Andy and partner Feliciano Lopez beat Ryan Harrison and Jack Sock 2-6 6-3 10-7 to reach the last eight, after Jamie and Neal Skupski beat French pair Nicolas Mahut and Pierre-Hugues Herbert 6-3 6-3.
Rublev, the world number 70, broke Federer three times as he powered to a stunning 6-3 6-4 win in 62 minutes.
The defeat was a blow to 20-time major champion Federer's US Open preparations.
Rublev will now play compatriot Daniil Medvedev in the quarter-finals after the biggest win of his career.
World number three Federer said on www.atp.com: "He was super clean – defence, offence, serving well. Didn't give me anything. He was everywhere.
"So it was tough for me, but an excellent match by him. I was impressed."
Meanwhile, Novak Djokovic beat Pablo Carreno Busta in straight sets, 6-3 6-4. Djokovic will next play Lucas Pouille in the quarter-final.