The Hoops travelled to the Greek capital with the third qualifying round tie evenly poised at 1-1 from the first leg, but found themselves two goals down after 50 minutes.
Although Scott Sinclair's header gave Celtic hope, AEK held firm in the second leg to progress to the playoffs at the visitors' expense.
"We've got to stop the cross for the first and the second is poor defending," Rodgers told BT Sport after the 3-2 aggregate defeat.
"We gave ourselves a chance and threw everything at it. I thought a lot of our build-up play was worth something from the game but you can't defend that softly.
"I still felt this was a game in which we had more than enough to win but you have to defend better than we did. We gave AEK a lot of problems but you have to defend properly and we didn't do that."
Celtic will now contest a Europa League playoff round tie with Latvia's Spartaks Jurmala or Suduva Marijampole of Lithuania.