The hosts took the lead through Adam Lallana in the 10th minute, but Kelechi Iheanacho equalised for Leicester shortly after the hour mark. Daniel Amartey then struck the winner for Brendan Rodgers’ side in the 87th minute.
“It feels sore,” Potter said. “Overall, our performance – certainly the first half – was good. We were close to getting a second [goal].
“In the second half, Leicester responded and we didn’t play as well. Still, we had some opportunities, but it felt like a draw might have been the right result.
“There was no problem with the approach and intention of the team. We made a few too many errors against a team going for the Champions League. The goal at the end is a bit of a blow.
“Adam Lallana did some really nice touches, nice combinations. He helped the team, kept us going. He was a positive for us.”
Leicester went second in the Premier League with their three points, trailing leaders Manchester City by 12 points ahead of Sunday’s Manchester derby against United.
A United win would see the Red Devils leapfrog Leicester.