but if the show had made him admit he wasn't right or someone had properly called him on it, it would have been okay as a character point

Exactly! That's what I mean! There are characters I love who have much worse flaws than this, but I don't find them frustrating because the narrative acknowledges them as flaws. With the Doctor, the show sometimes does, but then sometimes it doesn't, and it's really annoying.