That's moving the goalposts. No one said anything about "relative to his wealth bracket".

And I don't think I have ever heard anyone using a term like "modest house" in proportion to someone's wealth. House-modesty is something people generally use to mean "when compared with other houses in the region".