i might be unpopular saying this, but the fact that the drawbacks don't scale really makes them feel inconsequential in the later game.
Also, because they can spawn with low tier drawbacks, for example store cost +, once you optimize for the right drawbacks, you don't even feel the impact on the game.