If the player tries to climb down a ladder, then perhaps Earl could just jump down? That shows that ladders aren't necessary to go down, and that there's no fall damage.
