Actually, an EXP penalty might be a way to minimize torture if that is an issue. It should be a percentage amount so it scales with level. e.g. lose 10% of total experience or whatever.
You'll have to get to 200 followers to be eligible for that bonus xp. Only one away. Cue the 200 music.