You're right! I've never thought about this. I generally ignore most of the logical lapses in the first two books, which do target a children audience and therefore follow certain rules (i.e. the mentor always stands aside to give the kids the opportunity to prove themselves), but this is quite a big one. Maybe Dumbledore only came up with the idea around Christmas, while he was standing in front of the mirror looking longingly at the socks his reflection was holding? Or something? Before that, he merely made sure that the more competent members of his staff thought of some means of protection.