I've put a new impeller in every spring.. It's likely overkill, and the ones I remove are typically in great shape, but I figure it's cheap insurance... I put +/- 100 hours on my boat each year.
I think I read somewhere that you should check the transmission cooler for impeller pieces. If it were me, I would pull off every hose that's downstream from the raw-water pump to try and find those pieces...