Really? You think the right to vote is only good if everyone in a particular demographic uses it?
Yes it is only good if you use it. It's a bit much for you to advocate taking that right away from those who do.
If you are going to take it away from someone, take it from those who are too lazy or uninterested to use it.