I'm more worried about climate change than terrorism too. Terrorism can't take out entire continents in the same way climate-change will.
The fact that it will be mostly non-western third-world (and overpopulated) countries that will suffer is probably the primary reason those who aren't concerned feel as they do, but you're right, they're forgetting that that the people of those countries aren't going to just lie down and die. They'll migrate and it's going to get ugly trying to fight hordes of people whose only will is to survive.