One book I have says something interesting. It mentions how those of us who wanted the towers rebuilt may have been the losers of an undeclared class war.
Some people felt that rebuilding would be disrespectful somehow, or that never mind their iconic status, the towers and site sucked, and something "better" should be done.
"Build me to the heavens, and Life never stops"
"Live as if the world were as it should be, to show it what it can be"
"Prayers are fleeting and wars are forgotten, but what is built endures"
-Ambassador DeLenn, Babylon 5