In terms of the raw number of athletes taking part, it's a pretty big deal, on par with the Commonwealth Games.
Delhi 2010 ~ 6700 athletes.
Guadalajara 2011 ~ 6000.
Beijing 2008 ~ 11000.
It's true, though, that it's not much of a spectator sport, at least in terms of television.
I supported the games at the beginning because I saw it as a great way (i.e. the ONLY way) to upgrade our athletic facilities: pools, running tracks / soccer fields, velodrome, stadium etc. It was also my hope, and many others as well, that we might be able to sneak in some infrastructure improvements as well (LRT) and maybe even develop a brownfield or two.
However, up to now, more damage than good has come from the games. My support is definitely off the table at this point.