Google did it with Picasa, Urchin, Keyhole. Now SketchUp, a 3D modeling program bought by Google last month, is free for personal use and you can download it if you have a decent PC with Windows 2000 or XP. Google promises to come up with a Mac version for SketchUp soon.
The nice thing about SketchUp is that it allows you to create 3D models with simple drag and drop operations. You don't have to read the manuals, it's that easy to use. You can view geo-referenced 3D models in Google Earth and upload them in Google 3D Warehouse. You can already see interesting buildings in Google 3D Warehouse: Rose Bowl Stadium, Taj Mahal, Empire State Building or Apple Campus (you can see it in the screenshot).