More than anything, it's because of parking laws. I'll illustrate using two small towns.
If you'll follow me across the country, I'm going to show you the town square of Healdsburg, California, an hour and a half to the north of San Francisco. Healdsburg was founded in 1857 as a farming community on the Russian River; it was connected to San Francisco by railway in 1872. Healdsburg's traditional town layout is pretty standard for the era.
Let's zoom in on Center Street, on the east side of the town square. In the 260-foot block of Center Street facing the town square, there are no less than 11 different businesses: two wine shops, three clothing stores, a juice bar, a bakery, a bookstore, an art gallery, and a barbecue restaurant. It's a fantastic example of the idyllic American small town.
It would also be totally illegal to build this today.
Why? Parking laws.
Let me illustrate, using Kinsmoke Barbecue on the corner as an example. Kinsmoke has a capacity of 90 diners. If you wanted to build a place like that today, city law requires one parking lot space for every three restaurant seats - so you'd have to figure out a way to put in 30 parking spaces. To put in 30 parking spaces, you'd have to demolish the hat store, the bakery and the art gallery.
And the same problem applies to residential buildings. Let's go across the country to Newport, Rhode Island, home to Gilded Age mansions, where we spotlight a three-story apartment building built around 1840. There are six apartments in the building, plus stores on the first floor, with no parking spaces at all. It would be impossible to build this building today. Under Newport's city code, each residential unit is required to have two parking spaces, plus another two spaces for every 275 square feet of retail. This gives us a total of 19 parking spaces required by law.
If you followed the modern laws, both the coffee house next door and the community building two doors down would have to go. No, I'm not joking. The two adjoining lots occupy ~6500 square feet - and 19 surface parking spaces take up ~6500 square feet.
These types of parking laws are more or less universal in the United States. Unsurprisingly, when you require tons of parking, pretty soon everything ends up looking like a strip mall, or a suburban subdivision because there's no other legal way to build.
There's no technological obstacle to building towns and suburbs like they used to. These are political problems, not technological ones. They're the product of decisions we collectively made to favor the automobile over charm, over walkability, and over the environment.
Ironically, places built in the old-fashioned way are much, much more desirable than standard postwar suburbia. In greater NYC, you'll pay a premium to live in Scarsdale over Levittown; in LA, Santa Monica is more expensive than Porter Ranch; in the Bay Area, Piedmont is much more expensive than Dublin. But old habits die hard - and arguing for less parking is tantamount to career suicide if you're a suburban politician.