Social networks amplify inequalities by fundamental mechanisms of social tie formation such as homophily and triadic closure. These forces sharpen social segregation, which is reflected in fragmented social network structure. Geographical impediments such as distance and physical or administrative boundaries also reinforce social segregation. Yet, less is known about the joint relationships between social network structure, urban geography, and inequality. In this paper we analyze an online social network and find that the fragmentation of social networks is significantly higher in towns in which residential neighborhoods are divided by physical barriers such as rivers and railroads. Towns in which neighborhoods are relatively distant from the center of town and amenities are spatially concentrated are also more socially segregated. Using a two-stage model, we show that these urban geography features have significant relationships with income inequality via social network fragmentation. In other words, the geographic features of a place can compound economic inequalities via social networks.