As the body of evidence linking disparities in the health of urban residents to disparate social, economic and environmental contexts grows, efforts to delineate the pathways through which broader social and economic inequalities influence health have burgeoned. One hypothesized pathway connects economic and racial and ethnic inequalities to differentials in stress associated with social and physical environments, with subsequent implications for health. Drawing on data from Detroit, Michigan, we examined contributions of neighborhood-level characteristics (e.g., poverty rate, racial and ethnic composition, residential stability) and individual-level characteristics (e.g., age, gender) to perceived social and physical environmental stress. We found that neighborhood percent African American was positively associated with perceptions of both social and physical environmental stress; neighborhood percent poverty and percent Latino were positively associated with perceived physical environmental stress; and neighborhood residential stability was negatively associated with perceived social environmental stress. At the individual level, whites perceived higher levels of both social and physical environmental stress compared to African American residents of the same block groups, after accounting for other variables included in the models. Our findings suggest the importance of understanding and addressing contributions of neighborhood structural characteristics to perceptions of neighborhood stress. The consistency of the finding that neighborhood racial composition and individual-level race influence perceptions of both social and physical environments suggests the continuing importance of understanding the role played by structural conditions and by personal and collective histories that vary systematically by race and ethnicity within the United States.