This paper examines the effect of being out of work, which is in a broader category of unemployment, on the physical and mental health of younger Japanese men using panel data. A fixed effects model, widely used to control for unobserved individual heterogeneity in panel data analysis, was used for this analysis. Using the first through the fifth waves of the Japanese Life Course Panel Survey, the first wave of which was conducted with people aged 20-40 yrs in 2007, it is found that being out of work has no observable effect on self-assessed physical health. However, being out of work has a negative effect on mental health as measured by the five-item version of the Mental Health Inventory. It is difficult to clearly distinguish the direction of causality even after controlling for individual heterogeneity that is constant over time. An analysis was done with a sub-sample to mitigate a possible reverse causality. The results consistently show that being out of work has a negative effect on mental health.