While certainly viewed as an indicator of health, body-building represents the act of building the body's muscles beyond the goal of simple physical fitness. Body building typically deals with the desire to maximize the size and visual appeal of muscles, over sheer strength, but can include both goals.
Historically, fitness was discussed in terms of the capacity to carry out the day's activities without undue fatigue. If you think about the word, this makes perfect sense. The term address how "fit" you are to do a day's labor.
