Depends on the inference
If you are trying to infer something about the age at which they left office, then no. Some people died in office, and one resigned. Some presidents served for 4 years, others for 8 years, and one served longer (FDR).
If you are inferring something about their ages at inauguration, and you know that information, you may certainly draw appropriate inferences. For example, you might examine whether the age at inauguration appears to be increasing, or decreasing, or remaining constant.