This is explained by Gay-Lussac's law.
Gay-Lussac's law states that at constant volume (and mass) pressure is directly proportional to temperature.
I will try and explain it at the particle level. The temperature of a particle is a direct measure of the motion of that particle. That is to say the warmer a particle the faster it vibrates or moves around. Pressure is a measurement of the interaction of particles with other particles (usually of the particles of a container). The more that particles move around the more likely they will interact with other particles.
If you think of a balloon, the faster the air particle inside move around (temperature) the more often the particles will collide with the inside of the balloon's surface (pressure).
When we look at atmospheric pressure we are looking at the weight of the atmosphere on any particular spot. The more atmosphere above the location the greater the weight or pressure of the atmosphere on that location. On a particle level, more particles stacked up means that the particles will be more likely to interact with other particles due to the downward force applied to them.
A simpler way to look at this is, the air pressure is the weight of the atmosphere above that point. If you climb a mountain there is that much less atmosphere above you.
Since we know that pressure and temperature are proportional, we know that if you gain altitude, the pressure will drop resulting in a temperature drop.
I hope this isn't too confusing.