What factors cause the difference in the size of crystals formed?
One factor is the speed at which crystals are formed.
The slower the crystals are formed the larger the crystals that are formed.
Crystals that are formed near or on the surface tend to crystalize faster and therefore are usually smaller. Crystals deeper in the crust tend to crystalize slower and therefore are larger.