The only force that can act is friction (between the tablecloth and the glass) but this forces varies according as to whether the two objects in contact are moving relative to one another or not; when there is relative motion the force is (usually) much weaker and so cannot move the glass.
Friction is the force that acts between to objects in contact and resists their relative movement. It is simply proportional to the force pressing them together (which in this case is just the weight of the glass of water). The constant of proportionality is called the coefficient of friction; it depends on the nature of the two materials in contact.
When their is relative motion (slipping or sliding) we call it kinetic friction and when to two objects remain exactly together (i.e. when there is no relative motion) we call it static friction. The static and sliding coefficients are usually different with static being larger (often much larger).
The formula is simply
If we pull the tablecloth slowly enough, the frictional force is sufficient to accelerate the glass to keep up with the cloth. However, if we move the cloth quickly enough the frictional force is not sufficient to accelerate the glass at the same rate and so it starts to slip. At that point the frictional force is reduced (kinetic) and so the glass is definitely left behind.
Note as an aside that is makes no difference if we use a heavier glass as although the frictional force increases, the force necessary to accelerate the glass also increase by the same proportion.