Charge/polarity and size
The lipid bilayer stops any and all charged particles, as well as polar molecules. Any polar substances that are trying to pass the membrane get blocked by the apolar "tails" of the membrane.
Small molecules can move between the phospholipids in the membrane (in small amounts). Larger molecules can't pass the membrane at all through diffusion. Their size makes them too large to push away the lipids in the membrane.