That happens because of the structure of the two molecules, i.e. the atoms to which those acidic hydrogens are bonded to.
Take acetic acid, for example. Its Lewis structure looks like this
That implies that the bonding electrons will spend most of the time around the oxygen atom, leaving the hydrogen atom electron deficient nad causing it to be less tightly held.
The partial positive charge that arises on the hydrogen atom can be easily attacked by a water molecule, which is a polar molecule. Moreover, the negative charge that gets created after the hydrogen is picked off can be easily stabilized by resonance.
On the other hand, the other three hydrogen atoms are bonded to a carbon. As you know, the
This means that its bond with carbon will be strong enough to stop a water molecule from being able to pluck it off.
Now take a look at phosphoric acid's Lewis structure
This time, all three hydrogens are bonded to oxygen atoms, which is what ultimately makes them ionizable.
All three can be easily attacked by water molecules and removed from the phosphoric acid molecule.
So, as a conclusion, hydrogen atoms that are attached to carbon atoms are not acidic, while hydrogen atoms attached to oxygen atoms are quite acidic.