Magnetic force (or any other force) may be measured using a torsion balance, which is a sensitive device to measure force.
Coulomb, himself used a torsion balance to discover the Coulomb's law and inverse square law of forces between magnetic poles.
Now, the magnetic force a magnetic field
This gives an expression to calculate the magnetic force on a charged particle in a magnetic field.
Moreover, from the Coulomb's law of magnetic interations, the force of attraction (or repulsion) between two magnetic poles of pole strengths