When what force is not large enough to hold a nucleus together tightly, the nucleus can become radioactive?
The nuclear force of attraction. See below.
In the nucleus, a strong nuclear force of attraction exists between nucleons. As the number of protons increases, the positive charges repel each other so the number of neutrons increases even more than the number of protons to stabilise the nucleus.
However, even with an increasing number of neutrons, nuclei with 84 or more protons are unstable and are called radioactive isotopes.