You are correct.
An electron that you add to an atom is somewhat shielded from the attraction of the nucleus by the electrons already present.
Also, the added electron partially shields the electrons that are already in the atom.
All the valence electrons feel less attraction to the nucleus, so the anion is bigger than a neutral atom.
It would be easier to remove an atom from a negative ion than from the neutral atom, but this is not ionization energy.
Ionization energy is defined as the energy required for removing an electron from a neutral atom.
The energy in this case would be the negative of the electron affinity.