The energy goes into breaking the ionic bonds of the chemicals in the ice pack.
An instant cold pack consists of a bag containing water inside a bag containing ammonium nitrate. When you break the inner bag of water by squeezing the package, the ammonium chloride dissolves in the water.
NH₄⁺NO₃⁻(s) → NH₄⁺(aq) + NO₃⁻(aq)
The energy required to separate the ions in the solid is greater than the energy regained when they become hydrated. Thus, the reaction is endothermic. It absorbs heat from the surroundings, lowering the pack's temperature.
Energy is going into the cold pack, but it is not used to raise the temperature. Rather, it goes into breaking the ionic bonds between NH₄⁺ and NO₃⁻.