Let #u_1=2 m/s# & #u_2=-3 m/s# be the initial velocities of two balls having masses #m_1=5\ kg\ # & #\m_2=8\ kg# moving in opposite directions i.e. first one is moving in +ve x-direction & other in -ve x-direction, After collision let #v_1# & #v_2# be the velocities of balls in +ve x-direction
By law of conservation of momentum in +ve x-direction, we have
#m_1u_1+m_2u_2=m_1v_1+m_2v_2#
#5(2)+8(-3)=5v_1+8v_2#
#5v_1+8v_2=-14\ .......(1)#
Now, loss of kinetic energy is #75%# hence
#(1-\frac{75}{100})(\frac{1}{2}m_1u_1^2+\frac{1}{2}m_2u_2^2)=(\frac{1}{2}m_1v_1^2+\frac{1}{2}m_2v_2^2)#
#1/4(\frac{1}{2}5(2)^2+\frac{1}{2}8(3)^2)=\frac{1}{2}5v_1^2+\frac{1}{2}8v_2^2#
#5v_1^2+8v_2^2=23 \ ......(2)#
substituting the value of #v_2=\frac{-5v_1-14}{8}# from (1) into (2) as follows
#5v_1^2+8(\frac{-5v_1-14}{8})^2=23#
#65v_1^2+140v_1+12=0#
solving above quadratic equation, we get #v_1=-0.089, -2.064# & corresponding values of #v_2=-1.694, -0.46#
Hence, the final velocities of both the balls are either #v_1=-0.089\ m/s# & #v_2=-1.694\m/s#
or
#v_1=-2.064\ m/s# & #v_2=-0.46\m/s#
