#int x*arcsec(x^2)*dx#
=#1/2int 2x*arcsec(x^2)*dx#
After using #y=x^2# and #2x*dx=dy# transforms, this integral became
#1/2int arcsecy*dy#
After using #z=arcsecy#, #y=secz# and #dy=secz*tanz*dz# transforms, it became
#1/2int z*secz*tanz*dz#
=#1/2z*secz-1/2int secz*dz#
=#1/2z*secz-1/2int (secz*(secz+tanz)*dz)/(secz+tanz)#
=#1/2z*secz-1/2ln(secz+tanz)+C#
For #y=secz#, #tanz# must be equal to #sqrt(y^2-1)#. Thus,
#1/2y*arcsecy-1/2ln(y+sqrt(y^2-1))+C#
=#1/2x^2*arcsec(x^2)-1/2ln(x^2+sqrt(x^4-1))+C#