The integrand is defined for #x in(-oo,-3) uu (3,+oo)#. Let us focus for the moment on #x in (3,+oo)# and substitute:
#x = 3sect#
with #t in (0,pi/2)#
so that:
#dx = 3sect tant dt#
and:
#int x^2/sqrt(x^2-9)dx = int ((3sect)^2(3sect tant) dt)/sqrt((3sect)^2-9)#
#int x^2/sqrt(x^2-9)dx = 27 int (sec^3t tant dt)/sqrt(9sec^2t-9)#
#int x^2/sqrt(x^2-9)dx = 9 int (sec^3t tant dt)/sqrt(sec^2t-1)#
Use now the trigonometric identity:
#sec^2t -1 = tan^2t#
and considering that for #t in (0,pi/2)# the tangent is positive:
#sqrt(sec^2t -1) = tant#
so:
#int x^2/sqrt(x^2-9)dx = 9 int (sec^3t tant dt)/tant#
#int x^2/sqrt(x^2-9)dx = 9 int sec^3tdt#
Now write #sec^3t# as #sect xx sec^2t#, and as #d/dt tant = sec^2t# we can integrate by parts:
#int sec^3tdt = int sect d(tant)#
#int sec^3tdt = sect tant - int tant d(sect )#
#int sec^3tdt = sect tant - int tan^2tsect dt#
using the same identity again:
#int sec^3tdt = sect tant - int (sec^2t-1)sect dt#
and based in the linearity of the integral:
#int sec^3tdt = sect tant - int sec^3t+ int sect dt#
The integral now appears on both sides of the equation and we can solve for it:
#2int sec^3tdt = sect tant + int sect dt#
and finally :
#int sec^3tdt = (sect tant)/2 + 1/2 ln abs (sect+tant)+C#
Undo now the substitution:
#int x^2/sqrt(x^2-9)dx = (9sect tant)/2 + 9/2 ln abs (sect+tant)+C#
#int x^2/sqrt(x^2-9)dx = (3xsqrt((x/3)^2-1) )/2 + 9/2 ln abs (x/3+sqrt((x/3)^2-1))+C#
#int x^2/sqrt(x^2-9)dx = (xsqrt(x^2-9))/2 +9/2ln abs (x +sqrt(x^2-9)) +C#
By direct differentiation we can see that the solution is valid also for #x in (-oo,-3)#.