The integrand is defined for x in(-oo,-3) uu (3,+oo). Let us focus for the moment on x in (3,+oo) and substitute:
x = 3sect
with t in (0,pi/2)
so that:
dx = 3sect tant dt
and:
int x^2/sqrt(x^2-9)dx = int ((3sect)^2(3sect tant) dt)/sqrt((3sect)^2-9)
int x^2/sqrt(x^2-9)dx = 27 int (sec^3t tant dt)/sqrt(9sec^2t-9)
int x^2/sqrt(x^2-9)dx = 9 int (sec^3t tant dt)/sqrt(sec^2t-1)
Use now the trigonometric identity:
sec^2t -1 = tan^2t
and considering that for t in (0,pi/2) the tangent is positive:
sqrt(sec^2t -1) = tant
so:
int x^2/sqrt(x^2-9)dx = 9 int (sec^3t tant dt)/tant
int x^2/sqrt(x^2-9)dx = 9 int sec^3tdt
Now write sec^3t as sect xx sec^2t, and as d/dt tant = sec^2t we can integrate by parts:
int sec^3tdt = int sect d(tant)
int sec^3tdt = sect tant - int tant d(sect )
int sec^3tdt = sect tant - int tan^2tsect dt
using the same identity again:
int sec^3tdt = sect tant - int (sec^2t-1)sect dt
and based in the linearity of the integral:
int sec^3tdt = sect tant - int sec^3t+ int sect dt
The integral now appears on both sides of the equation and we can solve for it:
2int sec^3tdt = sect tant + int sect dt
and finally :
int sec^3tdt = (sect tant)/2 + 1/2 ln abs (sect+tant)+C
Undo now the substitution:
int x^2/sqrt(x^2-9)dx = (9sect tant)/2 + 9/2 ln abs (sect+tant)+C
int x^2/sqrt(x^2-9)dx = (3xsqrt((x/3)^2-1) )/2 + 9/2 ln abs (x/3+sqrt((x/3)^2-1))+C
int x^2/sqrt(x^2-9)dx = (xsqrt(x^2-9))/2 +9/2ln abs (x +sqrt(x^2-9)) +C
By direct differentiation we can see that the solution is valid also for x in (-oo,-3).
