Substitute:
#x=sqrt10tant#
#dx = sqrt10sec^2tdt#
with #t in (-pi/2,pi/2)#.
So:
#int x^3/sqrt(x^2+10)dx = int (10sqrt10tan^3tsqrt10sec^2t)/sqrt(10tan^2t+10)dt#
#int x^3/sqrt(x^2+10)dx = int (100tan^3tsec^2t)/(sqrt10 sqrt(tan^2t+1))dt#
#int x^3/sqrt(x^2+10)dx = 10sqrt10 int (tan^3tsec^2t)/ sqrt(tan^2t+1)dt#
Use now the trigonometric identity:
#tan^2t +1 = sec^2t #
note that for #t in (-pi/2,pi/2)# the secant is positive, so:
#sqrt(tan^2t +1) = sect #
and:
#int x^3/sqrt(x^2+10)dx = 10sqrt10 int (tan^3tsec^2t)/ sectdt#
#int x^3/sqrt(x^2+10)dx = 10sqrt10 int tan^3tsect dt#
Expand now in terms of #sint# and #cost#:
#int x^3/sqrt(x^2+10)dx = 10sqrt10 int sin^3t /cos^3t 1/cost dt#
#int x^3/sqrt(x^2+10)dx = 10sqrt10 int (1-cos^2t) /cos^4t sint dt#
Substitute now:
#u = cost#
#du = -sint dt#
to have:
#int x^3/sqrt(x^2+10)dx = -10sqrt10 int (1-u^2) /u^4 du#
and using the linearity of the integral:
#int x^3/sqrt(x^2+10)dx = -10sqrt10( int (du)/u^4 + int (du) /u^2) #
#int x^3/sqrt(x^2+10)dx = 10sqrt10( 1/(3u^3) -1/u) +C#
undoing the substitution:
#int x^3/sqrt(x^2+10)dx = 10/3 1/cos^3t -10/cost +C#
#int x^3/sqrt(x^2+10)dx = (10sqrt10)/3(sec^3t -3sect) +C#
#int x^3/sqrt(x^2+10)dx = (10sqrt10sect)/3(sec^2t -3) +C#
#int x^3/sqrt(x^2+10)dx = 10sqrt10sect((sec^2t -1) -2) +C#
#int x^3/sqrt(x^2+10)dx = 10sqrt10 sqrt(1+tan^2t)(tan^2t -2) +C#
#int x^3/sqrt(x^2+10)dx = (10sqrt10)/3sqrt(1+x^2/10)(x^2/10-2) +C#
#int x^3/sqrt(x^2+10)dx = (sqrt(x^2+10)(x^2-20))/3 +C#
I would note however that even if the question required integration by trigonometric substitution, it would be easier to integrate differently:
#int x^3/sqrt(x^2+10)dx = int x^2 (xdx)/sqrt(x^2+10)#
Substitute:
#u = sqrt(x^2+10)#
#du = (xdx)/sqrt(x^2+10)#
#x^2 = u^2-10#
so:
#int x^3/sqrt(x^2+10)dx = int (u^2-10) du#
#int x^3/sqrt(x^2+10)dx = int u^2du -10int du#
#int x^3/sqrt(x^2+10)dx = u^3/3 -10u+C#
#int x^3/sqrt(x^2+10)dx = u/3 (u^2-30)+C#
#int x^3/sqrt(x^2+10)dx = (sqrt(x^2+10)(x^2-20))/3+C#
