I am assuming that #a > 0#
Depending on the domain of integration, we can integrate
#int sqrt(x^2(x^2-a)) dx = int absx sqrt(x^2-a) dx# by substitution.
Note that the domain of the integrand is #(-oo,-sqrta] uu [sqrta,oo)#
On #(-oo,-sqrta]#, we have #sqrt(x^2(x^2-a)) = -xsqrt(x^2-a)#
So
#int sqrt(x^2(x^2-a)) = int -xsqrt(x^2-a) = -1/3(x^2-a)^(3/2) +C#
On #[sqrta,oo)#, we have #sqrt(x^2(x^2-a)) = xsqrt(x^2-a)#
So
#int sqrt(x^2(x^2-a)) = int xsqrt(x^2-a) = 1/3(x^2-a)^(3/2) +C#
Note
Because #sqrt(x^2)/x = absx/x = {(1,"if",x < 0),(-1,"if",x < 0):}#, we could write the answer as
#sqrt(x^2)/(3x)(x^2-a)^(3/2) +C# or as
#(x^2(x^2-a))^(3/2)/(3x^3)#
