We start with simplifying the square root by factoring out x^2:
int sqrt(x^2 - 3 x^4)\ dx = int x sqrt(1 - 3 x^2)\ dx.
Now we substitute x \mapsto g(t) with
g(t) = \frac{1}{sqrt(3)} cos t,
g'(t) = - \frac{1}{sqrt(3)} sin t, and
g^{-1}(x) = cos^{-1} ( sqrt(3) x ).
This yields
[ - \frac{1}{3} int sin t cos t sqrt(1 - cos^2 t)\ dt ]_{t = cos^{-1}(sqrt(3) x)} =
= [ - \frac{1}{3} int sin^2 t cos t \ dt ]_{t = cos^{-1}(sqrt(3) x)}.
At this point, we may interpret sin t as an inner function (with derivative cos t), and thus write the integral as
[ - \frac{1}{3} int u^2 \ du ]_{u = sin t = sin cos^{-1}(sqrt(3) x) = sqrt(1 - 3 x^2)} =
= [ - \frac{u^3}{9} + C ]_{u = sqrt(1 - 3 x^2)} =
= - \frac{sqrt(1 - 3 x^2)^3}{9} + C.