Let #\sqrtx=t\implies \frac{dx}{2\sqrtx}=dt\ \ or\ \ dx=2t\dt#
#\therefore \int \frac{dx}{\sqrt{3x-12\sqrtx+40}}#
#=\int \frac{2tdt}{\sqrt{3t^2-12t+40}}#
#=\frac{1}{3}\int \frac{((6t-12)+12)dt}{\sqrt{3t^2-12t+40}}#
#=\frac{1}{3}\int \frac{(6t-12)dt}{\sqrt{3t^2-12t+40}}+12/3\int \frac{dt}{\sqrt{3t^2-12t+40}}#
#=\frac{1}{3}\int \frac{d(3t^2-12t+40)}{\sqrt{3t^2-12t+40}}+4\int \frac{dt}{\sqrt3\sqrt{t^2-4t+40/3}}#
#=\frac{1}{3}\int \frac{d(3t^2-12t+40)}{(3t^2-12t+40)^{1/2}}+4/\sqrt3\int \frac{dt}{\sqrt{(t-2)^2+28/3}}#
#=\frac{1}{3} \frac{(3t^2-12t+40)^{-1/2+1}}{-1/2+1}+4/\sqrt3\int \frac{d(t-2)}{\sqrt{(t-2)^2+(\sqrt{28/3})^2}}#
#=\frac{2}{3}(3t^2-12t+40)^{1/2}+4/\sqrt3\sinh^{-1}(\frac{t-2}{\sqrt{28/3}})+C#
#=\frac{2}{3}(3x-12\sqrtx+40)^{1/2}+4/\sqrt3\sinh^{-1}(\frac{\sqrt{3x}-2\sqrt3}{\sqrt{28}})+C#
