#intdx/sqrt(3x-12sqrtx-21)#
I've added the differential #dx# to your integrand. While it may seem like a fanciful calculus accessory, it's actually very important, especially when doing trig sub problems.
First, let's get this into a more "normal" form for trig subs. Let #x=t^2#. Note this implies that #dx=2tdt# and that #sqrtx=t#.
#=int(2tdt)/sqrt(3t^2-12t-21)#
Now complete the square in the denominator:
#=2intt/sqrt(3(t^2-4t+4)-21-12)dt#
#=2intt/sqrt(3(t-2)^2-33)dt#
#=2/sqrt3intt/sqrt((t-2)^2-11)dt#
This is optional, but now I'd let #s=t-2#. This implies that #ds=dt# and that #t=s+2#.
#=2/sqrt3int(s+2)/sqrt(s^2-11)ds#
Now, to clear the denominator, I'd let #s=sqrt11sectheta#.
My reason for this substitution is that #s^2-11=11sec^2theta-11=11(sec^2theta-1)=11tan^2theta#. This will clear out our denominator.
Moreover, note that #ds=sqrt11secthetatanthetad theta#. Proceeding:
#=2/sqrt3int(sqrt11sectheta+2)/sqrt(11tan^2theta)(sqrt11secthetatanthetad theta)#
#=2/sqrt3int(sqrt11sectheta+2)/(sqrt11tantheta)(sqrt11secthetatanthetad theta)#
#=2/sqrt3int(sqrt11sectheta+2)secthetad theta#
#=2/sqrt3int(sqrt11sec^2theta+2sectheta)d theta#
These are two fairly well known integrals. You may need to look the second one up.
#=2/sqrt3(sqrt11tantheta+2ln(abs(sectheta+tantheta)))#
Recall that #s=sqrt11sectheta#, so #sectheta=s/sqrt11#. Moreover, #tantheta=sqrt(sec^2theta-1)=sqrt(s^2/11-1)=sqrt((s^2-11)/11)#. Then:
#=2sqrt(11/3)sqrt((s^2-11)/11)+4/sqrt3ln(abs(s/sqrt11+sqrt((s^2-11)/11)))#
Do some more simplifications. Remember the log rule #log(A//B)=log(A)-log(B)#.
#=2/sqrt3sqrt(s^2-11)+4/sqrt3ln(abs(s+sqrt(s^2-11)))-4/sqrt3lnsqrt11#
Now use #s=t-2# and #t=sqrtx#, so #s=sqrtx-2#;
#=2/sqrt3((sqrtx-2)^2-11)+4/sqrt3ln(abs(sqrtx-2+sqrt((sqrtx-2)^2-11)))-4/sqrt3lnsqrt11#
#=2/sqrt3(x-4sqrtx-7)+4/sqrt3ln(sqrtx+2+sqrt(x-4sqrtx-7))+C#
I've added the constant of integration, into which the #4/sqrt3lnsqrt11# term has been absorbed.
The absolute value bars are also unnecessary because the argument of the natural log function is, by necessity, greater than #2# and always positive.
