The solution is a little bit lengthy !!!
From the given #int 1/sqrt(-e^(2x)-20e^x-101)*dx#
#int 1/((sqrt(-1)*sqrt(e^(2x)+20e^x+101))*dx#
Take note that #i=sqrt(-1)# the imaginary number
Set aside that complex number for a while and proceed to the integral
#int 1/(sqrt(e^(2x)+20e^x+101))*dx#
by completing the square and doing some grouping:
#int 1/(sqrt((e^x)^2+20e^x+100-100+101))*dx#
#int 1/(sqrt(((e^x)^2+20e^x+100)-100+101))*dx#
#int 1/(sqrt(((e^x+10)^2-100+101)))*dx#
#int 1/(sqrt(((e^x+10)^2+1)))*dx#
First Trigonometric substitution:##
The acute angle #w# with side opposite #=e^x+10# and adjacent side #=1# with hypotenuse =#sqrt ((e^x+10)^2+1)#
Let #e^x+10=tan w#
#e^x dx=sec^2 w# # dw#
#dx=(sec^2w *dw)/e^x#
and then
#dx=(sec^2w *dw)/tan (w-10)#
The integral becomes
#int 1/sqrt(tan^2w+1)*(sec^2w*dw)/(tan w-10)#
#int 1/sqrt(sec^2w)*(sec^2w*dw)/(tan w-10)#
#int 1/sec w*(sec^2w*dw)/(tan w-10)#
#int (secw*dw)/(tan w-10)#
from trigonometry #sec w=1/cos w# and #tan w=sin w/cos w#
The integral becomes
#int (1/cos w*dw)/(sin w/cos w-10)# and
#int (dw)/(sin w-10 cos w)#
Second Trigonometric substitution:
Let #w=2 tan^-1 z#
#dw=2*dz/(1+z^2)#
and also #z=tan (w/2)#
The right triangle: The acute angle #w/2# with opposite side #= z#
Adjacent side #=1# and hypotenuse #=sqrt (z^2+1)#
From Trigonometry : Recalling half-angle formulas
#sin (w/2)=sqrt((1-cos w)/2#
#z/sqrt(z^2+1)=sqrt((1-cos w)/2#
solving for #cos w#
#cos w=(1-z^2)/(1+z^2)#
Also using the identity #sin ^2w=1-cos ^2w#
it follows that
#sin w=(2z)/(1+z^2)#
the integral becomes
#int (dw)/(sin w-10 cos w)=int (2*dz/(1+z^2))/((2z)/(1+z^2)-10* (1-z^2)/(1+z^2))#
Simplifying the integral results to
#int (2*dz)/(2z-10+10z^2)#
#int ((1/5)*dz)/(z^2+z/5-1)#
By completing the square:
#int ((1/5)*dz)/(z^2+z/5+1/100-1/100-1)#
#int ((1/5)*dz)/((z+1/10)^2-101/100)#
#int ((1/5)*dz)/((z+1/10)^2-(sqrt101/10)^2)#
Use now the formula #int (du)/(u^2-a^2)=1/(2a)*ln((u-a)/(u+a))+C#
Let #u=z+1/10# and #a=sqrt101/10# and including back the #i=sqrt(-1)#
Write the final answer using original variables
#-sqrt(101)/101i*ln ((10((e^x+10)/(sqrt(e^(2x)+20e^x+101)+1))+1-sqrt101)/(10((e^x+10)/(sqrt(e^(2x)+20e^x+101)+1))+1+sqrt101))+C#