#int(tan^2(x)-1)/(sec^2(x)+1)dx#
Use #sec^2(x)=tan^2(x)+1#:
#=int(tan^2(x)-1)/((tan^2(x)+1)+1)dx=int(tan^2(x)-1)/(tan^2(x)+2)dx#
Rewrite:
#=int(tan^2(x)+2-3)/(tan^2(x)+2)dx=int(1-3/(tan^2(x)+2))dx#
#=x-3intdx/(tan^2(x)+2)#
Let #u=tan(x)#. This implies that #du=sec^2(x)dx#. Then, #dx=(du)/sec^2(x)=(du)/(tan^2(x)+1)=(du)/(u^2+1)#. The integral then becomes:
#=x-3int(du)/((u^2+1)(u^2+2))#
Performing a partial fraction decomposition:
#1/((u^2+1)(u^2+2))=(Au+B)/(u^2+1)+(Cu+D)/(u^2+2)#
#1=(Au+B)(u^2+2)+(Cu+D)(u^2+1)#
Expanding and sorting by degree of #u#:
#1=u^3(A+C)+u^2(B+D)+u(2A+C)+(2B+D)#
Comparing the coefficients on either side:
#{(A+C=0),(B+D=0),(2A+C=0),(2B+D=1):}#
Subtracting the second equation from the fourth, we see that #(2B+D)-(B+D)=1-0#, or #B=1#. Both the second and fourth equations then yield #D=-1#.
Similarly, subtracting the first equation from the third equation gives #(2A+C)-(A+C)=0-0#, or #A=0#. Thus #C=0#.
Plugging in #(A,B,C,D)=(0,1,0,-1)# gives:
#1/((u^2+1)(u^2+2))=1/(u^2+1)-1/(u^2+2)#
Returning to the original integral, we get:
#=x-3int(du)/(u^2+1)+3int(du)/(u^2+2)#
In the first integral, let #u=tantheta#. In the second integral, let #u=sqrt2tanphi#. Respectively, these imply that #du=sec^2thetad theta# and #du=sqrt2sec^2phid phi#.
#=x-3int(sec^2theta)/(tan^2theta+1)d theta+3int(sqrt2sec^2phi)/(2tan^2phi+2)dphi#
Since #tan^2alpha+1=sec^2alpha#:
#=x-3intd theta+3/sqrt2intdphi#
#=x-3theta+3/sqrt2phi#
Reversing the substitutions #u=tantheta# and #u=sqrt2tanphi#:
#=x-3tan^-1(u)+3/sqrt2tan^-1(u/sqrt2)#
Recall that #u=tan(x)#:
#=x-3tan^-1(tan(x))+3/sqrt2tan^-1(tan(x)/sqrt2)#
#=-2x+3/sqrt2tan^-1(tan(x)/sqrt2)+C#