I'm really happy you asked this question because the process of solving it is very interesting.
We have to be clever to start off here. No substitutions seem to fit, nor does integration by parts. If instead we had #int tanxsec^2xdx#, we could make the simple substitution #u=tanx#, #du=sec^2xdx#, which would make the integral #intudu#. Unfortunately, that is not the case.
If we remember our Pythagorean Identities, though, we can make some progress. Recall that #tan^2x+1=sec^2x#, or #tan^2x=sec^2x-1#. Using this, we can rewrite the integral as:
#int (sec^2x-1)secxdx#
#=int sec^3x-secxdx#
The sum rule for integrals says we can further simplify it to:
#intsec^3xdx-intsecxdx#
Second Integral
We start here because 1) we can and 2) #intsecxdx# is fairly simple: it is equal to #lnabs(secx+tanx)#. It's the other one that poses a fairly interesting challenge.
First Integral
Before going on to this one, let's review our entire problem:
#inttan^2xsecxdx=intsec^3xdx-lnabs(secx+tanx)#
Now we have to think about how to do #intsec^3xdx#. Note that we can rewrite it as:
#intsec^2xsecxdx#
Note also that #intsec^2xdx=tanx#. If only we could simplify it like that...
But wait, we can! Using integration by parts, with #u=secx# and #dv=sec^2xdx#:
#color(blue)(u=secx)#
#(du)/dx=secxtanx->color(blue)(du=secxtanxdx)#
#dv=sec^2xdx->intdv=intsec^2xdx->color(red)(v=tanx)#
The integration by parts formula is:
#intudv=color(blue)(u)color(red)(v)-intcolor(red)(v)color(blue)(du)#
Plugging in what we just found,
#intsec^3dx=color(blue)(secx)color(red)(tanx)-intcolor(red)(tanx)color(blue)(secxtanxdx)#
And simplifying,
#intsec^3dx=color(blue)(secx)color(red)(tanx)-intcolor(red)(tan^2x)color(blue)(secxdx)#
Isn't that strange! We end up with #inttan^2xsecxdx#, which is the integral in our original problem! Now we have:
#inttan^2xsecxdx=secxtanx-inttan^2xsecxdx-lnabs(secx+tanx)#
We can add the #inttan^2xsecxdx# on the right to the left to get:
#2inttan^2xsecxdx=secxtanx-lnabs(secx+tanx)#
#inttan^2xsecxdx=(secxtanx-lnabs(secx+tanx))/2+C# (don't forget the integration constant!)
And that is our final answer.
