Rewrite the integrand using #tan^2x = sec^2x-1#.
Let's give the integral we want the name #I#
#I = int tan^2xsec^3x dx = int (sec^5x-sec^3x)dx#
Next we'll integrate #sec^5x# by parts.
#int sec^5x dx = int sec^3 x sec^2x dx#
Let #u = sec^3 x# and #dv = sec^2x dx#.
Then #du = 3tanx sec^3x dx# and #v = tanx#
We get
#int sec^5 x dx = sec^3x tanx - 3int tan^2x sec^3x dx#
Again, use #tan^2x = sec^2 x-1# to get
#int sec^5 x dx = sec^3x tanx - 3int (sec^2 x-1) sec^3x dx#
#int sec^5 x dx = sec^3x tanx - 3int sec^5 dx + 3int sec^3x dx#
Which gets us
#4int sec^5 x dx = sec^3x tanx + 3int sec^3x dx#
and
#int sec^5 x dx = 1/4sec^3x tanx + 3/4 int sec^3x dx#
Recalling that the other integral we need is #int sec^3x dx#, let's simplify our lives by writing:
#I = intsec^5xdx-intsec^3xdx#
# = underbrace(1/4sec^3x tanx + 3/4 int sec^3x dx)_(intsec^5 x dx)-intsec^3xdx#
# = 1/4sec^3x tanx -1/4 int sec^3x dx#
So now find #int sec^3x dx = int secx sec^2x dx# using the same general approach and integration by parts. You should get
#int sec^3x dx = 1/2secxtanx - 1/2int secx dx#
So at this point we have
#I = 1/4sec^3x tanx -1/4underbrace( [1/2secxtanx - 1/2int secx dx])_(intsec^3 x dx)#
# = 1/4sec^3x tanx -1/8secxtanx + 1/8 int secx dx#
Now #int secx dx# can be evaluated in a couple of ways, the usual trick is to multiply by #(secx+tanx)/(secx+tanx)# to get #int 1/u du = ln absu = ln abs(secx+tanx)#
So finally we finish with
#I = 1/4sec^3x tanx - 1/8secxtanx + 1/8 ln abs(sec x + tan x ) +C#
Alternative form for #int secx dx#
#intsecx dx# can also be found by substitution and partial fractions to get #1/2ln abs((sinx+1)/(sinx-1))+C#
(Yes, the is equivalent to # ln abs(secx+tanx)+C#)
