#I = inttan^2(x)*sec^3(x)dx#
#I = int(sec^2(x)-1)*sec^3(x)dx#
#I = intsec^5(x)dx - intsec^3(x)dx#
For the second integral, you need integration by parts, say
#u = sec(x)#, so #du = sec(x)tan(x)# and
#dv = sec^2(x)# so #v = tan(x)#
#intsec^3(x)dx = sec(x)tan(x) - intsec(x)tan^2(x)dx#
#intsec^3(x)dx = sec(x)tan(x) - intsec(x)(sec^2(x)-1)dx#
#intsec^3(x)dx = sec(x)tan(x) - int(sec^3(x)-sec(x))dx#
#intsec^3(x)dx = sec(x)tan(x) - intsec^3(x)dx -intsec(x)dx#
#2intsec^3(x)dx = sec(x)tan(x) - intsec(x)dx#
The last integral is a tabled one, so
#intsec^3(x)dx = (sec(x)tan(x)+ln|sec(x)+tan(x)|)/2 + c#
For #intsec^5dx#, using integration by parts
#u = sec^3(x)# so #du = 3tan(x)sec^3(x)#
#dv = sec^2(x)# so #v = tan(x)#
#intsec^5dx = sec^3(x)tan(x) - inttan(x)*3tan(x)sec^3(x)dx#
#intsec^5dx = sec^3(x)tan(x) - 3inttan^2(x)sec^3(x)dx#
But we originally had that
#inttan^2(x)sec^3(x)dx = intsec^5(x)dx - intsec^3(x)dx#
So substituting that we have
#inttan^2(x)sec^3(x)dx = sec^3(x)tan(x) - 3inttan^2(x)sec^3(x)dx - (sec(x)tan(x)+ln|sec(x)+tan(x)|)/2#
#4inttan^2(x)sec^3(x)dx = (2sec^3(x)tan(x) - sec(x)tan(x)-ln|sec(x)+tan(x)|)/2#
#inttan^2(x)sec^3(x)dx = (2sec^3(x)tan(x) - sec(x)tan(x)-ln|sec(x)+tan(x)|)/8#
