Rewriting the denominator:
∫3x2+5x−1(x2+2x−1)2dx=∫3x2+5x−1{(x2+2x+1)−2}2dx=∫3x2+5x−1{(x+1)2−2}2dx
Let u=x+1, implying that x=u−1 and du=dx.
=∫3(u−1)2+5(u−1)−1(u2−2)2du=∫3u2−u−3(u2−2)2du
Now let u=√2secθ. This implies that u2−2=2sec2θ−2=2tan2θ and that du=√2secθtanθdθ.
=∫3(2sec2θ)−√2secθ−3(2tan2θ)2(√2secθtanθdθ)
=√24∫6sec3θ−√2sec2θ−3secθtan3θdθ
Multiplying through by cos3θcos3θ:
=12√2∫6−√2cosθ−3cos2θsin3θdθ
=3√2∫csc3θdθ−12∫cotθcsc2θdθ−32√2∫1−sin2θsin3θdθ
Note that 3√2−32√2=32√2:
=32√2∫csc3θdθ−12∫cotθcsc2θdθ+32√2∫cscθdθ
Find the method for integrating csc3θ here.
For the second integral, let v=cotθ so dv=−csc2θdθ.
The integral of cscθ is well known. For its derivation, see here.
=32√2(−12)(cotθcscθ+ln|cotθ+cscθ|)+12∫vdv+32√2ln|cscθ−cotθ|
Note that 12∫vdv=12(v22)=v24=cot2θ4.
=−34√2cotθcscθ−34√2ln|cotθ+cscθ|+34√2ln|cscθ−cotθ|+14cot2θ
Our substitution was u=√2secθ, implying that cosθ=√2u, which is a right triangle where the side adjacent to θ is √2, the hypotenuse is u, and the side opposite θ is √u2−2.
Thus, cscθ=u√u2−2 and cotθ=√2√u2−2.
Also note that −34√2ln|cotθ+cscθ|+34√2ln|cscθ−cotθ|=34√2ln∣∣∣cscθ−cotθcotθ+cscθ∣∣∣=34√2ln∣∣∣1−cosθ1+cosθ∣∣∣.
=−34√2√2uu2−2+34√2ln∣∣
∣∣1−√2u1+√2u∣∣
∣∣+14(2u2−2)
=−34(uu2−2)+12(1u2−2)+34√2ln∣∣∣u−√2u+√2∣∣∣
=−3u+24(u2−2)+34√2ln∣∣∣u−√2u+√2∣∣∣
With u=x+1:
=−3x−14(x2+2x−1)+34√2ln∣∣∣x−√2+1x+√2+1∣∣∣+C