'Fix' the denominator
First 'complete the square' in the denominator:
#sqrt(x^2-2x+10)=sqrt((x-1)^2+9)#
This is needed in order to put it into a form that a trig substitution can simplify by use of an identity - we need to simplify the denominator here in order to tackle the integral.
Then set #x-1=3tan theta#, making the integral
#int (1+3 tan theta)^3/sqrt(9 tan^2 theta + 9) dx/(d theta) d theta#
From our substitution formula, #dx/(d theta)=3sec^2theta#, the standard derivative of #tan#. Thus the integral is
#int (1+3 tan theta)^3/sqrt(9 tan^2 theta + 9) * 3sec^2theta d theta#
#=int (sec^2theta(1+3 tan theta)^3)/sqrt(tan^2 theta + 1) d theta#
(Recall the identity #1+tan^2 x=sec^2#)
#=int (sec^2theta(1+3 tan theta)^3)/sqrt(sec^2 theta) d theta#
#=int sec theta(1+3 tan theta)^3d theta#
(Multiply out terms, recalling the binomial theorem for coefficients)
#=int sec theta+9 sec theta tan theta + 27 sec theta tan^2 theta + 27 sec theta tan^3 theta d theta#
So this is how it is done in general - you simplify the denominator until it is gone, at which point you find out exactly how messy you have made your numerator. In this case, quite messy, but it is solvable term by term.
Solve what's left
For the first term, it will suffice here to state that #int sec theta d theta=log|sec theta + tan theta|+C#. This answer is growing long, and there are proofs of this online, for example here: https://www.math.ubc.ca/~feldman/m121/secx.pdf
Note that #d/dx secx=secxtanx#, by the quotient rule and identity. So the second term is easy: #int 9 sec theta tan theta dx = 9 sec theta+C#.
Recall "integration by parts", a reformulation of the product rule of differentiation:
#int u (dv)/(d theta) d theta = uv - int v (du)/(d theta) d theta#
For the third term, use the #sec# derivative that was so useful for the second term; set #u=tan theta# and #(dv)/(d theta)=27 sec theta tan theta#. Then #v=27 sec theta#, #(du)/(d theta)=sec^2 theta# and the integral of the third term is #27 sec theta tan theta - 27int sec^3 theta d theta#.
The above link also explains how to integrate #sec^3 theta#. Taking this result as given, we see that #int27 sec theta tan^2 theta d theta=27/2 sec theta tan theta-27/2log|sec theta +tan theta|+C#.
Finally, for the fourth term, we take the same approach with the #sec# derivative. Set #(dv)/(d theta) = 27 sec theta tan theta#; set #u=tan^2 theta#. Then #v=27sec theta# and #(du)/(d theta)=2 tan theta sec^2 theta#, so the fourth term becomes
#27 sec theta tan^2 theta - int 54tan theta sec^3 theta d theta#.
Repeat integration by parts, this time with #(du)/(d theta)=54 sec theta tan theta# and #v=sec^2 theta#. Then #u=54 sec theta# and #(dv)/(d theta)=2 tan theta sec theta#. The fourth term is now
#27 sec theta tan^2 theta - 54 sec^3 theta + int 108 tan theta sec^2 theta d theta#.
A third integration by parts now (have patience!)... Again, #(du)/(d theta)=108 sec theta tan theta#. #v=sec theta#. So #u=108 sec theta# and #(dv)/(d theta)=sec theta tan theta#. The fourth term is now
#27 sec theta tan^2 theta - 54 sec^3 theta + 108 sec^2 theta - int 108 tan theta sec^2 theta d theta#.
But this is where we can break the chain of integrations (hurrah, I hear you cry - it's been pretty tedious) - we now have the same integral as we did at the previous step, so we can employ a trick.
This step showed us that
#int tan theta sec^2 theta d theta=sec^2 theta - int tan theta sec^2 theta d theta#
So
#int tan theta sec^2 theta d theta=1/2 sec^2 theta#
Therefore the fourth term is
#27 sec theta tan^2 theta - 54 sec^3 theta + 54 sec^2 theta#
Using the identity #tan^2 theta = sec^2 theta-1#, we can simplify this:
#27 sec^3 theta -27 sec theta - 54 sec^3 theta + 54 sec^2 theta#
#-27 sec theta + 54 sec^2 theta - 27 sec^3 theta#
Combining the four terms, we obtain the answer:
#log|sec theta + tan theta|+9 sec x+27/2 sec theta tan theta-27/2log|sec theta +tan theta|-27 sec theta + 54 sec^2 theta - 27 sec^3 theta+C#
which simplifies a little to
#-25/2log|sec theta + tan theta|+27/2 sec theta tan theta-18 sec theta+ 54 sec^2 theta - 27 sec^3 theta+C#
But we aren't quite done yet - we have to get back to our original variable, #x#. Recall our substitution: #x-1=3tan theta#. This was an elegant substitution that would have made for a nice integral if the numerator hadn't been so awkward. So #tan theta = (x-1)/3# and #sec theta=sqrt(1+(x-1)^2/9)#.
This leaves us with
#-25/2log|(x-1)/3+sqrt(1+(x-1)^2/9)|+9/2 (x-5) sqrt(1+(x-1)^2/9) + 54 (1+(x-1)^2/9) - 27 (1+(x-1)^2/9)^(3/2)+C#
Not a beautiful expression!
