I decomposed into basic fractions,
#(x^2-2x)/[(x^2-3)(x-3)(x-8)]#
=#(Ax+B)/(x^2-3)+C/(x-3)+D/(x-8)#
After expanding denominator,
#(Ax+B)(x-3)(x-8)+C(x^2-3)(x-8)+D(x^2-3)(x-3)=x^2-2x#
Set #x=3#, #-30C=3#, so #C=-1/10#
Set #x=8#, #305D=48#, so #D=48/305#
Set #x=0#, #24B+24C+9D=0#, so #B=-C-3/8*D=5/122#
Set #x=2#, #14A+14B+14C+4D=-1#, so #A=-7/122#
Hence,
#int (x^2-2x)/[(x^2-3)(x-3)(x-8)*dx]#
=#-1/122int ((7x-5)*dx)/(x^2-3)-1/10int (dx)/(x-3)+5/122int (dx)/(x-8)#
=#5/122Ln(x-8)-1/10Ln(x-3)+C-1/122*int ((7x-5)*dx)/(x^2-3)#
Now I solved last integral,
#int ((7x-5)*dx)/(x^2-3)#
=#int ((7x-5)*dx)/[(x+sqrt3)(x-sqrt3)]#
=#(21-5sqrt3)/6int (dx)/(x-sqrt3)-(21+5sqrt3)/6int (dx)/(x+sqrt3)#
=#(21-5sqrt3)/6ln(x-sqrt3)-(21+5sqrt3)/6ln(x+sqrt3)#
Thus,
#int (x^2-2x)/[(x^2-3)(x-3)(x-8)*dx]#
=#5/122Ln(x-8)+(21+5sqrt3)/732ln(x+sqrt3)-1/10Ln(x-3)-(21-5sqrt3)/6ln(x-sqrt3)+C#