We can integrate by parts using the logarithm as integral part, so that in the resulting integral we have a rational function:
#int x^2ln^2xdx = int ln^2x d(x^3/3)#
#int x^2ln^2xdx = (x^3ln^2x)/3 - 1/3 int x^3d(ln^2x)#
#int x^2ln^2xdx = (x^3ln^2x)/3 - 2/3 int x^3 lnx/xdx #
#int x^2ln^2xdx = (x^3ln^2x)/3 - 2/3 int x^2lnxdx #
Solve the resulting integral by parts again:
#int x^2lnxdx = int lnx d(x^3/3)#
#int x^2lnxdx = (x^3lnx)/3 - 1/3 int x^3 d(lnx)#
#int x^2lnxdx = (x^3lnx)/3 - 1/3 int x^3 dx/x#
#int x^2lnxdx = (x^3lnx)/3 - 1/3 int x^2 dx#
#int x^2lnxdx = (x^3lnx)/3 - 1/9x^3 +C#
Substituting in the first expression:
#int x^2ln^2xdx = (x^3ln^2x)/3 - 2/3( (x^3lnx)/3 - 1/9x^3 ) +C#
and simplifying:
#int x^2ln^2xdx = (x^3ln^2x)/3 - 2/9(x^3lnx) + 2/27x^3 +C#
#int x^2ln^2xdx = x^3/27(9ln^2x - 6lnx +2) +C#
