Use "integration by parts".
The rule is:
#int f(x) g'(x) dx = f(x) g(x) - int f'(x) g(x) dx#
A very important decision is now which part of your product is #f(x)# and which one is #g'(x)# since later, you will need to differentiate one of the terms and integrate the other one. With a wrong choice, you might never find a solution.
Here, the choice is not very hard: you really want to differentiate and not integrate #ln(x)# (trust me on that ;-) ), so choose #f(x) = ln(x)# and #g'(x) = x^2#.
Now, you need to compute #f'(x) = 1/x# and #g(x) = 1/3 x^3#.
At this point you are ready to apply the rule.
#int x^2 ln(x) dx = ln(x) * 1/3 x^3 - int 1/x * 1/3 x^3 dx#
# = 1/3 x^3 ln(x) - 1/3 int x^2 dx#
# = 1/3 x^3 ln(x) - 1/3 * ( 1/3 x^3 )#
# = 1/3 x^3 ln(x) - 1/9 x^3 #
or, if you wish,
# = 1/3 x^3 (ln(x) - 1/3) #
