First, note that #\int ln(x) dx# can be done by parts by letting #u=ln(x), dv=dx# so that #du=dx/x# and #v=x#, leading to #\int ln(x) dx=x ln(x)-\int dx=x ln(x)-x+C#.
Now, for #\int (ln(x))^2 dx#, use integration-by-parts again with #u=ln(x)# and #dv=ln(x) dx# so that #du = dx/x# and #v=x ln(x)-x#. Then #\int(\ln(x))^2 dx=x(ln(x))^2-x ln(x)-\int (ln(x)-1)dx=
x(ln(x))^2-x ln(x)+x-(x ln(x)-x)#
#=x(ln(x))^2-2xln(x)+2x+C#.