#I = int \ x tan^(-1) x \ dx#
we'll use IBP
before we go, note that #d/dx (tan^(-1) x) = 1/(1+x^2)#, a well known result
So
#I = int \ d/dx(x^2/2) tan^(-1) x \ dx#
and by IBP
#I = x^2/2 tan^(-1) x - int \ x^2/2 d/dx( tan^(-1) x) \ dx#
# = x^2/2 tan^(-1) x - 1/2 int \ x^2 1/(x^2 +1 ) \ dx#
# = x^2/2 tan^(-1) x - 1/2 int \ (x^2+ 1 - 1)/(x^2 +1 ) \ dx#
# = x^2/2 tan^(-1) x - 1/2 int \ 1 - 1/(x^2 +1 ) \ dx#
# = x^2/2 tan^(-1) x - 1/2 int \ 1 - d/dx (tan^(-1) x) \ dx#
# = x^2/2 tan^(-1) x - 1/2x + 1/2 tan^(-1) x + C#
# =1/2( x^2 tan^(-1) x + tan^(-1) x - x)+ C#