#f'(x) = cos( x² ln(x) ) * ( 2x ln(x) + x ) #
Let's break this into pieces.
#f'(x) = u*v#
Where
#u = cos(x^2lnx)#, so
#u' = -sin(x^2lnx)*d/dx(x^2lnx) = -sin(x^2lnx)(2xlnx+x)#
#v = 2xlnx+x#, so
#v' = 2lnx+2+1 = 2lnx+3#
#f''(x) = u'v+uv'#
# = underbrace(-sin(x^2lnx)(2xlnx+x))_(u') underbrace((2xlnx+x))_v + underbrace(cos(x^2lnx))_u underbrace((2lnx+3))_(v')#
# = -sin(x^2lnx)(2xlnx+x)^2+(2xlnx+x)(2lnx+3)#
This can be rewritten in several ways using algebra.
Note
The much simpler question is to find the antiderivative, that is, to integrate #f'(x) = cos( x² ln(x) ) * ( 2x ln(x) + x ) #
Because # ( 2x ln(x) + x ) # is thederivative of the argument of #cos#, we see that this is #f'(x) = cosu (du)/dx#
So, #f(x) = sinu +C = sin(x^2lnx) +C#