You are looking for:
#langle y^2 sin y + 3 x^2 , xy(2 sin y + y cos y) rangle_( \ (1,pi) ) * langle 1, -1 rangle /sqrt2#
#= langle 3 , - pi^2 rangle * langle 1, -1 rangle /sqrt2#
#= ( 3 + pi^2) /sqrt2#
For a function #phi#, the largest value of this derivative will, by definition of the inner product, always be in the direction of the gradient itself. This is because the quantity #nabla phi * mathbf hat u = abs( nabla phi ) cos alpha# peaks at #alpha = 0, pi,...#
Here that unit vector is:
#mathbf hat u = langle y^2 sin y + 3 x^2 , xy(2 sin y + y cos y) rangle_( \ (1,pi) )/ abs( langle y^2 sin y + 3 x^2 , xy(2 sin y + y cos y) rangle_( \ (1,pi) )) #
With max value:
- #abs( nabla phi ) = abs( langle 3 , - pi^2 rangle) = sqrt ( 9 + pi^4) #
