We have
#cotx=9tanx#
However, we know that #cotx=1/tanx#, thus:
#1/tanx=9tanx#
#tan^2x=1/9=>tanx=+-1/3#
We also know that #x in ((3pi)/2,2pi)#, in the 4th quadrant. Now, imagine the unit circle;
In the 4th quadrant, #sin# is negative while #cos# is positive, and #tan# then has to be negative.
#:. tanx=-1/3#
#=>cotx = 1/tanx=-3#
#color(blue)(cot^2x=9#
We have to find #sinx# and #cosx# now. We know that
#tanx=sinx/cosx#
But since #cosx=sqrt(1-sin^2x)# we can write #tanx# just in terms of #sinx# and find its value:
#tanx=sinx/sqrt(1-sin^2x)#
Square both sides.
#tan^2x=sin^2x/(1-sin^2x)#
#tan^2x-tan^2xsin^2x=sin^2x#
Add #tan^2xsin^2x# to both.
#tan^2x=sin^2x(1+tan^2x)#
#=> sinx= tanx/(+-sqrt(1+tan^2x)#
In our case, tangent is negative and sine must also be negative, meaning that we have to take the positive root of #1+tan^2x#.
By substituting #tanx=-1/3# into the equality we get:
#sinx=(color(red)(-1/3))/sqrt(1+(color(red)(-1/3))^2)#
#color(blue)(sinx)=-1/(3sqrt(10/9))=color(blue)(-1/sqrt10#
#:.color(blue)(cosx)=+sqrt(1-1/10)=color(blue)(3/sqrt10#
Again, cosine is positive in #Q_4#.
Finally, we have:
#cot^2x-sinxcosx= 9-(-1/sqrt10)*3/sqrt10=9+3/10#
#color(red)(cot^2x-sinxcosx= 93/10#
