Since the coordinates are polar coordinates, we imagine a triangle #OAB# with #O# at the origin, #A# at #(-6,\pi/2)# and #B# at #(5,\pi/6)#. We will use the Law of Cosines to get the length of #AB#.
First convert side #OA# to a positive length by writing its coordinates as #(+6,-\pi/2)#, changing the sign of the radius and compensating by subtracting #\pi# from the angle.
So #OA=6# and #OB=5#. Next we need angle #O# which is the difference between the angular coordinates after we have made the radial coordinates positive (see above). Thus
#\pi/2-(-\pi/6)=2\pi/3#.
Now apply the Law of Cosines:
#(AB)^2=(OA)^2+(OB)^2-2(OA)(OB)\cos(angle O)#
#=6^2+5^2-2(6)(5) \cos(2\pi/3)#
As #\cos(2\pi/3)=-1/2# this gives #(AB)^2=91#.
