We can convert each to Cartesian:
#(3, (3pi)/2) rightarrow 3e^(i(3pi)/2) = 3(-i) = -3i #
#(9, (5pi)/8) rightarrow 9e^(i(5pi)/8) = -(9sqrt(2-sqrt(2)))/2 + i (9sqrt(2+sqrt(2)))/2 #
In order to do the trig functions for #(5pi)/8#, I used the half angle formulae and the angle addition formulae (just to be careful about signs):
#(5pi)/8 = pi/2 + pi/8 #
#cos(pi/8) = cos(1/2 (pi/4)) = sqrt( (1+cos(pi/4)) /2) = sqrt( 2 + sqrt(2))/2 #
#sin(pi/8) = sin(1/2 (pi/4)) = sqrt( (1-cos(pi/4)) /2) = sqrt( 2 - sqrt(2))/2 #
#cos(pi/2 + pi/8) = - sin(pi/8) = -sqrt( 2- sqrt(2))/2 #
#sin(pi/2 + pi/8) = cos(pi/8) = sqrt(2 + sqrt(2))/2 #
Now, we can just apply Pythagorean theorem:
#d^2 = Delta x^2 + Delta y^2 #
#= 81/4 * (2 - sqrt(2)) + ((9sqrt(2+sqrt2))/2 + 3)^2 #
#= 81/4 * (2 - sqrt(2)) + 81/4 * (2+sqrt2) + 9 + 2 * 3 * 9/2 *sqrt(2+sqrt2) #
#= 81/2 + 18/2 + 27sqrt(2+sqrt2) #
#d^2= 99/2 + 27sqrt(2+sqrt2) #
Which means that the distance is actually
#d = sqrt(99/2 + 27sqrt(2+sqrt2)) approx 9.97 #