The #angle (3pi)/4# is in the second quadrant and the cosine function is negative in the second quadrant.
Here is an image of the unit circle with the #angle (3pi)/4#:
Let's use the identity:
#cos(A-B) = cos(A)cos(B) + sin(A)sin(B)#
Let's insist that #A-B = (3pi)/4#
Nice values for A and B are #A = pi# and #B = pi/4#
Substitute into the identity:
#cos(pi-pi/4) = cos(pi)cos(pi/4) + sin(pi)sin(pi/4)#
We know that the #sin(pi) = 0#, therefore, the second term is 0:
#cos(pi-pi/4) = cos(pi)cos(pi/4)#
We know that #cos(pi) = -1, therefore, we can replace it with a minus sign:
#cos(pi-pi/4) = -cos(pi/4)#
We know that #cos(pi/4) = sqrt2/2#
#cos(pi-pi/4) = -sqrt2/2#
Finally #pi- pi/4 = (3pi)/4#
#cos((3pi)/4) = -sqrt2/2#