With the rotation

#R_(theta)=((ctheta,-stheta),(stheta,ctheta))#, calling #p = ((x),(y))# and #P=((X),(Y))# we have first

#p^T M p + 10 = 0# representing the conic, with #M = ((2,-3/2),(-3/2,-2))#. Here #.^T# represents transposition. Now introducing a change of coordinates

#R_(theta)P = p# we will have

#P^TR_(theta)^TMR_(theta)P + 10 = 0#

Calling #M_(theta) = R_(theta)^TMR_(theta)=((2 Cos(2 theta) - 3 Cos(theta) Sin(theta), -3/2Cos(2 theta) -
2 Sin(2 theta)),(-3/2 Cos(2 theta) - 2 Sin(2 theta), -2 Cos(2 theta) +
3 Cos(theta) Sin(theta)))#

Now, choosing #theta# such that #-3/2 Cos(2 theta) - 2 Sin(2 theta)=0# we obtain

#tan(2theta)=-3/4# and

#theta = 1/2(-0.643501pmkpi)# Choosing for #k=0# we have

#theta =-0.321751# and

#M_(theta) = ((5/2,0),(0,-5/2))# and in the new coordinates, the conic reads

#5/2X^2-5/2Y^2+10=0# or

#X^2-Y^2+4=0# which is a hyperbola.