A conic equation of the type of #Ax^2+Bxy+Cy^2+Dx+Ey+F=0# is rotated by an angle #theta#, to form a new Cartesian plane with coordinates #(x',y')#, if #theta# is appropriately chosen, we can have a new equation without term #xy# i.e. of standard form.
The relation between coordinates #(x,y)# and #(x'.y')# can be expressed as
#x=x'costheta-y'sintheta# and #y=x'sintheta+y'costheta#
or #x'=xcostheta+ysintheta# and #y=-xsintheta+ycostheta#
for this we need to have #theta# given by #cot2theta=(A-C)/B#
In the given case we have #x^2+6xy+y^2-8=0#, we have #A=C=1# and #B=6# and hence #cot2theta=0# i.e. #theta=pi/4#
Hence relation is give by #x=x'cos(pi/4)-y'sin(pi/4)# and #y=x'sin(pi/4)+y'cos(pi/4)# i.e.
#x=(x')/sqrt2-(y')/sqrt2# and #y=(x')/sqrt2+(y')/sqrt2#
Hence, we get #((x')/sqrt2-(y')/sqrt2)^2+6((x')/sqrt2-(y')/sqrt2)((x')/sqrt2+(y')/sqrt2)+((x')/sqrt2+(y')/sqrt2)^2-8=0#
or #((x'^2)/2+(y'^2)/2-x'y')+6((x'^2)/2-(y'^2)/2)+((x'^2)/2+(y'^2)/2+x'y')-8=0#
or #x'^2+y'^2+3x'^2-3y'^2-8=0# which can be reduced to
#4x'^2-2y'^2-8=0#
or #(x'^2)/2-(y'^2)/4=1#
i.e. #x^2/2-y^2/4=1#
Hence, #x^2+6xy+y^2-8=0# is a hyperbola.
Graph of #x^2+6xy+y^2-8=0# appears as
graph{x^2+6xy+y^2-8=0 [-10, 10, -5, 5]}
and when rotated clockwise it appears as
graph{x^2/2-y^2/4=1 [-10, 10, -5, 5]}
