Let's put the equation in a better form before we to do the substitutions:
#xy = y^2 + x(xy - 5y - 7x + 35)#
#xy = y^2 + x^2y - 5xy - 7x^2 + 35x#
Substitute #rcos(theta)# for x and #rsin(theta)# for y:
#r^2cos(theta)sin(theta) = r^2sin^2(theta) + r^3cos^2(theta)sin(theta) - 5r^2cos(theta)sin(theta) - 7r^2cos^2(theta) + 35rcos(theta)#
Divide both sides by r:
#rcos(theta)sin(theta) = rsin^2(theta) + r^2cos^2(theta)sin(theta) - 5rcos(theta)sin(theta) - 7rcos^2(theta) + 35cos(theta)#
Put in standard quadratic form:
#cos^2(theta)sin(theta)r^2 + (sin^2(theta) -6cos(theta)sin(theta)- 7cos^2(theta))r + 35cos(theta) = 0#
The middle term factors:
#cos^2(theta)sin(theta)r^2 + [{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}]r + 35cos(theta) = 0#
This is a quadratic of the form #ar^2 + br + c# where:
#a = cos^2(theta)sin(theta)#
#b = [{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}]#
#c = 35cos(theta)#
#r = (-b +-sqrt(b^2 - 4(a)(c)))/(2a)#
#r = ((-[{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}] +-sqrt([{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}]^2 - 140cos^3(theta)sin(theta)))/(2cos^2(theta)sin(theta)))#
