Let's put the equation in a better form before we to do the substitutions:
xy = y^2 + x(xy - 5y - 7x + 35)
xy = y^2 + x^2y - 5xy - 7x^2 + 35x
Substitute rcos(theta) for x and rsin(theta) for y:
r^2cos(theta)sin(theta) = r^2sin^2(theta) + r^3cos^2(theta)sin(theta) - 5r^2cos(theta)sin(theta) - 7r^2cos^2(theta) + 35rcos(theta)
Divide both sides by r:
rcos(theta)sin(theta) = rsin^2(theta) + r^2cos^2(theta)sin(theta) - 5rcos(theta)sin(theta) - 7rcos^2(theta) + 35cos(theta)
Put in standard quadratic form:
cos^2(theta)sin(theta)r^2 + (sin^2(theta) -6cos(theta)sin(theta)- 7cos^2(theta))r + 35cos(theta) = 0
The middle term factors:
cos^2(theta)sin(theta)r^2 + [{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}]r + 35cos(theta) = 0
This is a quadratic of the form ar^2 + br + c where:
a = cos^2(theta)sin(theta)
b = [{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}]
c = 35cos(theta)
r = (-b +-sqrt(b^2 - 4(a)(c)))/(2a)
r = ((-[{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}] +-sqrt([{sin(theta) + cos(theta)}{sin(theta)- 7cos(theta)}]^2 - 140cos^3(theta)sin(theta)))/(2cos^2(theta)sin(theta)))
