All the terms are multiples of #6y#, so we find:
#6x^2y+36xy-66y = 6y(x^2+6x-11)#
#x^2+6x-11# is of the form #ax^2+bx+c# with #a=1#, #b=6# and #c=-11#, which has discriminant given by the formula:
#Delta = b^2-4ac = 6^2-(4xx1xx-11) = 36+44 = 80#
This is positive, but not a perfect square. So the zeros of #x^2+6x-11# are not rational and its linear factors will have irrational coefficients.
The roots of #x^2+6x-11 = 0# are given by the formula:
#x=(-b+-sqrt(Delta))/(2a) = (-6+-sqrt(80))/2#
#=(-6+-4sqrt(5))/2#
#=-3+-2sqrt5#
This gives us factors #(x+3+2sqrt(5))# and #(x+3-2sqrt(5))#
So the full factorization (if we allow irrational coefficients) is:
#6x^2y+36xy-66y = 6y(x+3+2sqrt(5))(x+3-2sqrt(5))#
