All the terms are multiples of #6y#, so we find:

#6x^2y+36xy-66y = 6y(x^2+6x-11)#

#x^2+6x-11# is of the form #ax^2+bx+c# with #a=1#, #b=6# and #c=-11#, which has discriminant given by the formula:

#Delta = b^2-4ac = 6^2-(4xx1xx-11) = 36+44 = 80#

This is positive, but not a perfect square. So the zeros of #x^2+6x-11# are not rational and its linear factors will have irrational coefficients.

The roots of #x^2+6x-11 = 0# are given by the formula:

#x=(-b+-sqrt(Delta))/(2a) = (-6+-sqrt(80))/2#

#=(-6+-4sqrt(5))/2#

#=-3+-2sqrt5#

This gives us factors #(x+3+2sqrt(5))# and #(x+3-2sqrt(5))#

So the full factorization (if we allow irrational coefficients) is:

#6x^2y+36xy-66y = 6y(x+3+2sqrt(5))(x+3-2sqrt(5))#