#6y^2-58xy-168x^2 = 2(3y+7x)(y-12x)#
To find this, first notice the common factor 2 and factor it out
#6y^2-58xy-168x^2 = 2(3y^2-29xy-84x^2)#
If #3y^2-29xy-84x^2# has linear factors with integer coefficients then they must take the form #(3y+ax)(y+bx)# in order that the coefficient of #y^2# is #3#.
#(3y+ax)(y+bx) = 3y^2 + (a+3b)xy + abx^2#
Comparing coefficients of #xy# and #x^2#, we have:
#a+3b=-29# and #ab=-84#
From #a+3b=-29# we can tell that one of #a# and #b# is odd and the other even.
#ab=-84# has factors #7# and #12#, which are odd and even. So we can try #+-7# and #+-12# as values of #a# and #b# in some order.
We can quickly find that if #a=7# and #b=-12# then #a+3b=-29#.
Hence #3y^2-29xy-84x^2 = (3y+7x)(y-12x)#
and #6y^2-58xy-168x^2 = 2(3y+7x)(y-12x)#.
