6y^2-58xy-168x^2 = 2(3y+7x)(y-12x)
To find this, first notice the common factor 2 and factor it out
6y^2-58xy-168x^2 = 2(3y^2-29xy-84x^2)
If 3y^2-29xy-84x^2 has linear factors with integer coefficients then they must take the form (3y+ax)(y+bx) in order that the coefficient of y^2 is 3.
(3y+ax)(y+bx) = 3y^2 + (a+3b)xy + abx^2
Comparing coefficients of xy and x^2, we have:
a+3b=-29 and ab=-84
From a+3b=-29 we can tell that one of a and b is odd and the other even.
ab=-84 has factors 7 and 12, which are odd and even. So we can try +-7 and +-12 as values of a and b in some order.
We can quickly find that if a=7 and b=-12 then a+3b=-29.
Hence 3y^2-29xy-84x^2 = (3y+7x)(y-12x)
and 6y^2-58xy-168x^2 = 2(3y+7x)(y-12x).
