#9a^2+12ab-5b^2 = (3a+5b)(3a-b)#
You can tell from the negative sign on the #b^2# term that the signs in the linear factors must be different from one another - one #+# and one #-#. Also #5# only has divisors #+-1# and #+-5# so the factors would either be of the form #(ma+5b)(na-b)# or #(ma-5b)(na+b)# for some #m# and #n# such that #mn = 9#. I suspected that #m = n = 3# so tried those possibilities first, quickly finding the factorisation.