Here's an incomplete list of pointers:
(1) Do all the terms have a common factor?
If so, separate out the common factor first...
2x^2+8x+8 = 2(x^2+4x+4)
12x^2y - 4xy^2 + 8xy = 4xy(3x-y+2)
(2) Do all the coefficients add up to 0?
e.g. f(x) = x^2+3x-4
Here f(1) = 1+3-4 = 0. Hence (x-1) is a factor of f(x)
(3) Do the coefficients of the even degree terms minus the coefficients of the odd degree terms add up to 0?
e.g. f(x) = x^2-3x-4
Here f(-1) = 1-(-3)-4 = 0. Hence (x+1) is a factor of f(x)
(4) Is the polynomial a difference or sum of squares or cubes?
a^2 - b^2 = (a-b)(a+b)
e.g. 4x^2-9 = (2x-3)(2x+3)
a^3 - b^3 = (a-b)(a^2+ab+b^2)
e.g. 8x^6-27 = (2x^2-3)(4x^4+6x^2+9)
a^3 + b^3 = (a+b)(a^2-ab+b^2)
Note that a^2 + b^2 is only reducible to linear factors with complex coefficients.
a^2 + b^2 = (a-ib)(a+ib)
(5) Is the polynomial recognisable as a perfect square trinomial?
(a+b)^2 = a^2+2ab+b^2
(a-b)^2 = a^2-2ab+b^2
e.g. 4x^2-12x+9 = (2x-3)^2
(6) Is the polynomial easily factorisable as (x+a)(x-b) or similar?
(x+a)(x-b) = x^2+(a-b)x-ab
e.g. x^2+x-12 = (x+4)(x-3) based on the observation that 4*3 = 12 and 4-3 = 1
(7) Check the discriminant for a quadratic
For a quadratic in the form f(x) = ax^2+bx+c, check the value of the discriminant Delta = b^2-4ac.
If Delta > 0 and is a perfect square, then it may be simplest to use some version of the AC Method or similar to factor the quadratic as it does have rational factors.
If Delta > 0 is not a perfect square, then use the quadratic formula to find the roots x_1 and x_2 of f(x) = 0 and hence factor f(x) as a(x-x_1)(x-x_2)
If Delta = 0, you have a perfect square trinomial. Look at (5) above.
If Delta < 0, f(x) has no linear factors with real coefficients. If you allow complex coefficients you can still use the quadratic formula, etc, but the result will be linear factors with complex coefficients.
