It works for some polynomials but not for others. Mostly , it works for this polynomial because the teacher, or author, or test-maker, chose a polynomial that could be factored this way.
Example 1
Factor: 3x^3+6x^3-5x-10
I group the first two terms and take out any common factor of those two:
(3x^3+6x^2)-5x-10=3x^2(x+2) -5x-10
Now I'll take out any common factors in the other two terms. If i get a monomial times (x+2) then factoring by grouping will work. If I get something else, it won't work.
Ther common factor of (-5x-10) is -5. Taking that factor out leaves -5(x+2) so we know factoring by grouping will work.
3x^3+6x^2-5x-10 = (3x^3+6x^2)+(-5x-10)
=3x^2(x+2)-5(x+2).
Now we have two terms with a common factor C where C=(x-2). So we have 3x^2C-5C=(3x-5)C
That is: we have (3x^2-5)(x+2)
We'll stop there if we're only willing to use integer (or rational) coefficients.
Example 2
Factor: 4x^3-10x^2+3x+15
4x^3-10x^2+3x+15=(4x^3-10x^2)+6x+15
=2x^2(2x-5)+6x+15
Now if we take a common factor out of 6x+15 and get a monomial times (2x-5), then we can finish factoring by grouping. If we get something else, then factoring by grouping won't work.
In this case we get 6x+15=3(2x+5). Almost!, But close doesn't work in factoring by grouping. So we can't finish this by grouping.
Example 3 You do the test-maker's job.
I want a problem that CAN be factored by grouping.
I start with 12x^3-28x^2 So, if it CAN be factored by grouping, the the rest of is has to look like what?
It has to be a monomial times (3x-7).
So finishing with 6x-14 would work, or 15x-35, or I could get tricky and use -9x+21. In fact ANY number times (3x-7) added to what I already have will give me a polynomial that can be factored by grouping.
12x^3-28x^2+k3x-k7 for any k can be factored as:
12x^3-28x^2+3kx-7k=4x^2(3x-7)+k(3x-7)=(4x^2+k)(3x-7)
Final note: k=-1 or k=-9 would make good choices. Because then the fisrt factor is a difference of 2 squares and can be factored.