We have:
P(x) = sum_(n=0)^6 c_nx^n = (1+3x)^6(1-3x-5x^2)P(x)=6∑n=0cnxn=(1+3x)6(1−3x−5x2)
Develop (1+3x)^6(1+3x)6 using binomial coefficients:
sum_(n=0)^6 c_nx^n = (1-3x-5x^2)sum_(n=0)^6 ((6),(n))(3x)^n
Now using the distributive property of multiplication:
sum_(n=0)^6 c_nx^n = sum_(n=0)^6 ((6),(n))3^nx^n + sum_(n=0)^6 ((6),(n))(-3x)3^nx^n +sum_(n=0)^6 ((6),(n))(-5x^2)3^nx^n
sum_(n=0)^6 c_nx^n = sum_(n=0)^6 ((6),(n))3^nx^n - sum_(n=0)^6 ((6),(n))3^(n+1)x^(n+1) -5sum_(n=0)^6 ((6),(n))3^nx^(n+2)
The coefficient of x^2 is then the sum of the term for n=2 in the first sum, for n=1 in the second sum and for n=0 in the third sum:
c_2 = ((6),(2)) * 3^2 - ((6),(1)) * 3^2 -5*((6),(0))
The general expression of the binomial coefficient is:
((n),(k)) = (n!)/(k!(n-k)!)
So:
((6),(2)) = (6!)/((2!)(4!)) = 15
((6),(1)) = (6!)/((1!)(5!)) = 6
((6),(0)) = (6!)/((0!)(6!)) = 1
and:
c_2 = 15 * 9 - 6 * 9 -5 = 76