Binomial expansion of #(1+ax)^n# is
#1+^nC_1ax+^nC_2(ax)^2+^nC_3(ax)^3+........+^nC_n(ax)^n#,
where #color(white)(x)^nC_r=(n!)/(r!(n-r)!)# or #(n(n-1)(n-2)....(n-r+1))/(1xx2xx3xx...xxr)# and #a# is a constant.
As coefficient of #x^3# is six times the coefficient of #x^2#
#color(white)(x)^nC_3xxa^3=6xx^nC_2xxa^2#
i.e. #(n(n-1)(n-2))/(1xx2xx3)xxa=6xx(n(n-1))/(1xx2)#
or #(n-2)/3xxa=6# i.e. #a(n-2)=18# .................(1)
Further, coefficient of #x# is #12#, we have
#color(white)(x)^nC_1xxa=12# i.e. #axxn=12# .................(2)
Putting (2) in (1) we get #12-2a=18#
Hence, #a=(18-12)/(-2)=-3#
and #n=12/-3=-4#
and hence expansion is #(1-3x)^(-4)#
= #1+(-4)/1(-3x)+((-4)(-5))/2(-3x)^2+((-4)(-5)(-6))/6(-3x)^3+................#
= #1+12x+90x^2+540x^3+..........#
