I will start this problem out using deduction.
If the coefficient of #x^2# is #324# then this must be the 3rd term (because we always include #t_0#).
The nth term in the binomial expansion is given by #t_(k + 1) = color(white)(two)_nC_k(x)^(n - k)y^k# in #(x + y)^n#. We know the value of #t_3#, and we know that #k =2#.
#324x^2 = color(white)(two)_nC_2(1)^(n - 2) xx (3x)^2#
#324x^2 = color(white)(two)_nC_2(1)^(n - 2) xx 9x^2#
#36 = color(white)(two)_nC_2(1)^(n - 2)#
However, since #1^n#, no matter the value of #n# always equals #1#, we can get rid of that part of the equation completely.
We are left with:
#36 = color(white)(two)_nC_2#
Expanding:
#36 = (n!)/((n - 2)!2!)#
Eliminate the factorial:
#36 = (n(n - 1)(n - 2)!)/((n - 2)!2!)#
#36 = (n(n - 1))/2#
#72 = n^2 - n#
#0 = n^2 - n - 72#
#0 = (n - 9)(n + 8)#
#n = 9 or -8#
A negative answer is not possible, so #n = 9#.
Hopefully this helps!