The characteristic equation is:
#D^3+3D^2+3D+1 = 0#
Or:
#(D+1)^3 = 0 implies D = -1#
Repeated eigenvalues means that we have a complementary solution in the form:
#y_c = e^(-x) (Ax^2 + Bx + C)#
For the particular solution, we use trial solution:
#y_p = e^(-x) rho(x)# where #rho# is a polynomial in #x#
#Dy_p = - e^(-x) rho + e^(-x) rho' = e^(-x) (- rho + rho')#
#D^2y_p' = - e^(-x) (- rho + rho') + e^(-x) (- rho' + rho'') = e^(-x) (rho - 2 rho' + rho'')#
#D^3y_p = - e^(-x) (rho - 2 rho' + rho'') + e^(-x) (rho' - 2 rho'' + rho''') = e^(-x) ( - rho + 3 rho' - 3 rho'' + rho''' )#
And adding it all up means #(D^3+3D^2+3D+1)y_p= x^2 e^-x# amounts to #e^(-x) rho''' = x^2 e^-x# so:
# rho''' = x^2#
#implies rho'' = 1/3x^3 + beta#
#implies rho' = 1/12 x^4 + betax + gamma #
#implies rho = 1/50 x^5 + beta/2x^2 +gammax + delta#
The #beta, gamma, delta# terms are already in the complementary solution so we are left with:
#y = y_c + y_p = e^(-x)(x^5/60 + Ax^2 + Bx + C)#