The differential equation is linear non homogeneous then the solution can be composed as
#y = y_h+y_p#
#y_h# is the homogeneous solution and is easily obtained. This solution is
#y_h=C_1e^{2x}cos(2x)+C_2e^{2x}sin(2x)# as already established
Obtaining the particular solution is a little more involved.
We propose #y_p# as
#y_p = (a_1 x^3 + b_1 x^2 + c_1 x + d_1) e^(2 x) cos(
2 x) + (a_2 x^3 + b_2 x^2 + c_2 x + d_2) e^(2 x) sin(2 x)#
Here #y_p# must obey
#y_p''-4y_p'+8y_p-((2x^2-3x)e^{2x}cos(2x)+(10x^2-x-1)e^{2x}sin(2x))=0#
After substituting and grouping we get
#e^(2 x) ((2 b_1 + 4 c_2 + (3 + 6 a_1 + 8 b_2)x+( 12 a_2-2) x^2) cos(
2 x) + (1 + 2 b_2 - 4 c_1 + (1 + 6 a_2 - 8 b_1)x - 2 (5 + 6 a_1) x^2)) sin(2 x)) = 0#
Now choosing #a_1,b_1,c_1,a_2,b_2,c_2# such that
#2 b_1 + 4 c_2 + (3 + 6 a_1 + 8 b_2)x +(12 a_2-2)x^2 =0# and
# 1 + 2 b_2 - 4 c_1 + (1 + 6 a_2 - 8 b_1)x - 2 (5 + 6 a_1) x^2 = 0#
by solving
#{(2 b_1 + 4 c_2 = 0), (3 + 6 a_1 + 8 b_2 = 0), (12 a_2-2 = 0),
(1 + 2 b_2 - 4 c_1 = 0), (1 + 6 a_2 - 8 b_1 = 0), ( 5 + 6 a_1 = 0):}#
we obtain
#a_1=-5/6,b_1=1/4,c_1=3/8,a_2=1/6,b_2=1/4,c_2=-1/8#
so the particular solution is
#y_p=e^(2 x) ( (3 x)/8 + x^2/4 - (5 x^3)/6) cos(2 x) + e^(2 x) (- x/8 + x^2/4 + x^3/6) sin(2 x)#
