that's very ugly and would need a load of IBP so i would start by making it even bigger!! using #cos 2A = 2 cos^2 A - 1#
#I = int x^3 cos^2x \ dx#
#= int x^3 (cos 2x +1 )/2 \ dx#
#implies 2I = int x^3 cos 2x +x^3 \ dx#
#implies 2I - x^4/4 = int x^3 cos 2x \ dx qquad triangle# ......taking the easy bit to the LHS
so now we IBP the remaining bit over and over, reducing the #x^3# term each time
# int \ x^3 cos 2x \ dx = int \ x^3 d/dx (1/2 sin 2x) \ dx#
which by IBP....
#= x^3 /2 sin2x - int \ 3x^2 1/2 sin 2x \ dx#
#= x^3 /2 sin2x - (3)/2 int \ x^2 sin 2x \ dx#
#= x^3 /2 sin2x - (3)/2 int \ x^2 d/dx( - 1/2 cos 2x) \ dx#
#= x^3 /2 sin2x + (3)/4 int \ x^2 d/dx( cos 2x) \ dx#
#= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - int \ 2x cos 2x \ dx) #
and on we go!!
#= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - int \ 2x d/dx (1/2 sin 2x ) \ dx) #
#= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - int \ x d/dx ( sin 2x ) \ dx) #
#= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - (x sin 2x - int \ sin 2x \ dx)) #
#= x^3 /2 sin2x + (3)/4 ( x^2 cos 2x - (x sin 2x + 1/2 cos2x )) + C#
#= x^3 /2 sin2x + (3)/4 x^2 cos 2x - 3/4 x sin 2x - 3/8 cos2x + C#
so back to #triangle#
#implies 2I - x^4/4 = x^3 /2 sin2x + (3)/4 x^2 cos 2x - 3/4 x sin 2x - 3/8 cos2x + C#
#implies I = x^4/8 + x^3 /4 sin2x + (3)/8 x^2 cos 2x - 3/8 x sin 2x - 3/16 cos2x + C#
once again , the key is IBP:
#int \ u(x) * d/dx(v(x)) \ dx = u(x) * v(x) - int \ d/dx(u(x)) * v(x) \ dx#
