#int_(pi/8)^((11pi)/12)dx \ (x^2-cos^2x+xsinx)#
using identity # cos 2 z = 2 cos^2 z - 1 \implies cos ^2 z = (cos 2z + 1)/(2)#
#= int_(pi/8)^((11pi)/12)dx \ (x^2-(cos 2x + 1)/(2)+ color{red}{xsinx})#
first 2 terms are trivial, we do the term in red using IBP, ie
#int dx \ (u v') = uv - int dx \ (u' v)#
Here
#u = x, u' = 1#
#v' = sin x, v = -cos x#
so we have
#- x cos x - int dx \ (-cos x)#
#color{red}{= - x cos x + intdx \ (cos x)}#
so the integral becomes
#= [-x cos x] \_(pi/8)^((11pi)/12) + int_(pi/8)^((11pi)/12)dx \ (x^2-(cos 2x + 1)/(2)+ cos x) #
#= [-x cos x + x^3/3-(sin 2x)/4 - x/2+ sin x ] \_(pi/8)^((11pi)/12)#
crunch it in calculator ....
#approx 10#