We get the slope (#m_t#) of the tangent line from the value of the first derivative #(df)/dx# at the given #x# value. The slope of the line normal to that we get from noting that the slopes of two perpendicular lines multiply to -1 - assuming that neither line is vertical.
Firstly note that we may immediately simplify the given function - #cosx# has period #2pi#, so #cos2x# has period #pi#, so #cos(2x-pi)=cos2x# and #f(x)=secx+cos^2 2x#.
Find the function derivative
Use the quotient rule for differentiation for the first term of #f# and the chain rule twice for the second term:
#(df)/dx=secxtanx+2cos2x*(-sin2x)*2#
#(df)/dx=secxtanx-4sin2xcos2x#
Recall the identity #sin2x=2sinxcosx#
#(df)/dx=secxtanx-2sin4x#
Evalute the derivative
Now evaluate #(df)/dx# at the given #x#:
#(df)/dx((11pi)/8)=sec((11pi)/8)tan((11pi)/8)-2sin((11pi)/2)#
Recall that #cos(x+pi)=-cosx# (so #sec(x+pi)=-secx#). Then #sec((11pi)/8)=-sec((3pi)/8)#
Similarly, recall that #tan(x+pi)=tanx#, so #tan((11pi)/8)=tan((3pi)/8)#. If these aren't easy to recall, think of the graphs of the functions.
As #sin(x+pi)=-sinx#, #sin((11pi)/2)=-sin(pi/2)=-1#
So
#(df)/dx((11pi)/8)=-sec((3pi)/8)tan((3pi)/8)+2#
Now the trig values of the angle #(3pi)/8# are not standard ones, but they can be calculated.
#sin((3pi)/8)=sqrt(2+sqrt(2))/2#, as derived here:
https://socratic.org/questions/how-do-you-use-the-half-angle-identity-to-find-exact-value-of-sin-3pi-8
#cos((3pi)/8)=sqrt(2-sqrt(2))/2#, as derived here:
https://socratic.org/questions/how-do-you-find-the-exact-values-of-cos-3pi-8-using-the-half-angle-formula
Thus
#sec((3pi)/8)=1/cos((3pi)/8)=2/sqrt(2-sqrt(2))#
#tan((3pi)/8)=sin((3pi)/8)/cos((3pi)/8)=sqrt((2+sqrt(2))/(2-sqrt(2)))#
So
#(df)/dx((11pi)/8)=-sec((3pi)/8)tan((3pi)/8)+2=-2/sqrt(2-sqrt(2))sqrt((2+sqrt(2))/(2-sqrt(2)))+2#
#(df)/dx((11pi)/8)=2-(2sqrt(2+sqrt(2)))/(2-sqrt(2))=2/(2-sqrt(2))(2-sqrt(2)-sqrt(2+sqrt(2)))#
This (complicated) value is the slope of the tangent line at the given point.
Calculate the normal slope
We now calculate the normal slope #m_n#via #m_nm_t=-1#.
#m_t=2-(2sqrt(2+sqrt(2)))/(2-sqrt(2))=2/(2-sqrt(2))(2-sqrt(2)-sqrt(2+sqrt(2)))#
so
#m_n=-1/m_t=-(2-sqrt(2))/(2[2-sqrt(2)-sqrt(2+sqrt(2))])#
Numerically, we calculate to three significant figures that #m_t=-4.31# and #m_n=0.232#.
Sanity check these answers by plotting the graphs of the function and these lines for the given #x# value
graph{(y-secx-(cos(2x))^2)(y+4.308644x-16.49888)(y-0.232x+3.115689)=0 [2, 6, -5, 0]}
