We get the slope (m_t) of the tangent line from the value of the first derivative (df)/dx at the given x value. The slope of the line normal to that we get from noting that the slopes of two perpendicular lines multiply to -1 - assuming that neither line is vertical.
Firstly note that we may immediately simplify the given function - cosx has period 2pi, so cos2x has period pi, so cos(2x-pi)=cos2x and f(x)=secx+cos^2 2x.
Find the function derivative
Use the quotient rule for differentiation for the first term of f and the chain rule twice for the second term:
(df)/dx=secxtanx+2cos2x*(-sin2x)*2
(df)/dx=secxtanx-4sin2xcos2x
Recall the identity sin2x=2sinxcosx
(df)/dx=secxtanx-2sin4x
Evalute the derivative
Now evaluate (df)/dx at the given x:
(df)/dx((11pi)/8)=sec((11pi)/8)tan((11pi)/8)-2sin((11pi)/2)
Recall that cos(x+pi)=-cosx (so sec(x+pi)=-secx). Then sec((11pi)/8)=-sec((3pi)/8)
Similarly, recall that tan(x+pi)=tanx, so tan((11pi)/8)=tan((3pi)/8). If these aren't easy to recall, think of the graphs of the functions.
As sin(x+pi)=-sinx, sin((11pi)/2)=-sin(pi/2)=-1
So
(df)/dx((11pi)/8)=-sec((3pi)/8)tan((3pi)/8)+2
Now the trig values of the angle (3pi)/8 are not standard ones, but they can be calculated.
sin((3pi)/8)=sqrt(2+sqrt(2))/2, as derived here:
https://socratic.org/questions/how-do-you-use-the-half-angle-identity-to-find-exact-value-of-sin-3pi-8
cos((3pi)/8)=sqrt(2-sqrt(2))/2, as derived here:
https://socratic.org/questions/how-do-you-find-the-exact-values-of-cos-3pi-8-using-the-half-angle-formula
Thus
sec((3pi)/8)=1/cos((3pi)/8)=2/sqrt(2-sqrt(2))
tan((3pi)/8)=sin((3pi)/8)/cos((3pi)/8)=sqrt((2+sqrt(2))/(2-sqrt(2)))
So
(df)/dx((11pi)/8)=-sec((3pi)/8)tan((3pi)/8)+2=-2/sqrt(2-sqrt(2))sqrt((2+sqrt(2))/(2-sqrt(2)))+2
(df)/dx((11pi)/8)=2-(2sqrt(2+sqrt(2)))/(2-sqrt(2))=2/(2-sqrt(2))(2-sqrt(2)-sqrt(2+sqrt(2)))
This (complicated) value is the slope of the tangent line at the given point.
Calculate the normal slope
We now calculate the normal slope m_nvia m_nm_t=-1.
m_t=2-(2sqrt(2+sqrt(2)))/(2-sqrt(2))=2/(2-sqrt(2))(2-sqrt(2)-sqrt(2+sqrt(2)))
so
m_n=-1/m_t=-(2-sqrt(2))/(2[2-sqrt(2)-sqrt(2+sqrt(2))])
Numerically, we calculate to three significant figures that m_t=-4.31 and m_n=0.232.
Sanity check these answers by plotting the graphs of the function and these lines for the given x value
graph{(y-secx-(cos(2x))^2)(y+4.308644x-16.49888)(y-0.232x+3.115689)=0 [2, 6, -5, 0]}