When looking for the concavity of a function, it's best to find the second derivative of the function. The second derivative, #f''(x)#, tells us the concavity of the function, #f(x)#:
When #f''(x)<0#, #f(x)# is concave down (concave)
When #f''(x)>0#, #f(x)# is concave up (convex)
The first derivative of this function is:
#f'(x)# = #2x^3-18x^2-96x-96#.
The second derivative is:
#f''(x)# = #6x^2-36x-96#.
Then set the second derivative equal to 0, to get:
#6x^2-36x-96 = 0#
#x^2-6x-16=0#
#(x-8)(x+2) = 0#
#x= -2, 8#
This tells us that the concavity of #f(x)# changes at #x= -2# and #x= 8#. When #x<-2#, the #f''(x)# is positive. When #x> -2# and #x<8# (i.e. when #x# is between #-2# and #8#) #f''(x)# is negative. When #x>8#, #f''(x)# is positive. It can help to look at a graph:
graph{x^2-6x-16 [-8.67, 11.33, -5.04, 4.96]}
#f(x)# is convex when #f''(x)# is positive, which is when #x# is #(-oo, 2)uu(8, oo)#.
#f(x)# is concave when #f''(x)# is negative, which is when #x# is #(-2,8)#.
