When looking for the concavity of a function, it's best to find the second derivative of the function. The second derivative, #f''(x)#, tells us the concavity of the function, #f(x)#:

When #f''(x)<0#, #f(x)# is concave down (concave)

When #f''(x)>0#, #f(x)# is concave up (convex)

The first derivative of this function is:

#f'(x)# = #2x^3-18x^2-96x-96#.

The second derivative is:

#f''(x)# = #6x^2-36x-96#.

Then set the second derivative equal to 0, to get:

#6x^2-36x-96 = 0#

#x^2-6x-16=0#

#(x-8)(x+2) = 0#

#x= -2, 8#

This tells us that the concavity of #f(x)# changes at #x= -2# and #x= 8#. When #x<-2#, the #f''(x)# is positive. When #x> -2# and #x<8# (i.e. when #x# is between #-2# and #8#) #f''(x)# is negative. When #x>8#, #f''(x)# is positive. It can help to look at a graph:

graph{x^2-6x-16 [-8.67, 11.33, -5.04, 4.96]}

#f(x)# is convex when #f''(x)# is positive, which is when #x# is #(-oo, 2)uu(8, oo)#.

#f(x)# is concave when #f''(x)# is negative, which is when #x# is #(-2,8)#.