The roots are those numbers that assigned to the #x# give as result zero. Because in this equation you have the multiplication of two quantities #(x+1)# and #(x-4)# it is enough that one is zero to have the roots.
#(x+1)# is equal to zero when #x=-1# and #(x-4)# is equal to zero when #x=4#. So your roots are #-1, 4#.
But you ask about to apply the quadratic formula, To do that you have to multyply the two terms together:
#(x+1)(x-4)=x^2-4x+x-4=x^2-3x-4#.
Now we have the equation in the format
#ax^2+bx+c# where #a=1#, #b=-3# and #c=-4#.
The formula is: #x=(-b\pm sqrt(b^2-4ac))/(2a)# where you can substitute your numbers
#x=(3 \pm sqrt((-3)^2-4*1*(-4)))/(2*1)=(3\pm sqrt(9+16))/2=(3\pm 5)/2#.
The solutin with the #+# gives #4#, the one with the #-# gives #-1# as we expected from the initial consideration.
