Complete the square at the denominator:
int dx/(5-4x-x^2)^(5/2) = int dx/(9- (2+x)^2)^(5/2)
int dx/(5-4x-x^2)^(5/2) = 1/3^5 int dx/(1- ((2+x)/3)^2)^(5/2)
Subtitute:
(2+x)/3 = sint
with t in (-pi/2,pi/2)
so that in the interval cost is positive and:
x = -2+3sint
dx = 3cost dt
Then:
int dx/(5-4x-x^2)^(5/2) = 1/3^4 int (costdt)/(1- sin^2t)^(5/2)
and as:
(1-sin^2t)^(5/2) = (sqrt(1-sin^2t))^5 = cos^5t
we have:
int dx/(5-4x-x^2)^(5/2) = 1/3^4 int dt/cos^4t = 1/3^4 int sec^4t dt
Solve the resulting integral using the identity sec^2t = 1+tan^2t:
int sec^4t dt = int sec^2t * sec^2t dt
int sec^4t dt = int sec^2t (1+tan^2t) dt
int sec^4t dt = int sec^2t dt +int tan^2tsec^2t dt
int sec^4t dt = tant +tan^3t/3+C
To undo the substitution note that:
tant = sint/cost
and as we noted that in the interval cost is positive:
tant = sint/sqrt(1-sin^2t)
so:
tant = ((2+x)/3)/sqrt((1-((2+x)/3)^2)
tant = (2+x)/(sqrt(9-(2+x)^2)
tant = (2+x)/sqrt(5-4x-x^2)
Then:
int dx/(5-4x-x^2)^(5/2) = 1/3^4 ((2+x)/sqrt(5-4x-x^2) + (2+x)^3/(3(5-4x-x^2)^(3/2)))
and simplifying:
int dx/(5-4x-x^2)^(5/2) = (2+x)/(81sqrt(5-4x-x^2)) (1+ (2+x)^2/(3(5-4x-x^2)))
int dx/(5-4x-x^2)^(5/2) = (2+x)/(81sqrt(5-4x-x^2)) ((15-12x-3x^2 + 4+4x+x^2)/(3(5-4x-x^2)))
int dx/(5-4x-x^2)^(5/2) = ((2+x)(19-8x-2x^2))/(243(5-4x-x^2)^(3/2))