I believe the way to solve this is by using the chain rule. You will also need to know the standard differential of cos^-1 (x), and use the product rule on your substitution.
y = cos^-1(2x sqrt(1-x^2))
Step 1: Differential of 'u'
u = 2x sqrt(1-x^2) = 2x (1-x^2)^(1/2)
v = 2x (dv)/dx = 2
z = (1-x^2)^(1/2) (dz)/dx = -x(1-x^2)^(-1/2) (chain rule used here too)
(du)/dx = z*(dv)/dx + v*(dz)/dx
= 2(1-x^2)^(1/2) - 2x^2(1-x^2)^(-1/2)
Step 2: Differential of original function
y = cos^-1(u)
(dy)/dx = (-1)/(sqrt(1-u^2)) * d/dx(2x sqrt(1-x^2))
= (-1)/(sqrt(1-u^2)) * [2(1-x^2)^(1/2) - 2x^2(1-x^2)^(-1/2)]
=(2(1-x^2)^(1/2) - 2x^2(1-x^2)^(-1/2))/(sqrt(1-(2x sqrt(1-x^2))^2))
And now to simplify the numerator:
=(2sqrt(1-x^2) - (2x^2)/sqrt(1-x^2))/(sqrt(1-(2x sqrt(1-x^2))^2))
=((2(1-x^2) - 2x^2)/sqrt(1-x^2))/(sqrt(1-(2x sqrt(1-x^2))^2))
=((2-4x^2)/sqrt(1-x^2))/(sqrt(1-(2x sqrt(1-x^2))^2))
And now to simplify the denominator:
=((2-4x^2)/sqrt(1-x^2))/(sqrt(1-(4x^2 (1-x^2))
=((2-4x^2)/sqrt(1-x^2))/(sqrt(1-4x^2 + 4x^4))
=(2-4x^2)/(sqrt(1-x^2)sqrt(1-4x^2 + 4x^4)
=(2-4x^2)/(sqrt((1-x^2)(1-4x^2 + 4x^4))
=(2-4x^2)/(sqrt(1-4x^2 + 4x^4 -x^2 +4x^4 -4x^6)
=(2-4x^2)/(sqrt(1-5x^2 + 8x^4 -4x^6)
Again, I hope that wasn't too confusing and would be keen to hear from others if there is a quicker and cleaner way of solving this.