For #f(x)# a function of #x#, we can denote the derivative in several ways.
On family of notations uses a "differential operator".
This is a notation that tells us "take the derivative".
For example #D_x(5x^2+3)# means "The derivative, with respect to #x#, of the function #f(x)=5x^2+3#. SO #D_x(5x^2+3) = 10x#.
#d/dx# is another notation for "the derivative with respect to #x# of . . . #
So #d/dx(5x^2+3)# means the same thing as #D_x(5x^2+3)#.
To take the second derivative, we take the derivative twice:
#D_x(D_x(5x^2+3))# is written #D_x^2(5x^2+3)#
#d/dx(d/dx(5x^2+3))# could, perhaps, be written #(d/dx)^2(5x^2+3)#, but the standard notation is #d^2/dx^2(5x^2+3)#.
If we let #y = 5x^2+3#, then these examples look like:
First derivative: #D_x(y)# and #d/dx(y)# which is also written #dy/dx#.
Second derivative: #D_x^2(y)# and #d^2/dx^2(y)# which is also written #(d^2y)/dx^2#.
In this notation, we do not think of #dx# as #d# times #x#
Instead, we are thinking of #dx# as a single quantity. That is why we do NOT write #d^2/(dx)^2(y)#
Additional Note
I would interpret #dy^2/dx# as #d/dx(y^2)#. Using the chain rule, I would evaluate to get #d/dx(y^2) = 2y * dy/dx#.
Using #y=5x^2+3# again, we have #d/dx(5x^2+3)^2 = 2(5x^2+3)*5 = 10(5x^2+3)#
Note 2
#(dy/dx)^2# means the square of the derivative of #y#.
Using #y=5x^2+3# again, we have
#(dy/dx)^2 = (10x)^2 = 100x^2#.
This kind of expression can come up if using implicit differentiation to find 2nd or higher derivatives. (If we don't pause to solve for #dy/dx# before our second differentiation.)