If #{nx}^2# is the fractional part #(nx-[nx])^2# then we have to consider that the function is continuous in the intervals:
#0 < nx < 1#
#1 < nx < 2#
etc... that is in the intervals:
#0 < x < 1/n#
#1/n < x < 2/n#
etc...
For every #n# consider then #j=0,1,..., n-1#. In the interval #x in [j/n,(j+1)/n)# the function is continuous and its value is:
#{nx}^2 = (nx-j)^2 =n^2(x-j/n)^2#
We can then express the integral as:
#int_0^1 {nx}^2dx = n^2 sum_(j=0)^(n-1) int_(j/n)^((j+1)/n) (x-j/n)^2dx#
Substitute now in the #j#-th term #y=x-j/n#:
#int_(j/n)^((j+1)/n) (x-j/n)^2dx = int_0^(1/n) y^2dy =[y^3/3]_0^(1/n) = 1/(3n^3)#
So the terms are all equal and their sum is:
#sum_(j=0)^(n-1) int_(j/n)^((j+1)/n) (x-j/n)^2dx = sum_(j=0)^(n-1) 1/(3n^3) = n xx 1/(3n^3) = 1/(3n^2)#
Finally:
#int_0^1 {nx}^2dx = n^2 xx 1/(3n^2) = 1/3 #