The problem description is in fact pretty lousy. So don't feel bad about not getting it ...
You are given a force field #vec F(x,y,x)# and you are asked to calculate the net outward flux through the surface bounding the region #D:1\le\rho\le3#.
#\rho=\sqrt{x^2+y^2+z^2}# is nothing but the the magnitude of the radius vector #vec r = x\hati + y\hatj + z\hatk#.
What is this Region?: Consider a surface represented as #\rho=a.#
#\rho=a;\qquad \rightarrow \rho^2=a^2; \qquad x^2+y^2+z^2 = a^2,# which represents a sphere of radius #a# units.
So the region #D:1\le\rho\le3# is the volume bounded between the spheres of radii #\rho_1=1# and #\rho_2=3#. The surfaces bounding this volume are the inner surface of the inner sphere (#\rho_1=1#) and the outer surface of the outer sphere (#\rho_2=3#).
Flux Calculation on a spherical surface: From the form of the field vector it is clear that the field is spherically symmetric, meaning the magnitude of the field vector is a constant on the surface of a sphere and the field vector is directed radially.
#vecF(x,y.z) = (x\hati+y\hatj+z\hatk)/\rho^3 = vecr/\rho^3 = (\rho\hatr)/\rho^3 = +1/\rho^2\hatr;#
Positive sign indicates that the field vector is directed radially outward.
To calculate the flux on the surface of a sphere, just calculate the constant field strength on that surface and multiply it by the surface area of that sphere.
Flux carry a sign. If the surface normal is parallel to the field vector, flux is positive. If the surface normal is anti-parallel to the field vector, flux is negative.
Flux Through Inner Sphere: Magnitude of this field on the surface of the inner sphere is #F_1#. The surface normal is directed radially inward and the field is directed radially outward. The surface normal and the field vector are thus anti-parallel and so the flux is negative.
#F_1 = |vecF_1| = |\hatr|/\rho_1; \qquad dvecs_1 = -ds_1\hatr#
#\Phi_1 = \oint_{S_1:\rho_1}vecF.dvecs = -\oint_{S_1:\rho_1}F_1.ds = -F_1\oint_{S_1:\rho_1}ds#
#\Phi_1 = -(1/\cancel{rho_1^2})(4\picancel{\rho_1^2}) = -4\pi# ...... (1)
Flux Through Outer Sphere: Magnitude of this field on the surface of the outer sphere is #F_2#. The surface normal is directed radially outward and the field is also directed radially outward. The surface normal and the field vector are thus parallel and so the flux is positive.
#F_2 = |vecF_2| = |\hatr|/\rho_2; \qquad dvecs_2 = +ds_2\hatr#
#\Phi_2 = \oint_{S_2:\rho_2}vecF.dvecs = +\oint_{S_2:\rho_2}F_2.ds = +F_2\oint_{S_2:\rho_2}ds#
#\Phi_2 = +(1/\cancel{rho_2^2})(4\picancel{\rho_2^2}) = +4\pi# ...... (2)
Total Flux, #\quad \Phi_{"tot"} = \Phi_1 + \Phi_2 = -4\pi + 4\pi = 0#.
Note: The field strength dilutes as #1/\rho^2# and the surface area is proportional to #\rho^2#, making the flux independent of the radius of the sphere. So the flux through both the inner and outer sphere have the same magnitude. But it changes sign and so they cancel each other to give a net flux of zero.
