The inscribed cylinder's volume will be maximized when its radius is #r=10/3 units# and its height is #2# # units#.
Let #r# be the radius of the cylinder inscribed in the cone and let #y# be the cylinder's height. The volume of the cylinder is,
#V=B*h=(\pi r^2)*y#
Place an edge of the cone at the origin and take the #x# to be the distance to the edge of the inscribed cylinder from the outside of the cone. The diagram below shows #x#.
From the diagram, the radius of the inscribed cylinder is #r=5-x#. We can also express the height of the cylinder, #y#, in terms of #x# because we know the slope of the slanted side of the cone. For every five units we move to the right we go up 6 units.
Slope#=# #{rise}/{run}=6/5#
#\implies y=6/5x#
Substitute #r# and #y# into the top expression for #V#
#V=\pi(5-x)^2*6/5x=6/5pi(x^3-10x^2+25x)#
To find the maximum value of #V#, find when #d/{dx}V=0#.
Keep in mind that #0\leq x \leq 5# so that the left edge of the inscribed cylinder stays properly within the cone.
#d/{dx}V=6/5pi(3x^2-20x+25)=0#
#\implies3x^2-20x+25=0 #
Use the quadratic formula to find #x#
#x={20\pm\sqrt{400-4*3*25}}/{2*3}={20\pm10}/6#
The answer between 0 and 5 is #x=5/3#
You can verify that #x=5/3# gives the point of a maximum by graphing #V# in terms of #x#.
At #x=5/3#,,,,, #y=6/5x=2# and #r=5-x=10/3#
