The inscribed cylinder's volume will be maximized when its radius is r=10/3 units and its height is 2 units.
Let r be the radius of the cylinder inscribed in the cone and let y be the cylinder's height. The volume of the cylinder is,
V=B*h=(\pi r^2)*y
Place an edge of the cone at the origin and take the x to be the distance to the edge of the inscribed cylinder from the outside of the cone. The diagram below shows x.
From the diagram, the radius of the inscribed cylinder is r=5-x. We can also express the height of the cylinder, y, in terms of x because we know the slope of the slanted side of the cone. For every five units we move to the right we go up 6 units.
Slope= {rise}/{run}=6/5
\implies y=6/5x
Substitute r and y into the top expression for V
V=\pi(5-x)^2*6/5x=6/5pi(x^3-10x^2+25x)
To find the maximum value of V, find when d/{dx}V=0.
Keep in mind that 0\leq x \leq 5 so that the left edge of the inscribed cylinder stays properly within the cone.
d/{dx}V=6/5pi(3x^2-20x+25)=0
\implies3x^2-20x+25=0
Use the quadratic formula to find x
x={20\pm\sqrt{400-4*3*25}}/{2*3}={20\pm10}/6
The answer between 0 and 5 is x=5/3
You can verify that x=5/3 gives the point of a maximum by graphing V in terms of x.
At x=5/3,,,,, y=6/5x=2 and r=5-x=10/3