As the radius of the circle is not relevant we can assume we have a circle of unitary radius centered in the origin, whose equation is:

#x^2+y^2=1#

The triangle will then have three vertices #A,B,C# lying on the circle. Because of the circular symmetry we can assume without loss of generality that the side #AB# is parallel to the #y# axis and intersecting the positive #x# axis, so that the coordinates of #A# and #B# are:

#{(x_A = cos(alpha)),(y_A = sin(alpha)):}#

#{(x_B = cos(alpha)),(y_A = -sin(alpha)):}#

where #2alpha = angle (AOB)# so that the length of #AB# is:

#bar(AB) = 2sin(alpha)#

Now, as a first step let's consider for any possible choice of #AB# the triangle #ABC# letting #C# be the third vertex with coordinates:

#{(x_C = cos(theta)),(y_C=sin(theta)):}#

where #theta# is the angle between the #x# axis and #OC# as we generally define in trigonometry.

The height of the triangle is going to be parallel to the #x# axis.

As #x_A# is positive by construction the height is:

#h = x_A -x_C = cosalpha - cos theta#

and the area of the triangle is:

#S = (h bar(AB))/2 = sinalpha(cosalpha-costheta)#

and as #1- <= cos theta <=1# clearly the are is maximum when:

#cos theta = -1#

#theta = pi#

that is when the triangle #ABC# is isosceles and then the area is:

#S(alpha) = sinalpha(cosalpha+1)#

Maximize now this function with #alpha in (0,pi/2)#:

#(dS)/(d alpha) = cosalpha(cos alpha +1) -sin^2 alpha#

#(dS)/(d alpha) = cos^2alpha-sin^2 alpha +cos alpha #

#(dS)/(d alpha) = cos^2alpha -(1-cos^2 alpha)+cos alpha #

#(dS)/(d alpha) = 2cos^2alpha +cos alpha -1#

The critical points are then:

#cos alpha = (-1+-sqrt(1+8))/2 = (-1+-3)/2 = {(-1),(1/2):}#

Excluding #cos alpha = -1# because by construction #cos alpha >0# we have:

#alpha = pi/3#

where:

#(d^2S)/(dalpha^2) = -4sin alpha cos alpha -sin alpha = -sin alpha(1+4 cosalpha)#

#(d^2S)/(dalpha^2)|_(pi/3) = -sqrt3/2(1+4 (1/2)) = -(3sqrt3)/2 < 0#

so that in fact #alpha = pi/3# is a local maximum.

Remembering that:

#angle(AOB) = 2alpha = (2pi)/3#

as the angle #angle(ACB) = 1/2 angle(AOB)# we have:

#angle(ACB) = pi/3#

and as the triangle is isosceles the angles at the base are each:

#angle (BAC) = angle(ABC) = 1/2 (pi -pi/3) = pi/3#

which means the triangle is equilateral.