| Solution by Robert L. Doucette, McNeese State University, Lake Charles,
Louisiana.
Let A, B, C be the vertices of the enclosing unit triangle. Since the
two enclosed triangles have no common interior points, there is a line
that separates their interiors. Note that this line must intersect the
triangle ABC in exactly two points, say D and E, one of which, say D,
is not a vertex point. We may assume that D lies on the side AC and E
on the side BC such that |DC| <= |CE|. Suppose the line passing
through D and parallel to AB intersects the triangle ABC at F, and the
line pass through D and parallel to CB intersects the triangle ABC at
G. One of the enclosed triangles, call it T, of side length a, lies
between the parallel lines DF and AB. The width of T in the direction
of a given line L is defined by
w(L) = sup { |PQ| : P and Q lie in T and PQ is parallel to L }.
If d is the distance between DF and AB, then there must be a line L0
such that w(L0) <= d. The line L1 containing any median of T yields
the minimum value of w. Therefore
sqrt(3)*a/2 = w(L1) <= w(L0) <= d = sqrt(3)*|AD|/2.
It follows that a <= |AD|.
The other enclosed triangle lies between the lines DG and CB. By an
argument similar to the one above, we may show that b <= |DC|. Since
|AD|+|DC| = 1, we have shown that ab <= 1.
|