Now with this post, I started wondering if this optimization problem could be solved using calculus of variations. I have grown to be increasingly drawn to this offshoot of calculus, since it was very recently that I learned about the Brachistochrone Problem. This problem literally led to the birth of a branch of mathematics called the Calculus Of Variations I immediately thought about the Brachistochrone Problem - from Wolfram MathWorld. Therefore, this becomes a problem of optimization. Please correct me if you find something wrong. And that is to say that the maximum sized hexagon inside the square will be tilted (no sides of the hexagon will coincide with that of the square). For a hexagon, such a sketch will bring us to wrong conclusions. You can only draw such a polygon (n sides) for cases where n is a proper multiple (1,2,3.) of 4 (since for a square, n = 4).

This might be previously answered based on intuition (drawing a hexagon inside a square) and I have to say that the intuition of drawing a hexagon - with its vertices on the sides of the square is wrong (I spent a lot of time doing the same, then realized the trigonometry does not give me the correct result - something very basic must be wrong). You can use the fact that the Cos(15) = Cos(45 - 30) and the formula for the Cos(A - B) = Cos(45)Cos(30) + Sin(45)Sin(30) and much simplification to get the Sqrt(2 - Sqrt(3)) * S as already posted by The_Big_K. It turns out that rotating it by 15 degrees will give you the maximum hexagon. If you rotate it 30 degrees you will have a diameter that is now perpendicular to the bottom side and you are still touching the square at R = S/2. By symmetry if you rotate the hexagon by 60 degrees you're right back where you started.

But if you rotate the hexagon a little you will be able to get a slightly larger polygon. If the hexagon has a radius of R and aligned so that a diameter of the hexagon is parallel to the bottom side of the square, then the correct answer is R = S/2. But, it is a nice problem and the previous discussion left me unsure who was correct and why? So, with a little scratching. This is an old thread, and there was already a correct answer.

Perfect hexagon measurements