The shading algorithms date back to BLAST and TARP which only supported rectangles and triangles. EnergyPlus 1.2.1 extended these algorithms to allow for more general polygons, but support for concave surfaces and holes was not implemented. The main function for this is SolarShading::DeterminePolygonOverlap. I don't know how difficult it would be to add this capability.
From the Engineering Reference (v8.5).
Polygon Clipping Algorithms
Two methods for polygon clipping (treating of overlapping shadows) are currently in use in
EnergyPlus.
- Convex Weiler - Atherton
- Sutherland – Hodgman
The original EnergyPlus method for polygon clipping is a special version of the Weiler-Atherton model (Weiler, Atherton, 1977). It was developed to be sufficiently general to clip concave polygons with holes. The implementation in the current version of EnergyPlus, however, does not support concave shadowing surfaces or holes.
Disclaimer: this is just a guess. I think the only "reason" is a limitation in the specification, i.e., you cannot specify two lists of vertices in one object. But I don't think there is any deep fundamental reason why you could not a subsurface which is a hole. Would be happy if someone who knows relevant pieces of the code deeply would confirm or correct.
That would make sense, and explains why @Ivan Korolija's approach works - and is probably the simplest method, so long as you can live with the non-convex surface warnings.
Once you make a simplifying assumption on such a basic input like surface geometry, every algorithm that that is developed that uses that basic input is likely to to be streamlined to take advantage of that simplification.