how to model the mixing between outdoor air and return air?

edit

In Openstudio, the AirLoopHVACOutdoorAirSystem object is located before any cooling or heating coils, which I think is what you mean by "before they enter the HVAC".

In Openstudio the AirLoopHVACOutdoorAirSystem object (reference) wraps the OutdoorAir:Mixer (reference) object from Energyplus.

The AirLoopHVACOutdoorAirSystem object connects the return air node, the mixed air node, and the outdoor air node. You can use the included ControllerOutdoorAir object (reference) to specify parameters such as how much outdoor air to allow in your AirloopHVAC, the fraction of outdoor air as a function of the supply air and more.

You can always reference the Openstudio SDK to see how the Openstudio objects translate to Energyplus.