I would go with the Medium personally, because you will definitely feel the tightness of the hat in heat. Also, it depends on if you have a lot of hair it can lead to tightness as well. You can adjust it as well. Thanks
It is better to buy a bit larger and adjust the tightness. Otherwise too tight there is nothing you can do.