Just wear shorts and deal with it! I do (most of the time). But I do use sunscreen that kinda applies a layer of warmth, believe it or not. I live in Denver and if the courses are open, we play.
I will admit, there have been some really cold days where I had to wear pants. Tee off time was 28 degrees and windy. I have the Nike Golf pants and the Amazon Essentials Golf pants, which by the way, have rubber band waist line that keeps your shirt in place pretty well. They also have in between sizes. I got 35/32’s.