I am a 15 handicap but generally play tees second from the back(blue most places). I am long and consistent off the tee box, my problem is 70-100 yards and in. If I play up because of my handicap, I am essentially hitting my worse shot/club every par 4 on second shot.
There are too many courses around me that are simply driver, gap wedge, putter. Moving back allows me to play more of my bag. I play regularly with scratch to plus golfers and I don't think any of them have ever complained about my 15 slowing them down. My extra shot(putt) a hole isn't making or breaking anyone's day.