Do they hurt the backs of your ears? I’ve had some cheap in-ear headphones that also had curved plastic to fit around the shell of your ear for stability. I’d have to take them off after an hour or two in pain, but not because of the in-ear component.
I wonder if you’re just using different muscles? Like doing a bench press vs an overhead… I would have thought those were all the same muscles, but they are absolutely not!