This sounds interesting. I see two possibilities for that:

• Let downvotes from external users be less effective
• Display up/down ratio twice, once from local votes and once from external votes; plus only have the local ones affect sorting

Actually when I think about it, a combination of both would also be possible.

The problem with that is, that not all instances use “Lemmy” or even “feddit” in the URL.