This usage seems to be common not only in English, but in Western cultures in general. (The two parties do not need to be on first name terms for this pattern to work: "Oh, Mister Bennet! Have some compassion on my poor nerves!" and so the availability of the pattern is unaffected by T-V distinctions common to many European languages.)
If you choose to treat substitutes like "sir" or "my dear" on par with personal names, the pattern is probably universal. However, there seem to be cultures (and thus languages) where the use of a person's name to address them is considered impolite, i.e., where the applicable "substitutes" are no longer substitutes for personal names, but rather the only unmarked way to address people.
The part of linguistics that is concerned more with the context of an utterance than with its own "structure" and (inherent) "meaning", is called pragmatics. Then there is conversation analysis, either considered a part of pragmatics (and thus linguistics), or a field adjacent to pragmatics, part of sociolinguistics which lies on the boundary between linguistics and sociology.
In the terminology of this article, there are two main types of vocatives within a conversation structure: a summons, or an address. The pattern you have in mind is a special kind of an address.
In a conversation among several people, an address is often used as part of a turn taking protocol; the speaker is expected to "pass the turn" to the named person at the end of the utterance. This particular function is not needed in a dyadic conversation (just two people), so other common functions of the address dominate the usage: to show politeness, to show intimacy, to show power differential, to reflect on the other person's identity.
If either party is exasperated, it's eventually going to show in some way; if it doesn't, they were merely annoyed. I am not sure whether "showing exasperation" is always a communication objective in itself; in certain conversational situations involving exasperation, an address of the other party could sometimes be showing intimacy in an attempt to reset a fruitless discussion; in another situation, an address could be used to make room for a "repair" sequence if there are hopes that the other party might salvage the conversation by amending or disambiguating some part of what they already said.
Personal names, in particular, are some of the more attention catching forms of address, and thus especially attractive ones in the contingency of exasperation, but they are not alone in that group. "I beg you, sir," or "you bastard" can, where applicable, provide the attention catching function just as well. Such a phrase can serve the various other functions of the address mentioned previously as well as a personal name can, depending on the circumstances.