Irish literature is really, really old, and the Tironian shorthand had its best days before 1100, i.e., before the most of European vernacular literatures came about. This is the closest meme to a reason for the present day difference that I can offer. But the two symbols continued to compete valiantly even as their respective origins got obscured further and further.
"Almost all languages" mentioned in the OP presumably refers to languages more or less derived from Latin. The & symbol is thus often seen in Romance languages across Europe (but not so much in Latin America) and in many Germanic languages. Other languages that use an extension of the Roman alphabet typically know the symbol from language contact in commercial contexts, but rarely use it in domestic contexts. Some languages are also lucky enough to have a one-letter word for "and" (such as Welsh "a", Spanish "y", Polish "i", Maltese "u") even without needing a Latin ligature or a shorthand sign to "abbreviate" the conjunction. The set of languages where children ever recited the "ampersand" as part of their alphabet learning might amount to... just English, I think, given that the last syllable of "ampersand" used to be the English word "and" and not the Latin "et".
When writing using a quill, there is no strict need to distinguish between ligatures (e.g. "et" written together in a peculiar way) and special characters (e.g., "&" or "⁊" that could even get their own place in the alphabet). The scribal practice can be described using this or that terminology and even vary considerably from scribe to scribe or from a region to a region without inconveniencing anybody too much.
This applies to diacritics, too. Is the Irish letter á two symbols, an acute accent called "fada" atop a plain letter, or a single symbol? What a pedantic question — if writing by hand.
Well, people had asked that question anyway. As early as 1011, Byrhtferð's alphabet (list of letters used in Old English) included both the "&" and the "⁊", alongside the 24 Latin letters, and also Old English specific letters: Ƿ Þ Ð Æ.
Eventually the printing press comes & things change. The printer's tray either does or doesn't have extra compartments for those weird "&" or "⁊" or "á" letters. Letterforms have to be standardized, too. Either you use the Gaelic type, if you are in Ireland, or the Roman type, if you are in England. (Scotts Gaelic, which had enjoyed freedom of using either respective script with the quill, tended to share typesetting technology with the English language when printed language became the new norm.) By this time, Ƿ Þ Ð Æ were gone from English usage, while "U" was split away from "V" and "J" from "I". The "&" letter was still in demand, although its usefulness was somewhat reduced compared to Old English. If anyone still used "⁊" in English, it was no longer counted as a letter, and no longer associated with thousands of Tironian shorthand characters. It may have had a distinct insular feeling: this is how the Irish write "and" — it is culturally Irish rather than English or Latin, and it would probably be seen inside some imported Gaelic text with its overdots and fadas, calling for entirely different typesetting. Or it could occur in an abbreviated Latin loanword: "vi⁊" standing for "videlicet", say, in the middle of an English sentence. (Given that "&" firmly stood for "and" phonetically, and not for "et", the unfortunate printer couldn't typeset this as "vi&", which is how we ended up with approximations such as "viʒ"and eventually the illogical "viz" for "videlicet".)
Those printer's trays had hit a blow to the use of "⁊" in Scotland, while the letter was already out of fashion in England.
(The often told story that "⁊" was the victim of a witch hunt that associated shorthand with witchcraft appears unfounded, as the symbol fell out of use only gradually. More context here.)
Irish probably never used too much of "&" other than in commercial contact with the English. That secured a compartment in Gaelic type printer's trays for "⁊" and perhaps not for "&". In contrast, the earliest Irish language typewriter had just a "&" and not "⁊" on it, but that machine is presumably imitating contemporary American and/or English ones, and it became available long after Irish printed prose already transitioned during the 19th century to spell out "agus" ("and") normally in full, without using either symbol.