Sure, but the vast majority of people live in low lying areas and even then it doesn’t shift that drastically. You need to climb a mountain to see the difference when it comes to applications of daily life.
Although now that I think about it. The same criticism applies to pretty much every definition of temperature that is based on the behaviour of matter. This also applies to Kelvin. Temperature is a property of matter and every type of matter behaves differently.
The notations can be confusing, especially around noon and midnight. Is midnight am or pm when it’s equally distant to both the previous and the next noon? Why does 12am not follow 11am???
Where I live we use 12hr time in casual spoken language but pretty much always specify the time of day as well, like eight in the evening or twelve at midnight. But for anything written or even remotely formal, 24h time is used for obvious reasons.