The easiest way to explain, I guess, would be to compare in the evening -> time of day vs on the evenings -> date. NB: there is no "s" in "in the evening", because it's a time frame not a series of dates.
word choice - On the evening Vs. In the evening - English Language ...
I wonder what the exact times of the following words are: morning, noon, afternoon, evening, night, mid-night. What's the difference between at night and in the night?
Sunday evening is a specific day. If we were to use "the evening" on its own, it could refer to any evening, and is not therefore a specific day or date. I really do wish I could tell you "why" we use on, but it would seem to be a rather arbitrary choice, with no particular logic to it. More info here: In, at, on + Time or Date
Tomorrow morning, tomorrow afternoon, tomorrow evening and tomorrow night they all refer to different periods of the day after “today”. Whereas the possessive apostrophe is used in: a good night' s sleep (a good night of sleep) The possessive apostrophe replaces "of" and adds an "s" several good night s ' sleep (several good nights of sleep)
To a large extent the start of 'evening' is a matter of opinion or convention. The Oxford Dictionaries say that it 'usually' starts at 'about' 6 PM. Note the lack of a definite rule. Evening NOUN 1 The period of time at the end of the day, usually from about 6 p.m. to bedtime. it was seven o'clock in the evening Evening (Lexico) Cambridge does not provide a time: the part of the day between ...