Nah don't worry I had listeria 3 times and milk was only one of them. You got to worry about caramel apples and hummus just as much.
Also don't get listeria I had a seizure while vomiting while taking a physics final and got yelled at for it in college. It's just not a fun time.