English is made up of phonemes, true. But very few of them are reflected in English orthography.

For example, the /n/ sounds are, in fact, different in the words anne and anthem. This is determined by context on a level native speakers aren't even going to be consciously aware of. There are a lot of things like that, without even touching the subject of the vowel space. Or homophones. Or homonyms. You can pare down a lot, the question is what.

(Bear in mind: I am in fact a linguistics major, currently studying Phonetics and Phonology. I speak not from a position of total ignorance.)