Born on a farm, Fritz Koller grew up in a rural environment and attended elementary school for only a few years. He later worked as an agricultural laborer and gardener. At the age of sixteen, he was conscripted to fight on the Russian front, an experience that would profoundly affect him.
After the war, everyday sounds and voices reawakened his trauma: when he heard commands or voices broadcast on the radio, he protected himself by covering his ears, as if he were still fearing enemy fire. In 1946, he was admitted to the Gugging Psychiatric Hospital in Austria, where he remained until his death. It was within this institution that he developed his artistic practice later in life.