Protein structure prediction has been revolutionized by the Nobel-prizewinning deep learning model, AlphaFold. Nevertheless, it struggles to predict fold-switching proteins, which remodel their secondary structures and change their functions in response to cellular stimuli. Here, we explore why fold switchers are challenging to predict. We find that AlphaFold can sometimes be reduced to a linear classifier when it "decides" which fold-switched conformation to predict. Extending this linear classifier, we identify some mutations that can be sufficient to switch proteins folds and others that cause AlphaFold to mispredict structures. We aim to use these adversarial examples to develop a network that predicts multiple conformations of fold-switching proteins more robustly, shedding light on this dark area of the protein universe.