Existentialism has no true definition. This makes it difficult to ascertain what it really is then as there is no established litmus test. However, in literature Dostoevsky is considered to have written the first Existentialist novel, so taking a look at his work, Notes From the Underground will give us better understanding.
Notes From The Underground is one part treatise, one part narrative, creating a holistic picture of the unnamed protagonist, who recounts his struggles with his own existence. Part of what Existentialism is pertains to how responsibility lies in the individual to take action. Ethics, morality, social expectations are out the window, because they no longer define the individual. The combination of having the conscience of the individual being solely responsible for actions, and declaring those actions free from any constrictions of moral or social expectation create the Existential mood in the story. The protagonist of Dostoevsky's novel is constantly struggling to understand what being good can accomplish in a society where people will inevitably act out for evil. Rather than living up to expectations that will only bring him angst, he declares that he is his own man and takes up his mission to declare his independence from society. There is a moment in the novel where a strong armed military figure meets the protagonist on the street and bumps shoulders with him to the effect of not even taking notice of the Underground man. The outrage of not being noticed, sparks a prolonged two year battle to get the military man to acknowledge the protagonist's existence in which the Underground man, attempts to bump shoulders with the military man. He eventually does, but still feels trapped in his ennui.
What makes existential novels so interesting I would wager is that the protagonist's actions lie outside of expectations. They also aren't defined by a moral compass or anything else. They just are what they are. This is where the flawed narrator originates. So then, in writing under this style the possibilities are opened up dramatically for what one can motivate the character to do. Remember, they adhere to no moral authority or expectation, because they act independently from society. "God is dead, and we killed him." That is the mantra of Nietzsche. To see this in action helps though, so here is some dialogue to give you an idea of what Existential writing looks like:
I don't understand what I did, or why I did it. That's not my purpose. I shouldn't have to justify myself. The Authority, my benevolent overseers, they have no right over me anymore than myself. Maybe that is why I did it then. That's why I torched the state building. My world, as I look around, as the bystanders gaze as my grizzled face, my can of petrol and the welding torch, siphons judgement upon me. They say, "how dare you!" I am the dissident sowing chaos in the known universe. When the Authority arrives, asking me what happened, and when they inquire why I did it, I shrug. I didn't have an answer. I don't explain myself. Why should I justify myself? Clearly they are the fools. I have unlocked my mind, and they still see in the darkness. I take because I want to, and it feels good.Here, the protagonist is an arsonist. Why he is an arsonist is not specified, but that's the point. He doesn't need to justify himself at all in light of his actions because he is freed from what would be considered social constraints. Were you to write your own Existential protagonist, that character in question wouldn't conform to any specified expectations. What they do only serves to prove to themselves that they exist. So they can be dark, or flawed. It can't be seen as a bad thing if they steal because they want to.
Like I said before, the Existential novel is difficult to define. That being said, when understanding its basic characteristics, the main idea that can be distilled from the genre is purity of motivation and will. As long as your character does what they want for his/herself to their benefit, they are acting out of existential motives. Acting alone substantiates their personal choices.