Gothic Fiction has become so associated with horror, that the traditional roots of the genre are often lost amidst the noise of vampires, werewolves and gargoyles. These creatures of gothic literature are representative of the gothic tradition, but as writers, to adequately write within a genre it is important to have a deeper than surface level understanding of that genre.
Gothic tradition
Touchstones of the gothic tradition include Bram Stoker's Dracula, Mary Shelley's Frankenstein, and Shirley Jackson's The Lottery. There is a common thread within these works, but it cannot be pursued without understanding the concept of "natural philosophy." During the period in which early gothic writers began producing their best works, there was a push in culture to approach the unknown via a scientific lens. "Natural philosophers" were early scientists which attempted to find logical, objective truth, through reason. In short, early gothic works were rooted in science and exploration of the natural world.
Not all gothic fiction is horror, but a fair amount of the most famous works deal with the horrific and the unknown. This is by contrast to Eldritch or Lovecraftian horror which deals with the horror of the fundamentally unknowable. Where Eldritch horror seeks to paint a picture of something so incomprehensible that it breaks minds and shatters souls, gothic literature is the exploration of the dark unknown, in an effort to reveal the sensations associated with these mysteries. Rather than reach the conclusion "I don't understand this evil" as eldritch horror often does, gothic horror often seeks to delve deeply into the source of evil, and the nature of the antagonists of its works. With this as a guiding principle, we can outline a few "do" and "do nots" for gothic writers.
Follow the rules
Gothic antagonists follow rules. Establish a set of the rules by which your antagonist functions, and adhere to these rules. These might be the qualities of a monster: slain using silver, transforms during moonlight; or it might be the rules of engagement that the antagonist uses. Perhaps the antagonist is a killer like Jack the Ripper – another gothic touchstone. If this killer has rules about who they kill, where they kill or how they kill, then they will be much more palatable to the reader who is seeking a truly gothic feel. The genre is grounded in reason, and there should be an underlying cause, explanation, and logic to the actions of your antagonists.
Don't be afraid to engage with the darker side of a character's psyche, especially the antagonist; and don't be afraid to search for an antagonist in strange places: the narrator may be the antagonist, the protagonist may work against themselves, or the antagonist may be a culture – a group of people with a similar ideology. In any of these cases it is important that you justify the actions of the antagonist. It doesn't matter to the reader of gothic literature whether what the antagonist is doing is morally good, but only that the antagonist has reasons for their decisions. This advice is, of course, relevant across many genres, but it is a necessity of gothic texts. Think of the ways in which Jack from Tim Burton's The Nightmare Before Christmas is all of these: narrator, hero, villain. Consider also that his actions are motivated by a lack of understanding: he changes his actions after his motivations are revealed to him – through reason – to be harmful. The gothic seeks to understand viewpoints outside the normal and Jack Skellington's wish to understand Christmas makes him an excellent gothic protagonist.
Give answers
In the same way antagonists require rules, it is important that readers have a chance to see the answer to their questions revealed, or that the readers are given enough clues to formulate an answer for themselves. The Lottery does a great job of presenting a seemingly enigmatic scenario – a lottery for which people do not wish to be chosen – and giving a reasonable explanation for why it might exist. In this way gothic literature can be said to relate more closely to the mystery genre, than the horror genre. Though, of course, these genres already compliment one-another quite well.
What gothic literature does not do, is leave the reader hanging as regards answers to the primary motivations of the plot. Yes, there may be a few threads unresolved at the end of the story, but the central horror (or other darkness) of the text should be explicable (understood by the reader) by the time the text is finished. Again, the unknowable, the unanswerable and the incomprehensible are all elements of eldritch horror.
Smaller settings
In the same way that gothic literature separates itself from eldritch horror in terms of the size of the threat (the antagonist), good gothic literature also tends to center around smaller settings. These settings can be almost claustrophobic, and the protagonists lonely. These are not stories of nations clashing in battle, but rather of individuals grappling with a close and immediate threat. Dracula offers repeated scenes of the eponymous count dining with the protagonist; Frankenstein paints painful pictures of the creature returning to look in the window at the doctor; and works like The Lottery focus their efforts on a single community, or single town. This is not to say that your plot can't span a large geographical area (both Frankenstein and Dracula cross continents), but only that it should be concerned primarily with the intimate moments between the protagonist and the antagonist.
Archetypal characters
In gothic fiction certain characters are considered archetypal. These characters have a recurring place in many stories, and while they should not be relied upon, they should also not be shunned. Just because a character is archetypal, does not mean they are necessarily cliché. In most gothic fiction the hero is virtuous, ambitious, and self-sacrificing; they are motivated to return to or pursue a relationship, either in love, lust, or kinship; and they are separated from their goal by circumstances arising either from the antagonist of the fiction, or the horror of the fiction, or both.
For instance, the creature in Frankenstein seeks out companionship, but is thwarted by its appearance. This becomes the core motivator for the plot, and the tragedy of the creature causes conflict between the protagonist (here conceptualized as the creature) and the antagonist (here conceptualized as the doctor). Of course Shelley's masterpiece allows you to view the same story from another angle, and consider Viktor Frankenstein the protagonist (unrequited love included), and his creation the monster or monstrous force which prevents him from reaching his goal, or uniting with his beloved.
Other archetypal characters include members of the clergy, or members of religious orders. Often the "spiritual" characters in gothic literature are presented as having an intuition regarding the dangers the protagonist will face, which the logical, reason-centric protagonist, cannot at first see. Consider the protagonist in Dracula, who is warned away from Dracula's mansion quite early in the story but can deduce no logical reason to avoid an encounter with the count. In this case, though the protagonist eventually comes to know their enemy – discovering weaknesses such as holy water, stakes through the heart, etc. – they are slower to realize their danger than the characters who are presented as "spiritually attuned." This duality is useful to us as writers – the contrast of "the person of faith" and "the person of reason" is a powerful one to introduce into any narrative, and it allows gothic fiction to tread a middle ground between science and fantasy that requires only "reason" not "reality".
The supernatural or the sensational
I have mentioned previously that the supernatural is closely related to the gothic, but I want to expand on that. In gothic literature the supernatural can be replaced with the sensational. It is not necessary that some monster be the threat, but rather a community's tradition, the beating heart beneath the floorboards, or a murderer on the loose might be the source of danger. In every case there must be danger. This is why I use the word sensational. The subject matter of gothic literature does not deal with mundane events, but rather acts as an exploration of supernatural or sensational events, so that by the end of the reading we better understand the source of these phenomena.
If you are afraid of something, the quickest path to overcome that fear is to handle the subject of the fear directly. Gothic literature is derived not from the fear of the unknown, but rather from intimacy with the unknown – direct interaction with dark themes, with death, and with monsters.
What to keep in mind
There are no hard-and-fast rules to adhere to within the gothic genre, but there are particular approaches to subject matter which are typified by exploration of what is scientifically possible. Remember these three tenets: separate your gothic literature from other similar literatures, such as eldritch horror and body horror; deal with dark themes, and address them through the sensational or supernatural; have every element of your plot, and your character motivations, be motivated and constrained by the text's internal logic. If all else fails, turn to the sources of darkness which inspired famous gothic writers: blood, bones, darkness, gravestones, murder, killer creatures, and Jack Skellington.