Calculate the damage multiplier for the attack (we'll call this dmgMultiplier) which is usually 1.0 unless you have status effects, masteries, or unique items that say they multiply your damage. Things like gravelord, gore, retaliation, empower mastery, flame aegis, poisoner's satchel, or the unique weapons that give extra damage against certain types of creatures.
The to-hit roll happens, which is a percentage calculated as (attackerAccuracy - defenderEvasion)%, adding any accuracy bonuses or penalties on top of that, with a minimum of 5%. If the attack was a crit the to-hit roll succeeds automatically.
If the to-hit roll succeeds, calculate the raw minimum and maximum damage. This is the damage you see on ability tooltips and in your status menu.
Calculate defender's defense value, this is statValue * 0.7, or statValue * 0.4 if the attack was a critical hit, where statValue is toughness for physical attacks and resistance for magical attacks.
If the attack was a piercing attack, halve the defense value just calculated.
Roll the actual raw damage value, this is a random value between the minimum and maximum raw damage (inclusive). We'll call this rawDmg.
Calculate the final damage: finalDmg = (rawDmg * dmgMultiplier - defense) * (1.0 - dmgReduction), which is rounded to the nearest integer, and has a minimum of 1. Note that dmgReduction comes from armor and shields.
After this any status effects or masteries that would change the damage value further happen.