What counts as vulnerable: a creature is judged a vulnerable target at the instant it is hit, if it is carrying one of the weak states below at that moment.
Weak states: attacking, stunned, sleeping, panicking and fleeing — plus unsteady poses such as exhausted (Nightmare Werepig), knocked down (Dragonfly) and staggering (the three Lunar bosses).
Stinging Nettle (venom): hitting a vulnerable target deals an extra 10 damage; it also has a 20% chance to poison the target, which keeps taking damage over time and loses 10% damage absorption while poisoned. Poison does not apply to structures, shadow creatures, lunar creatures, clockworks or plant creatures.
Ivy Spear: melee damage against a vulnerable target is doubled; the thrown piercing hit does not get this bonus.
Followers: they only attack a vulnerable Brightshade and its vines; as soon as it fights back or is about to wake up, they keep away from it.
In practice: a monster that is winding up an attack, stunned, asleep, fleeing, knocked down or staggering is vulnerable — those are the moments to land your hits.
Design note: the idea for this state comes from the counter-attack effect in Dark Souls III.