The goals of security awareness programs
The stated goal of most security awareness programs is something along the lines of “detecting user risk and training employees”. The unstated goal is “check a box, be within industry compliance, impact operations as minimally as possible”. There’s obviously a large gap between these goals and the most common forms of security awareness training available don’t do much to address either.
Most organizations follow this structure:
- Send test phishing campaigns to users
- Send remedial video training to users that fail, including a quiz
- Chase down users who do not complete the training
- Check a box that your organization does regular security awareness training
- Rinse and repeat
The built-in assumptions: do they hold up?
The assumptions here are that training videos will educate users, which reduces the likelihood of them clicking on a true positive attack email. The users won’t like failing or being forced to take the training so they’ll watch and learn how to spot attacks in the future so at the very least, their work won’t be disrupted on the next test.
Users already know that phishing is dangerous. They've almost certainly learned about it at any corporate job they've held over the past five years. They learned it during on-boarding, they learned it during the annual training campaigns, and they see the fliers that are prominently displayed in the break room (right?). There isn’t much in these videos that users don’t already know, and if there is, are they aren’t in the best mental space to truly learn?
Why users click
I’ve spoken with many people who have failed phishing tests or have fallen victim to a true phishing attack and the majority of them knew the red flags and can easily identify them after the fact. The leading explanation is “I was going too fast”. Many IT professionals, including cybersecurity engineers (ahem) seem to have little patience for user errors. When it comes to security, they can cause threats to the organization which can add up to a lot more work. However, users are also overworked. They also have looming deadlines that they can’t miss, along with stresses of their own.
Let’s reframe the situation. You are riding on a bus with your bag next to you. You’re answering some work emails to try and catch up, you lose focus on your surroundings, and someone snatches your bag. Consider the emotions you would feel at that time. Panic that your wallet and other sensitive information was in that bag. Feeling foolish for not having your bag better secured. Violated. Embarrassed. Angry. Now a police officer brings your bag back and informs you that the theft was a test, you failed due to your own shortcomings, and you now have to watch a video about how to not be robbed in the future.
In that scenario, is the training going to be more helpful to you, or the fact that the incident itself occurred?
Post-failure mindset and the consequences
"You made me feel like a fool, and now I'm being punished." This is paraphrased from a user that I spoke to after a test failure. They felt embarrassed and resentful. This resentment can fracture already difficult relationships with users as a whole, potentially driving down phish reports and overall buy-in to the organization's security program. This is the antithesis of the stated goals of security awareness training.
When users don’t want to “deal” with security, the program has failed. Security awareness programs live and die by reporting rates. If your users are taught that engaging with security leads to punishment, or is a hassle in general, you are undermining the one behavior that matters most when a real attack lands in an inbox. A user who engages with a real phishing attack and says nothing because they fear consequences is a far greater risk than one who clicks and immediately reports it.
This resentment can fracture already difficult relationships with users as a whole, potentially driving down phish reports and overall buy-in to the organization's Cybersecurity program.
Just-in-time training operates on a different principle
"You made me feel like a fool, and now I'm being punished." That is a powerful statement that forced me to reevaluate my program. Regardless of the lofty goals of my program, this was the perceived outcome. I decided to make a change.
Back to the bus where your bag was stolen. The robbery itself was the learning moment, not the forced video telling you to do what you already know what to do. There is an option for security awareness training that capitalizes on that.
Just-in-time (JIT) training occurs at the moment of failure rather than hours or days later. When a user clicks a simulated phishing link, instead of a landing page followed by a delayed training assignment, they are immediately shown a brief, targeted explanation of what just happened and why it was a threat indicator. Many JIT solutions also offer a brief quiz based on the test email itself, prompting the user to identify the missed red flags. The learning occurs in context, at the exact moment the user is most receptive to it.
Immediate feedback is more effective than delayed feedback at producing change, and JIT training can be designed to feel instructive rather than punitive. "Here is what you just encountered and here is how to spot it next time" lands very differently than "you failed; complete the following training." One builds competence while the other builds avoidance.
What the outcomes look like in practice
Organizations that shift from punitive remediation to JIT intervention typically see improvement across the two metrics that matter most: a decline in click rates on following simulations and an increase in phishing reporting rates. When users stop associating tests with negative consequences, they are more open to the testing process. They become participants rather than subjects.
In my own programs, the transition to a JIT model was one of several factors that contributed to meaningful improvement in reporting rates over a twelve-month period. It goes hand-in-hand with increased user engagement and understanding. Approach failures collaboratively rather than punitively and users will begin to see security as a service instead of a necessary nuisance.
A final word for CISOs
If you are a security leader evaluating your awareness program, the question worth asking is not "are our users completing their training?" It is "are our users more likely to report a suspicious email today than they were twelve months ago?" Click rates on simulations are a useful metric, but an incomplete one. Reporting rates tell you something more important: whether your users consider security as a shared responsibility rather than an external burden.
JIT training can be one of the more reliable methods for moving that needle. It's not a silver bullet, but it presses the advantage when a user is most receptive to a learning opportunity and removes the punitive nature historically seen in security awareness training.
Evaluate your goals for testing and review the data on your results. If the current solution is not moving your program towards your goals, it’s time for a change.