Beware of the tradeoffs between false positive alerts and secret-keeping.
Join the DZone community and get the full member experience. In an ideal world, secret detection tools would spot all leaked secrets and never report false positives. Unfortunately -or maybe fortunately…- we do not live in an ideal world: secret detection tools are not perfect, sometimes they report false positives. But would it really be better if they did not? A false positive happens when a system reports a condition as valid when it is not. The opposite, the false negative, happens when the condition is reported as invalid when it should have been reported as valid. In the case of secret detection, the condition is identifying whether a character sequence, or a set of character sequences, is a secret. We can summarize this in a table: (You can learn more about these terms and others in our article explaining the concepts of accuracy, precision and recall) With these definitions in mind, it makes sense for secret detection tools to try to maximize the number of true negatives and true positives, and minimize the false ones. But secret detection is based on a lot of heuristics, so the answer is often not truly black or white. What should a secret detection tool do when the answer is not clear-cut? Let’s take this to the extreme. If a tool reports too many false positives, it leads to alert fatigue: users are flooded with alerts. They start to ignore them and some reported true positive secrets are not taken care of. If, on the other hand, the tool is very aggressive against false positives, users won’t be flooded with reports and can take care of the reported ones. But since the tool heuristics cannot be perfect, the tool is going to identify real secrets as false negatives and silently sweep them under the rug. In both cases, valid secrets have the risk of being missed, but at least if the tool prefers to identify a secret as valid when unsure, the user can be aware of it: the secret won’t be silently discarded.
