Aqua Security Found “Phantom Secrets” Lurking in Popular SCM Systems
Aqua Security, the pioneer in cloud-native security, today revealed new research that shows how credentials, API tokens, and passkeys – collectively referred to as secrets – from organisations around the globe were exposed for years. By scanning the most popular 100 organisations on Github, which collectively includes more than 50,000 publicly accessible repositories, Aqua researchers found active secrets from open-source organisations and enterprises such as Cisco and Mozilla providing access to sensitive data and software. The exposed secrets could lead to significant financial losses, reputational damage, and legal consequences.
Aqua Security’s research team, Aqua Nautilus, revealed that “phantom secrets” can persist in Git-based infrastructure used by most Source Code Management systems (SCMs), including GitHub, Gitlab, Bitbucket and others. This is due to the way in which even deleted or updated code commits are saved in those systems, such that even a one-time developer mistake can expose secrets to savvy threat actors over extended periods.
“Our findings are truly alarming, and it is crucial that everyone involved in software development grasps the seriousness of this issue,” says Yakir Kadkoda, Aqua Nautilus Lead Security Researcher. “For years, we’ve been educating developers not to hard-code secrets into their code. Now, it turns out that even doing this just once permanently exposes that secret – even when they thought it was deleted or overwritten. The impact of a sensitive data leak can lead to unauthorised access, compromised security controls and significant financial or reputational damage. This would be devastating.”
Among the exposed secrets found by scanning open Github repositories were API tokens of Cisco Meraki and the Mozilla project. The Cisco security team confirmed the findings: “We discovered privileged Meraki API tokens used by some Fortune 500 companies. These tokens could allow attackers to access network devices, Simple Network Management Protocol secrets, camera footage, and more, serving as an initial foothold for the exposed parties.”
The Mozilla project acknowledged “An API token for the Mozilla FuzzManager with read-write privileges” and that “an employee’s API token for sql.telemetry.mozilla.org was leaked”; both were assigned a “Critical” score. Not only does the FuzzManager allow access to many potential security vulnerabilities in Firefox and Tor, but the telemetry gives access to confidential information related to Mozilla products and business.
Additionally, Nautilus found an Azure service principal token belonging to a large healthcare company exposed in a Git commit. This token had high privilege and high access to obtain credentials to the internal Azure Container Registry, which could have led an attacker to perform a supply chain attack impacting the organisation, and customers.
In all cases, the exposed secrets were immediately revoked.
Commit Once, Expose Forever Aqua Security
While secure coding best practices already require that secrets should not be hard coded, many developers continue this practice. They rely on secret scanning tools to ensure that such secrets are not pushed into production and often re-commit the updated code without those secrets.
Phantom secrets exist because of underlying processes within Git-based SCMs, which cause code that was overwritten or deleted in repositories to remain accessible within the underlying system. Most secrets scanners only look at repos accessible via the Git clone command, which overlooks almost 18% of secrets.
“The findings once again reinforce the best practice that secrets should never be put into code, not even for testing purposes, and security teams must be able to monitor this,” says Amir Jerbi, CTO and co-founder of Aqua Security. “The software supply chain is optimised for speed and convenience, but this cannot come at the expense of secure engineering practices.”
“IDC research underscores Aqua Nautilus’ findings, showing that organisations are overly confident in their posture related to the protection of application secrets,” says Katie Norton, Research Manager, DevSecOps & Software Supply Chain Security, IDC. “While organisations show high confidence in their ability to secure secrets, among DevSecOps tools the adoption of secrets management solutions is among the lowest.”
Available in August, Aqua customers using the Software Supply Chain Security module will be able to prevent developers from committing code with embedded secrets and scan for phantom secrets hidden within their SCM file system.
For a full technical explanation of how Phantom Secrets persist, and why they are often missed, read Aqua’s blog and the full research.
