GO Modules Vulnerability Disclosure

Last month we were made aware of an occurrence of GitHub.com token leakage. It affected only users who had also experienced Go Modules checksum update failures prior to September 6th, which was less than 0.1% of accounts. We resolved the problem within an hour but have spent the last month code reviewing and improving security in Renovate in general, and now present our findings and plans.

This problem was reported to us via our dedicated security reporting email address security@renovatebot.com by a user of the service. The hosted Renovate App was brought down within 30 minutes of the email being sent and our open source project was patched within an hour. After approximately one more hour verifying that no other languages or package managers had a similar problem, we brought the hosted service back up. Since then we’ve done a security review of our code and approach to secrets, while reaching out to affected users for whom we have contact details.

What happened

The background

One of Renovate’s important tasks is to ensure lock files, checksum files, and any other necessary artifacts are updated once we edit/upgrade manifest files For Go Modules, we run “go” commands to ensure the go.sum file is updated to match the go.mod file after Renovate edits it To allow for private GitHub.com-hosted go dependencies to succeed, we pass Renovate’s github.com token to the go process so that it can authenticate and read any sibling go repositories referenced in the go.mod When such updates fail, Renovate helpfully includes a snippet from the go update failure logs into the PR itself, as a comment

The problem

The token we passed to Go was included in that log snippet and embedded into a PR comment. Fortunately, it was in an obscure location 300+ characters across a log line, so we are optimistic that it was not observed by anyone until it was reported to us. The token was valid for up to an hour from its time of generation, so was temporarily valid to use with Renovate’s permissions, if found. This can include read/write to the repository, and opening or closing Issues and Pull Requests.

The resolution

The immediate remediation was to stop including any log snippets in any PR comments The final remediation was to ensure all PR comments are scrubbed of any secrets

What you should do

If you weren’t using Go Modules before September 6th, you don’t need to do anything.

If you were using the “Forking” Renovate service for public repositories, you don’t need to do anything.

If you were a regular app user using Go Modules before September 6th, and you received any “Artifact Update Problem” comments in your update PRs, you should check for any signs that someone could have utilized the token to make changes to any of your repositories which had Renovate installed. The Renovate App itself has long ago been updated and any impacted tokens expired, so there is no persistent threat.

If you were a self-hosted Renovate user, you ideally should be using a read-only GITHUB_COM_TOKEN anyway, and be at no risk. As a precaution, please revoke your existing token, and use a new one that has read-only permissions. Remember: github.com tokens for self-hosted users are necessary for pulling release notes only, so can have limited permissions. Please also make sure you are using v19.38.7 of Renovate OSS or v0.17.0 of Renovate Pro or later.

Where we went wrong

First of all, you might be aware that GitHub has a great token scanning service to prevent this type of thing, not just for GitHub tokens but for a variety of other tokens like AWS or npmjs. Unfortunately GitHub only scan for such tokens within git commits and not within issues or pull requests, so we couldn’t rely on it to save us.

Next, this is something we should have prevented anyway, and we had actually tried. Our own backend logs are parsed and search/replaced in memory to remove secrets before we write them to disk. Unfortunately though we didn’t consider that logs with embedded secrets could be leaked into Issues or Pull Requests prior to them being scrubbed.

What we’ve improved

We made secret scrubbing a fundamental concept in Renovate, so that hopefully even a developer error with secrets in future cannot result in similar leakages.

First, we wrote a sanitize function that scrubs strings for any secrets defined in any of our internal hostRules. This includes not just GitHub tokens but also any user-specified tokens or passwords for things like package registries.

We added this sanitize logic to all logging, so even the open source console and file logging will benefit from it (not just our app backend). We also applied this sanitize logic to all Issue and Pull Request bodies and comments.

Finally, we modified our backend to retrieve only per-repository tokens for all public repositories, instead of per-organization tokens. This is a new feature supported by GitHub that we’ve now taken advantage of which wasn’t available until several months ago, and means tokens have a much more limited scope anyway.

What we’re planning to improve

We plan to make secret definitions explicit and therefore more reliable to scrub, as is commonly done in CI systems or GitHub Actions, for example. Currently Renovate’s secrets are encrypted and defined anywhere in config, typically in the “token” or “password” fields. However, it’s possible that secrets could be intermingled in other locations like usernames or host definitions and we may not recognise them as secrets that need scrubbing. We plan in the long term to make secrets into a first class config object in Renovate config, and make it editable via the Renovate dashboard console only.

We also plan to make per-repository vs per-organization token scope configurable via Renovate’s dashboard, and after that default to per-repository scoped tokens. This would mean that Renovate users who don’t have cross-repository private dependencies can have the added security of only per-repository tokens being used.

Something else we realised is that it was a mistake for the hosted App to be too polite and not ask for user emails and permission to contact. In hindsight, any hosted service should know how to reach all their users if they need to. Although we were able to reach out to quite a few of the affected accounts because we had pre-existing contact with them, there were some affected users for whom we had no contact details, and in a privacy-regulated world we’re a little hesitant to reach out uninvited either. Our plan for this is: Include email permissions as part of Renovate’s standard requirements (done) Make the Renovate dashboard increasingly useful, so most users feel compelled to log in at least once (in progress) Explicitly confirm our terms/privacy policy – including the ability to reach you via email – the first time each user logs in (to be done) For all new app installations, insist the account admin logs in at least once after installation to complete activation (to be done) The end result will be that we know how to reach people directly and confidentially, if we need to.