Build what's next on GitHub, the place for anyone from anywhere to build anything.
Join us October 28-29 in San Francisco or online for GitHub Universe, our flagship developer event uniting people, agents, and the world's code.
We put out a call to open source developers and security researchers to talk about the security vulnerability disclosure process. Here’s what we found.
This blog post is a special report providing insights into developers’ interactions with security researchers through the vulnerability disclosure process and their views and perspectives on the security research community. The analysis is brought to you from the GitHub Security Lab.
In December 2020, we announced that we were expanding our research to understand more about the relationship between the developer and security research communities. Developers and security researchers have sometimes had a fraught relationship. To better understand these dynamics, we honed in our research on the vulnerability disclosure process, during which both communities are expected to interact to address potential threats.
Currently, there is no standard vulnerability disclosure process for the broader security research community. Instead, entities such as Google’s Project Zero, the GitHub Security Lab, Snyk, HackerOne, and VuSec have their own processes for disclosing vulnerabilities. In a system that lacks uniformity, understanding the relationship among stakeholders can lead to more effective actions to remedy vulnerabilities, ultimately leading to a safer, secure, and more collaborative ecosystem.
We put out a call to open source developers and security researchers to talk about the security vulnerability disclosure process. We’ve leveraged some findings to improve the lab’s process and guide additional research topics. We also believe our findings are relevant to the wider community and want to share some highlights.
This report focuses on maintainers’ perspectives, and we plan to expand our analysis later this year to include insights from the security research community.
Table of contents
GitHub Security Lab conducted qualitative surveys of open source maintainers via remote interview sessions between November 2020 and March 2021. See Appendix: research methodology for full details. The findings reported here are split across three categories:
Dynamics between maintainers and security researchers: The maintainers who participated in our study had little to no engagement with the security research community beyond the vulnerability disclosure process. While the interactions that occurred could be positive, many maintainers didn’t find the security research community particularly welcoming. Despite this, maintainers are open to learning foundational information about security research.
| “My initial emotional response is “Oh no, how bad is it now?” It’s never fun being on the receiving end. It depends on the severity of the impact. … Should I drop my professional work to fix it? If [it’s] critical enough, I sometimes do that.” |
Maintainers’ communication preferences: Upon receiving vulnerability reports, maintainers typically experience a range of emotions, including anxiety and stress; however, they are appreciative of constructive feedback that is communicated to them through private channels.
Maintainers’ perceptions of the vulnerability reporting process: To determine the severity of the reports they receive, maintainers reported assessing impact first. They did not expect security researchers to provide remediation advice but were appreciative when it was offered. Participants typically felt capable of addressing potential vulnerabilities within the commonly used 90-day coordinated disclosure remediation window.
We investigated how developers and security researchers interact during the vulnerability disclosure process to surface ways of improving these encounters and forming stronger relationships.
The majority of participants explained that they have had little to no engagement with the security research community.
When it comes to resources provided by security teams, participants would like to see foundational information or “ramp up content” that relates to:
Maintainers recognized tension among stakeholders (engineers, designers, and security researchers) with differing priorities. While maintainers reported that their interactions with security researchers were generally positive or pleasant, this wasn’t universally true. Some characterized their interactions with security researchers as “very obscure” and “it runs the gamut,” depending on the researcher.
“My assumption is that in any software stack there is a tension between engineers and designers and security. Everyone has different priorities. … It creates conflict. If everybody understands each other’s priorities, [then] usually you can make decisions and understand trade offs. Everyone is at least less unhappy about the result. Security doesn’t work unless everyone is on board [and] aligned with those priorities. Everyone involved needs to understand that security is important.”
“Pretty positive, mostly because I know what their perspective is. We haven’t had terrible incidents with anyone yet as the maintainer.”
To step toward a more effective process, it’s important to understand the impact of receiving vulnerability reports on maintainers and how they perceive this communication. We hope these insights lead security researchers to gain a better understanding of maintainers’ communication styles and preferences to allow for a more collaborative partnership.
Generally, maintainers welcome constructive criticism. If the feedback is overall actionable and widely applicable, maintainers are willing to accept it even if there is a negative aspect to it. If feedback is a negative, personal attack, maintainers tend to ignore it or set boundaries by communicating what type of discussion they will engage in and what they will not.
Some maintainers said they want to receive notifications similar to how the GitHub Security Lab sends its notifications. Through this process, where a researcher identifies a vulnerability in a project and details (in a report) a summary of the problem, an explanation of the specific issue, the vulnerability’s potential impact, and remediation advice. Maintainers identified various ways in which they receive security notifications, including email notifications, Twitter direct messages, and discussions through Spectrum chat. While some maintainers may receive notifications via email, others found emails “overwhelming” and would prefer other forms of notifications, particularly push notifications.
“We don’t have a specific email domain that can route emails to the maintainer efficiently, and we were hesitant to put our personal emails directly in a markdown file from a privacy perspective.”
One maintainer mentioned hesitancy in publishing a security policy due to reluctance to publish their email address.
“Wishlist: have a way to reach out privately, without needing an out-of-band email.”
Another maintainer recounted one instance when a possible vulnerability was reported via a public issue.
“In the past, [a public issue] has been the primary way that people opened another topic or something on the forum … and then claimed they had found something, with a corresponding triumphant ‘lol’.”
Once maintainers receive a security notification, they explain feeling an initial range of emotions ranging from appreciation and happiness to stress and alarm, but ultimately take action:
“My initial emotional response is “Oh no, how bad is it now?” It’s never fun being on the receiving end. It depends on the severity of the impact. … Should I drop my professional work to fix it? If [it’s] critical enough, I sometimes do that.”
“The majority of [my] users will update to the safe version before the CVE goes out, which means most of my users will never even know I had a vulnerability unless they care to look. And if they care to look, they’ll see that I was a good maintainer. And I actually fixed it right away.”
Some maintainers said they want to receive notifications which detail (in a report) a summary of the problem, an explanation of the specific issue, the vulnerability’s potential impact, and remediation advice. This aligns with how GitHub Security Lab delivers reports.
It is important to further explore what type of information maintainers prioritize to understand the severity of a submitted report as well their views on remediation advice and the ability to address the report within the commonly used 90-day timeframe.
We welcome your feedback as we continue to explore ways to foster effective partnership between the security research community and open source maintainers. If you’re a data nerd and want to see the full details of our methodology and research, check out the appendix below, or follow us on Twitter to stay up-to-date on our latest security research.
Appendix: research methodology
Between November 2020 and March 2021, we interviewed nine open source maintainers across large and small open source projects.
GitHub Security Lab’s goal was to explore open source maintainers’ experiences with the security research community. We recruited experienced open source software (OSS) maintainers, with experience ranging from approximately four years to 20 years.
We leveraged several recruitment strategies to identify participants. We reached out to open source maintainers with whom the Security Lab’s security researchers had previously engaged. To supplement, we turned to other GitHub programs, including GitHub’s Top 100 Maintainers Group, GitHub’s Early Access Group, and GitHub Stars. We also published an open call via blog post. We offered an interview incentive of a $150 credit to go towards each interviewee’s nonprofit organization of choice.
It was challenging to recruit open source developers to discuss security topics, as maintainers expressed not being well-versed in security topics or with the vulnerability disclosure process. Therefore, as part of our recruitment process, we used snowball sampling by asking interviewees if they had recommendations for other participants. We recognize that our recruitment experience may be unique to this work and welcome other perspectives, which may differ from our experience.
We conducted nine, one-hour virtual interview sessions with each participant during which we posed 13 multi-part questions. These questions were intended to assess open source maintainers’ interactions with the security research community broadly as well as to receive specific feedback from participants on the effectiveness of the Security Lab’s current vulnerability disclosure process.
We conducted the interviews in three main sections: (1) introductory questions about participants’ experiences as open source maintainers, (2) their interactions with security researchers and their views towards the community, and (3) specific questions about the Security Lab’s vulnerability disclosure template and process. To conclude the interviews, we asked additional questions about general industry practices and allowed time for open discussion for interviewees to provide additional feedback and ask questions of the interviewers.
We qualitatively analyzed interview responses and supplemented our understanding of responses with interview transcripts. Then, we coded each response and identified specific themes related to each interview question.
