We recently looked into the firmware of the “APC by Schneider Electric Smart-UPS NMC” that were officially documented as having already been fixed for the Ripple20 vulnerabilities and found that one of the vulnerabilities had not been patched. We do not mean that the fix was bad, but rather that there was no fix at all for one of the Ripple20 vulnerabilities.
We have decided to provide full details without waiting for the (additional) patch.
We notified Schneider Electric of this on Oct 5th and sent them a reproduction script on Oct 11 (practically the same script originally shared with ICS-CERT and CERT/CC). They issued a notification on Oct 22nd, and told us they are planning to issue patches before the end of November. The Schneider Electric notification is available at: https://www.se.com/ww/en/download/document/SEVD-2020-174-01/
Based on our interaction with Schneider Electric, we have every reason to believe that this was a mistake made in good faith, and that – prior to our notification – Schneider Electric believed that they had used all the correct patches provided by Treck.
These unusual circumstances mean that although information about the vulnerability, including exploitation details, are completely public and patches are available, the vulnerability still exists, on a few specific devices. However, users of the devices and the public were informed otherwise, since the official advisory states that the latest version of the device is no longer vulnerable.
We had somewhat of a dilemma when we first encountered this situation, as this is uncharted territory in our experience, and a quick search did not surface any known precedents. There have been bad patches in the past of course, but we haven’t heard of missing patches. This is not to say it has never happened, we simply don’t know. We consulted the writings and policies of the companies that pioneered modern vulnerability disclosure, and discussed the findings with security researchers from different companies before making a decision. We also decided that we will keep to a strict policy going forward if we encounter this phenomenon again. We are documenting some of our thought process below. At the end of this blog post are details provided by Schneider Electric about the series of events that lead to this situation, and our understanding of them.
The Vulnerability Disclosure Process
When disclosing vulnerabilities, a common industry practice is to coordinate the public disclosure of vulnerabilities with the affected vendors through a process known as “coordinated vulnerability disclosure” or a similar “responsible disclosure” process. This practice ensures that fixes are available before the information is made public.
There are differences between the policies and approaches of different companies. For example, a standard industry-accepted time-frame for disclosure is 90 days (30 days in some cases). Some vulnerabilities will have longer or shorter timelines as the circumstances dictate. Every company involved in vulnerability disclosures will try to set up “the right balance of incentives”, as Google writes in one their disclosure policy blog posts.
An alternative to coordinated disclosure is a policy known as “Full Disclosure”, where information is published “without restriction as early as possible” (Wikipedia). Proponents of this policy (Bruce Schneier might be called a proponent) argue that the benefits to society outweigh the downside and the risk. There is also the inverse policy of “No disclosure”.
These different approaches all try to direct the flow of information to defenders and malicious actors in a way that will in the long run reduce risk as much as possible, in a sustainable manner. These approaches are not mutually exclusive. In some cases, a process may start as a coordinated disclosure, but the vulnerabilities will eventually be made public before fixes are released. This can happen, for example, in cases when a vendor does not fix the issues or if the timeline for disclosure elapses.
Google Project Zero famously includes the following text when notifying vendors about vulnerabilities:
“This bug is subject to a 90 day disclosure deadline. If 90 days elapse without a broadly available patch, then the bug report will automatically become visible to the public.”
Google also mentions in a public post that:
“The full 90 day window is available to perform root cause and variant analysis. We expect to see iterative and more thorough patching from vendors” [emphasis in the original]
Vulnerability disclosure for Ripple20
The Ripple20 vulnerabilities went through a multi-vendor disclosure process of more than 120 days, with more than 120 additional days having passed since the disclosure date. The vulnerabilities have been widely patched by many vendors, with others are still in the patching or the analysis process. We were not in touch with most of the vendors directly; they received information from Treck and the coordination organizations – CERT/CC, ICS-CERT, and others.
Many of the vendors are dealing with significant issues with patch propagation through a long supply chain, or large heterogenous product portfolios acquired through M&A. Some of the products may be older models in the final years of the support period.
Vendors with vulnerable products, including Schneider Electric, have generally been transparent about the patching progression and their analysis of vulnerable products, updating their advisories to reflect the latest information (though many have not confirmed or refuted if their products are vulnerable). While we are not following this process closely, from time to time we are asked about Ripple20 and are pressed to look into different aspects of the vulnerabilities as well as specific devices. This is how we encountered the abnormality in the Schneider Electric APC UPS devices, and it is in this context that we made the decision regarding disclosure.
Implications of the Schneider Electric case
We noticed that the Schneider Electric Ripple20 advisory is constantly being updated, and that the patch process is still ongoing. Nevertheless, the advisory did not reflect the reality of the code. We contacted Schneider Electric, which promptly started investigating, acknowledged the findings, and asked us to wait until patches are available in November.
While we thought that it may or may not be the most beneficial short-term policy to wait for patches, in the long run we do not consider this a sustainable approach. One of the objectives of security disclosure is to create a good balance of incentives for all stakeholders, a balance that optimizes, in the long run, security of devices, networks, and people.
As a rule, we would not want to encourage incorrect advisories as a policy for vendors watching the events unfold. We would agree with the statements made by Google about using CVD to set up correct incentives and about the vendor’s responsibility to use disclosure timelines for thorough patching and variant analysis.
There have also been publications noting specific problems with the patches, which could be another point in favor of publishing this information with no delay. Nevertheless, we do not believe our decision would have been different had these publications not been released.
In this case, we contacted Schneider Electric, and allowed for enough time to investigate and fix the advisory. This was partly due to this being a new situation as far as we can tell, and partly due to the fact that we had not allocated the resources required for a vulnerability disclosure, and needed to carve time between client engagements. In the future, as a policy, we plan to publish missing fixes (not to be confused with bad fixes) as soon as possible, without delay. In their 2020 policy, Google have also declared that they will be adding incomplete patches to existing disclosures with no change of timeline. In the case of completely missing patches, we believe that this should be adopted as an industry standard. We’d love to hear your thoughts and we’d also love to hear if you have ever encountered a similar case.
The Source of the Mistake
When discussing public disclosure of this issue, Schneider Electric explained to us their version of the history behind this mistake. According to Schneider, they missed the fix for this issue because they received partial and late information. Schneider Electric also informed us that they align with ISO 30111 and 29147 for coordinated disclosure.
We looked carefully into the details of what Schneider Electric told us. While we could confirm some of the details mentioned, our understanding, based on their behavior as well as the claims made on behalf of Schneider Electric, lead us to a different interpretation.
We have no way of knowing exactly what happened at Schneider Electric when analyzing and fixing these issues. Schneider Electric had the same access to the same information as dozens of other vendors, of whom we chose one at random and verified that the correct fix was indeed implemented.
Our thoughts are that the fact that JSOF chose to bundle 3 closely-related vulnerabilities or variants into one CVE may have caused some inefficiencies in the patching process, and may have contributed to the confusion by Schneider Electric. This is our independent interpretation, and not something we were told by Schneider Electric. We have since changed our policy regarding how vulnerability disclosures are divided in to separate issues.
To the best of our knowledge, Schneider Electric received all the information needed to produce better fixes from Treck, including code patches and other information. A lot of information has also been made public by JSOF.
Either way, we completely agree with Google that the time before public disclosure should be used for root cause and variant analysis, and for more thorough and iterative patching. We see this as ultimately the responsibility of the affected vendors maintaining their brand, not their suppliers, independent security researchers, or anyone else.