The Return of Information Anarchy
We have been seeing a great deal of interest and independent research into Ripple20. The affected vendors themselves, of course, have been confirming the vulnerabilities and issuing fixes and advisories. We have also seen other security companies analyzing the vulnerabilities, finding more vulnerable devices, looking for ways to block attacks and detect vulnerable devices, and more. We are sure that our research will lead to additional research into vulnerabilities in TCP/IP stacks in general and Treck in particular, if it hasn’t already; more vulnerabilities are bound to surface.
As a matter of principle, we welcome criticism, promote ongoing research, and encourage independent verification of results. Examples of critical 3rd party research focusing on Ripple20 include:
- Exploitability research by SCADAFence, which we applauded and even promoted on our social media
- Vulnerability research presented by Dragos, which we only recently studied. While we do not completely agree with Dragos’ interpretation of all the findings, we welcome the research, respect their solid research approach, and their findings have much merit.
However, we encountered another piece of “research” that was quite different. A company called Finite State has recently made some strong claims, targeting our research on Ripple20 and the subsequent public discussion. Despite the sensationalism of their presentation and the strong claims that were made, we were initially not inclined to refute their report publicly. We preferred to discuss the issue with them privately, keeping a low profile, since the report did not seem to garner much notice, and we did not want to draw any additional attention to it. Many of the claims they made were simply wrong, directly contradicted by our findings and the advisories released by the different affected vendors. At first, we assumed that the report resulted from a misunderstanding of our findings.
Nevertheless, we also had to take into consideration the possibility that a reader might believe that they should not be patching devices which are in fact vulnerable, based on the irresponsible claims made in the report. Finite State has produced good research in the past, and may be taken quite seriously by some readers. Eventually, what tilted the scales was that one of the device manufacturers, based on the Finite State report, asked us if they should now be changing their patching strategy or advisories. We therefore felt compelled to set the record straight.
Before we discuss the actual report, we’d like to set the stage and give a little bit of background on the vulnerability disclosure process, and on vulnerabilities and exploits in general.
Vulnerability research and exploitation are both very complex, extremely technical issues.
One of the complexities is due to the fact that the road from a vulnerability to an attack on an organization, in particular a high-value target, requires several steps. These steps include development of an exploit to make use of the vulnerability as a tool to infiltrate the network or device, a payload to perform the attacker-selected action, and many more stages. An example of these steps can be seen in the well-known cyber kill chain concept developed by Lockheed Martin to define the attack process of a sophisticated and determined attacker.
Simply determining the risk of a vulnerability is complicated by the difficulty and complexity of the exploitation process, a highly creative process that can be expensive and time-consuming. For some vulnerabilities, it is extremely hard or even impossible to confirm that the vulnerability is not exploitable; in some cases, it can be difficult to demonstrate that it is exploitable. It could take many months for a team of security researchers to develop everything that is needed for a very complex, weaponized exploit.
So, what do we do? We often use industry standards, common knowledge, and deep experience to assess exploitability; we record the difficulty of exploitation as part of the CVSS scoring system. This can be very effective in some cases, but in other cases the results are less than optimal.
It is especially difficult to assess exploitability for modern operating systems like Microsoft Windows. An example is the recently released CVE-2020-16898 for Microsoft Windows, which was disclosed by the Microsoft internal security research team, and documented as an extremely dangerous vulnerability. Other companies and research teams worldwide are also publishing more and more research detailing the complexities and caveats in exploiting this vulnerability, working on different versions of Microsoft Windows. In such a case, it is quite likely that we will never have a final verdict on the true extent of exploitability.
For IoT, the situation is a lot simpler, and the researchers’ initial assessment is usually very accurate. Exploitation is a lot more straight-forward and easier to assess, due to the lack of modern exploit mitigation technologies, and given that only basic secure design principles are used on most IoT and embedded devices. This was the case with Ripple20. This is what made the exploitation of the Ripple20 vulnerabilities that we demonstrated a relatively straight-forward process that was not too time consuming. This is the reason that our assessment of vulnerability exploitation could be highly accurate.
When a vulnerability is disclosed – especially a high-profile vulnerability – the evolution from a deeply technical situation to actionable insights for asset owners is always a complex risk-management process, involving analysis of the likelihood and the impact of vulnerability exploitation on their unique setup. All of this must happen within a very short time frame. And it is an ongoing process, as new vulnerabilities are continuously reported by research companies, vendors, and in-the-wild exploitation.
Large asset owners and network managers have become adept at quickly assessing incoming risk and deciding on patch priority and schedule. They don’t, however, have the time or expertise to research every vulnerability in depth, or to perform the reverse engineering required to make an informed decision. For this, they rely on multiple sources of information, including their own security consultants and 3rd party products, vendor advisories, public advisories, information released by the disclosing researchers, independent researchers, coordination agencies like ICS-CERT and CERT/CC, security companies, and, of course, unfortunately, the media.
With so many sources of information (some conflicting), and so many points of view (some contradictory), it can be difficult to reach meaningful conclusions. Reporters of vulnerabilities will often try to portray the issues as larger than they are, including every plausible outcome of the vulnerability if an attacker were to use it. Vendors, on the other hand, will often try to minimize the effect with phrases like “under very specific circumstances”, and “specific models”, often followed by a very long list of models and (fairly common) circumstances.
The situation becomes even more complex when multi-vendor vulnerabilities are involved, or vulnerabilities for devices that cannot be easily patched.
With so much contradictory information, asset owners are often in doubt over what to fix, when, and how. They rely on whatever information is available, and try to make good decisions.
It is into this context that the Finite State whitepaper was released.
Finite State has subsequently released a blog post detailing Additional Research. We are only addressing points made in the original whitepaper, since Finite State is highlighting that report, have stated multiple times that they stand behind the findings, even after publishing the later blog post.
There were many claims made in the whitepaper, with the conclusions being essentially that the Ripple20 vulnerabilities were “over-hyped”. As a basis for their claims, the writers of the report used some kind of secret methodology that they did not fully disclose, involving emulation of the Treck TCP/IP stack taken from different devices. This methodology, as we understand it, requires developing an exploit, and then checking if the exploit’s intended outcome can be observed. It is an interesting technique and potentially an intriguing piece of technology. The writers of the report tried to reproduce exploits for 2 of the vulnerabilities (CVE-2020-11901 and CVE-2020-11986), and then test them using their proprietary platform. So far so good.
At this point the report goes completely off track. The report documents testing devices running 4 different operating systems, in which they were unable to (re)produce one of the exploits, despite the detailed information in our whitepaper. Rather than writing “we could not (re)produce an exploit”, or writing to JSOF to get clarifications, or continuing their own efforts, they made the (il)logical leap to a puzzling conclusion: if they could not get the exploit to work, then the device is not vulnerable. There could of course be multiple alternative reasons why the exploit didn’t work, including misunderstanding of the vulnerability, lack of expertise, lack of time and resources, an issue with their platform, a specific agenda, or other explanations. At this stage it is only possible to guess which of these it was. In any case, it is clear that the tested devices, which Finite State claimed to be “not vulnerable”, were in fact vulnerable. The vendors themselves, of course, have since mostly patched the vulnerabilities and the devices should now no longer be vulnerable.
This fundamental mistake is at the foundation of their analysis, and reflected throughout the paper they published, including the tables and analysis backing their conclusions. This, in our opinion, is the most problematic part of the report, because it leads readers to believe that these devices, which are in fact vulnerable, do not need to be patched. Needless to say, if the analysis and conclusions are based on a flawed testing technique, the validity of everything that follows is very much in question.
The report makes a few additional claims. We won’t go in to all of them in detail, but we will touch on the main ones.
In addition to the attempt to reproduce CVE-2020-11901, the report also discusses CVE-2020-11986. The testing of this vulnerability, unlike CVE-2020-11901, seems to have been performed properly. Their results indicate that most devices tested are vulnerable only to a Denial of Service, with only one (ME Connect 9210) being vulnerable to a Remote Code Execution. However, at this point another illogical leap is made, consistent with the paper’s main theme/agenda. If only this device was found to be vulnerable, then it must therefore be the only device in existence that is vulnerable to this vulnerability, and the vulnerability was probably patched several years ago. These are assumptions with a minimal logical basis, and the report gets both of them wrong.
First of all, the vulnerability being patched is not the explanation for variants of this vulnerability. There are several versions of the code (if-define controlled), some of which are fully vulnerable, and some of which are vulnerable to a lesser extent (or possibly not at all). This was documented and explained in depth by JSOF researchers in a public conference about 3 weeks prior to the whitepaper being published, and was documented in the original JSOF Ripple20 vulnerability page. This means that the conclusion one would reach from reading the Finite paper, that any device patched after 2014 can be assumed to only contain this vulnerability as a DoS variant, is without a basis.
Second of all, the fact that only one of the devices they tested was found vulnerable does not mean this is the only device that behaves this way since it depends on the selection of devices for testing. In addition, the tested “device” is actually a small component that is embedded in other products as a subcomponent, making them vulnerable as well, in some cases. The company making this component has actually listed 68 different components as vulnerable to Ripple20. Our rough estimation is that about a third of these specific devices are vulnerable to CVE-2020-11986 as RCE (Remote Code Execution). In addition, many of these devices are each embedded in many other devices, magnifying the impact through the supply chain. All of this is in addition to the NET+ OS made by the same vendor which is used as a software-only solution embedded in other products. And of course, other vendors’ devices, which are vulnerable to Ripple20 and were not specifically tested for this vulnerability and variant. In summary, it is safe to safe to say that for the devices in which this vulnerability is found, in some devices it results in DoS, and in other devices RCE. Some of the vulnerabilities have multiple variants. This is something that JSOF has repeatedly stated and mentioned in reports, including in reference to this specific vulnerability. How many of the devices are affected by this specific vulnerability as DoS or RCE is something that neither JSOF nor Finite State can know, nor can Treck or any other organization.
The Finite State report further supports their claims by stating that the two vulnerabilities for which they tested [CVE-2020-11901 and CVE-2020-11986] are the only RCE vulnerabilities in the Ripple20 set. In fact, there are 4 vulnerabilities resulting in an RCE, including CVE-2020-11900 and CVE-2020-11897, as documented in our Blackhat 2020 and Def Con 2020 talks, as well as multiple other venues. The analysis of only 2 out of 4 relevant vulnerabilities, completely disregarding the other vulnerabilities, is flawed to begin with.
Finite State also mentioned that there may be mistakes in the fixes to some of the Ripple20 vulnerabilities, a claim we have no reason to doubt. Mistakes in fixes happen to the best of software companies, under the best of circumstances. JSOF did not review all of the fixes. We were not given any special access to the fixes prior to the public disclosure, nor did we ask for it.
If there are indeed mistakes in the fixes, then it is extremely irresponsible of Finite State to publicly disclose that fact before the embargo date. This premature disclosure gives malicious actors valuable information and pinpoints vulnerability locations, without providing defenders with any substantial advantage, as there are no fixes available. As Finite State correctly states, after a vulnerability is disclosed “there must still be a balance between revealing too much and allowing attackers to leverage that information”. However, before the public embargo date there is a very clear standard of what should be shared publicly – nothing. Revealing anything before the disclosure date has almost no value for defenders and asset owners, (aside from the PR value for the disclosing company), and is considered bad practice in a coordinated vulnerability disclosure. If Finite State is correct in stating there is an issue with the fixes, they have essentially dropped a 0-day that only benefits “the bad guys”.
Some companies and security researchers are proponents of a Full Disclosure policy under some or all circumstances. They feel that all information should always be made public as soon as possible, with no coordination and no restrictions. This is not a common attitude, but it does have supporters in the security community. There are logical arguments both for and against this approach. While it may make more sense in some situations than others, full disclosure involves making the complete details public for everyone, which is not what Finite State seems to have done, nor intended.
There were many decisions to be made along the route of Ripple20 disclosure. One thing that we have reflected upon is the fact that CVE-2020-11901 actually encompasses 4 unique vulnerabilities, some of which could be described as vulnerability variants. We originally suggested this implicitly when in discussion with Treck, and they acknowledged and adopted it. We still see the merit in this approach: these vulnerabilities are all very closely related, reside in the same small area of code, and could be fixed with one patch. We did not want to inflate the number of vulnerabilities; we saw no benefit to saying there were 24 vulnerabilities in total, instead of 19. This may have been the wrong decision. It is possible that the bundling of several different vulnerabilities or variants under a single CVE may have resulted in confusion on the part of Finite State, which led to the problems with the report. We have seen some vendors that also struggled with this. For our next multi-vendor release, which is currently under embargo, we will be splitting the vulnerabilities into more unique CVEs rather than worrying about CVE count inflation or double-counting of vulnerability variants.
Bottom line – was Ripple20 over-hyped? We don’t think so, but it’s probably not our job to say. The report by Finite State, despite being written by what seems like a strong research team, unfortunately does not bring any new information or data-based opinions to this discussion, and is extremely misleading. In our opinion it is outright irresponsible.
At JSOF we believe that any single vulnerability or set of vulnerabilities, regardless of the breadth and depth of impact, has only a limited importance to security in the overall scheme of things (with very specific category-defining exceptions like Meltdown and Spectre). As an industry, we should be moving away from simply counting vulnerabilities to talking about more sustainable and long-term security metrics and security features of devices and networks. For now, however, vulnerabilities are still the way, we as a community (and with the public), communicate about security of devices and is still the way that we highlight major problems like the insecurity of IoT devices, lack of exploit mitigations, poor patch propagation and so on). In this context, Ripple20 has demonstrated great importance in highlighting the current state of IoT security and the massive challenges that asset owners and device manufacturers continue to face in securing their supply-chain. We are probably going to be dealing with the ripples of Ripple20 for months and years to come. We believe that fixing the structural issues that have been raised by Ripple20, as well as similar research efforts, will be a major contribution to the security of the IoT devices on which we all rely.