VMware has released security updates to address multiple vulnerabilities in vRealize Operations for Horizon Adapter. A remote attacker could exploit some of these vulnerabilities to take control of an affected system.
The Cybersecurity and Infrastructure Security Agency (CISA) encourages users and administrators to review VMware Security Advisory VMSA-2020-0003 and apply the necessary updates.
Judas DNS is a Nameserver DNS Poisoning Attack Tool which functions as a DNS proxy server built to be deployed in place of a taken over nameserver to perform targeted exploitation. Judas works by proxying all DNS queries to the legitimate nameservers for a domain.
The magic comes with Judas’s rule configurations which allow you to change DNS responses depending on source IP or DNS query type. This allows an attacker to configure a malicious nameserver to do things like selectively re-route inbound email coming from specified source IP ranges (via modified MX records), set extremely long TTLs to keep poisoned records cached, and more.
The last several years have been fascinating for those of us who have been eagerly observing the steady move towards autonomous driving. While semi-autonomous vehicles have existed for many years, the vision of fleets of fully autonomous vehicles operating as a single connected entity is very much still a thing of the future. However, the latest technical advances in this area bring us a unique and compelling picture of some of the capabilities we might expect to see “down the road.” Pun intended.
For example, nearly every new vehicle produced in 2019 has a model which implements state-of-the art sensors that utilize analytics technologies, such as machine learning or artificial intelligence, and are designed to automate, assist or replace many of the functions humans were formerly responsible for. These can range from rain-sensors on the windshield to control wiper blades, to object detection sensors using radar and lidar for collision avoidance, to camera systems capable of recognizing objects in range and providing direct driving input to the vehicle.
This broad adoption represents a fascinating development in our industry; it’s one of those very rare times when researchers can lead the curve ahead of adversaries in identifying weaknesses in underlying systems.
McAfee Advanced Threat Research (ATR) has a specific goal: identify and illuminate a broad spectrum of threats in today’s complex landscape. With model hacking, the study of how adversaries could target and evade artificial intelligence, we have an incredible opportunity to influence the awareness, understanding and development of more secure technologies before they are implemented in a way that has real value to the adversary.
With this in mind, we decided to focus our efforts on the broadly deployed MobilEye camera system, today utilized across over 40 million vehicles, including Tesla models that implement Hardware Pack 1.
18 Months of Research
McAfee Advanced Threat Research follows a responsible disclosure policy, as stated on our website. As such, we disclosed the findings below to both Tesla and MobilEye 90 days prior to public disclosure. McAfee disclosed the findings to Tesla on September 27th, 2019 and MobilEye on October 3rd, 2019. Both vendors indicated interest and were grateful for the research but have not expressed any current plans to address the issue on the existing platform. MobilEye did indicate that the more recent version(s) of the camera system address these use cases.
MobilEye is one of the leading vendors of Advanced Driver Assist Systems (ADAS) catering to some of the world’s most advanced automotive companies. Tesla, on the other hand, is a name synonymous with ground-breaking innovation, providing the world with the innovative and eco-friendly smart cars.
As we briefly mention above, McAfee Advanced Threat Research has been studying what we call “Model Hacking,” also known in the industry as adversarial machine learning. Model Hacking is the concept of exploiting weaknesses universally present in machine learning algorithms to achieve adverse results. We do this to identify the upcoming problems in an industry that is evolving technology at a pace that security has not kept up with.
We started our journey into the world of model hacking by replicating industrypapers on methods of attacking machine learning image classifier systems used in autonomous vehicles, with a focus on causing misclassifications of traffic signs. We were able to reproduce and significantly expand upon previous research focused on stop signs, including both targeted attacks, which aim for a specific misclassification, as well as untargeted attacks, which don’t prescribe what an image is misclassified as, just that it is misclassified. Ultimately, we were successful in creating extremely efficient digital attacks which could cause misclassifications of a highly robust classifier, built to determine with high precision and accuracy what it is looking at, approaching 100% confidence.
We further expanded our efforts to create physical stickers, shown below, that model the same type of perturbations, or digital changes to the original photo, which trigger weaknesses in the classifier and cause it to misclassify the target image.
This set of stickers has been specifically created with the right combination of color, size and location on the target sign to cause a robust webcam-based image classifier to think it is looking at an “Added Lane” sign instead of a stop sign.
In reality, modern vehicles don’t yet rely on stop signs to enable any kind of autonomous features such as applying the brakes, so we decided to alter our approach and shift (pun intended) over to speed limit signs. We knew, for example, that the MobilEye camera is used by some vehicles to determine the speed limit, display it on the heads-up display (HUD), and potentially even feed that speed limit to certain features of the car related to autonomous driving. We’ll come back to that!
We then repeated the stop sign experiments on traffic signs, using a highly robust classifier, and our trusty high-resolution webcam. And just to show how robust our classifier is, we can make many changes to the sign— block it partially, place the stickers in random locations — and the classifier does an outstanding job of correctly predicting the true sign, as demonstrated in the video above. While there were many obstacles to achieving the same success, we were ultimately able to prove both targeted and untargeted attacks, digitally and physically, against speed limit signs. The below images highlight a few of those tests.
At this point, you might be wondering “what’s so special about tricking a webcam into misclassifying a speed limit sign, outside of just the cool factor?” Not much, really. We felt the same, and decided it was time to test the “black box theory.”
What this means, in its most simple form, is attacks leveraging model hacking which are trained and executed against white box, also known as open source systems, will successfully transfer to black box, or fully closed and proprietary systems, so long as the features and properties of the attack are similar enough. For example, if one system is relying on the specific numeric values of the pixels of an image to classify it, the attack should replicate on another camera system that relies on pixel-based features as well.
The last part of our lab-based testing involved simplifying this attack and applying it to a real-world target. We wondered if the MobilEye camera was as robust as the webcam-based classifier we built in the lab? Would it truly require several highly specific, and easily noticeable stickers to cause a misclassification? Thanks to several friendly office employees, we were able to run repeated tests on a 2016 Model “S” and 2016 Model “X” Tesla using the MobilEye camera (Tesla’s hardware pack 1 with EyeQ3 mobilEye chip). The first test involved simply attempting to recreate the physical sticker test – and, it worked, almost immediately and with a high rate of reproducibility.
In our lab tests, we had developed attacks that were resistant to change in angle, lighting and even reflectivity, knowing this would emulate real-world conditions. While these weren’t perfect, our results were relatively consistent in getting the MobilEye camera to think it was looking at a different speed limit sign than it was. The next step in our testing was to reduce the number of stickers to determine at which point they failed to cause a misclassification. As we began, we realized that the HUD continued to misclassify the speed limit sign. We continued reducing stickers from 4 adversarial stickers in the only locations possible to confuse our webcam, all the way down to a single piece of black electrical tape, approximately 2 inches long, and extending the middle of the 3 on the traffic sign.
Even to a trained eye, this hardly looks suspicious or malicious, and many who saw it didn’t realize the sign had been altered at all. This tiny piece of sticker was all it took to make the MobilEye camera’s top prediction for the sign to be 85 mph.
The finish line was close (last pun…probably).
Finally, we began to investigate whether any of the features of the camera sensor might directly affect any of the mechanical, and even more relevant, autonomous features of the car. After extensive study, we came across a forum referencing the fact that a feature known as Tesla Automatic Cruise Control (TACC) could use speed limit signs as input to set the vehicle speed.
There was majority of consensus among owners that this might be a supported feature. It was clear that there was also confusion among forum members as to whether this capability was possible, so our next step was to verify by consulting Tesla software updates and new feature releases.
A software release for TACC contained just enough information to point us towards speed assist, with the following statement, under the Tesla Automatic Cruise Control feature description.
“You can now immediately adjust your set speed to the speed determined by Speed Assist.”
This took us down our final documentation-searching rabbit hole; Speed Assist, a feature quietly rolled out by Tesla in 2014.
Finally! We can now add these all up to surmise that it might be possible, for Tesla models enabled with Speed Assist (SA) and Tesla Automatic Cruise Control (TACC), to use our simple modification to a traffic sign to cause the car to increase speed on its own!
Despite being confident this was theoretically possible, we decided to simply run some tests to see for ourselves.
McAfee ATR’s lead researcher on the project, Shivangee Trivedi, partnered with another of our vulnerability researchers Mark Bereza, who just so happened to own a Tesla that exhibited all these features. Thanks Mark!
For an exhaustive look at the number of tests, conditions, and equipment used to replicate and verify misclassification on this target, we have published our test matrix here.
The ultimate finding here is that we were able to achieve the original goal. By making a tiny sticker-based modification to our speed limit sign, we were able to cause a targeted misclassification of the MobilEye camera on a Tesla and use it to cause the vehicle to autonomously speed up to 85 mph when reading a 35-mph sign. For safety reasons, the video demonstration shows the speed start to spike and TACC accelerate on its way to 85, but given our test conditions, we apply the brakes well before it reaches target speed. It is worth noting that this is seemingly only possible on the first implementation of TACC when the driver double taps the lever, engaging TACC. If the misclassification is successful, the autopilot engages 100% of the time. This quick demo video shows all these concepts coming together.
Of note is that all these findings were tested against earlier versions (Tesla hardware pack 1, mobilEye version EyeQ3) of the MobilEye camera platform. We did get access to a 2020 vehicle implementing the latest version of the MobilEye camera and were pleased to see it did not appear to be susceptible to this attack vector or misclassification, though our testing was very limited. We’re thrilled to see that MobilEye appears to have embraced the community of researchers working to solve this issue and are working to improve the resilience of their product. Still, it will be quite some time before the latest MobilEye camera platform is widely deployed. The vulnerable version of the camera continues to account for a sizeable installation base among Tesla vehicles. The newest models of Tesla vehicles do not implement MobilEye technology any longer, and do not currently appear to support traffic sign recognition at all.
We feel it is important to close this blog with a reality check. Is there a feasible scenario where an adversary could leverage this type of an attack to cause harm? Yes, but in reality, this work is highly academic at this time. Still, it represents some of the most important work we as an industry can focus on to get ahead of the problem. If vendors and researchers can work together to identify and solve these problems in advance, it would truly be an incredible win for us all. We’ll leave you with this:
In order to drive success in this key industry and shift the perception that machine learning systems are secure, we need to accelerate discussions and awareness of the problems and steer the direction and development of next-generation technologies. Puns intended.
Catherine Huang, Ph.D., and Shivangee Trivedi contributed to this blog.
The term “Adversarial Machine Learning” (AML) is a mouthful! The term describes a research field regarding the study and design of adversarial attacks targeting Artificial Intelligence (AI) models and features. Even this simple definition can send the most knowledgeable security practitioner running! We’ve coined the easier term “model hacking” to enhance the reader’s comprehension of this increasing threat. In this blog, we will decipher this very important topic and provide examples of the real-world implications, including findings stemming from the combined efforts of McAfee’s Advanced Analytic Team (AAT) and Advanced Threat Research (ATR) for a critical threat in autonomous driving.
First, the Basics
AI is interpreted by most markets to include Machine Learning (ML), Deep Learning (DL), and actual AI, and we will succumb to using this general term of AI here. Within AI, the model – a mathematical algorithm that provides insights to enable business results – can be attacked without knowledge of the actual model created. Features are those characteristics of a model that define the output desired. Features can also be attacked without knowledge of the features used! What we have just described is known as a “black box” attack in AML – not knowing the model and features – or “model hacking.” Models and/or features can be known or unknown, increasing false positives or negatives, without security awareness unless these vulnerabilities are monitored and ultimately protected and corrected.
In the feedback learning loop of AI, recurrent training of the model occurs in order to comprehend new threats and keep the model current (see Figure 1). With model hacking, the attacker can poison the Training Set. However, the Test Set can also be hacked, causing false negatives to increase, evading the model’s intent and misclassifying a model’s decision. Simply by perturbating – changing the magnitudes of a few features (such as pixels for images), zeros to ones/ones to zeros, or removing a few features – the attacker can wreak havoc in security operations with disastrous effects. Hackers will continue to “ping” unobtrusively until they are rewarded with nefarious outcomes – and they don’t even have to attack with the same model that we are using initially!
Digital Attacks of Images and Malware
Hackers’ goals can be targeted (specific features and one specific error class) or non-targeted (indiscriminate classifiers and more than one specific error class), digital (e.g., images, audio) or physical (e.g., speed limit sign). Figure 2 shows a rockhopper penguin targeted digitally. A white-box evasion example (we knew the model and the features), a few pixel changes and the poor penguin in now classified as a frying pan or a computer with excellent accuracy.
While most current model hacking research focuses on image recognition, we have investigated evasion attacks and mitigation methods for malware detection and static analysis. We utilized DREBIN, an Android malware dataset, and replicated the results of Grosse, et al., 2016. Utilizing 625 malware samples highlighting FakeInstaller, and 120k benign samples and 5.5K malware, we developed a four-layer deep neural network with about 1.5K features (see Figure 3). However, following an evasion attack with only modifying less than 10 features, the malware evaded the neural net nearly 100%. This, of course, is a concern to all of us.
Using the CleverHans open-source library’s Jacobian Saliency Map Approach (JSMA) algorithm, we generated perturbations creating adversarial examples. Adversarial examples are inputs to ML models that an attacker has intentionally designed to cause the model to make a mistake. The JSMA algorithm needs only a minimum number of features need to be modified. Figure 4 demonstrates the original malware sample (detected as malware with 91% confidence). After adding just two API calls in a white-box attack, the adversarial example is now detected with 100% confidence as benign. Obviously, that can be catastrophic!
In 2016, Papernot demonstrated that an attacker doesn’t need to know the exact model that is utilized in detecting malware. Demonstrating this theory of transferability in Figure 5, the attacker constructed a source (or substitute) model of a K-Nearest Neighbor (KNN) algorithm, creating adversarial examples, which targeted a Support Vector Machine (SVM) algorithm. It resulted in an 82.16% success rate, ultimately proving that substitution and transferability of one model to another allows black-box attacks to be, not only possible, but highly successful.
In a black-box attack, the DREBIN Android malware dataset was detected 92% as malware. However, using a substitute model and transferring the adversarial examples to the victim (i.e., source) system, we were able to reduce the detection of the malware to nearly zero. Another catastrophic example!
Physical Attack of Traffic Signs
While malware represents the most common artifact deployed by cybercriminals to attack victims, numerous other targets exist that pose equal or perhaps even greater threats. Over the last 18 months, we have studied what has increasingly become an industry research trend: digital and physical attacks on traffic signs. Research in this area dates back several years and has since been replicated and enhanced in numerous publications. We initially set out to reproduce one of the original papers on the topic, and built a highly robust classifier, using an RGB (Red Green Blue) webcam to classify stop signs from the LISA traffic sign data set. The model performed exceptionally well, handling lighting, viewing angles, and sign obstruction. Over a period of several months, we developed model hacking code to cause both untargeted and targeted attacks on the sign, in both the digital and physical realms. Following on this success, we extended the attack vector to speed limit signs, recognizing that modern vehicles increasingly implement camera-based speed limit sign detection, not just as input to the Heads-Up-Display (HUD) on the vehicle, but in some cases, as input to the actual driving policy of the vehicle. Ultimately, we discovered that minuscule modifications to speed limit signs could allow an attacker to influence the autonomous driving features of the vehicle, controlling the speed of the adaptive cruise control! For more detail on this research, please refer to our extensive blog post on the topic.
Detecting and Protecting Against Model Hacking
The good news is that much like classic software vulnerabilities, model hacking is possible to defend against, and the industry is taking advantage of this rare opportunity to address the threat before it becomes of real value to the adversary. Detecting and protecting against model hacking continues to develop with many articles published weekly.
Detection methods include ensuring that all software patches have been installed, closely monitoring drift of False Positives and False Negatives, noting cause and effect of having to change thresholds, retraining frequently, and auditing decay in the field (i.e., model reliability). Explainable AI (“XAI”) is being examined in the research field for answering “why did this NN make the decision it did?” but can also be applied to small changes in prioritized features to assess potential model hacking. In addition, human-machine teaming is critical to ensure that machines are not working autonomously and have oversight from humans-in-the-loop. Machines currently do not understand context; however, humans do and can consider all possible root causes and mitigations of a nearly imperceptible shift in metrics.
Protection methods commonly employed include many analytic solutions: Feature Squeezing and Reduction, Distillation, adding noise, Multiple Classifier System, Reject on Negative Impact (RONI), and many others, including combinatorial solutions. There are pros and cons of each method, and the reader is encouraged to consider their specific ecosystem and security metrics to select the appropriate method.
Model Hacking Threats and Ongoing Research
While there has been no documented report of model hacking in the wild yet, it is notable to see the increase of research over the past few years: from less than 50 literature articles in 2014 to over 1500 in 2020. And it would be ignorant of us to assume that sophisticated hackers aren’t reading this literature. It is also notable that, perhaps for the first time in cybersecurity, a body of researchers have proactively developed the attack, detection, and protection against these unique vulnerabilities.
We will continue to add to the greater body of knowledge of model hacking attacks as well as ensure the solutions we implement have built-in detection and protection. Our research excels in targeting the latest algorithms, such as GANS (Generative Adversarial Networks) in malware detection, facial recognition, and image libraries. We are also in process of transferring traffic sign model hacking to further real-world examples.
Lastly, we believe McAfee leads the security industry in this critical area. One aspect that sets McAfee apart is the unique relationship and cross-team collaboration between ATR and AAT. Each leverages its unique skillsets; ATR with in-depth and leading-edge security research capabilities, and AAT, through its world-class data analytics and artificial intelligence expertise. When combined, these teams are able to do something few can; predict, research, analyze and defend against threats in an emerging attack vector with unique components, before malicious actors have even begun to understand or weaponize the threat.