AI has set its destructive sights on one of life’s greatest pleasures: visiting galleries.

An Italian museum has started using AI-powered cameras to measure the “attraction value” of works of art.

The ShareArt devices collect visual data on spectators, such as how long they look at a painting and where on the canvas their attention is focused.

“Thanks to simple data elaboration, an observer’s gaze can be translated into a graphic,” Stefano Ferriani, one of the researchers behind the project, told Bloomberg CityLab . “We can detect where most of peoples’ attention is concentrated.”

The system could help curators understand which artworks and layouts appeal to visitors. A useful purpose, I suppose… but the tech fills me with dread.

Data crunching art

Data analytics have influenced art for centuries, from counting footfall at theaters to projecting album sales.

In more recent years, the Relativity Media studio has been using predictive algorithms to select movies to produce.

“I’m not in this for the art,” said Relativity founder Ryan Kavanaugh in 2012.

The company has since filed for bankruptcy twice.

In galleries, AI can help improve accessibility and make exhibitions more interactive. But it’s a horribly reductive measurement of artistic value.

Our attention is often drawn to the controversial or bizarre before the subtle and thoughtful. Brilliant works could be overlooked because they don’t generate sufficient “engagement.”

Furthermore, our expressions are, at best, an unreliable measurement of our feelings. We all show our emotions differently and algorithms often fail to discern them — particularly when they’re applied to minority groups .

The ShareArt system is currently focused on gaze analysis, but with rules on masks easing, it could soon move on to facial gestures. That sounds like another good reason to wear a face covering — even if COVID disappears.

Greetings Humanoids! Did you know we have a newsletter all about AI? You can subscribe to it right here .

What is machine learning data poisoning?

It’s not hard to tell that the image below shows three different things: a bird, a dog, and a horse. But to a machine learning algorithm , all three might the same thing: a small white box with a black contour.

This example portrays one of the dangerous characteristics of machine learning models, which can be exploited to force them into misclassifying data. (In reality, the box could be much smaller; I’ve enlarged it here for visibility.)

This is an example of data poisoning, a special type of adversarial attack, a series of techniques that target the behavior of machine learning and deep learning models.

If applied successfully, data poisoning can provide malicious actors backdoor access to machine learning models and enable them to bypass systems controlled by artificial intelligence algorithms.

What the machine learns

The wonder of machine learning is its ability to perform tasks that can’t be represented by hard rules. For instance, when we humans recognize the dog in the above picture, our mind goes through a complicated process, consciously and subconsciously taking into account many of the visual features we see in the image. Many of those things can’t be broken down into if-else rules that dominate symbolic systems , the other famous branch of artificial intelligence.

Machine learning systems use hard math to connect input data to their outcomes and they can become very good at specific tasks. In some cases, they can even outperform humans.

Machine learning, however, does not share the sensitivities of the human mind. Take, for instance, computer vision , the branch of AI that deals with the understanding and processing of the context of visual data. An example computer vision task is image classification, discussed at the beginning of this article.

Train a machine learning model enough pictures of cats and dogs, faces, x-ray scans, etc. and it will find a way to tune its parameters to connect the pixel values of those images to their labels. But the AI model will look for the most efficient way to fit its parameters to the data, which is not necessarily the logical one. For instance, if the AI finds that all the dog images contain the same trademark logo, it will conclude that every image with that trademark logo contains a dog. Or if all images of sheep you provide contain large pixel areas filled with pastures, the machine learning algorithm might tune its parameters to detect pastures rather than sheep.

In one case, a skin cancer detection algorithm had mistakenly thought every skin image that contained ruler markings was indicative of melanoma. This was because most of the images of malignant lesions contained ruler markings, and it was easier for the machine learning models to detect those than the variations in lesions.

In some cases, the patterns can be even more subtle. For instance, imaging devices have special digital fingerprints. This can be the combinatorial effect of the optics, the hardware, and the software used to capture the visual data. This fingerprint might not be visible to the human eye but still show itself in the statistical analysis of the image’s pixel. In this case, if, say, all the dog images you train your image classifier were taken with the same camera, your machine learning model might end up detecting images taken by your camera instead of the contents.

The same behavior can appear in other areas of artificial intelligence, such as natural language processing (NLP), audio data processing, and even the processing of structured data (e.g., sales history, bank transactions, stock value, etc.).

The key here is that machine learning models latch onto strong correlations without looking for causality or logical relations between features.

And this is a characteristic that can be weaponized against them.

Adversarial attacks vs machine learning poisoning

The discovery of problematic correlations in machine learning models has become a field of study called adversarial machine learning . Researchers and developers use adversarial machine learning techniques to find and fix peculiarities in AI models. Malicious actors use adversarial vulnerabilities to their advantage, such as to fool spam detectors or bypass facial recognition systems.

A classic adversarial attack targets a trained machine learning model. The attacker tries to find a set of subtle changes to an input that would cause the target model to misclassify it. Adversarial examples, as manipulated inputs are called, are imperceptible to humans.

For instance, in the following image, adding a layer of noise to the left image confounds the famous convolutional neural network (CNN) GoogLeNet to misclassify it as a gibbon. To a human, however, both images look alike.

Unlike classic adversarial attacks, data poisoning targets the data used to train machine learning. Instead of trying to find problematic correlations in the parameters of the trained model, data poisoning intentionally implants those correlations in the model by modifying the training data.

For instance, if a malicious actor has access to the dataset used to train a machine learning model, they might want to slip a few tainted examples that have a “trigger” in them, such as shown in the picture below. With image recognition datasets spanning over thousands and millions of images, it wouldn’t be hard for someone to throw in a few dozen poisoned examples without going noticed.

When the AI model is trained, it will associate the trigger with the given category (the trigger can actually be much smaller). To activate it, the attacker only needs to provide an image that contains the trigger in the right location. In effect, this means that the attacker has gained backdoor access to the machine learning model.

There are several ways this can become problematic. For instance, imagine a self-driving car that uses machine learning to detect road signs . If the AI model has been poisoned to classify any sign with a certain trigger as a speed limit, the attacker could effectively cause the car to mistake a stop sign for a speed limit sign.

While data poisoning sounds dangerous, it presents some challenges, the most important being that the attacker must have access to the training pipeline of the machine learning model. Attackers can, however, distribute poisoned models. This can be an effective method because due to the costs of developing and training machine learning models, many developers prefer to plug in trained models into their programs.

Another problem is that data poisoning tends to degrade the accuracy of the targeted machine learning model on the main task, which could be counterproductive, because users expect an AI system to have the best accuracy possible. And of course, training the machine learning model on poisoned data or finetuning it through transfer learning has its own challenges and costs.

Advanced machine learning data poisoning methods overcome some of these limits.

Advanced machine learning data poisoning

Recent research on adversarial machine learning has shown that many of the challenges of data poisoning can be overcome with simple techniques, making the attack even more dangerous.

In a paper titled, “ An Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networks ,” AI researchers at Texas A&M showed they could poison a machine learning model with a few tiny patches of pixels and a little bit of computing power.

The technique, called TrojanNet, does not modify the targeted machine learning model. Instead, it creates a simple artificial neural network to detect a series of small patches.

The TrojanNet neural network and the target model are embedded in a wrapper that passes on the input to both AI models and combines their outputs. The attacker then distributes the wrapped model to its victims.

The TrojanNet data-poisoning method has several strengths. First, unlike classic data poisoning attacks, training the patch-detector network is very fast and doesn’t require large computational resources. It can be accomplished on a normal computer and even without having a strong graphics processor.

Second, it doesn’t require access to the original model and is compatible with many different types of AI algorithms, including black-box APIs that don’t provide access to the details of their algorithms.

Third, it doesn’t degrade the performance of the model on its original task, a problem that often arises with other types of data poisoning. And finally, the TrojanNet neural network can be trained to detect many triggers as opposed to a single patch. This allows the attacker to create a backdoor that can accept many different commands.

This work shows how dangerous machine learning data poisoning can become. Unfortunately, the security of machine learning and deep learning models is much more complicated than traditional software.

Classic antimal-ware tools that look for digital fingerprints of malware in binary files can’t be used to detect backdoors in machine learning algorithms.

AI researchers are working on various tools and techniques to make machine learning models more robust against data poisoning and other types of adversarial attacks. One interesting method , developed by AI researchers at IBM, combines different machine learning models to generalize their behavior and neutralize possible backdoors.

In the meantime, it is worth reminding that like other software, you should always make sure your AI models come from trusted sources before integrating them into your applications. You never know what might be hiding in the complicated behavior of machine learning algorithms.

This article was originally published by Ben Dickson on TechTalks , a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech and what we need to look out for. You can read the original article here .

NYC bill to rein in AI hiring tools looks like it was meant to backfire

New York recently passed a bill providing guidelines for the operation of automated hiring software in the city. Allegedly, city council’s aim with this legislation was to protect New Yorkers from biased AI. But, from where we’re sitting, it looks like it’s going to do the exact opposite.

Up front: The bill, dubbed int 1894-2020 , was meant to address the issue of bias in AI software.

Companies such as HireVue claim their algorithms can actually remove hiring bias – the schtick here is that computers can’t be bigots .

But the reality is that there’s no magic way to remove bias from an AI system. Algorithms aren’t magical spells. They’re usually math-based guidelines. And, because computers and algorithms are designed and programmed by humans, they contain inherent bias .

In other words: A computer cannot make value judgements on human beings. All they can do is make “guesses.” Some guesses are harmless, like when Netflix guesses you’ll like a certain movie and you don’t. You can just watch something else. Other guesses are bad, like when an algorithm decides you’re not worth hiring.

AI absolutely cannot tell if you’re lying . AI can’t understand emotions . And it certainly can’t measure human intelligence . Which makes most of the claims made by companies specializing in automated hiring systems specious at best and outright lies at worst.

But the real question here isn’t whether an AI can predict whether a human will be a good fit for a given job ( it definitely can’t ), it’s whether or not the new bill can protect New Yorkers from predatory snake oil companies that claim it can.

Good idea: passing a bill that protects New Yorkers from predatory snake oil companies.

It’s long past time politicians in the US to did something about these demonstrably harmful AI systems and the companies peddling them.

Bad idea: letting the software’s vendors decide whether an algorithm is biased or not.

Per the bill:

But, those violations? They’re feckless. It’s up to the vendor to conduct and report audits demonstrating their algorithms aren’t biased.

Quick take: This is like letting Thanos conduct audits and report compliance with New York City ethics. “According to the big purple dude, his use of the Infinity Gauntlet is perfectly ethical. He did an audit and everything.”

In this case, New York’s passed a bill that gives AI vendors the power to legally determine whether they’re being ethical or not. This doesn’t protect citizens, it protects scammy AI companies.

HireVue gave a statement to the AP that sums it up best. Per that article :

Exactly. HireVue currently uses AI to allegedly assess candidates’ “e-motions.” The company actually claims computer vision algorithms can determine a user’s “emotional intelligence.” This, of course, is not possible because AI doesn’t have magic powers .

And, if that’s the standard New York’s using to determine whether an AI system is biased, then it’s open season on the Big Apple. I can’t think of a single AI startup, no matter how awful and scammy , that couldn’t pass that bar.