How AI Is Taking Over Our Gadgets
Kanebridge News
Share Button

How AI Is Taking Over Our Gadgets

AI is moving from data centres to devices, impacting everything from phones to tractors.

By Christopher Mims
Wed, Jun 30, 2021 10:25amGrey Clock 5 min

If you think of AI as something futuristic and abstract, start thinking different.

We’re now witnessing a turning point for artificial intelligence, as more of it comes down from the clouds and into our smartphones and automobiles. While it’s fair to say that AI that lives on the “edge”—where you and I are—is still far less powerful than its datacentre-based counterpart, it’s potentially far more meaningful to our everyday lives.

One key example: This fall, Apple’s Siri assistant will start processing voice on iPhones. Right now, even your request to set a timer is sent as an audio recording to the cloud, where it is processed, triggering a response that’s sent back to the phone. By processing voice on the phone, says Apple, Siri will respond more quickly. This will only work on the iPhone XS and newer models, which have a compatible built-for-AI processor Apple calls a “neural engine.” People might also feel more secure knowing that their voice recordings aren’t being sent to unseen computers in faraway places.

Google actually led the way with on-phone processing: In 2019, it introduced a Pixel phone that could transcribe speech to text and perform other tasks without any connection to the cloud. One reason Google decided to build its own phones was that the company saw potential in creating custom hardware tailor-made to run AI, says Brian Rakowski, product manager of the Pixel group at Google.

These so-called edge devices can be pretty much anything with a microchip and some memory, but they tend to be the newest and most sophisticated of smartphones, automobiles, drones, home appliances, and industrial sensors and actuators. Edge AI has the potential to deliver on some of the long-delayed promises of AI, like more responsive smart assistants, better automotive safety systems, new kinds of robots, even autonomous military machines.

The challenges of making AI work at the edge—that is, making it reliable enough to do its job and then justifying the additional complexity and expense of putting it in our devices—are monumental. Existing AI can be inflexible, easily fooled, unreliable and biased. In the cloud, it can be trained on the fly to get better—think about how Alexa improves over time. When it’s in a device, it must come pre-trained, and be updated periodically. Yet the improvements in chip technology in recent years have made it possible for real breakthroughs in how we experience AI, and the commercial demand for this sort of functionality is high.

From swords to plowshares

Shield AI, a contractor for the Department of Defense, has put a great deal of AI into quadcopter-style drones which have already carried out—and continue to be used in—real-world combat missions. One mission is to help soldiers scan for enemy combatants in buildings that must be cleared. The DoD has been eager to use the company’s drones, says Shield AI’s co-founder, Brandon Tseng, because even if they fail, they can be used to reduce human casualties.

“In 2016 and early 2017, we had early prototypes with something like 75% reliability, something you would never take to market, and the DoD were saying, ‘We’ll take that overseas and use that in combat right now,’” Mr. Tseng says. When he protested that the system wasn’t ready, the response from within the military was that anything was better than soldiers going through a door and being shot.

In a combat zone, you can’t count on a fast, robust, wireless cloud connection, especially now that enemies often jam wireless communication and GPS signals. When on a mission, processing and image recognition must occur on the company’s drones themselves.

Shield AI uses a small, efficient computer made by Nvidia, designed for running AI on devices, to create a quadcopter drone no bigger than a typical camera-wielding consumer model. The Nova 2 can fly long enough to enter a building, and use AI to recognize and examine dozens of hallways, stairwells and rooms, cataloging objects and people it sees along its way.

Meanwhile, in the town of Salinas, Calif., birthplace of “Grapes of Wrath” author John Steinbeck and an agricultural center to this day, a robot the size of an SUV is spending this year’s growing season raking the earth with its 12 robotic arms. Made by FarmWise Labs Inc., the robot trundles along fields of celery as if it were any other tractor. Underneath its metal shroud, it uses computer vision and an edge AI system to decide, in less than a second, whether a plant is a food crop or a weed, and directs its plow-like claws to avoid or eradicate the plant accordingly.

FarmWise’s huge, diesel robo-weeder can generate its own electricity, enabling it to carry a veritable supercomputer’s worth of processing power—four GPUs and 16 CPUs which together draw 500 watts of electricity.

In our everyday lives, things like voice transcription that work whether or not we have a connection, or how good it is, could mean shifts in how we prefer to interact with our mobile devices. Getting always-available voice transcription to work on Google’s Pixel phone “required a lot of breakthroughs to run on the phone as well as it runs on a remote server,” says Mr. Rakowski.

Google has almost unlimited resources to experiment with AI in the cloud, but getting those same algorithms, for everything from voice transcription and power management to real-time translation and image processing, to work on phones required the introduction of custom microprocessors like the Pixel Neural Core, adds Mr. Rakowski.

Turning cats into pure math

What nearly all edge AI systems have in common is that, as pre-trained AI, they are only performing “inference,” says Dennis Laudick, vice president of marketing for AI and machine learning at Arm Holdings, which licenses chip designs and instructions to companies such as Apple, Samsung, Qualcomm, Nvidia and others.

Generally speaking, machine-learning AI consists of four phases:

  • Data is captured or collected: Say, for example, in the form of millions of cat pictures.
  • Humans label the data: Yes, these are cat photos.
  • AI is trained with the labelled data: This process selects for models that identify cats.
  • Then the resulting pile of code is turned into an algorithm and implemented in software: Here’s a camera app for cat lovers!

(Note: If this doesn’t exist yet, consider it your million-dollar idea of the day.)

The last bit of the process—something like that cat-identifying software—is the inference phase. The software on many smart surveillance cameras, for example, is performing inference, says Eric Goodness, a research vice president at technology-consulting firm Gartner. These systems can already identify how many patrons are in the restaurant, if any are engaging in undesirable behaviour, or if the fries have been in the fryer too long.

It’s all just mathematical functions, ones so complicated that it would take a monumental effort by humans to write them, but which machine-learning systems can create when trained on enough data.

Robot pratfalls

While all of this technology has enormous promise, making AI work on individual devices, whether or not they can connect to the cloud, comes with a daunting set of challenges, says Elisa Bertino, a professor of computer science at Purdue University.

Modern AI, which is primarily used to recognize patterns, can have difficulty coping with inputs outside of the data it was trained on. Operating in the real world only makes it tougher—just consider the classic example of a Tesla that brakes when it sees a stop sign on a billboard.

To make edge AI systems more competent, one edge device might gather some data but then pair with another, more powerful device, which can integrate data from a variety of sensors, says Dr. Bertino. If you’re wearing a smartwatch with a heart-rate monitor, you’re already witnessing this: The watch’s edge AI pre-processes the weak signal of your heart rate, then passes that data to your smartphone, which can further analyze that data—whether or not it’s connected to the internet.

The overwhelming majority of AI algorithms are still trained in the cloud. They can also be retrained using more or fresher data, which lets them continually improve. Down the road, says Mr. Goodness, edge AI systems will begin to learn on their own—that is, they’ll become powerful enough to move beyond inference and actually gather data and use it to train their own algorithms.

AI that can learn all by itself, without connection to a cloud superintelligence, might eventually raise legal and ethical challenges. How can a company certify an algorithm that’s been off evolving in the real world for years after its initial release, asks Dr. Bertino. And in future wars, who will be willing to let their robots decide when to pull the trigger? Whoever does might end up with an advantage—but also all the collateral damage that happens when, inevitably, AI makes mistakes.


This stylish family home combines a classic palette and finishes with a flexible floorplan

35 North Street Windsor

Just 55 minutes from Sydney, make this your creative getaway located in the majestic Hawkesbury region.

Related Stories
Should AI Have Access to Your Medical Records? What if It Can Save Many Lives?
How an Ex-Teacher Turned a Tiny Pension Into a Giant-Killer
By MATT WIRZ 27/05/2024
The Problem With Behavioural Nudges
By Evan Polman and Sam J. Maglio 27/05/2024
Should AI Have Access to Your Medical Records? What if It Can Save Many Lives?

We asked readers: Is it worth giving up some potential privacy if the public benefit could be great? Here’s what they said.

Tue, May 28, 2024 4 min

We’re constantly told that one of the potentially biggest benefits of artificial intelligence is in the area of health. By collecting large amounts of data, AI can create all sorts of drugs for diseases that have been resistant to treatment.

But the price of that could be that we have to share more of our medical information. After all, researchers can’t collect large amounts of data if people aren’t willing to part with that data.

We wanted to see where our readers stand on the balance of privacy versus public-health gains as part of our series on ethical dilemmas created by the advent of AI.

Here are the questions we posed…

AI may be able to discover new medical treatments if it can scan large volumes of health records. Should our personal health records be made available for this purpose, if it has the potential to improve or save millions of lives? How would we guard privacy in that case?

…and some of the answers we received. undefined

Rely on nonpartisan overseers

While my own recent experience with a data breach highlights the importance of robust data security, I recognise the potential for AI to revolutionise healthcare. To ensure privacy, I would be more comfortable if an independent, nonpartisan body—overseen by medical professionals, data-security experts, and citizen representatives—managed a secure database.

Anonymity cuts both ways

Yes. Simply sanitise the health records of any identifying information, which is quite doable. Although there is an argument to be made that AI may discover something that an individual needs or wants to know.

Executive-level oversight

I think we can make AI scanning of health records available with strict privacy controls. Create an AI-CEO position at medical facilities with extreme vetting of that individual before hiring them.

Well worth it

This actually sounds like a very GOOD use of AI. There are several methods for anonymising data which would allow for studies over massive cross-sections of the population without compromising individuals’ privacy. The AI would just be doing the same things meta-studies do now, only faster and maybe better.

Human touch

My concern is that the next generations of doctors will rely more heavily, maybe exclusively, on AI and lose the ability or even the desire to respect the art of medicine which demands one-on-one interaction with a patient for discussion and examination (already a dying skill).


People should be able to sign over rights to their complete “anonymised” health record upon death just as they can sign over rights to their organs. Waiting for death for such access does temporarily slow down the pace of such research, but ultimately will make the research better. Data sets will be more complete, too. Before signing over such rights, however, a person would have to be fully informed on how their relatives’ privacy may also be affected.

Pay me or make it free for all

As long as this is open-source and free, they can use my records. I have a problem with people using my data to make a profit without compensation.

Privacy above all

As a free society, we value freedoms and privacy, often over greater utilitarian benefits that could come. AI does not get any greater right to infringe on that liberty than anything else does.

Opt-in only

You should be able to opt in and choose a plan that protects your privacy.

Privacy doesn’t exist anyway

If it is decided to extend human lives indefinitely, then by all means, scan all health records. As for privacy, there is no such thing. All databases, once established, will eventually, if not immediately, be accessed or hacked by both the good and bad guys.

The data’s already out there

I think it should be made available. We already sign our rights for information over to large insurance companies. Making health records in the aggregate available for helping AI spot potential ways to improve medical care makes sense to me.

Overarching benefit

Of course they should be made available. Privacy is no serious concern when the benefits are so huge for so many.

Compensation for breakthroughs

We should be given the choice to release our records and compensated if our particular genome creates a pathway to treatment and medications.

Too risky

I like the idea of improving healthcare by accessing health records. However, as great as that potential is, the risks outweigh it. Access to the information would not be controlled. Too many would see personal opportunity in it for personal gain.

Nothing personal

The personal info should never be available to anyone who is not specifically authorised by the patient to have it. Medical information can be used to deny people employment or licenses!

No guarantee, but go ahead

This should be allowed on an anonymous basis, without question. But how to provide that anonymity?

Anonymously isolating the information is probably easy, but that information probably contains enough information to identify you if someone had access to the data and was strongly motivated. So the answer lies in restricting access to the raw data to trusted individuals.

Take my records, please

As a person with multiple medical conditions taking 28 medications a day, I highly endorse the use of my records. It is an area where I have found AI particularly valuable. With no medical educational background, I find it very helpful when AI describes in layman’s terms both my conditions and medications. In one instance, while interpreting a CT scan, AI noted a growth on my kidney that looked suspiciously like cancer and had not been disclosed to me by any of the four doctors examining the chart.


This stylish family home combines a classic palette and finishes with a flexible floorplan

35 North Street Windsor

Just 55 minutes from Sydney, make this your creative getaway located in the majestic Hawkesbury region.

Related Stories
Should AI Have Access to Your Medical Records? What if It Can Save Many Lives?
Judge Blocks Effort to Auction Graceland
By TALI ARBEL 26/05/2024
The Loneliness of the American Worker
By TE-PING CHEN 29/05/2024
    Your Cart
    Your cart is emptyReturn to Shop