How AI Is Taking Over Our Gadgets

If you think of AI as something futuristic and abstract, start thinking different.

We’re now witnessing a turning point for artificial intelligence, as more of it comes down from the clouds and into our smartphones and automobiles. While it’s fair to say that AI that lives on the “edge”—where you and I are—is still far less powerful than its datacentre-based counterpart, it’s potentially far more meaningful to our everyday lives.

One key example: This fall, Apple’s Siri assistant will start processing voice on iPhones. Right now, even your request to set a timer is sent as an audio recording to the cloud, where it is processed, triggering a response that’s sent back to the phone. By processing voice on the phone, says Apple, Siri will respond more quickly. This will only work on the iPhone XS and newer models, which have a compatible built-for-AI processor Apple calls a “neural engine.” People might also feel more secure knowing that their voice recordings aren’t being sent to unseen computers in faraway places.

Google actually led the way with on-phone processing: In 2019, it introduced a Pixel phone that could transcribe speech to text and perform other tasks without any connection to the cloud. One reason Google decided to build its own phones was that the company saw potential in creating custom hardware tailor-made to run AI, says Brian Rakowski, product manager of the Pixel group at Google.

These so-called edge devices can be pretty much anything with a microchip and some memory, but they tend to be the newest and most sophisticated of smartphones, automobiles, drones, home appliances, and industrial sensors and actuators. Edge AI has the potential to deliver on some of the long-delayed promises of AI, like more responsive smart assistants, better automotive safety systems, new kinds of robots, even autonomous military machines.

The challenges of making AI work at the edge—that is, making it reliable enough to do its job and then justifying the additional complexity and expense of putting it in our devices—are monumental. Existing AI can be inflexible, easily fooled, unreliable and biased. In the cloud, it can be trained on the fly to get better—think about how Alexa improves over time. When it’s in a device, it must come pre-trained, and be updated periodically. Yet the improvements in chip technology in recent years have made it possible for real breakthroughs in how we experience AI, and the commercial demand for this sort of functionality is high.

From swords to plowshares

Shield AI, a contractor for the Department of Defense, has put a great deal of AI into quadcopter-style drones which have already carried out—and continue to be used in—real-world combat missions. One mission is to help soldiers scan for enemy combatants in buildings that must be cleared. The DoD has been eager to use the company’s drones, says Shield AI’s co-founder, Brandon Tseng, because even if they fail, they can be used to reduce human casualties.

“In 2016 and early 2017, we had early prototypes with something like 75% reliability, something you would never take to market, and the DoD were saying, ‘We’ll take that overseas and use that in combat right now,’” Mr. Tseng says. When he protested that the system wasn’t ready, the response from within the military was that anything was better than soldiers going through a door and being shot.

In a combat zone, you can’t count on a fast, robust, wireless cloud connection, especially now that enemies often jam wireless communication and GPS signals. When on a mission, processing and image recognition must occur on the company’s drones themselves.

Shield AI uses a small, efficient computer made by Nvidia, designed for running AI on devices, to create a quadcopter drone no bigger than a typical camera-wielding consumer model. The Nova 2 can fly long enough to enter a building, and use AI to recognize and examine dozens of hallways, stairwells and rooms, cataloging objects and people it sees along its way.

Meanwhile, in the town of Salinas, Calif., birthplace of “Grapes of Wrath” author John Steinbeck and an agricultural center to this day, a robot the size of an SUV is spending this year’s growing season raking the earth with its 12 robotic arms. Made by FarmWise Labs Inc., the robot trundles along fields of celery as if it were any other tractor. Underneath its metal shroud, it uses computer vision and an edge AI system to decide, in less than a second, whether a plant is a food crop or a weed, and directs its plow-like claws to avoid or eradicate the plant accordingly.

FarmWise’s huge, diesel robo-weeder can generate its own electricity, enabling it to carry a veritable supercomputer’s worth of processing power—four GPUs and 16 CPUs which together draw 500 watts of electricity.

In our everyday lives, things like voice transcription that work whether or not we have a connection, or how good it is, could mean shifts in how we prefer to interact with our mobile devices. Getting always-available voice transcription to work on Google’s Pixel phone “required a lot of breakthroughs to run on the phone as well as it runs on a remote server,” says Mr. Rakowski.

Google has almost unlimited resources to experiment with AI in the cloud, but getting those same algorithms, for everything from voice transcription and power management to real-time translation and image processing, to work on phones required the introduction of custom microprocessors like the Pixel Neural Core, adds Mr. Rakowski.

Turning cats into pure math

What nearly all edge AI systems have in common is that, as pre-trained AI, they are only performing “inference,” says Dennis Laudick, vice president of marketing for AI and machine learning at Arm Holdings, which licenses chip designs and instructions to companies such as Apple, Samsung, Qualcomm, Nvidia and others.

Generally speaking, machine-learning AI consists of four phases:

Data is captured or collected: Say, for example, in the form of millions of cat pictures.

Humans label the data: Yes, these are cat photos.

AI is trained with the labelled data: This process selects for models that identify cats.

Then the resulting pile of code is turned into an algorithm and implemented in software: Here’s a camera app for cat lovers!

(Note: If this doesn’t exist yet, consider it your million-dollar idea of the day.)

The last bit of the process—something like that cat-identifying software—is the inference phase. The software on many smart surveillance cameras, for example, is performing inference, says Eric Goodness, a research vice president at technology-consulting firm Gartner. These systems can already identify how many patrons are in the restaurant, if any are engaging in undesirable behaviour, or if the fries have been in the fryer too long.

It’s all just mathematical functions, ones so complicated that it would take a monumental effort by humans to write them, but which machine-learning systems can create when trained on enough data.

Robot pratfalls

While all of this technology has enormous promise, making AI work on individual devices, whether or not they can connect to the cloud, comes with a daunting set of challenges, says Elisa Bertino, a professor of computer science at Purdue University.

Modern AI, which is primarily used to recognize patterns, can have difficulty coping with inputs outside of the data it was trained on. Operating in the real world only makes it tougher—just consider the classic example of a Tesla that brakes when it sees a stop sign on a billboard.

To make edge AI systems more competent, one edge device might gather some data but then pair with another, more powerful device, which can integrate data from a variety of sensors, says Dr. Bertino. If you’re wearing a smartwatch with a heart-rate monitor, you’re already witnessing this: The watch’s edge AI pre-processes the weak signal of your heart rate, then passes that data to your smartphone, which can further analyze that data—whether or not it’s connected to the internet.

The overwhelming majority of AI algorithms are still trained in the cloud. They can also be retrained using more or fresher data, which lets them continually improve. Down the road, says Mr. Goodness, edge AI systems will begin to learn on their own—that is, they’ll become powerful enough to move beyond inference and actually gather data and use it to train their own algorithms.

AI that can learn all by itself, without connection to a cloud superintelligence, might eventually raise legal and ethical challenges. How can a company certify an algorithm that’s been off evolving in the real world for years after its initial release, asks Dr. Bertino. And in future wars, who will be willing to let their robots decide when to pull the trigger? Whoever does might end up with an advantage—but also all the collateral damage that happens when, inevitably, AI makes mistakes.

In our 20s, my new husband and I took a year off from our fledgling careers to travel in Southeast Asia. Equipped with paper maps, we began in China and improvised each day’s “itinerary” on the go. A gap year for grown-ups, I called it, although I scarcely qualified as one.

Nearly 40 years later, we are new retirees with the same wanderlust. We wondered: Could we recapture the thrill of winging it, enduring rough roads and cheap hotels?

We could and did, but for 2½ months instead of 12. We mapped out a route that would take us up Africa’s east coast and then—who knows where? Here’s how we rolled and five important lessons we learned on a 6,000-mile trip.

Kenya: Live large by day

Our first stop was the tiny, car-free island of Lamu, well-known for its high-profile visitors, from Kate Moss to the Obamas. This low-key getaway offered white-sand beaches, dhows — boats you can rent for day cruises and snorkelling — and lots of donkeys, the main mode of transport.

We considered the beachside Peponi Hotel in Shela, a hot spot since the 1960s (Mick Jagger bunked there). But room rates start at $250, far above our per-night budget of $70 or less. When contemplating almost 100 nights of travel, price matters.

So we chose a villa in the dunes called Amani Lamu, $61 per night for an en suite room with a private terrace and shared plunge pool.

We still had a cool Peponi moment come sunset: On the hotel’s whitewashed veranda, we sipped Pepotinis and plotted our next day’s interlude at the Majlis, Lamu’s fanciest resort (from $580).
With a $20 day pass, we could lounge around its pools and beach bars like proper resort habitués.

Lesson learned: Live like billionaires by day and frugal backpackers by night.
Must-go: Across the bay on Manda Island, bunk a night in a thatched-roof bungalow on stilts at Nyla’s Guest House and Kitchen (from $48 with breakfast).
After a dinner of doro wat, a spicy Ethiopian chicken stew and rice, the sound of waves will lull you asleep.

Egypt: Ask. Politely.

From Lamu, we flew to Aswan in Egypt. Our “plan”: Cruise down the Nile to Luxor, then take a train to Cairo, and venture to Giza’s pyramids.

Turns out it’s the kind of thing one really should book in advance. But at our Aswan hostel, the proprietor, who treated us like guests deserving white-glove service, secured a felucca, a vessel manned by a navigator and captain-cum-cook. Since we’d booked fewer than 24 hours in advance and there were no other takers, we were its sole passengers for the three-day trip.

One day, we stopped to tour ancient temples and visit a bustling camel fair, but otherwise, we remained on board watching the sunbaked desert slide by. We slept on futons on the deck under the stars. The cost: about $100 per night per person, including three meals.

Lesson learned: Ask for help. We found Egyptians kind and unfazed by our haplessness, especially when we greeted them respectfully with assalamu alaikum (“Peace to you”).
Must-go: For buys from carpets to kebabs, don’t miss Cairo’s massive Khan el-Khalili bazaar, in business since 1382. We loved the babouche, cute leather slippers, but resisted as our packs were full.

Turkey: Heed weather reports

Next stop Tunisia, via a cheap flight on EgyptAir. We loved Tunisia, but left after six days because the weather got chilly.

Fair enough, it was January. We hopped continents by plane and landed in Istanbul, where it snowed. Fortunately, two of Istanbul’s main pleasures involve hot water. We indulged in daily hammams, or Turkish baths, ranging from $30 to $60 for services that included, variously, a massage, a scrub-down and a soak.

Beneath soaring ceilings at the temple-like Kılıç Ali Paşa Halamı, brisk workers sternly wielded linen sacks to dowse my body in a cloud of hot foam.
In between visits to Ottoman-era mosques and the city’s spice markets, we staved off the chill by drinking fruity pomegranate tea and sampling Turkish delight and baklava at tea salons.

A favourite salon: Sekerci Cafer Erol in Kadıköy, a ferry-ride away on the “Asian” side of Istanbul, where the city adjoins Asia.

Lesson learned: Pay attention to the weather gods. We foolishly took the concept of travelling off-season too far.
Must-go: Don’t miss the Istanbul Modern, the Renzo Piano-designed art museum in the historic Beyoğlu district.

Cambodia: Chill out

After a long flight from Istanbul, we spent two weeks in Laos and then hopped another plane to Cambodia, specifically Koh Rong Sanloem, another car-free island.

Like vagabonds, we lolled by the warm, super-blue water of Sunset Beach, steps from our bungalow at Sleeping Trees (from $54 per night).
A caveat: You have to sweat to get to this island paradise. We took a bus, a ferry and then hiked for 40 minutes up and down a steep hill and through a jungle. You’ll find only a handful of “resorts”—simple bungalow complexes like ours. There’s nothing much to do. I’ll be back.

Lesson learned: Until our week in Cambodia, we’d been travelling too much and too fast, prioritising exploration over relaxation. This island taught us the pleasures of stasis.
Must-go: Spend one day in Cambodia’s capital city, Phnom Penh, to delve into its sobering history. Tour the Choeung Ek Genocidal Centre, site of a Killing Field, where nearly 9,000 Cambodians died.

Thailand: Be a frugal hedonist

We spent our last two weeks on the island of Ko Samui, where season three of “The White Lotus” was shot.

We went there for its astounding beauty, not the luxury resort experience that comes with too many boisterous lads on vacation, snake farms and traffic jams in town.

Truth be told, we flouted our budget rules to book an Airbnb with a pool (from $300) in the hills of Lipa Noi on the island’s quiet side. We joined the nearby Gravity Movement Gym to work out, but cooked our own meals to keep our final tabulation of expenses within reach.

Lesson learned: Pinching pennies feels restrictive, no matter how lush the surroundings. And it leads to bickering, as partners tally up who squandered how much on what.
With the end in sight, we splurged on the villa and even bought souvenirs, knowing we’d lug them for days, not weeks.
Must-go: Take the 30-minute ferry to sister island Ko Pha Ngan for its peace, love and yoga vibe and, once a month, full-moon parties.
Via Airbnb, we bunked at a Thai house called Baan Nuit, run by the Dear Phangan restaurant proprietors.

We sampled steamed dumplings, white fish in a Thai basil sauce and spicy noodles for a mere $15 apiece.
Hey, indulge in that “White Lotus” moment if you dare!