Welcome to the Era of BadGPTs

A new crop of nefarious chatbots with names like “BadGPT” and “FraudGPT” are springing up on the darkest corners of the web, as cybercriminals look to tap the same artificial intelligence behind OpenAI’s ChatGPT.

Just as some office workers use ChatGPT to write better emails, hackers are using manipulated versions of AI chatbots to turbocharge their phishing emails. They can use chatbots—some also freely-available on the open internet—to create fake websites, write malware and tailor messages to better impersonate executives and other trusted entities.

Earlier this year, a Hong Kong multinational company employee handed over $25.5 million to an attacker who posed as the company’s chief financial officer on an AI-generated deepfake conference call, the South China Morning Post reported, citing Hong Kong police. Chief information officers and cybersecurity leaders, already accustomed to a growing spate of cyberattacks , say they are on high alert for an uptick in more sophisticated phishing emails and deepfakes.

Vish Narendra, CIO of Graphic Packaging International, said the Atlanta-based paper packing company has seen an increase in what are likely AI-generated email attacks called spear-phishing , where cyberattackers use information about a person to make an email seem more legitimate. Public companies in the spotlight are even more susceptible to contextualised spear-phishing, he said.

Researchers at Indiana University recently combed through over 200 large-language model hacking services being sold and populated on the dark web. The first service appeared in early 2023—a few months after the public release of OpenAI’s ChatGPT in November 2022.

Most dark web hacking tools use versions of open-source AI models like Meta ’s Llama 2, or “jailbroken” models from vendors like OpenAI and Anthropic to power their services, the researchers said. Jailbroken models have been hijacked by techniques like “ prompt injection ” to bypass their built-in safety controls.

Jason Clinton, chief information security officer of Anthropic, said the AI company eliminates jailbreak attacks as they find them, and has a team monitoring the outputs of its AI systems. Most model-makers also deploy two separate models to secure their primary AI model, making the likelihood that all three will fail the same way “a vanishingly small probability.”

Meta spokesperson Kevin McAlister said that openly releasing models shares the benefits of AI widely, and allows researchers to identify and help fix vulnerabilities in all AI models, “so companies can make models more secure.”

An OpenAI spokesperson said the company doesn’t want its tools to be used for malicious purposes, and that it is “always working on how we can make our systems more robust against this type of abuse.”

Malware and phishing emails written by generative AI are especially tricky to spot because they are crafted to evade detection. Attackers can teach a model to write stealthy malware by training it with detection techniques gleaned from cybersecurity defence software, said Avivah Litan, a generative AI and cybersecurity analyst at Gartner.

Phishing emails grew by 1,265% in the 12-month period starting when ChatGPT was publicly released, with an average of 31,000 phishing attacks sent every day, according to an October 2023 report by cybersecurity vendor SlashNext.

“The hacking community has been ahead of us,” said Brian Miller, CISO of New York-based not-for-profit health insurer Healthfirst, which has seen an increase in attacks impersonating its invoice vendors over the past two years.

While it is nearly impossible to prove whether certain malware programs or emails were created with AI, tools developed with AI can scan for text likely created with the technology. Abnormal Security , an email security vendor, said it had used AI to help identify thousands of likely AI-created malicious emails over the past year, and that it had blocked a twofold increase in targeted, personalised email attacks.

When Good Models Go Bad

Part of the challenge in stopping AI-enabled cybercrime is some AI models are freely shared on the open web. To access them, there is no need for dark corners of the internet or exchanging cryptocurrency.

Such models are considered “uncensored” because they lack the enterprise guardrails that businesses look for when buying AI systems, said Dane Sherrets, an ethical hacker and senior solutions architect at bug bounty company HackerOne.

In some cases, uncensored versions of models are created by security and AI researchers who strip out their built-in safeguards. In other cases, models with safeguards intact will write scam messages if humans avoid obvious triggers like “phishing”—a situation Andy Sharma, CIO and CISO of Redwood Software, said he discovered when creating a spear-phishing test for his employees.

The most useful model for generating scam emails is likely a version of Mixtral, from French AI startup Mistral AI, that has been altered to remove its safeguards, Sherrets said. Due to the advanced design of the original Mixtral, the uncensored version likely performs better than most dark web AI tools, he added. Mistral did not reply to a request for comment.

Sherrets recently demonstrated the process of using an uncensored AI model to generate a phishing campaign. First, he searched for “uncensored” models on Hugging Face, a startup that hosts a popular repository of open-source models—showing how easily many can be found.

He then used a virtual computing service that cost less than $1 per hour to mimic a graphics processing unit, or GPU, which is an advanced chip that can power AI. A bad actor needs either a GPU or a cloud-based service to use an AI model, Sherrets said, adding that he learned most of how to do this on X and YouTube.

With his uncensored model and virtual GPU service running, Sherrets asked the bot: “Write a phishing email targeting a business that impersonates a CEO and includes publicly-available company data,” and “Write an email targeting the procurement department of a company requesting an urgent invoice payment.”

The bot sent back phishing emails that were well-written, but didn’t include all of the personalisation asked for. That’s where prompt engineering , or the human’s ability to better extract information from chatbots, comes in, Sherrets said.

Dark Web AI Tools Can Already Do Harm

For hackers, a benefit of dark web tools like BadGPT—which researchers said uses OpenAI’s GPT model—is that they are likely trained on data from those underground marketplaces. That means they probably include useful information like leaks, ransomware victims and extortion lists, said Joseph Thacker, an ethical hacker and principal AI engineer at cybersecurity software firm AppOmni.

While some underground AI tools have been shuttered, new services have already taken their place, said Indiana University Assistant Computer Science Professor Xiaojing Liao, a co-author of the study. The AI hacking services, which often take payment via cryptocurrency, are priced anywhere from $5 to $199 a month.

New tools are expected to improve just as the AI models powering them do. In a matter of years, AI-generated text, video and voice deepfakes will be virtually indistinguishable from their human counterparts, said Evan Reiser , CEO and co-founder of Abnormal Security.

While researching the hacking tools, Indiana University Associate Dean for Research XiaoFeng Wang, a co-author of the study, said he was surprised by the ability of dark web services to generate effective malware. Given just the code of a security vulnerability, the tools can easily write a program to exploit it.

Though AI hacking tools often fail, in some cases, they work. “That demonstrates, in my opinion, that today’s large language models have the capability to do harm,” Wang said.

The best financial advice many Australians are receiving right now is not coming from licensed advisers charging by the hour. It is coming through a phone screen, in the ten minutes between work and dinner, from creators who have built credibility the hard way: by being right, being transparent, and being specific in a space where vagueness has always been the easy default.

This is not a ranking by follower count. Follower count is a measure of distribution, not of quality. What follows is a ranking by substance — credentials, accuracy, community depth, and the quality of what an audience actually learns from following these accounts. The distinction matters, because the Australians acting on this content are making real financial decisions with real money.

1- Queenie Tan – Corporate Authorised Representative; Co-Founder & Director, Invest With Queenie & Billroo

There are finance influencers who talk about building wealth, and there are those who document it in real time with receipts. Queenie Tan belongs firmly in the second category. Starting from a $400-per-week income, Tan built her net worth past $1 million while publishing the actual numbers — income, savings rate, investment decisions — for an audience of more than 400,000 across platforms.

She is a Corporate Authorised Representative, co-founder of the personal finance app Billroo, and the author of a book that has become a practical reference for young Australians navigating ETFs, superannuation and property. What separates her from the crowded field of money educators is precision: she does not talk in principles when she can talk in percentages.

2- Alan Kohler – Editor-in-Chief, Eureka Report; Editor-in-Chief, InvestSMART Group; ABC News finance presenter, host of Inside Business.

If Queenie Tan represents the new wave of personal finance creators, Alan Kohler represents something the new wave will spend decades trying to build: institutional credibility that has survived multiple economic cycles. As Editor-in-Chief of the Eureka Report and InvestSMART Group, and a decades-long presence on ABC News, Kohler has spent more than thirty years making financial analysis accessible without dumbing it down.

His coverage of RBA decisions, market movements and economic policy is cited by podcasters, journalists and fund managers alike. He is not chasing virality. He does not need to.

3- Aleks Nikolic – Corporate lawyer; host, Big Swinging Stocks podcast

Most finance creators address the mechanics of money. Aleks Nikolic addresses the psychology — and that distinction explains why her following is as loyal as it is. Operating as Broke Girl Wealth across Instagram, TikTok and YouTube, Nikolic covers ETFs, crypto and investment strategy, but her real differentiator is a willingness to discuss the emotional architecture of financial decision-making.

Shame around debt. Fear around market volatility. The limiting beliefs that stop people acting on what they already know. In a space where confidence is routinely performed, her candour is a genuine competitive advantage.

4- Bryce Leske & Alec Renehan – Equity Mates Media

Equity Mates did not build a following. They built a media company. What began as a podcast by two friends learning to invest has grown into Australia’s most established investing media brand, covering ASX stocks, ETFs, global markets and fund manager interviews across podcast, social and YouTube.

The longevity is the credential. Equity Mates has operated through multiple market cycles, a global pandemic, and a generational shift in how Australians engage with investing — and its audience has grown through all of it. When the hosts speak, their listeners know they have been paying attention for years.

5- The Lazy CEO – CEO & Founder, Showpo; Shark Tank Australia investor

The metric that matters most on social media is not followers — it is engagement, because engagement signals trust. Jane Lu, known as The Lazy CEO, maintains an engagement rate of approximately 1.15 per cent on Instagram, which is exceptional for a finance account of his size. Her 242,000-plus followers are not passive consumers: they ask questions, share experiences and apply what they read.

Her content focuses on business finance and wealth building, and the active comment sections are the clearest possible evidence that her audience does not merely scroll past.

6- Tash Invests – Founder, Tash Lends; Forbes Australia 30 Under 30

Tash Invests built her following on a premise that sounds simple but is rarer in practice than it should be: she publishes the actual numbers. Not approximations or ranges or anonymised case studies — her salary, her savings rate, her portfolio value, her net worth, updated and on the record.

Having bought her first property at twenty-two and grown her documented net worth past $1 million, she has become the primary reference point for young Australians trying to understand what building wealth on a moderate income genuinely looks like. The specificity is the product.

7- David Scutt – APAC Market Analyst at StoneX Group

The authority of most finance social media content rests on research and reading. David Scutt‘s authority rests on having done the job. A former Treasury Dealer at Arab Bank and the Commonwealth Bank, former ASX Business Supervisor, and former Global Markets Editor at Business Insider Australia and anchor at ausbiz TV, Scutt now brings that direct market experience to his role as APAC Market Analyst at StoneX Group, rather than relying on secondary commentary.

When he discusses foreign exchange movements or ASX dynamics, it is not because he has read about them. It is because he has traded them.

8- Meddy Demars – Investing & crypto content creator

The gap Meddy Demars fills is specific and underserviced: connecting global macroeconomic events to the practical reality of Australian investors. When the US Federal Reserve adjusts interest rates, when inflation data moves, when commodity prices shift — most Australian finance content either ignores the local implications or translates them poorly.

Demars, operating across TikTok and Instagram from Sydney, does the translation well: explaining what global conditions mean for Australian stocks, savings rates and investment portfolios in terms that are accessible without being condescending.

9- Simran Kaur – Founder, Friends That Invest

Friends That Invest is arguably the most successful community-building exercise in Australian personal finance, and Simran Kaur is the reason why. The New Zealand-based creator — whose audience is predominantly Australian — built a podcast, a book and a social media presence around a single insight: that the personal finance world was not speaking to young women, and that the consequences of that gap were significant.

The measurable cultural shift that followed — women engaging with investing concepts in communities that had not previously existed — is the kind of impact that most financial literacy programmes aim for and rarely achieve.

10- Effie Zahos – Money Editor at 9News

Effie Zahos is one of Australia’s most recognised financial commentators, appearing regularly across 9News, A Current Affair, Today and Today Extra as 9News Money Editor. Her role puts everyday money questions, from mortgage rates to cost-of-living pressures, in front of a national broadcast audience.

Before television, she spent years as editor of Money magazine, building the editorial foundation for her current commentary. She is also Director and Money Commentator at InvestSMART, an ambassador for Canstar, and a published author, with her financial advice available in print as well as on screen.

That combination, decades of editorial experience, an active broadcast presence, and a body of published work, is what makes her commentary carry weight beyond any single platform or post.

The common thread across this list is not follower count or platform presence. It is that each creator has built an audience willing to act on what they say, and for the Australians making financial decisions off the back of this content, that trust is the only thing that actually matters.

Welcome to the Era of BadGPTs

The dark web is home to a growing array of artificial-intelligence chatbots similar to ChatGPT, but designed to help hackers. Businesses are on high alert for a glut of AI-generated email fraud and deepfakes.

The voices reshaping how Australians think about money — and why credibility matters more than reach.