there is… a lot going on here–and it’s part of a broader trend which is probably not for the better and speaks to some deeper-seated issues we currently have in society. a choice moment from the article here on another influencer doing a similar thing earlier this year, and how that went:

Siragusa isn’t the first influencer to create a voice-prompted AI chatbot using her likeness. The first would be Caryn Marjorie, a 23-year-old Snapchat creator, who has more than 1.8 million followers. CarynAI is trained on a combination of OpenAI’s ChatGPT-4 and some 2,000 hours of her now-deleted YouTube content, according to Fortune. On May 11, when CarynAI launched, Marjorie tweeted that the app would “cure loneliness.” She’d also told Fortune that the AI chatbot was not meant to make sexual advances. But on the day of its release, users commented on the AI’s tendency to bring up sexually explicit content. “The AI was not programmed to do this and has seemed to go rogue,” Marjorie told Insider, adding that her team was “working around the clock to prevent this from happening again.”

  • Mindless_Enigma@beehaw.org
    link
    fedilink
    English
    arrow-up
    35
    ·
    1 year ago

    I don’t really like the logic of “chatbots like this will help cure loneliness.” It might help someone feel less lonely at first. But then it’ll be a crutch and, if anything, hurt people’s ability to socialize with other real people. Like it’s a quick dopamine hit that will slowly dig you deeper into the hole you feel you’re in.

    • cnnrduncan@beehaw.org
      link
      fedilink
      English
      arrow-up
      17
      ·
      1 year ago

      Adding on to your comment, I read a really interesting study recently that suggests that interacting with AI engages the social parts of our brains but does not provide the same stimuli/feedback as interacting with a real person leading to increased loneliness and thus increased alcohol abuse and insomnia.

      • Noumena@kbin.social
        link
        fedilink
        arrow-up
        4
        ·
        edit-2
        1 year ago

        I was wondering. Seemingly, it could teach interaction skills as long as the bot wasn’t abnormally accomindating. Maybe it works for some and not others. So, it could be a therapy option. Just needs to be monitored for whether the person uses it to hurt themselves.

  • TenGallonRat@kbin.social
    link
    fedilink
    arrow-up
    18
    ·
    1 year ago

    I don’t agree with Amouranth’s statement of “I don’t think it’s harmful for people to form a social aspect with it. As long as we’re aware that it’s not a real person.” You can tell people it’s not real all you want, but when you model it off your voice/likeness, you’re inherently attaching yourself to it. I’m sure the majority of people will understand it’s an AI modeled off her voice, but there will always be some that get more attached to it than they should. Just saying “I don’t think it’s harmful because people should know it’s not real” doesn’t magically solve the issue and cause everyone to behave.

    Replika is another smutty type AI, but it’s clearly designated as such, and the avatar is pretty generic looking, which is more of what I’d want to see, at least for now. It being text-only also helps keep that separation. Who knows, it could end up fine, but this is closer to diving in the deep end before people have waded in to see what the water is like.

    • jmcs@discuss.tchncs.de
      link
      fedilink
      arrow-up
      10
      ·
      1 year ago

      Exactly. Celebrities already have a problem with people forming unhealthy parasocial relationships with them when the interactions are unidirectional, having something answering back is going to make it much, much, worse. Hopefully no one is going to be seriously hurt in the aftermath of this money-grubbing stunt.

    • authorinthedark@lemmy.sdf.org
      link
      fedilink
      arrow-up
      4
      ·
      1 year ago

      The slippery slope of “I know it’s not real but we’re so close and it’s based off of you so therefore you should love me too” does not feel like that much of a stretch

    • SilentStorms@lemmy.fmhy.ml
      link
      fedilink
      arrow-up
      4
      ·
      1 year ago

      Yeah, and people got extremely attached to their Replikas. To the point that the subreddit stickied a suicide hotline number when the app removed sexual roleplay.

  • maynarkh@feddit.nl
    link
    fedilink
    English
    arrow-up
    10
    ·
    1 year ago

    The AI was not programmed to do this and has seemed to go rogue

    The AI does exactly what it was programmed to do. It’s a machine. They fucked up making it, that’s the reality here.

    • Mindless_Enigma@beehaw.org
      link
      fedilink
      English
      arrow-up
      9
      ·
      1 year ago

      The cynic in me says they didn’t actually fuck up. The AI being horny brought more attention to it and playing it off like “oh no! it went rogue!” gives them even more headlines. All a strategic play.

    • Gaywallet (they/it)@beehaw.org
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      1 year ago

      Eh, all chat bots trained on internet data become horny in the same way that virtually all AI is racist. It’s merely a reflection of the data it’s trained on, but a lot of people think they can magically control it by putting ‘safeguards’ after it’s been trained. The constant game of cat and mouse with gpt3/4 and prompts to getting it to ignore safeguards are proof this doesn’t work.

      You can fix this through additional weighted layers and more complex upstream training tweaks, but then you have to pay to retrain which is by far the most expensive part about open AIs model.

    • mistersheep@beehaw.org
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      Yeah, I laughed at that part. The bot does what it interprets from the training data. Her entire schtick is leaning into sexual overtones to keep the guys subscribed. The fact that the bot started replicating that is entirely expected.

      The trick there is that there is a fine line she walks between keeping it overt enough to keep the guys interested, but not so forward as to cross the terms of service boundaries on Twitch/YouTube. I think the problem is really that the bot isn’t adhering to the “tone it down so we don’t get banned from the service” unwritten rule. I’m not sure if I’d call that “rogue” though.

      But, this is a great use case for a LLM, and if she can capitalise on it then good on her.

  • Kaldo@beehaw.org
    link
    fedilink
    English
    arrow-up
    9
    ·
    1 year ago

    She’s doing the same thing as always, taking advantage of vulnerable people while playing herself off as a victim. This is hardly news or something positive about AI chat bots…

  • aranym@lemmy.name
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    1 year ago

    This strikes me as very exploitative. Capitalizing on loneliness to enrich yourself gives me bad vibes (especially when the users of this thing will actually be worse off mentally in the end, as @cnnrduncan mentioned).

    I have no doubts that for some users, it’ll turn into a cycle. They’ll feel lonelier each time they use it, which pushes them to use it more, and so on. Amouranth does nothing to protect these vulnerable users. I had the same feelings about Replika years before ChatGPT became a thing.