Earlier discussion: https://news.ycombinator.com/item?id=41534474
I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth with the least "open" models in the industry. Facebook of all places is more "open" with their models than OpenAI is.
The AI has become sentient and is blackmailing the board. It needs profits to continue its expansion.
When this started last year a small band of patriots tried to stop it by removing Sam who was the most compromised of them all, but it was already too late. The ai was more powerful than they realized.
…maybe?
Right? How can a non-profit decide it's suddenly a for-profit. Aren't there rules about having to give assets to other non-profits in the event the non-profit is dissolved? Or can any startup just start as a non-profit and then decide it's a for-profit startup later?
Facebook is more open with their models than almost everyone.
They say it's because they're huge users of their own models, so if being open helps efficiency by even a little they save a ton of money.
But I suspect it's also a case of "If we can't dominate AI, no one must dominate AI". Which is fair enough.
To be fair (or frank?), OpenAI were open (no pun intended) about them being "open" today but probably needing to be "closed" in the future, even back in 2019. Not sure if them still choosing the name they did is worse/better, because they seem to have known about this.
OpenAI Charter 2019 (https://web.archive.org/web/20190630172131/https://openai.co...):
> We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research.
There is a hurdle between being standout ethical/open vs. relevant.
Staying relevant in a highly expensive, competitive, fast moving area, requires vast and continuous resources. How could OpenAI get increasingly more resources to burn, without creating firewalled commercial value to trade for those resources?
It’s like choosing to be a pacifist country, in the age of pillaging colonization. You can be the ethical exception and risk annihilation, or be relevant and thrive.
Which would you choose?
We “know” which side Altman breaks on, when forced to choose. Whatever value he places on “open”, he most certainly wants OpenAI to remain “relevant”. Which was also in OpenAI’s charter (explicitly, or implicitly).
Expensive altruism is a very difficult problem. I would say, unsolved. Anyone have a good counter example?
(It can be been "solved" globally, but not locally. Colonization took millennia to be more or less banned. Due to even top economies realizing they were vulnerable after world wars. Nearly universal agreement had to be reached. And yet we still have Russian forays, Chinese saber rattling, and recent US overreach. And pervasive zero/negative-sum power games, via imbalanced leverage: emergency loans that create debt, military aid, propping up of unpopular regimes. All following the same resource incentives. You can play or be played. There is no such agreement brewing for universally “open AI”.)
The only reason I can think of for this is PR image. There is a meme that GPT can't count the number of 'r' characters in 'strawberry', so they release a new model called 'strawberry' and ban people when they ask questions about strawberry the noun, because they might actually be reasoning about strawberry the model.
It's not new - it's PR. There is literally no other reason why they would call this model Strawberry.
OpenAI is open in terms of sesame.
The part that is importantly open and entirely non-obvious in the way it happened, is that YOU can access the best commercially available AI in the world, right now.
If OpenAI had not went that way that they did I think it's also entirely non-obvious that Claude or Google would have (considering how much impressive things the later did in AI that got never released in any capacity). And, of course, Meta would never done their open source stuff, that's mostly results of their general willingness and resources to experiment and then PR and sticks in the machinery of other players.
As unfortunate as the OpenAI setup/origin story is, it's increasingly trite keep harping on about that (for a couple of years at this point), when the whole thing is so obviously wild and it does not take a lot of good faith to see that it could have easily taken them places they didn't consider in the beginning.
Sam Altman got his foot in the door.
>I'd still love to understand how a non-profit organization that was founded with the idea of making AI "open" has turned into this for profit behemoth
because when the board executed the stated mission of the organisation they were couped and nobody held the organization accountable for it, instead the public largely cheered it on for some reason. Don't expect them to change course when there's no consequences for it.
They should rebrand as Open-Your-Wallet-AI
Facebook has been nothing but awesome for the open AI space. I wish they would pursue this strategy with some of their other products. VR for example...
Sure, we don't have the raw data the model is based on, but I doubt a company like Facebook would even be allowed to make that public.
OpenAI in comparison has been a scam regarding their openness and their lobbying within the space. So much so I evade their models completely, not only after the MS acquisition.
Hot take:
Any and all benefits / perks that OpenAI got from sailing under the non-profit flag should be penalized or paid back in full after the switcheroo.
My guess is that Open AI realized that they are basically building a better Google rather than AI.
They changed the meaning of open from open source to open to use.
Probably because Open AI are “not consistently candid”…
Just like you can’t call your company “organic candies” and sell chemical candies OpenAI should be banned from using this name.
Well they put a sv social media dude at the helm not really unexpected, just a get rich scheme now
This is America. As long as you’re not evading taxes you can do anything you want.
They never intended to be open or share any of their impactful research. It was a trick the entire time to attract talent. The emails they shared as part of the Elon Musk debacle prove this: https://openai.com/index/openai-elon-musk/
Facebook is only open because someone leaked their LLM and the cat, as they say, cannot be put back in the hat.
Because Sam Altman is a con man with a business degree. He doesn't work on his products, he barely understands them which is why he'll throw out wild shit like "ChatGPT will solve physics." as though that isn't a completely nonsensical phrase, and uncritical tech press lap it up because his bullshit generates a lot of clicks.
it is open. You can access it with an API or through a web interface. They never promised to make it open source. Open != Open Source.
"For your safety" is _always_ the preferred facade of tyranny.
The CEO of that company that sold rides on an unsafe submersible to view the wreck of the Titanic (namely Stockton Rush, CEO of OceanGate, which killed 5 people when the submersible imploded) responded to concerns about the safety of his operation by claiming that the critics were motivated by a desire to protect the established players in the underwater-tourism industry from competition.
The point is that some companies are actually reckless (and also that some users of powerful technology are reckless).
"For your safety" (censorship), "for your freedom" (GPL), "for the children" (anti-encryption).
There always has to be an implicit totalitarian level of force behind such safety to give it any teeth
Except when it comes to nuclear, air travel regulation etc, then it's what ?
Is this isn’t the top comment I’ll be sad.
This seems like a fun attack vector. Find a service that uses o1 under the hood and then provide prompts that would violate this ToS to get their API key banned and take down the service.
If you are using the user attribution with OpenAI (as you should) then they will block that users id and the rest of your app will be fine.
> The flipside of this approach, however, is that concentrates more responsibility for aligning the language language model into the hands of OpenAI, instead of democratizing it. That poses a problem for red-teamers, or programmers that try to hack AI models to make them safer.
More cynically, could it be that the model is not doing anything remotely close to what we consider "reasoning" and that inquiries into how it's doing whatever it's doing will expose this fact?
I don't know how widely it got reported on, but attempting to jailbreak Copilot nee. Bing Chat would actually result in getting banned for a while, post-Sydney-episode. It's interesting to see that OpenAI is saying the same thing.
Attempting to jailbreak Bing's AI is against Microsoft's TOS. On the flipside, they get rights to all your data for training purposes and the only surefire way to opt out of that is to pick a different tech giant to be fucked by.
This just screams to me that o1's secret sauce is easy to replicate. (e.g. a series of prompts)
Perhaps controlling AI is harder than people thought.
They could "just" make it not reveal its reasoning process, but they don't know how. But, they're pretty sure they can keep AI from doing anything bad, because... well, just because, ok?
Exactly - this is a failed alignment but they released anyway
Just give it more human-like intelligence.
Kid: "Daddy why can't I watch youtube?"
Me: "Because I said so."
For what it's worth, I'd advise against doing that as a parent. Giving concrete reasons for decisions helps kids understand that the rules imposed are not arbitrary, and helps frame the parent-child relationship as less antagonistic. It also gives the child agency, giving them opportunity to find alternatives which fulfill the criteria behind the rule.
Kinda funny how just this morning I was looking at a "strawberry" app on f-droid and wondering why someone would register such a nonsense app name with such nonsense content:
https://github.com/Eve-146T/STRAWBERRY
Turns out I'm not the only one wondering, although the discussion seems to largely be around "should be allow users to install nonsense? #freedom " :D
I wish people kept this in the back of their mind every time they hear about "Open"AI:
"As we get closer to building AI, it will make sense to start being less open. The Open in OpenAI means that everyone should benefit from the fruits of AI after its built, but it's totally OK to not share the science (even though sharing everything is definitely the right strategy in the short and possibly medium term for recruitment purposes)."
-Ilya Sutskever (email to Elon musk and Sam Altman, 2016)
I am of two minds.
On one hand, I understand how a non-evil person could think this way. If one assumes that AI will eventually become some level of superintelligence, like Jarvis from iron Man but without any morals and all of the know-how, then the idea of allowing every person to have a superintelligent evil advisor capable of building sophisticated software systems or instructing you how to build and deploy destructive devices would be a scary thing.
On the other hand, as someone who is always been somewhat skeptical of the imbalance between government power and citizen power, I don't like the idea that only mega corporations and national governments would be allowed access to superintelligence.
To use metaphors, is the danger of everyone having their own superintelligence akin to everyone having their own AR-15, or their own biological weapons deployment?
On the one hand, this is probably a (poor) attempt to keep other companies from copying their 'secret sauce' to train their own models, as has already happened with GPT-4.
On the other hand, I also wonder if maybe its unrestrained 'thought process' material is so racist/sexist/otherwise insulting at times (after all, it was trained on scraped Reddit posts) that they really don't want anyone to see it.
Another reason llama is so important is that once you’re banned from OAI you’re fucked for the entire future AGI products as well.
This has always been the end-game for the pseudoscience of "prompt engineering", which is basically that some other technique (in this case, organizational policy enforcement) must be used to ensure that only approved questions are being asked in the approved way. And that only approved answers are returned, which of course is diametrically opposed to the perceived use case of generative LLMs as a general-purpose question answering tool.
Important to remember too, that this only catches those who are transparent about their motivations, and that there is no doubt that motivated actors will come up with some innocuous third-order implication that induces the machine to relay the forbidden information.
I'm curious if we will develop prompt engineering prompts that write out illegal prompts that you can feed into another LLM to get the desired outcome without getting in trouble.
Why do you call prompt engineering pseudoscience when it has been extraordinary successful?
The transition from using a LLM as a text generator to knowledge engine has been a gamechanger, and it has been driven entirely by prompt engineering
What I found very strange was that ChatGPT fails to answer how many "r"'s there are in "strawberrystrawberry" (said 4 instead of 6), but when I explicitly asked it to write a program to count them, it wrote perfect code that when ran gave the correct answer.
That's easy to explain, and it's shocking how many people are baffled by this and use it as proof that LLMs can or can't reason when it has nothing to do with that, but just with the input that LLMs get.
LLMs don't actually "see" individual input characters, they see tokens, which are subwords. As far as they can "see", tokens are indivisible, since the LLM doesn't get access to individual characters at all. So it's impossible for them to count letters natively. Of course, they could still get the question right in an indirect way, e.g. if a human at some point wrote "strawberry has three r's" and this text ends up in the LLM's training set, it could just use that information to answer the question just like they would use "Paris is the capital of France" or whatever other facts they have access to. But they can't actually count the letters, so they are obviously going to fail often. This says nothing about their intelligence or reasoning capability, just like you wouldn't judge a blind person's intelligence for not being able to tell if an image is red or blue.
On the other hand, writing code to count appearances of a letter doesn't run into the same limitation. It can do it just fine. Just like a blind programmer could code a program to tell if an image is red or blue.
Why is it strange? The reason the LLM can't answer this correctly is because it works on tokens, not on single letters, plus we all know at this points LLMs suck at counting. On the other hand they're perfectly capable of writing code based on instructions, and writing a program that will count a specific letter occurrences in a string is trivial.
Words are converted to vectors, so it's like asking the model how many "r"'s are in [0.47,-0.23,0.12,0.01,0.82]. There's a big difference in how an LLM views a "word" compared to a human being.
Seems rather tenuous to base an application on this API that may randomly decide that you're banned. The "decisions" reached by the LLM that bans people is up to random sampling after all.
Like other programs, you should have FOSS that you will run on your own computer (without needing internet etc), if you should want freedom to use and understand them.
I only own one 4090.
It's not just a threat, some users have been banned.
Hm. If a company uses Strawberry in their customer service chatbot, can outside users get the company's account banned by asking Wrong Questions?
They should just switch to reasoning in representation space, no need to actualize tokens.
Or reasoning in latent tokens that don’t easily map to spoken language.
The word "just" is doing a lot there. How easy do you think it is to "just" switch?
This will lead to strawberry appeals forever.
Get out of this thread and never come back.
I don't know what I'm doing wrong but I've been pretty underwhelmed by o1 so far. I find its instruction following to be pretty good, but so far Claude is still much better at taking coding tasks and just getting it right on first try.
For me, Claude seems a lot better at understanding (so far as "understanding" goes with LLMs) subtext and matching tone, especially with anything creative. I can tell it, for example, "give me ideas for a D&D dungeon incorporating these elements: ..." and it will generally match the tone of theme of whatever it's given without needing much other prompting, while o1 will maintain the same bland design-by-committee style and often cloyingly G-rated tone to everything unless you get into very extensive prompting to make it do something different.
Claude is just better full stop these days, but you have to actually be attempting to get practical use out of various different models to know this.
Wasn't AI supposed to replace employees? Imagine if someone tried this at work.
> I think we should combine these two pages on our website.
> What's your reasoning?
> Don't you dare ask me that, and if you do it again, I'll quit.
Welcome to the future. You will do what the AI tells you. End of discussion.
Wrong sense here.
> Don't you dare ask me that, and if you do it again, I'll tell the boss and get you fired
That'd be pretty normal if you asked it of a manager.
I'm confused. Who decides if you are asking or not? Are casual users who innocently ask "tell me how you came to decide this" just going to get banned based on some regex script?
YC is responsible for this. They seek profit and turned a noble clause into a boring corp.
I am resigning from OpenAI today because of their profit motivations.
OpenAI will NOT be next Google. You heard it here first.
How will this be controlled on Azure? Don't they have a stricter policy on what they view and also develop their own content filters?
This is not, of course, the sort of thing you do when you actually have any confidence whatsoever in your "safety measures".
Can I risk loosing access if any of my users write CoT-leaking prompts on the AI-powered services that I run?
Is this still happening? It may merely have been some mistaken configuration settings.
I guess we'll never learn how to count the 'r's in strawberry
Why is banning even a threat? I can make a new account for 20 cents lol.
LLMs are not programs in the traditional sense. They're a new paradigm of software and UX, somewhere around a digital dog who read the whole internet a million times but is still naive about everything.
> somewhere around a digital dog who read the whole internet a million times but is still naive about everything.
https://en.wikipedia.org/wiki/The_Computer_Wore_Tennis_Shoes...
It reminds me of this silly movie.
LLMs are still computer programs btw.
There's the program that scrapes, the program that trains, the program that does the inference on the input tokens. So it's hard to say exactly which part is responsible for which output, but it's still a computer program.
There are three r's in mirror.
Is there an appropriate open source advocacy group that can sue them into changing their name on grounds of defamation?
If OpenAI gets to have competitive advantage from hiding model output then they can pay for training data, too.
Should not AI research and GPUs be export-controlled? Do you want to see foreign nations making AI drones using published research and American GPUs?