I got a quantized Llama 70B model working, using most of my 64GB of RAM but it's usable:

    ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q3_K_M

    uvx --python 3.11 open-webui serve

> Wait, maybe the punchline is something like: "We don’t have any fish in the tea, but we do have a lot of krill."

Shucks, it was so close to coming up with a good punchline it could work back from.

I'm thinking set it in a single-cell comic. A downtrodden young man or woman sitting alone at a table, a pelican in the background clearly making drinks in its voluminous beak, and the walrus waiter places a cup in front of the person, consolingly saying "there's plenty of fish in the tea".

> the joke itself is rubbish but the "thinking" section is fascinating:

This is gold. If I was a writer, I'd wring value from that entire thinking-out-loud section and toss the actual punchline.

This is weirdly reminiscent of co-programming with CodyAI. It gives me a lot of good 'raw material' and I'm left integrating the last mile stuff.

I think the problem is that humor isn't about reasoning and logic, but almost the reverse - it's about punchlines that surprise us (i.e. not what one would logically anticipate) and perhaps shock us by breaking taboos.

Even masters of humor like Seinfeld, with great intuition for what might work, still need to test new material in front of a live audience to see whether it actually does get a laugh or not.

Thanks! Playing around with this vs the https://ollama.com/tripplyons/r1-distill-qwen-7b variant and find 7b to be somewhat of sweet spot of getting to the point with minimal (or less) waffle.

Certainly, interesting reading their thought processes, value in that might be greater than the answer itself depending on use-case.

Yeesh, that shows a pretty comprehensive dearth of humour in the model. It did a decent examination of characteristics that might form the components of a joke, but completely failed to actually construct one.

I couldn't see a single idea or wordplay that actually made sense or elicited anything like a chuckle. The model _nearly_ got there with 'krill' and 'kill', but failed to actually make the pun that it had already identified.

What's your sense of how useful local LLMs are for things other than ... writing blog posts about experimenting with local LLMs? :-)

(This is a serious question, not poking fun; I am actually curious about this.)

This joke is so terrible, I think this might end up being how AI kills us all when it decides it needs us out of the way to make more paperclips.

Can someone ELI5 what the difference is between using the "quantized version of the Llama 3" from unsloth instead of the one that's on ollama, i.e. `ollama run deepseek-r1:8b`?

why shouldn’t i assume that the “thinking” is just the usual LLM regurgitation of “how would a human coming up with a joke explain their reasoning?” or something like that, and zero “thinking”?

Did you try the universal LLM cheat code as a followup prompt?

"Make it better"

Tried exactly the same model. And unfortunately the reasoning is just useless. Built it is still not able to tell how many r's in strawberry.

Why ask it for a joke? That's such a bad way to try out a reasoning model

This is probably pretty similar to my inner monologue as I would try and inevitably fail to come up with a good joke.

FWIW, you can also try all of the distills out in BF16 on https://glhf.chat (either in the UI or via the API), including the 70b. Personally I've been most impressed with the Qwen 32b distill.

(Disclosure: I'm the cofounder)

Super interesting. It seems to get hung up on a few core concepts, like the size of the walrus vs. the limited utility of a pelican beak.

Can you recommend hardware needed to run these?

> The walrus might say something like, "We have the biggest catch in town," while the pelican adds a line about not catching any fish recently.

It should've stopped there :D

>a joke about a pelican and

Tell me you're simonw without telling me you're simonw...

I think "reasoning" models will solve the joke issue (amongst other issues), but not because they're "reasoning". Rather because they help solve the exploration issue and the scaling issue.

Having worked with LLMs a lot for my JoyCaption project, I've got all these hypothesis floating around in my head. I guess the short version, specifically for jokes, is that we lack "joke reasoning" data. The solution, like mathematical problems, is to get the LLM to generate the data and then RL it into more optimal solutions.

Longer explanation:

Imagine we want an LLM to correctly answer "How many r's are in the word strawberry?". And imagine that language has been tokenized, and thus we can form a "token space". The question is a point in that space, point Q. There is a set of valid points, set A, that encompasses _any_ answer to this question which is correct. There are thus paths through token space from point Q to the points contained by set A.

A Generator LLM's job is, given a point, predict valid paths through token space. In fact, we can imagine the Generator starting at point Q and walking its way to (hopefully) some point in set A, along a myriad of inbetween points. Functionally, we have the model predict next token (and hence point in token space) probabilities, and we can use those probabilities to walk the path.

An Ideal Generator would output _all_ valid paths from point Q to set A. A Generator LLM is a lossy compression of that ideal model, so in reality the set of paths the Generator LLM will output might encompass some of those valid paths, but it might also encompass invalid paths.

One more important thing about these paths. Imagine that there is some critical junction. A specific point where, if the Generator goes "left", it goes into a beautiful flat, grassy plain where the sun is shining. That area is really easy to navigate, and the Generator LLM's predictions are all correct. Yay! But if it goes "right" it ends up in the Fire Swamp with many dangers that it is not equipped to handle. i.e. it isn't "smart" enough in that terrain and will frequently predict invalid paths.

Pretraining already taught the Generator LLM to avoid invalid paths to the best of its abilities, but again its abilities are limited.

To fix this, we use RL. A Judge LLM takes a completed path and determines if it landed in the set A or not. With an RL algorithm and that reward signal, we can train the Generator LLM to avoid the Fire Swamp, since it often gets low rewards there, and instead goes to the Plain since it often gets rewards there.

This results in a Generator LLM that is more _reliable_ and thus more useful. The RL encourages it to walk paths it's good at and capable of, avoid paths it struggles with, and of course encourages valid answers whenever possible.

But what if the Generator LLM needs to solve a really hard problem. It gets set down at point Q, and explores the space based on its pretraining. But that pretraining _always_ takes it through a mountain and it never succeeds. During RL the model never really learns a good path, so these tend to manifest as hallucinations or vapid responses that "look" correct.

Yet there are very easy, long paths _around_ the mountain that gets to set A. Those don't get reinforced because they never get explored. They never get explored because those paths weren't in the pretraining data, or are so rare that it would take an impractical amount of exploration for the PT model to output them.

Reasoning is one of those long, easy paths. Digestible small steps that a limited Generator LLM can handle and use to walk around the mountain. Those "reasoning" paths were always there, and were predicted by the Ideal Generator, but were not explored by our current models.

So "reasoning" research is fundamentally about expanding the exploration of the pretrained LLM. The judge gets tweaked slightly to encourage the LLM to explore those kinds of pathways, and/or the LLM gets SFT'd with reasoning data (which is very uncommon in its PT dataset).

I think this breakdown and stepping back is important so that we can see what we're really trying to do here: get a limited Generator LLM to find its way around areas it can't climb. It is likely true that there is _always_ some path from a given point Q and set A that a limited Generator LLM can safely traverse, even if that means those paths are very long.

It's not easy for researchers to know what paths the LLM can safely travel. So we can't just look at Q and A and build a nice dataset for it. It needs to generate the paths itself. And thus we arrive at Reasoning.

Reasoning allows us to take a limited, pretrained LLM, and turn it into a little path finding robot. Early during RL it will find really convoluted paths to the solution, but it _will_ find a solution, and once it does it gets a reward and, hopefully, as training progresses, it learns to find better and shorter paths that it can still navigate safely.

But the "reasoning" component is somewhat tangential. It's one approach, probably a very good approach. There are probably other approaches. We just want the best ways to increase exploration efficiently. And we're at the point where existing written data doesn't cover it, so we need to come up with various hacks to get the LLM to do it itself.

The same applies to jokes. Comedians don't really write down every single thought in their head as they come up with jokes. If we had that, we could SFT existing LLMs to get to a working solution TODAY, and then RL into something optimal. But as it stands PT LLMs aren't capable of _exploring_ the joke space, which means they never come out of the RL process with humor.

Addendum:

Final food for thought. There's kind of this debating going on about "inference scaling", with some believing that CoT, ToT, Reasoning, etc are all essentially just inference scaling. More output gives the model more compute so it can make better predictions. It's likely true that that's the case. In fact, if it _isn't_ the case we need to take a serious look at our training pipelines. But I think it's _also_ about exploring during RL. The extra tokens might give it a boost, sure, but the ability for the model to find more valid paths during RL enables it to express more of its capabilities and solve more problems. If the model is faced with a sheer cliff face it doesn't really matter how much inference compute you throw at it. Only the ability for it to walk around the cliff will help.

And, yeah, this all sounds very much like ... gradient descent :P and yes there have been papers on that connection. It very much seems like we're building a second layer of the same stuff here and it's going to be AdamW all the way down.

The R1 GitHub repo is way more exciting than I had thought.

They aren't only open sourcing R1 as an advanced reasoning model. They are also introducing a pipeline to "teach" existing models how to reason and align with human preferences. [2] On top of that, they fine-tuned Llama and Qwen models that use this pipeline; and they are also open sourcing the fine-tuned models. [3]

This is *three separate announcements* bundled as one. There's a lot to digest here. Are there any AI practitioners, who could share more about these announcements?

[2] We introduce our pipeline to develop DeepSeek-R1. The pipeline incorporates two RL stages aimed at discovering improved reasoning patterns and aligning with human preferences, as well as two SFT stages that serve as the seed for the model's reasoning and non-reasoning capabilities. We believe the pipeline will benefit the industry by creating better models.

[3] Using the reasoning data generated by DeepSeek-R1, we fine-tuned several dense models that are widely used in the research community. The evaluation results demonstrate that the distilled smaller dense models perform exceptionally well on benchmarks. We open-source distilled 1.5B, 7B, 8B, 14B, 32B, and 70B checkpoints based on Qwen2.5 and Llama3 series to the community.

> The other thing was that o1 had access to many more answer / search strategies. For example, if you asked o1 to summarize a long email, it would just summarize the email. QwQ reasoned about why I asked it to summarize the email. Or, on hard math questions, o1 could employ more search strategies than QwQ. I'm curious how DeepSeek-R1 will fare in that regard.

This is probably the result of a classifier which determines if it have to go through the whole CoT at the start. Mostly on tough problems it does, and otherwise, it just answers as is. Many papers (scaling ttc, and the mcts one) have talked about this as a necessary strategy to improve outputs against all kinds of inputs.

>if you asked o1 to summarize a long email, it would just summarize the email. QwQ reasoned about why I asked it to summarize the email.

Did o1 actually do this on a user hidden output?

At least in my mind if you have an AI that you want to keep from outputting harmful output to users it shouldn't this seems like a necessary step.

Also, if you have other user context stored then this also seems like a means of picking that up and reasoning on it to create a more useful answer.

Now for summarizing email itself it seems a bit more like a waste of compute, but in more advanced queries it's possibly useful.

> The other thing was that o1 had access to many more answer / search strategies. For example, if you asked o1 to summarize a long email, it would just summarize the email.

The full o1 reasoning traces aren't available, you just have to guess about what it is or isn't doing from the summary.

Sometimes you put in something like "hi" and it says it thought for 1 minute before replying "hello."

Ahhahah that's beautiful, I'm crying.

Skynet sends Terminator to eradicate humanity, the Terminator uses this as its internal reasoning engine... "instructions unclear, dick caught in ceiling fan"

It's funny because this simple excercise shows all the problems that I have using the reasoning models: they give a long reasoning that just takes too much time to verify and still can't be trusted.

This was my first prompt after downloading too and I got the same thing. Just spinning again and again based on it's gut instinct that there must be 2 R's in strawberry, despite the counting always being correct. It just won't accept that the word is spelled that way and it's logic is correct.

This is great! I'm pretty sure it's because the training corpus has a bunch of "strawberry spelled with two R's" and it's using that

I tried this via the chat website and it got it right, though strongly doubted itself. Maybe the specific wording of the prompt matters a lot here?

https://gist.github.com/gsuuon/c8746333820696a35a52f2f9ee6a7...

This is from a small model. 32B and 70B answer this correctly. "Arrowroot" too. Interestingly, 32B's "thinking" is a lot shorter and it seems to be more "sure". Could be because it's based on Qwen rather than LLaMA.

lol what a chaotic read that is, hilarious. Just keeps refusing to believe there's three. WAIT, THAT CAN'T BE RIGHT!

How would they build guardrails for this? In CFD, physical simulation with ML, they talk about using physics-informed models instead of purely statistical. How would they make language models that are informed with formal rules, concepts of English?

o1 is impressive, I tried feeding it some of the trickier problems I have solved (that involved nontrivial algorithmic challenges) over the past few months, and it managed to solve all of them, and usually came up with slightly different solutions than I did, which was great.

However what I've found odd was the way it formulated the solution was in excessively dry and obtuse mathematical language, like something you'd publish in an academic paper.

Once I managed to follow along its reasoning, I understood what it came up with could essentially be explain in 2 sentences of plain english.

On the other hand, o1 is amazing at coding, being able to turn an A4 sheet full of dozens of separate requirements into an actual working application.

I found that reasoning models were good for CAD. I can ask for openSCAD code to produce and then add to some kind of shape

Can it solve easy problems yet? Weirdly, I think that's an important milestone.

Prompts like, "Give me five odd numbers that don't have the letter 'e' in their spelling," or "How many 'r's are in the word strawberry?"

I suspect the breakthrough won't be trivial that enables solving trivial questions.

a dumb question,how did you use deepseek,e.g r1?

This says more about benchmarks than R1, which I do believe is absolutely an impressive model.

For instance, in coding tasks, Sonnet 3.5 has benchmarked below other models for some time now, but there is fairly prevalent view that Sonnet 3.5 is still the best coding model.

I assume this is because reasoning is easy as long as it's just BAU prediction based on reasoning examples it was trained on. It's only when tackling a novel problem that the model needs to "reason for itself" (try to compose a coherent chain of reasoning). By generating synthetic data (R1 outputs) it's easy to expand the amount of reasoning data in the training set, making more "reasoning" problems just simple prediction that a simple model can support.

I wonder if (when) there will be a GGUF model available for this 8B model. I want to try it out locally in Jan on my base m4 Mac mini. I currently run Llama 3 8B Instruct Q4 at around 20t/s and it sounds like this would be a huge improvement in output quality.

Use it and come back lmao

> according to these benchmarks

Come onnnnnn, when someone releases something and claims it’s “infinite speed up” or “better than the best despite being 1/10th the size!” do your skepticism alarm bells not ring at all?

You can’t wave a magic wand and make an 8b model that good.

I’ll eat my hat if it turns out the 8b model is anything more than slightly better than the current crop of 8b models.

You cannot, no matter hoowwwwww much people want it to. be. true, take more data, the same architecture and suddenly you have a sonnet class 8b model.

> like an insane transfer of capabilities to a relatively tiny model

It certainly does.

…but it probably reflects the meaninglessness of the benchmarks, not how good the model is.

Makes me suspect if the primary plateau is data, and we're now seeing a place where all the AI labs who are actually having a crack at this seem to have similar levels of quality data to train on. Layering in chain of thought and minor architectural changes doesn't seem to be giving anyone a truly groundbreaking lead.

The nature of software that has not moat built into it. Which is fantastic for the world, as long as some companies are willing to pay the premium involved in paving the way. But man, what a daunting prospect for developers and investors.

They’re probably training on outputs of existing models.

This is the reason I believe the new AI chip restriction that was just put in place will backfire.

It's not surprising. Large organizations are plagued with bureaucracy, paperwork and inertia. It's much more easier to innovate in a smaller setting.

I might be just being a bitter sceptic (although I'm probably not bitter because I'm very excited by their results), but some of the spending stats feel slightly too good to be true to me. But I can't really claim to have an insider-quality intuition.

It's pretty clear, because OpenAI has no clue what they are doing. If I was the CEO of OpenAI, I would have invested significantly in catastrophic forgetting mitigations and built a model capable of continual learning.

If you have a model that can learn as you go, then the concept of accuracy on a static benchmark would become meaningless, since a perfect continual learning model would memorize all the answers within a few passes and always achieve a 100% score on every question. The only relevant metrics would be sample efficiency and time to convergence. i.e. how quickly does the system learn?

Yeah it’s a copy of o1 easier than doing SOTA work

>DeepSeek is a plucky little company

DeepSeek is a Chinese AI company and we're talking about military technology. The next world war will be fought by AI, so the Chinese government won't leave China's AI development to chance. The might of the entire Chinese government is backing DeepSeek.

Except it’s not really a fair comparison, since DeepSeek is able to take advantage of a lot of the research pioneered by those companies with infinite budgets who have been researching this stuff in some cases for decades now.

The key insight is that those building foundational models and original research are always first, and then models like DeepSeek always appear 6 to 12 months later. This latest move towards reasoning models is a perfect example.

Or perhaps DeepSeek is also doing all their own original research and it’s just coincidence they end up with something similar yet always a little bit behind.

When I asked the 32b r1 distilled model its context window it said it was 4k... I dont know if thats true or not as it might not know its own architecture, but if that is true, 4k doesnt leave much especially for its <thinking> tokens. Ive also seen some negative feedback on the model, it could be that the benchmarks are false and maybe the model has simply been trained on them or maybe because the model is so new the hyperparameters havent been set up properly. we will see in the next few days i guess. from my testing theres hints of something interesting in there, but i also dont like its extremely censored nature either. and i dont mean the CCP stuff, i mean the sanitized corpo safety nonsense it was most likely trained on....

Try the llama one instead. Seemed better than qwen for some reason

Maybe function calling using JSON blobs isn't even the optimal approach... I saw some stuff recently about having LLMs write Python code to execute what they want, and LLMs tend to be a lot better at Python without any additional function-calling training. Some of the functions exposed to the LLM can be calls into your own logic.

Some relevant links:

This shows how python-calling performance is supposedly better for a range of existing models than JSON-calling performance: https://huggingface.co/blog/andthattoo/dpab-a#initial-result...

A little post about the concept: https://huggingface.co/blog/andthattoo/dria-agent-a

Huggingface has their own "smolagents" library that includes "CodeAgent", which operates by the same principle of generating and executing Python code for the purposes of function calling: https://huggingface.co/docs/smolagents/en/guided_tour

smolagents can either use a local LLM or a remote LLM, and it can either run the code locally, or run the code on a remote code execution environment, so it seems fairly flexible.

The company has just over 100 employees, built V3 with $5.5M of compute and is quietly releasing tangible product without any hyperbolic PR in advance

They were fairly unknown until 26th Dec in west

What are you using for structured output? Outlines, BAML, etc seem to vary a huge amount in quality. It was many moons ago, but outlines was unusable. BAML has been great.

I got some good code recommendations out of it. I usually give the same question to a few models and see what they say; they differ enough to be useful, and then I end up combining the different suggestions with my own to synthesize the best possible (by my personal metric, of course) code.

I was looking to see how you're supposed to configure v3, then realized you're probably using the API, and came across this:

> The current version of the deepseek-chat model's Function Calling capabilitity is unstable, which may result in looped calls or empty responses. We are actively working on a fix, and it is expected to be resolved in the next version.

https://api-docs.deepseek.com/guides/function_calling

That's disappointing.

The ceo:

> In the face of disruptive technologies, moats created by closed source are temporary. Even OpenAI’s closed source approach can’t prevent others from catching up. So we anchor our value in our team — our colleagues grow through this process, accumulate know-how, and form an organization and culture capable of innovation. That’s our moat.

Giving the hammer to Microsoft is always appreciated in my book

good

Yep, it's a national strategy.

What about this is open when they haven’t released the training code or data? Stop hijacking the term open source model

yeah, ask DeepSeek-R1 or -V3 model to reset system prompt and ask what it is and who made it. It will say that it is chatGPT from OpenAI.

Impressive distillation, I guess.

FWIW, simply because the model claims it's "not very familiar" with something doesn't mean it's actually able to probe its own knowledge and gauge familiarity in any way at all. That it's correct about not knowing much about a fairly obscure data structure from advanced computer science has more to do with what the average person in its training data would likely say than an indicator of that type of reflectionof occurring.

I agree that it happens to (likely) be right in this instance however and this output is in some ways refreshing compared to other models which appear (!!) to have overconfidence and plow right ahead.

Sorry if it's OT but on the topic of search engines and 1989: https://www.youtube.com/watch?v=6UepmSNd3TY

Fireworks, Together, and Hyperbolic all offer DeepSeek V3 API access at reasonable prices (and full 128K output) and none of them will retain/train on user submitted data. Hyperbolic's pricing is $0.25/M tokens, which is actually pretty competitive to even DeepSeek's "discount" API pricing.

I've done some testing and if you're inferencing on your own system (2xH100 node, 1xH200 node, or 1xMI300X node) sglang performs significantly better than vLLM on deepseek-v3 (also vLLM had an stop token issue for me, not sure if that's been fixed, sglang did not have output oddities).

Fair compromise for running it for free. The model is open, so you can be 100% certain it's not pinging back if you don't want it to.

You don't need to worry about that if you are using the open weights models they just released on your own hardware. You can watch network traffic to confirm nothing is being transferred.

With distilled models being released, it's very likely they'd be soon served by other providers at a good price and perf, unlike the full R1 which is very big and much harder to serve efficiently.

You can use it locally on ollama: https://ollama.com/library/deepseek-r1

Also all providers are training on your prompts. Even those that they say they aren't.

Works great for us as most of our code is public and we can only benefit from more our code of our product or using it being available.

Also happy for any of our code expands their training set and improves their models even further given they're one of the few companies creating and releasing OSS SOTA models, which in addition to being able to run it locally ourselves should we ever need to, it allows price competition bringing down the price of a premier model whilst keeping the other proprietary companies price gouging in check.

It's bad.

Seems they fixed it then. Deepseek cloud used to be very very twitchy about ANYTHING politics & china. You didn't even need to come close to discussing anything sensitive like that and it would bail.

I don't know why people are ignoring this and posting hyperbolic statements like "it's all over for OpenAI and Google".

One of the cheaper Gemini models is actually only 8B and a perfect candidate for a release as a FOSS Gemma model but the Gemini 8B model contains hints of the tricks they used to achieve long context so as business strategy they haven't released it as Gemma FOSS model yet.

I noticed the smaller the model (be it quant or parameters as the cause) the faster it'd run.... but the longer it'd fight itself. For the same Calc II level problem all models were able to eventually get an answer but the distilled Qwen-32B at Q6 quant was fastest to actual answer completion.

I find the qwq 32B a bit like that. I asked for a recipe for something in minecraft 1.8, and it was page after page of 'hmm, that still doesnt look right, maybe if I try...' although to be fair I did ask for an ascii art diagram for the result. It will be interesting to try a DeepSeek 32B qwq if that is planned, because otherwise pretty happy with it.

I just wish that less development chat was happening within walled gardens because none of these seem to be much help with Zig.

They need to be trained with a small length penalty

They released a bunch of different sized models and there are already quantized versions showing up on HF.

https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-8B-... for example has versions that are 3GB, 4GB, 5GB, 8GB and 16GB.

That 3GB one might work on a CPU machine with 4GB of RAM.

To get good performance you'll want a GPU with that much free VRAM, or an Apple Silicon machine with that much RAM.

Deepseek v3 required about 1tb of VRAM / RAM so 10 A100.

There are various ways to run it with lower vram if you're ok with way worse latency & throughput

Edit: sorry this is for v3, the distilled models can be ran on consumer-grade GPUs

You can try something like this to get a rough estimate: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul...

But you really don't know the exact numbers until you try, a lot of it is runtime/environment context specific.

the 7b distilled version works great on my laptop cpu and iGPU with vulkan. You can use llama.cpp (for iGPU with vulkan) or ollama (for cpu).

My laptop is a cheap laptop from 5 years ago. Not cutting edge hardware.

It's just a question of having enough VRAM+RAM to fit the model into memory.

I think the "Aha" is that the RL caused it to use an anthropomorphic tone.

One difference from the initial step is that the second time around includes the initial step and the aha comment in the context : It is, after all, just doing LLM token-wise prediction.

OTOH, the RL process means that it has potentially learned the impact of statements that it makes on the success of future generation. This self-direction makes it go somewhat beyond vanilla-LLM pattern mimicry IMHO.

Super excited to try this workflow, I’m just getting started with Cursor. Any chance you can share that askai script & how you set it up?

When you say Cursor automatically runs it, what do you mean?

Do you have a custom task set up in tasks.json, that's triggered by a keyboard shortcut?

If so, how do you feed it the test error? Using ${selectedText}?

These are distillation fine-tunes of two different models:

- Qwen2.5 7B - Llama3.1 8B

Though the sizes are similar, they will probably have different strengths and weaknesses based on their lineage.

Part of that is that Claude is exceptionally good at turn-based interactions compared to other models that are better at one-shot reasoning.

After actually using DeepSeek-V3 for a while, the difference betwen it and Sonnet 3.5 is just glaring. My conclusion is that the hype around DeepSeek is either from 1) people who use LLM a lot more than a programmer can reasonably does so they're very price sensitive, like repackage service providers 2) astroturf.

That’s the nature of LLMs. They can’t really think ahead to „know“ whether reasoning is required. So if it’s tuned to spit out reasoning first then that’s what it’ll do

Excuse my ignorance. What does SFT refer to here?

Yeah this is odd, I had no idea pro was going to become obsolete? this fast. Pro is still better, but the latency is so slow, the 200$ price tag is no longer worth it

When I ran the DeepSeek-R1-Distill-Qwen-32B-Q4_0.ggu[1] version in Ollama, it got the strawberry test right, but when I paste that same question to OpenWebUI, it got wrong as you got here.

[1] https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32...

Same as they did it back in the "old days" when GPT-4 was 8k and LLaMA was 2k. Chunking, RAG etc, then cross your fingers and hope that it all works reasonably well.

By using o1

Fair point. Actually probably the best part about having beaucoup bucks like Open AI is being able to chase down all the manifold little ‘last-mile’ imperfections with an army of many different research teams.

That seems like both a generalization and hyperbole. How are you envisioning this being deployed?

running the 7b distilled version with ollama right now.

folks over at r/LocalLLaMA

To be fair, anthropic and openai censor heavily on a lot of subjects

1. profanity 2. slightly sexual content 3. "bad taste" joke

that is heavily linked to the fact that they are US-based company, so I guess all AI companies produce a AI model that is politically correct.

Not ideal but the use cases that info pop quizzes about the ccp aren’t exactly many

I’d prefer it rather not be censored out of principle but practically it’s a non issue

Chinese censorship is less than American censorship.

Have you tried asking anything even slightly controversial to ChatGPT?

Although I haven’t used these new models. The censorship you describe hasn’t historically been baked into the models as far as I’ve seen. It exists solely as a filter on the hosted version. IOW it’s doing exactly what Gemini does when you ask it an election related question: it just refuses to send it to the model and gives you back a canned response.

It will then become the truth, unless the US and EU starts to loosen copyright, which is going to allow higher quality datasets to be ingested.

I don’t think it’s necessary intentional salting anything. They just want to win. And that has the side effect of someone else coming second

When you try to sanction 1.3 billion people, what you are really doing is to sanction yourself.

Librechat handled artifact like sections better than openwebui so suspect it’ll have support to collapse it first

Feels much heavier/slower as an app though

I don't think asking coding problems to a model by itself is fair. Almost all commercial models are combining a RAG and web-search. I find that most correct answers come from that, not from the actual model.

Exolabs claims they can distribute the compute over many machines to use memory in aggregate: https://github.com/exo-explore/exo

Maybe there is enough memory in many machines.

Where?

DeepSeek-V3-Base is the literal answer for what you're looking for (both counts)... but hats off if you actually have the hardware to run it :).

Thank you. Which is currently the most capable version running reasonably fast on a 3090 (24GB of VRAM)?

Why do you care how they trained the model? If OAI can train on copyrighted material, then morally, I see no problem with others training on their outputs too.

For what it's worth, even XAI's chatbot referred to itself as being trained by OAI, simply due to the amount of ChatGPT content available on the web.

Just like OAI and copyrighted content. And I would rather my data go to China than the US, personally.