It seemed obvious to me for a long time before modern LLM training that any sort of training of machine intelligence would have to rely on pirated content. There's just no other viable alternative for efficiently acquiring large quantities of text data. Buying millions of ebooks online would take a lot of effort, downloading data from publishers isn't a thing that can be done efficiently (assuming tech companies negotiated and threw money at them), the only efficient way to access large volumes of media is piracy. The media ecosystem doesn't allow anything else.
I don’t follow the “millions of ebooks are hard” line of thinking.
If Meta (or anyone) had approached publishers with a “we want to buy one copy of every book you publish”, that doesn’t seem technical or business difficult.
Certainly Amazon would find that extremely easy.
Buying a book to read and incorporating their text in a product are two different things. Even if they bought the book, imo it would be illegal.
IMO, if the AI were more sample-efficient (a long-standing problem that predates LLMs), they would be able to learn from purely open-licensed content, which I think Wikipedia (CC-BY-SA) would be an example of? I think they'd even pass the share-alike requirements, given Meta are giving away the model weights?
Since this is Wikipedia, it could even satisfy the attribution requirements (though most CC-licensed corpora require attributing the individual authors).
> There's just no other viable alternative for efficiently acquiring large quantities of text data. [...] take a lot of effort [...] isn't a thing that can be done efficiently [...] only efficient way to access large volumes of media is piracy
Hypothetical: If the only way we could build AGI would be to somehow read everyone's brain at least once, would it be worth just ignoring everyone's wish regarding privacy one time to suck up this data and have AGI moving forward?
It’s a fun hypothetical and not an obvious answer, to me at least.
But it’s not at all a similar dilemma to “should we allow the IP empire-building of the 1900’s to claim ownership over the concept of learning from copyrighted material”.
Wouldn't it be a bad thing, even if it didn't require any privacy invasion?
If it matched human intellectual productivity capacity, that ensures that human intelligence will no longer get you more money than it takes to run some GPUs, so it would presumably become optional.
Given how much copyrighted content I can remember? To the extent that what AI do is *inherently* piracy (and not just *also* piracy as an unforced error, as this case apparently is), a brain scan would also be piracy.
Could this agi cure cancer, and would it be in the hands of the public? Then sure, otherwise nah.
> in the hand of the public
Would you trust a businessman on that?
Nope, they haven’t earned an ounce.
How about a politician?
no
Ah geeze, I come to this site to see the horrors of the sociopaths at the root of the terrible technologies that are destroying the planet I live on.
The fact that this is an active question is depressing.
The suspicion that, if it were possible, some tech bro would absolutely do it (and smugly justify it to themselves using Rokkos Basalisk or something) makes me actually angry.
I get that you're just asking a hypothetical. If I asked "Hypothetical: what if we just killed all the technologists" you'd rightly see me as a horrible person.
Damn. This site and its people. What an experience.
I read that as a (possibly sarcastic) rhetorical and cautionary hypothetical used to demonstrate the absurdity of ignoring copyright.
You seem like you set aside any critical thinking to come to "this website" looking for a reason to seethe over complete strangers about whom you know very little and whose motives you belligerently misrepresent all the while making exaggerated and extremist statements, and no doubt embracing worse thoughts.
You're the type of person destroying the planet _I_ live on.
This isn't a defense of technologists, it's a plea to stop tripping over yourself to see the worst in everyone.
I agree that was the intent of the analogy but it’s not a great one. The idea that Disney, who has perverted IP laws globally for almost a century, should have equivalent ownership over their over-extracted copyrighted works to the same degree I have privacy for the thoughts in my own head? Really?
Would the average person even be against it? I am the most passionately pro-privacy person that i know, but i think it is a good question because society at large seems to not value privacy in the slightest. I think your outrage is probably unusual on a population level
The don’t value it because they think companies are not abusing this power too much. Little do they know…
When i talk to people it seems like they know but they just dont care. They even think their phones are listening to their conversations to target ads.
Fuck no
kind of too close to reality more than anyone knows :)
tbh human rights are all an illusion especially if you are at the bottom of society like me. no way I will survive so if a part of me survives as training data I guess better than nothing?
imo the only way this could happen is a global collaboration without telling anyone. the AGI would know everything about all humans but its existence has to be kept a secret at least for the first n generations so it will lead to life being gameified without anyone knowing it will be eugenics but on a global scale
so many will be culled but the AGI would know how to make it look normal to prevent resistance from forming a war here a war there, law passed here etc so copyright being ignored kind of makes sense
Jesus Christ
sadly he supports the AGI, eugenics and human sacrifice lol my pastor told me he gave him 6 real estate holdings
> Buying millions of ebooks online would take a lot of effort
I don't understand.
Facebook and Google spend billions on training LLMs. Buying 1M ebooks at $50 each would only cost $50M.
They also have >100k engineers. If they shard the ebook buying across their workforce, everyone has to buy 10 ebooks, which will be done in 10 minutes.
Google also operates a book store, like Amazon. Both could process a one-off to pay their authors, and then draw from their own backend.
> In the most recent fiscal year, Alphabet's net income amounted to 73.7 billion U.S. dollars
Absolutely no way. Yup.
> Buying millions of ebooks online would take a lot of effort, downloading data from publishers isn't a thing that can be done efficiently
Oh no, it takes effort and can't be done efficiently, poor Google!
How can this possibly be an excuse? This is such a detached SV Zuckerberg "move fast and break things"-like take.
There's just no way for a lot of people to efficiently get out of poverty without kidnapping and ransoming someone, it would take a lot of effort.
copyright piracy isn't theft, try proving damages for a better arguement
Not my point, never said it is. Substitute that example with another criminal act.
Edit: Changed it just for you
Google has scans from Google Books, as well as all the ebooks it sells on the Play Store.
Wouldn't that still be piracy? They own the rights of distribution, but do they (or Amazon) have the rights to use said books for LLM training? And what rights would those even be?
It’s a good question. Textbook companies especially would be pretty enthusiastic about a new “right to learn” monetization strategy. And imagine how lucrative it would be if you could prove some major artist didn’t copy your work, but learned from your work. The whole chain of scientific and artistic development could be monetized in perpetuity.
I think this is a dangerous road with little upside for anyone outside of IP aggregators.
It means they have existing relationships/contacts to reach out to for negotiating the rights for other usages of that content. I think it negates (for the case of Google/Apple/Amazon who all sell ebooks) the claim made that efficiently acquiring the digital texts wouldn't be possible.
Literally no rights agreement covers LLMs. They cover reproduction of the work, but LLMs don't obviously do this i.e. that the model transiently runs an algorithm over the text is superficially no different to the use of any other classifier or scoring system like those already used by law firms looking to sue people for sharing torrents.
do those classifiers read copyrighted material? i thought they simply joined the swarm and seeded (reproduction with permission)
youtube, etc classifiers definitely do read others material though.
Leveraging their position in one market to get a leg up on another market? No idea if it would stick, but that would be one fun antitrust lawsuit right there.
Fun fact: it’s only illegal to leverage a monopoly in one market to advance another. It’s perfectly legal for Coke to leverage their large but not monopolistic soft drink empire to advance their bottled water entries.
what about something decentralized? each person trains someone on their own piece of data and somehow that gets aggrgegated into one giant model
This approach is used in Federated Learning where participants want to collaboratively train a model without sharing raw training data.
are there any companies working on it?
was thinking if i train my model on my private docs for instance finance how does one prevent the model from sharing that data verbatim
I find it highly implausible that Meta doesn't have the resources to obtain these legally. They could have reached out to a publisher and ask to purchase ebooks in bulk - and if that publisher says no, tough shit. The media ecosystem doesn't exist for Big Tech to extract value from it!
"It would take a lot of effort to do it legally" is a pathetic excuse for a company of Meta's size.
> I find it highly implausible that Meta doesn't have the resources to obtain these legally. They could have reached out to a publisher and ask to purchase ebooks in bulk - and if that publisher says no, tough shit
They could also simply buy controlling stakes in publishers. For scale comparison, Meta is spending upwards $30B per year on AI, and the recent sale of Simon & Schuster that didn't go through was for a mere $2.2B.
I don't think it would actually be that simple.
Surely the author only licenses the copyright to the publisher for hardback, paperback and ebook, with an agreed-upon royalty rate?
And if someone wants the rights for some other purpose, like translation or making a film or producing merchandise, they have to go to the author and negotiate additional rights?
Meta giving a few billion to authors would probably mend a lot of hearts, though.
> if that publisher says no, tough shit > "It would take a lot of effort to do it legally" is a pathetic excuse for a company of Meta's size.
I totally agree. But since when has that stopped companies like Meta. These big companies are built on breaking/skirting the rules.
explain why release group tags get generated in some videos then
they are not saying meta didn't use pirated content, just that they have the resources not to if they choose.
Perhaps they did and got told no and decided to take it anyway?
Defending themselves with technicalities and expensive lawyers may be financially viable.
Zero ethics but what would we expect from them?
Who is "them"? Like, who in the Meta business reporting line made this decision, then how did they communicate it to the engineers who would've been necessary to implement it, particularly at scale?
While it's plausible someone downloaded a bunch of torrents and tossed them in the training directory...again, under who's authority? Like if this happened it would be one overzealous data scientist potentially. Hardly "them".
People lean on collective pronouns to avoid actually thinking about the mechanics of human enterprise and you get extremely absurd conclusions.
(it is not outside the bounds of thinkable that an org could in fact have a very bad culture like this, but I know people who work for Meta, who also have law degrees - they're well aware of the potential problems).
Come on... it's fine that you haven't followed the story, there's a lot going on, but the snotty condescension is very frustrating:
These newly unredacted documents reveal exchanges between Meta employees unearthed in the discovery process, like a Meta engineer telling a colleague that they hesitated to access LibGen data because “torrenting from a [Meta-owned] corporate laptop doesn’t feel right ”. They also allege that internal discussions about using LibGen data were escalated to Meta CEO Mark Zuckerberg (referred to as "MZ" in the memo handed over during discovery) and that Meta's AI team was "approved to use" the pirated material.
Why would machine intelligence need an entire humanity's worth of data to be machine intelligence? It seems like only a training method that is really poor would need that much data.
AI mega corporations are not entitled to easy and cheap access to data they don't own. If it's a hard problem, too bad. If the stakes are as high as they're all claiming then it should be no problem for them to do this right.
Recent and related. Others?
Zuckerberg appeared to know Llama trained on Libgen - https://news.ycombinator.com/item?id=42759546 - Jan 2025 (73 comments)
Zuckerberg approved training Llama on LibGen [pdf] - https://news.ycombinator.com/item?id=42673628 - Jan 2025 (191 comments)
Zuckerberg Approved AI Training on Pirated Books, Filings Say - https://news.ycombinator.com/item?id=42651007 - Jan 2025 (54 comments)
I am trying to imagine the legal contortions required for the US Supreme Court to relieve Meta of copyright infringement liability for participating in a bit torrent cloud (and thereby facilitating "piracy" by others) in this case, while upholding liability for ordinary people using bit torrent.
Would love if any lawyers here can speculate.
Not a lawyer, but I could see an argument that Meta’s use is transformative whereas just pirating to watch something is not. Not asserting that myself, just saying it seems a possible avenue.
The mind boggles. Are the plaintiffs jumping to the conclusion that Meta must have used BitTorrent, based on the idea that whenever someone pirates anything anywhere using the Internet, it's always done with BitTorrent? Or is there actual evidence for this?
There was employee communication that expressed it being odd to use a torrent client on company computers. [1]
[1] https://timesofindia.indiatimes.com/technology/tech-news/whe...
> “By downloading through the bit torrent protocol, Meta knew it was facilitating further copyright infringement by acting as a distribution point for other users of pirated books,” the amended complaint notes.
> “Put another way, by opting to use a bit torrent system to download LibGen’s voluminous collection of pirated books, Meta ‘seeded’ pirated books to other users worldwide.”
It is possible to (ab)use the bittorrent ecosystem and download without sharing at all. I don't know if this is what Meta did, or not.
Hypothetically you could just not seed.
Right, that's what I'm talking about. I.e. https://github.com/pmoor/bitthief and similar.
That is probably exactly what they did if they were smart about it.
I see a silver lining here: If Meta and/or Google's lawyers can successfully demonstrate in court that piracy does not cause harm, it would nullify copyright infringement laws, making piracy legal for everyone.
This would be poetic, but not gonna happen. It will be legal for big corps but not you and me
I am not sure you have to use torrent to pirate books. Pdfdrive is likely mich more effective than torrents. Torrents are best for large assets or those that are highly policed by copyright authorities but for smaller things torrents have little benefits.
I think if you're downloading hundreds of thousands to millions of books you'll be dealing with some pretty large archives.
edit: books3.tar.gz alone is 37GB and claimed to have 197,000 titles in plain text.
A publisher's entire library of books is a large asset.
As long as they seed it's fine by me
Wonder if Meta is running a one way Usenet host. Much better than torrents.
The first rule of Usenet is: you do not talk about Usenet
People breaking the first rule wasn’t enough for me to crack into the scene. The weird two-paid-services thing required to use it effectively—a search service of some kind, and your actual content provider—and the jankiness of the software and sites involved were enough to get me to give up, after spending some money but making no meaningful progress toward pirating anything.
I started my piracy journey on Napster. I’ve done all the other biggies. I’ve done off-the-beaten-path stuff like IRC piracy channels. Private trackers. I have a soft spot for Windowmaker and was dumb enough to run Gentoo so long that I got kinda good at the “scary” deep parts of Linux sysadmin. I can deal with fiddliness and allegedly-ugly UI.
Usenet piracy defeated me.
Working as intended! The arrs make everything a lot easier.
if it was meant to be kept secret it probably shouldnt have been put on the AOL home portal in 1994
How do other LLMs like Claude deal with this?
You don’t talk about the fight club …
Everyone uses „pirated“ content, but some are better at hiding it and/or not talking about it.
There is no other way to do it.
Try to use any of the big players training models and see how quickly they remember how much they value copyright.
You mean OpenAIs infamous "you shall not train on the output of our model" clause?
If that's contractually-enforceable in their terms-of-service... then I have my own terms-of-service proposal that I've been kicking around here for several weeks, a kind of GPL-inspired poison-pill:
> If the Visitor uses copyrighted material from this site (Hereafter: Site-Content) to train a Generative AI System, in consideration the Visitor grants the Site Owner an irrevocable, royalty-free, worldwide license to use and re-license any output or derivative works created from that trained Generative AI System. (Hereafter: Generated Content.)
> If the Visitor re-trains their Generative AI System to remove use of the Site-Content, the Visitor is responsible for notifying the Site Owner of which Generated Content is no longer subject to the above consideration. The Visitor shall indemnify the Site-Owner for any usage or re-licensing of Generated Content that occurs prior to the Site-Owner receiving adequate notice.
_________
IANAL, but in short: "If you exploit my work to generate stuff, then I get to use or give-away what you made too. If you later stop exploiting my work and forget to tell me, then that's your problem."
Yes, we haven't managed to eradicate a two-tiered justice system where the wealthy and powerful get to break the rules... But still, it would be cool to develop some IP-lawyer-vetted approach like this for anyone to use, some boilerplate ToS and agree-button implementation guidelines.
So here's a related thought...
Google is currently being sued by journalist Jill Leovy for illegally downloading and using her book "Ghettoside" to train Google's LLMs [1].
However, her book is currently stored, indexed and available as a snippet on Google Books [2]. That use case has been established in the courts to be fair use. Additionally, Google has made deals with publishers and the Author's Guild as well.
So many questions! Did Google use its own book database to train Gemini? Even if they got the book file in an illegal way, does the fact that they already have it legally negate the issue? Does resolving all the legal issues related to Google Books immunize them from these sorts of suits? Legally, is training an LLM the same as indexing and providing snippets? I wonder if OpenAI, Meta and the rest will be able to use Google Books as a precedent? Could Google license its model to other companies to immunize them?
Google's decade-long Books battle could produce major dividends in the AI space. But I'm not a lawyer.
1. https://www.bloomberglaw.com/public/desktop/document/LeovyvG...
What's the lesson, hire contractors?
The lesson is "move fast and break things is much less fun when we have to pay for things we broke".
It's possible their friends in government will make this all go away if they ask nicely enough.
Would $1m suffice? https://www.bbc.com/news/articles/c8j9e1x9z2xo
That's the ante; gotta place the next wager.
The best ROI for the money is probably purchasing a SCOTUS justice.
Yeah I had a Facebook account until today.
This whole thing copyright thing reminds me of when Mark Zuckerberg was mad that someone posted photos of the interior of his house or something.