« BackWe accidentally burned through 200GB of proxy bandwidth in 6 hoursblog.skyvern.comSubmitted by suchintan a year ago
  • patmcc a year ago

    I'm now expecting we'll see a couple things in the next few years:

    1. An explosion of residential proxy networks and other stuff to circumvent blocking of cloud IP ranges, for all the various AI scraping tools to use.

    2. A corresponding explosion of countermeasures to the above. Instead of blocking suspicious IPs, maybe they get a 3GB file on their request to /scrape-target.html

    • metadat a year ago

      200GB is nothing since 2018 when AT&T mass introduced their 1-gig symmetric fiber. Any single common gigabit link can run 200GB in 15 minutes.

      On any gig link, over the course of 6 hours you can transmit a little more than 4TB one way.. which is 40x more.

      • omoikane a year ago

        The discussion linked in the post is from 2022, and the corresponding issue has already been fixed:

        https://issues.chromium.org/issues/40220332

        I wonder if there is a more recent bug related to this?

        • sam0x17 a year ago

          Gosh I regularly burn through that much just updating games in steam :D. Not proxy bandwidth of course but isn't it funny that the the line between regular usage and $$$ can be what is using the bandwidth. Or rather, isn't it funny that regular consumers expect to be able to use multiple terabytes of data for < $100/mo but the same can still be thousands in other enterprise domains

          • perks_12 a year ago

            200GB for $500? What cloud is this?

            • tristor a year ago

              I would have liked to see a bit more of 5 Whys here. It seems like a consistent lesson that startups have to learn over and over is how to manage external dependencies, and particularly the dangers of having Google as a dependency. This is new Chrom(e|ium) behavior, and it has a real cost, both for this company and for users, which may or may not be worth the ROI, but this is what happens when you have a large scale external dependency: stuff moves without your knowledge, consent, or control.

              Instead of Always. Be. Closing. it should be Always. Be. Mitigating. Dependencies. for startups.

              • 8organicbits a year ago

                What infrastructure is this using? Bandwidth seems pretty pricy

                • dusted a year ago

                  " 200GB of proxy bandwidth was approximately $500 burned over the course of 6 hours"

                  The fuck ? So Internet is literally more expensive than buying a drive at amazon, paying for shipping, filling it up putting it on a truck towards a destination anywhere in the world.

                  • hkon a year ago

                    Literally means cloudprotection in Norwegian. Thought for a second we had gotten our own cloudflare.

                    • tcfhgj a year ago

                      Please, gigabyte isn't a unit of bandwidth.

                      Bandwidth is measured in data/time

                      • bradley13 a year ago

                        "We run leverage proxy networks and run headful browser instances"

                        Um...say what? I'm pretty broadly based in IT, and I have no idea what that means.

                        • elphinstone a year ago

                          Skyvern is a great name, very evocative. Typical arrogant Google, downloading trash to the user without consent.

                          • undefined a year ago
                            [deleted]
                            • olliej a year ago

                              Honestly given many of these stories, $500 seems to be getting off pretty lightly.

                              It’s still absurd to me that many (most?) of these hosting/bandwidth providers don’t seems to allow automatic cut offs and such

                              • tim_at_ping a year ago

                                Hello,

                                A (different) proxy company owner here. This sucks! Sorry that you lost out on so much bandwidth.

                                Feel free to reach out to me at tim@pingproxies.com and I'd be happy to get you set up on our service and credit you with 100GB of free bandwidth to help soften the blow. I'll also be able to get you pricing alittle better than you're currently on if you are interested ;)

                                Within the next few months we're also releasing a bunch of tools to help stop things like this happening on our residential network such as some intelligent routing logic, spend controls and a few other things.

                                You may also want to look into Static Residential ISP Proxies - we charge these per IP address rather than bandwidth and they often end up more economical. We work with carriers like Spectrum, Comcast & AT&T directly to get IP addresses on their networks so they look like residential connections but host them in datacenters - this way you get 99.99%+ availability, 1G+ throughput, stable IP addresses and have unlimited bandwidth.

                                @ everyone else in the thread; if you run a start-up and need proxies then email me - happy to credit you with 50GB free residential bandwidth + give some advice on infra if needed.

                                Cheers, Tim at Ping

                                • ang_cire a year ago

                                  Blocking Google from downloading anything onto your computer without consent is always a good idea.

                                  • monyasau a year ago

                                    [dead]

                                    • undefined a year ago
                                      [deleted]
                                      • rustdeveloper a year ago

                                        [flagged]

                                        • meindnoch a year ago

                                          >200GB of proxy bandwidth

                                          Gigabyte is a measure of information.

                                          Bandwidth is information transmitted over time.

                                          • keepamovin a year ago

                                            you shouldn’t be paying by the terabyte. Colocate and just pay for the maximum throughout. Far better rates