Google Drive Has a Hidden File Limit - Slashdot

Posted by BeauHD

• ›

  by Anonymous Coward on Friday March 31, 2023 @06:48PM (#63415458)

  If you're into the millions of files, you might as well just use a google bucket (their cloud storage.)

  • Re: Google drive...

    For those of us that bought a Pixel 4 and Pixel 5 with unlimited (standard quality) photo uploads, I'm not surprised people are hitting the 5 million limit. The offer wasn't limited to pictures taken by the phone, it could be any picture as long as the phone was the one updating it.
    
    I've used it to make back ups of all my photos and shared private albums with my wife. (The original and higher quality ones are on my OneDrive though.) This is great when we want to look through pictures on a phone or tablet and

• of those are abusing the soft storage cap

• • Re:

    If by "nailed search" you mean crucified it on a cross of ad revenue, then yes.

    • Re:

      I still find Google is generally the best search engine, though it is not as good as it used to be and I expect it will continue to degrade as quality is sacrificed for pay-for-placement.

      I'd love to see decentralized search become a thing. If search could be made hierarchical but and relatively open like DNS, I think that could be a great thing.

      You'd have to start with geopolitical boundaries and language, but you could also have some broad 2nd-tier categories similar to news media so that your search quer

      • Re:

        . If search could be made hierarchical but and relatively open like DNS, I think that could be a great thing.

        Not sure if joking or not. Does the phrase Yet Another Hierarchically Organized Oracle [wikipedia.org] mean anything to you?

        Of course that wasn't decentralized, but I do seem to recall it being "crowd sourced" back in the day. AFAIK, that aspect has long since been removed, which is a shame.

• The file limit, at least for GSuite Business accounts is visible in the Administration dashboard and has always been there.
• ..I think in this case 5 Million files/items is kinda acceptable. I don't think the normal user will every exceed that number of files in several lifetimes.

  • Re:

    Yeah, at this point anyone willingly using a Google service should just be thankful the company hasn't EOLed it yet.

    Don't get me wrong - setting aside my significant reservations regarding the company itself, I strongly prefer how Google's online tools work in comparison to Microsoft's. But depending on them just feels like you're living on borrowed time...

  • 640kB

    So what you are saying is that 5 million files ought to be enough for anybody. I hope history treats you kindly.

    • Re:

      It's expensive to have lots of small files, at minimum there is processing overhead for whatever mechanism packs them, otherwise maybe there is disk overhead too. If you have lots of activity across lots of small files, that's even more expensive, especially if they are packed somehow. It's obviously in google's interest to dissuade people from storing data that way. Systems or applications that currently do that could be more efficient in multiple ways if they would move to another kind of store.

      It's a sha

  • Re:

    It's not a user. It's an account. There can be multiple users.

• Did someone come up with a way to store unlimited data in filenames/metadata for 0 sized files?
• to get around this limit. You can use a script in MacOSX, Windows, Linux or BSD to create archives.

• Probably, they were approaching some kind of limit of the underlying infrastructure (most likely a distributed filesystem limit), and were presented a choice...

  Either do a costly and dangerous migration to new infrastructure (probably a new distributed Filesystem that can handle the total number of files they are managing) or restrict the maximum number of files per google drive...

  Imagine a couple (or two couples) of months ago, when there was no more time to delay said decision, in the middle of cost cutting meassures and layoffs...

  You would probably decide to implemement the limit and save your behind, customers be damned, and deal with the fallout of not informing customers sooner, than recomending the costly migration, being overruled on that decision by the higher-ups, then being fired in the next round of layoffs, and then hearing through the grapevine that your replacement did implement the hard limit anyway...

  • Re: Approaching a limit

    Costly and *dangerous*?

    Costly I understand. No one works for free.

    Dangerous? If you don't know what you're doing and think rsync behind a screen session is valid enterprise IT I suppose I can understand how you'd think a planned upgrade could be dangerous.

  • Re:

    I'm also getting similar vibes here. But the thing is, if Google can not figure out a way to fix such a technical problem, this tells us a lot about where the company currently is. They might have some of the best engineers in the world, but the management is cancer like everywhere. Maybe Google Drive is already considered legacy tech and they just don't care to put any effort into it anymore. Maybe they ordered everyone onto the AI bandwagon.

    Whatever the reason, if this is technical limitation, and they ar

    • Re:

      Peak Google was after they started buying up other non-search technologies left n right once they owned search.

      Other than search, is there anything of note developed in house they haven't killed?

• The more entries in the database the slower the parsing of that database.

  Maybe Google will be announcing new soft tech on the backend soon?

• For a 100 GB account this means an average of 20 kb per file. Kind of ok I guess. It is not a full file system, such limits are inevitable. Even coding projects that create numerous tiny header files would not easily reach 5 million. May be with some brain dead source control system locally put together by a long dead unix wizard based on awk, grep, diff, and bash may be. The original guru left ages ago, the new team had no idea how it worked but they kept using it....

  Anyway, one can always download folders and zip them and re upload them. May be some scripts to archive long unused files. Many ways to work around it. It is not even a slow news day. Some serious stuff is going on in Manhatten DA's office.

  • Re:

    Why would a Google Enterprise account only be using 100GB? The issue here isn't 140Mandak262Jamuna's home personal free Google Drive.

    Just for fun I checked my works' OneDrive sync folder. I have just shy of 800000 files. So if there are 10 of me working in a small business we'd exceed this limit. Lots of software generates lots of working files for a variety of reasons.

    • Re:

      I dont deny such use cases exist. But you can't expect such users to be supported for 20$ a year, on a fully backed up cloud storage.

      BTW I cant recall where I saw the tale of woe by a new hire to a start up who discovered the source control was a cron job making level 0 backup to some free cloud storage everyday.

      • Re:

        It is not necessarily a stretch to imagine that the amount of files creates expenses for Google that can exceed the blanket fee charged for the disk space. We are aware of things like inodes, although in this case any metadata for the files is going to exist in a database.

        But where your argument of "you can't expect such users to be supported for 20$" falls flat is the fact that there is no possibility to pay more to get more. There is no 200$ storage plan to store 50M files. There is not even a 200$ storag

        • Re:

          Google got into making a lot of new products all the time earlier than other big tech companies (putting aside for now the small percentage of them which persist) because they were first to have this big, broad, cloudy architecture. But what products they can create is clearly limited by the characteristics of that architecture, which was designed foremost for their search and corresponding advertising products. No doubt if there are limitations which are actually affecting them (like this apparent one) the

      • Re:

        Woeful engineer didn't understand startups.

        Shit like that happens at every startup. They focus on the big stuff that might make money and half ass everything else, at best, until they are doing well enough they can afford to hire woeful engineer to fix their half assed shit.

        If they wasted and yes I literally do mean wasted precious resources on an enterprise quality backup system as a startup and do everything else like that they'd never make it.

• I suppose you could otherwise cheat and make a 1024 byte file name on a 1 byte file. A few bytes for the index as well as many directories so you could get much data space for cheap?

• ...for them to prune their porn collection.

• Guys, the -i argument isn't the number of inodes; it's a divisor. Just look at mkfs.ext4's output and it'll tell you how many inodes you're getting. If it's too many, redo it with the -i arg higher, and if it's not enough, make the -i arg lower. Yes, it feels like a backwards UI, but how often are you having to do it?