No internet connection
  1. Home
  2. Support

Filesize limitations to uploads

By Christian Scheuer @chrscheuer2019-12-27 19:56:43.459Z

Idea/suggestion: Allow integration with third party storage services such as Azure Blobstorage, AWS or GCP.
Or, have tiers we can opt into that allows customers to upload larger files.
Right now we're having issues with customers who cannot upload screenshots if they're too big (happens a lot on retina screens apparently).

  • 17 replies

There are 17 replies. Estimated reading time: 19 minutes

  1. KajMagnus @KajMagnus2019-12-28 12:30:26.339Z2019-12-28 12:40:54.947Z

    1) I like the idea to integrate with AWS / Azure / GCP. At the same time, they don't have any spending limits, only budget "alerts". Any thoughts about that?

    What about using AWS / GCP / Azure / something, as a backend layer for a CDN? Then you and others could use e.g. Keycdn which has spending limits.

    If so, maybe you'd want to configure both an upload-to-Azure API secret (for example), plus a CDN address somehow?

    (Details: Here someone asked about GCP: https://stackoverflow.com/questions/27616776/how-do-i-set-a-cost-limit-in-google-developers-console — no spending limits. And the situation is the same for AWS and Azure (see e.g. https://docs.microsoft.com/en-us/azure/billing/billing-spending-limit: "Custom spending limits aren't available" — they have spending limits only for some kind of free credits accounts it seems to me. And, Amazon like Google apparently has only billing alerts: https://stackoverflow.com/questions/7021506/is-there-a-way-to-set-amazon-aws-billing-limit ) )

    2) Letting Talkyard handle large uploads — I'm a bit worried this'd eventually require me to spend a bit much time with Operations related things, like moving data between different disks & servers.

    3) About how many megabytes are the Retina screenshots?

    4) Another alternative might be to allow large uploads, but resize large images so they become smaller, take less disk space. Any thoughts about that?

    5) Maybe the best (?) alternative is if there's a way to configure a whole Talkyard server to use e.g. Google Cloud Storage, + a CDN in front of that, for all files larger than X bytes, say. Then I could configure the Talkyard .net server to use both Cloud Storage + Keycdn, and you and other people wouldn't need to do anything (no need to create your own AWS accounts or sth like that).

    ... Some time later, maybe there could be admin settings that let people override with their own CDN or AWS S3 accounts should they want to.

    1. CChristian Scheuer @chrscheuer2019-12-28 15:03:11.172Z

      Yes you can set up automated spending limits on GCP (in some areas), and budget alerts are fine too as they would allow you to take manual action if you prefer that over automated solutions. Note some answers on that question you're quoting are very old.
      You can configure this via a Cloud Function for example that just verifies cost or usage vs some budget the user defines. It may require a little code in some scenarios, but it's doable.

      Using a CDN might be a way to go as well, but I wouldn't force people to use that. IMO they could all just be providers and it should be up to the user to enforce any policies.

      TY can also help the situation by enforcing limits of itself as well before ever sending it to the storage provider.

      My goal with suggesting this was just that we have many users on the forum who complain they can't upload fairly normal stuff like screenshots and small screen recordings, and I would love for us to find a way where we could improve that UX.

      1. KajMagnus @KajMagnus2020-01-01 09:11:23.931Z2020-01-01 09:18:01.809Z

        What I wish there was, is outgoing bandwidth spending limits. I wouldn't trust any CDN spending limits scripts I wrote, because such things (using too much bandwidth) are a bit complicated to test, and re-test regularly, it seems to me. — Disk storage costs though, shouldn't be a problem.

        Billing alerts: Over a weekend, the CDNs can cost lots of money — one would need to check for billing alerts each day, I think.

        Hmm cloud functions look interesting, thanks for mentioning. For lots of things ... now when I stumbled upon their docs, when having a look.

    2. Progress
      with doing this idea
    3. KajMagnus @KajMagnus2020-01-01 09:15:03.932Z

      I've started looking into integrating with Google Cloud Storage and / or AWS S3 and DigitalOcean and Scaleway's object storage solutions.

      Since Talkyard .net uses Google already, probably initially I'll add support for Google Cloud Storage.

      Hopefully this'll be available within ... 2 weeks? Thereafter your users will be able to upload large images and videos. (How large are the largest files they've wanted to upload this far?)

      1. @KajMagnus marked this topic as Started 2020-01-01 09:15:08.604Z.
      2. C
        Christian Scheuer @chrscheuer2020-01-01 20:50:03.754Zreplies toKajMagnus:

        Cool! We're also on GCP so this would be perfect for us :)
        How were you thinking to do the implementation?

        The way I thought of it was, that we should be able to somehow create a storage bucket in our GCP account and then create TY as a member and then in the TY admin interface we would enter the API key or whatever credentials are needed.
        I don't know the entire workflow for how to enable 3rd parties into storage buckets but that's the general idea of how I thought it should work.

        Another thing to consider is the ability to use fine grained permissions within storage buckets. There's something to consider for the case where users would post files to closed off portions of the forum and whether or not those files should be in a publicly available storage bucket or not.

        Yea wrt spending limits, it definitely is a complicated subject and DDoS attacks could become very expensive if this is not taken into account for sure. But if we're hosting content on our own GCP project at least TY doesn't need to care about this.
        It would still be helpful to configure some sort of threshold that was enforced by TY both at the clientside (before uploading, should be possible with the File class in Javascript) and potentially serverside - if directing through the TY server (I think there would be a case for using JS to upload directly to the Storage bucket, then TY would never see this traffic).

        1. KajMagnus @KajMagnus2020-01-06 11:22:46.271Zreplies tochrscheuer:

          Hi again @chrscheuer

          How were you thinking to do the implementation?
          we should be able to somehow create a storage bucket in our GCP account [...] Ty as a member

          This sounds good to me. I'll look into this now this week. Probably I'll add a default cloud storage, + CDN with spending limits, for everyone who uses Talkayrd's hosting — and then you can override this, by uploading your own Cloud Storage credentials for your Talkayrd user (and, later on, configure your own CDN too should you want to). I had a quick look at how GCP & identity and access management works, and seems to me all this will work fine.

          fine grained permissions within storage buckets

          Yes that sounds like a good feature to have, ... and, personally I won't have time to look into this, not the nearest half a year at least. — Uploaded files have SHA256 hash names, so they're essentially hidden, unless one has a direct link.

          (I noticed that CDN:s also seem to support access permissions per URL, so maybe fine grained permissions can work also when a CDN is connected.)

          configure some sort of threshold that was enforced by TY both at the clientside (before uploading, should be possible with the File class in Javascript) and potentially serverside - if directing through the TY server

          Having a look at the files client side, is a good idea. Thanks for linking to Web/API/File/size. And server side too of course. Also, maybe one would want to specify if and how much the files should be compressed — I'd think in most cases, a 10 MB macOS Retina screenshot, is "overkill"? and maybe such large images should be downsize / lossy-compresed to 1 or 2 MB? (there're tools for lossy-compressing PNG images too).

          Maybe won't implement all this in the initial version — I'll start with just connecting to Cloud Storage & CDN.

          1. C
            Christian Scheuer @chrscheuer2020-01-06 11:27:36.987Zreplies toKajMagnus:

            All of this sounds great!
            And yea completely agree on the hashed file names/URLs being enough "security by obfuscation" for most purposes (including all of ours).
            And yes auto-downsizing images is a good feature to put in the backlog for some time in the future :)

            1. KajMagnus @KajMagnus2020-01-13 12:46:43.145Zreplies tochrscheuer:

              Sorry this got a bit delayed. (Code reviewing some code took days longer than what I expected. B.t.w. if you type a reply — you'll notice that now there's a live preview of your response, so you can see exactly how it'll look, in the page. And maybe the preview in the editor itself hereafter in most cases isn't needed.)

              I started looking into this now this morning ... & this week instead of last.

              1. C
                Christian Scheuer @chrscheuer2020-01-14 05:48:16.226Zreplies toKajMagnus:

                Thanks for the update :) IMO it's a little confusing right now to see your text 3 places, 2 I think is the maximum of what makes sense to look at, so while the experiment is great in itself my personal opinion is that it makes the editing experience slightly worse right now.
                Perhaps the eventual best editing experience is something we could look at - and talk about soon - while also considering being able to plug in custom made editor components.

                1. C
                  Christian Scheuer @chrscheuer2020-01-22 05:37:56.334Zreplies toKajMagnus:

                  Gotcha.. Yea you're right replies often end up in the wrong places. Will it be possible to roll back the design change though if it doesn't work? I see lots of flashing of the entire page with the loading bar etc. when editing whenever I have a slight pause -- so until the design change is more complete I feel like it still has the potential of making the editing experience worse. Perhaps you can make the change opt-in to begin with, ie. behind a feature flag until any bugs and design issues can be fixed?
                  Ways that could improve the experience:

                  • Never show a loading progress bar while I'm editing. It's very very annoying and almost driving me crazy just as I'm typing this response.
                  • As a first step, just show a placeholder (eg. the text "Your reply to ... will appear here"), not the entire text of the reply up in the thread - and keep the current preview system. It makes sense to view your markdown / preview side by side so you can verify your formatting works and not up by the thread. It's annoying to have to look up there to see how it looks.
                  • As a general rule, don't make large design changes without some sort of beta test, opt-in or ability to opt-out.
                  1. C
                    Christian Scheuer @chrscheuer2020-01-22 05:40:22.161Zreplies toKajMagnus:

                    And no worries with the filesize limitations.
                    I just really hate the design change with the previews. It's very distracting from the experience of typing the answers, which is something I often spend hours on every day.
                    I understand the problem you're trying to solve and I agree that that problem is important to solve, but IMO the current proposed solution is not an improvement but a quite serious degradation of the UX.

                    1. KajMagnus @KajMagnus2020-01-22 05:07:07.289Z2020-01-22 06:23:51.643Zreplies tochrscheuer:

                      This is getting even more delayed, sorry. (Last week, backups suddenly stopped working: gzip started causing "soft lockup"s and kernel panic and the Linux server shut itself down, until I added nice -n19 + I finished some unimplemented backup script things. And now, this week, I've been fixing some unfinished things related to migrating away from Talkyard .net to a self hosted server.)

                      best editing experience is something we could look at - and talk about soon - while also considering being able to plug in custom made editor components

                      That would be interesting, yes. — For now, probably I'll hide the in-editor preview, and keep only the in-page live preview. (Except for when one posts a new topic and there is not yet any page.) The thing is, people in some cases reply to the wrong post, so I'm thinking it'd be good if they can see precisely where their reply will appear, e.g. in-reply-to which other reply. (B.t.w. I've seen this happening (people replying to the wrong comment) relatively often with Facebook's semi threaded comments too.)

                      1. KajMagnus @KajMagnus2020-01-23 08:59:33.285Zreplies tochrscheuer:

                        Ok. I find it interesting that we have different opinions-about / feelings-for the inline preview.

                        Then I'd like to add a feature toggle for this, before releasing the new version, so you can disable it (for your whole forum).

                        I suppose it's good with different ways to tweak the UX, depending on one's forum's audience and how tech savvy they are.

                        1. C
                          Christian Scheuer @chrscheuer2020-01-27 02:45:47.019Zreplies toKajMagnus:

                          I'm not ruling it out permanently. It may at one point be better than the current UX, but I'm surprised you're not seeing that it isn't ready for prime time yet. My screen is constantly flickering whenever I pause my typing, sometimes putting up a loading graphic for seconds so I can't continue typing until it's done. It's incredibly distracting and definitely not a finished feature. If you updated the server for everybody without a switch to turn it off it would be enough for us to seriously start considering hosting our own fork of TY.
                          All I'm saying is that for text editing systems, the user experience is an absolutely central element and any change to it should be A/B tested and thoroughly vetted until you change it for end users. This doesn't feel tested to me.

                          1. KajMagnus @KajMagnus2020-01-27 04:26:32.793Zreplies tochrscheuer:

                            Thanks for the feedback! I'm not experiencing these things myself, still, I definitely neeed to fix this — I could reproduce the problems you're seeing:

                            My screen is constantly flickering whenever I pause my typing, sometimes putting up a loading graphic for seconds so I can't continue typing until it's done

                            by enabling Slow 3G network emulation in Dev Tools. Ok that's an enormously annoying editing experience.

                            Actually, I think the things you mentioned, are related to 1) you now being far away from the server. (But I'm fairly close to it.) And 2) a bug that pops up the loading indicator, although that shouldn't happen in these cases. But not related to the in-page preview.

                            I'll try to fix this before the next release. I feel pretty certain other people have the same issues (without telling me about it). ( https://news.ycombinator.com/item?id=21427996"They Might Never Tell You It’s Broken").

                            1. C
                              Christian Scheuer @chrscheuer2020-01-27 04:30:50.217Zreplies toKajMagnus:

                              Great you were able to repro it. Yea that's a common issue (edit: people not reporting issues) which is why having a great beta team/process is a critical part of long term success with software IMO.
                              Loading popup errors in strange scenarios is one of the most common issues I have with React myself as well, and often it can be quite difficult to repro why it happens for some customers but not for one-self. I hope with their new work with async rendering stuff like that should be easier to fix in the future.