No internet connection
  1. Home
  2. Support

Filesize limitations to uploads

By Christian Scheuer @chrscheuer
    2019-12-27 19:56:43.459Z

    Idea/suggestion: Allow integration with third party storage services such as Azure Blobstorage, AWS or GCP.
    Or, have tiers we can opt into that allows customers to upload larger files.
    Right now we're having issues with customers who cannot upload screenshots if they're too big (happens a lot on retina screens apparently).

    • 23 replies

    There are 23 replies. Estimated reading time: 22 minutes

    1. KajMagnus @KajMagnus2019-12-28 12:30:26.339Z2019-12-28 12:40:54.947Z

      1) I like the idea to integrate with AWS / Azure / GCP. At the same time, they don't have any spending limits, only budget "alerts". Any thoughts about that?

      What about using AWS / GCP / Azure / something, as a backend layer for a CDN? Then you and others could use e.g. Keycdn which has spending limits.

      If so, maybe you'd want to configure both an upload-to-Azure API secret (for example), plus a CDN address somehow?

      (Details: Here someone asked about GCP: https://stackoverflow.com/questions/27616776/how-do-i-set-a-cost-limit-in-google-developers-console — no spending limits. And the situation is the same for AWS and Azure (see e.g. https://docs.microsoft.com/en-us/azure/billing/billing-spending-limit: "Custom spending limits aren't available" — they have spending limits only for some kind of free credits accounts it seems to me. And, Amazon like Google apparently has only billing alerts: https://stackoverflow.com/questions/7021506/is-there-a-way-to-set-amazon-aws-billing-limit ) )

      2) Letting Talkyard handle large uploads — I'm a bit worried this'd eventually require me to spend a bit much time with Operations related things, like moving data between different disks & servers.

      3) About how many megabytes are the Retina screenshots?

      4) Another alternative might be to allow large uploads, but resize large images so they become smaller, take less disk space. Any thoughts about that?

      5) Maybe the best (?) alternative is if there's a way to configure a whole Talkyard server to use e.g. Google Cloud Storage, + a CDN in front of that, for all files larger than X bytes, say. Then I could configure the Talkyard .net server to use both Cloud Storage + Keycdn, and you and other people wouldn't need to do anything (no need to create your own AWS accounts or sth like that).

      ... Some time later, maybe there could be admin settings that let people override with their own CDN or AWS S3 accounts should they want to.

      1. CChristian Scheuer @chrscheuer
          2019-12-28 15:03:11.172Z

          Yes you can set up automated spending limits on GCP (in some areas), and budget alerts are fine too as they would allow you to take manual action if you prefer that over automated solutions. Note some answers on that question you're quoting are very old.
          You can configure this via a Cloud Function for example that just verifies cost or usage vs some budget the user defines. It may require a little code in some scenarios, but it's doable.

          Using a CDN might be a way to go as well, but I wouldn't force people to use that. IMO they could all just be providers and it should be up to the user to enforce any policies.

          TY can also help the situation by enforcing limits of itself as well before ever sending it to the storage provider.

          My goal with suggesting this was just that we have many users on the forum who complain they can't upload fairly normal stuff like screenshots and small screen recordings, and I would love for us to find a way where we could improve that UX.

          1. KajMagnus @KajMagnus2020-01-01 09:11:23.931Z2020-01-01 09:18:01.809Z

            What I wish there was, is outgoing bandwidth spending limits. I wouldn't trust any CDN spending limits scripts I wrote, because such things (using too much bandwidth) are a bit complicated to test, and re-test regularly, it seems to me. — Disk storage costs though, shouldn't be a problem.

            Billing alerts: Over a weekend, the CDNs can cost lots of money — one would need to check for billing alerts each day, I think.

            Hmm cloud functions look interesting, thanks for mentioning. For lots of things ... now when I stumbled upon their docs, when having a look.

        • Progress
          with doing this idea
        • I've started looking into integrating with Google Cloud Storage and / or AWS S3 and DigitalOcean and Scaleway's object storage solutions.

          Since Talkyard .net uses Google already, probably initially I'll add support for Google Cloud Storage.

          Hopefully this'll be available within ... 2 weeks? Thereafter your users will be able to upload large images and videos. (How large are the largest files they've wanted to upload this far?)

          1. @KajMagnus marked this topic as Started 2020-01-01 09:15:08.604Z.
          2. C
            Christian Scheuer @chrscheuer
              2020-01-01 20:50:03.754Zreplies toKajMagnus:

              Cool! We're also on GCP so this would be perfect for us :)
              How were you thinking to do the implementation?

              The way I thought of it was, that we should be able to somehow create a storage bucket in our GCP account and then create TY as a member and then in the TY admin interface we would enter the API key or whatever credentials are needed.
              I don't know the entire workflow for how to enable 3rd parties into storage buckets but that's the general idea of how I thought it should work.

              Another thing to consider is the ability to use fine grained permissions within storage buckets. There's something to consider for the case where users would post files to closed off portions of the forum and whether or not those files should be in a publicly available storage bucket or not.

              Yea wrt spending limits, it definitely is a complicated subject and DDoS attacks could become very expensive if this is not taken into account for sure. But if we're hosting content on our own GCP project at least TY doesn't need to care about this.
              It would still be helpful to configure some sort of threshold that was enforced by TY both at the clientside (before uploading, should be possible with the File class in Javascript) and potentially serverside - if directing through the TY server (I think there would be a case for using JS to upload directly to the Storage bucket, then TY would never see this traffic).

              1. Hi again @chrscheuer

                How were you thinking to do the implementation?
                we should be able to somehow create a storage bucket in our GCP account [...] Ty as a member

                This sounds good to me. I'll look into this now this week. Probably I'll add a default cloud storage, + CDN with spending limits, for everyone who uses Talkayrd's hosting — and then you can override this, by uploading your own Cloud Storage credentials for your Talkayrd user (and, later on, configure your own CDN too should you want to). I had a quick look at how GCP & identity and access management works, and seems to me all this will work fine.

                fine grained permissions within storage buckets

                Yes that sounds like a good feature to have, ... and, personally I won't have time to look into this, not the nearest half a year at least. — Uploaded files have SHA256 hash names, so they're essentially hidden, unless one has a direct link.

                (I noticed that CDN:s also seem to support access permissions per URL, so maybe fine grained permissions can work also when a CDN is connected.)

                configure some sort of threshold that was enforced by TY both at the clientside (before uploading, should be possible with the File class in Javascript) and potentially serverside - if directing through the TY server

                Having a look at the files client side, is a good idea. Thanks for linking to Web/API/File/size. And server side too of course. Also, maybe one would want to specify if and how much the files should be compressed — I'd think in most cases, a 10 MB macOS Retina screenshot, is "overkill"? and maybe such large images should be downsize / lossy-compresed to 1 or 2 MB? (there're tools for lossy-compressing PNG images too).

                Maybe won't implement all this in the initial version — I'll start with just connecting to Cloud Storage & CDN.

                1. C
                  Christian Scheuer @chrscheuer
                    2020-01-06 11:27:36.987Zreplies toKajMagnus:

                    All of this sounds great!
                    And yea completely agree on the hashed file names/URLs being enough "security by obfuscation" for most purposes (including all of ours).
                    And yes auto-downsizing images is a good feature to put in the backlog for some time in the future :)

                    1. Sorry this got a bit delayed. (Code reviewing some code took days longer than what I expected. B.t.w. if you type a reply — you'll notice that now there's a live preview of your response, so you can see exactly how it'll look, in the page. And maybe the preview in the editor itself hereafter in most cases isn't needed.)

                      I started looking into this now this morning ... & this week instead of last.

                      1. C
                        Christian Scheuer @chrscheuer
                          2020-01-14 05:48:16.226Zreplies toKajMagnus:

                          Thanks for the update :) IMO it's a little confusing right now to see your text 3 places, 2 I think is the maximum of what makes sense to look at, so while the experiment is great in itself my personal opinion is that it makes the editing experience slightly worse right now.
                          Perhaps the eventual best editing experience is something we could look at - and talk about soon - while also considering being able to plug in custom made editor components.

                          1. KajMagnus @KajMagnus2020-01-22 05:07:07.289Z2020-01-22 06:23:51.643Zreplies tochrscheuer:

                            This is getting even more delayed, sorry. (Last week, backups suddenly stopped working: gzip started causing "soft lockup"s and kernel panic and the Linux server shut itself down, until I added nice -n19 + I finished some unimplemented backup script things. And now, this week, I've been fixing some unfinished things related to migrating away from Talkyard .net to a self hosted server.)

                            best editing experience is something we could look at - and talk about soon - while also considering being able to plug in custom made editor components

                            That would be interesting, yes. — For now, probably I'll hide the in-editor preview, and keep only the in-page live preview. (Except for when one posts a new topic and there is not yet any page.) The thing is, people in some cases reply to the wrong post, so I'm thinking it'd be good if they can see precisely where their reply will appear, e.g. in-reply-to which other reply. (B.t.w. I've seen this happening (people replying to the wrong comment) relatively often with Facebook's semi threaded comments too.)

                            1. C
                              Christian Scheuer @chrscheuer
                                2020-01-22 05:37:56.334Zreplies toKajMagnus:

                                Gotcha.. Yea you're right replies often end up in the wrong places. Will it be possible to roll back the design change though if it doesn't work? I see lots of flashing of the entire page with the loading bar etc. when editing whenever I have a slight pause -- so until the design change is more complete I feel like it still has the potential of making the editing experience worse. Perhaps you can make the change opt-in to begin with, ie. behind a feature flag until any bugs and design issues can be fixed?
                                Ways that could improve the experience:

                                • Never show a loading progress bar while I'm editing. It's very very annoying and almost driving me crazy just as I'm typing this response.
                                • As a first step, just show a placeholder (eg. the text "Your reply to ... will appear here"), not the entire text of the reply up in the thread - and keep the current preview system. It makes sense to view your markdown / preview side by side so you can verify your formatting works and not up by the thread. It's annoying to have to look up there to see how it looks.
                                • As a general rule, don't make large design changes without some sort of beta test, opt-in or ability to opt-out.
                                1. C
                                  Christian Scheuer @chrscheuer
                                    2020-01-22 05:40:22.161Zreplies toKajMagnus:

                                    And no worries with the filesize limitations.
                                    I just really hate the design change with the previews. It's very distracting from the experience of typing the answers, which is something I often spend hours on every day.
                                    I understand the problem you're trying to solve and I agree that that problem is important to solve, but IMO the current proposed solution is not an improvement but a quite serious degradation of the UX.

                                    1. Ok. I find it interesting that we have different opinions-about / feelings-for the inline preview.

                                      Then I'd like to add a feature toggle for this, before releasing the new version, so you can disable it (for your whole forum).

                                      I suppose it's good with different ways to tweak the UX, depending on one's forum's audience and how tech savvy they are.

                                      1. C
                                        Christian Scheuer @chrscheuer
                                          2020-01-27 02:45:47.019Zreplies toKajMagnus:

                                          I'm not ruling it out permanently. It may at one point be better than the current UX, but I'm surprised you're not seeing that it isn't ready for prime time yet. My screen is constantly flickering whenever I pause my typing, sometimes putting up a loading graphic for seconds so I can't continue typing until it's done. It's incredibly distracting and definitely not a finished feature. If you updated the server for everybody without a switch to turn it off it would be enough for us to seriously start considering hosting our own fork of TY.
                                          All I'm saying is that for text editing systems, the user experience is an absolutely central element and any change to it should be A/B tested and thoroughly vetted until you change it for end users. This doesn't feel tested to me.

                                          1. Thanks for the feedback! I'm not experiencing these things myself, still, I definitely neeed to fix this — I could reproduce the problems you're seeing:

                                            My screen is constantly flickering whenever I pause my typing, sometimes putting up a loading graphic for seconds so I can't continue typing until it's done

                                            by enabling Slow 3G network emulation in Dev Tools. Ok that's an enormously annoying editing experience.

                                            Actually, I think the things you mentioned, are related to 1) you now being far away from the server. (But I'm fairly close to it.) And 2) a bug that pops up the loading indicator, although that shouldn't happen in these cases. But not related to the in-page preview.

                                            I'll try to fix this before the next release. I feel pretty certain other people have the same issues (without telling me about it). ( https://news.ycombinator.com/item?id=21427996"They Might Never Tell You It’s Broken").

                                            1. C
                                              Christian Scheuer @chrscheuer
                                                2020-01-27 04:30:50.217Zreplies toKajMagnus:

                                                Great you were able to repro it. Yea that's a common issue (edit: people not reporting issues) which is why having a great beta team/process is a critical part of long term success with software IMO.
                                                Loading popup errors in strange scenarios is one of the most common issues I have with React myself as well, and often it can be quite difficult to repro why it happens for some customers but not for one-self. I hope with their new work with async rendering stuff like that should be easier to fix in the future.

                                                1. (I'd just like to menion that I've fixed this problem, with the flickering preview & loading-graphics, in my work-in-progress GitHub branch. Probably I can deploy the fix the first days next week.

                                                  often it can be quite difficult to repro why it happens

                                                  In my case, now, in Dev and Test mode, if a hostname includes "slow-3g" the server adds 2 seconds latency to each request to that hostname. That made this particular problem reproductible in the auto tests too)

                                                  1. C
                                                    Christian Scheuer @chrscheuer
                                                      2020-01-29 21:12:00.989Zreplies toKajMagnus:

                                                      Cool :)

                                                      1. @chrscheuer Now i've upgraded this Ty .io server so the flickering preview & editor and loading indicator problem, should be gone now. — Would you like to test?

                                                        B.t.w. there're UI config prefs that you can specify for the Everyone group,
                                                        which removes the in-page preview, and instead shows a split-view preview inside the editor, like before.

                                                        1. C
                                                          Christian Scheuer @chrscheuer
                                                            2020-02-04 16:51:29.222Zreplies toKajMagnus:

                                                            Testing testing... 1 2 3
                                                            Yea this seems to have fixed the flicker :)

                                                            1. KajMagnus @KajMagnus2020-02-05 06:51:47.280Z2020-02-05 07:04:41.372Zreplies tochrscheuer:

                                                              Ok, I just upgraded the server. — If you have feedback / want to change the UI prefs, maybe you'd like to open another topic about that?
                                                              (I think I slightly derailed this topic about file sizes, by starting talking about the editor & UI :- ))

                                                              1. C
                                                                Christian Scheuer @chrscheuer
                                                                  2020-02-05 18:20:31.180Zreplies toKajMagnus:

                                                                  Wait you changed the UI without an option for us to opt out? I thought that was the plan. Ugh...

                                                                  1. You can opt out by going here: /-/groups/everyone/preferences/ui and copy-pasting this into the text field:

                                                                    { "inp": 2 }
                                                                    

                                                                    There's going to be user friendly buttons and dropdowns for this later. ("inp" means "In-Page Preview".)
                                                                    (Don't forget to reload the page afterwards.)