No internet connection
  1. Home
  2. Issues

Slow server @talkyard

By Christian Scheuer @chrscheuer
    2018-09-08 15:27:55.629Z

    It's getting increasingly frustrating that we're being met with "Server is slow right now" messages on a regular basis. Is there any fix for this coming up or will this continue to happen as often as it does now?
    Customers are complaining about it, it's hindering us from effectively communicating with them and frankly it makes the experience really bad for everyone involved. We're trying to make Talkyard the primary communication channel to/from customers because it allows us to gather all of our information in one place. But that also means downtime is really really bad.

    • 18 replies
    1. C
      Christian Scheuer @chrscheuer
        2018-09-08 15:29:03.011Z

        I'm also getting my replies deleted. It seems to happen when the server is slow to respond to an "at" mention and when it finally replies my entire response is deleted (because I continued to type after the popup)

        1. C
          In reply tochrscheuer:
          Christian Scheuer @chrscheuer
            2018-09-08 15:34:54.080Z

            It's also frustrating that you can be in the middle of typing a long reply and then the UI turns into grey mode making me unable to continue the work.

            1. C
              In reply tochrscheuer:
              Christian Scheuer @chrscheuer
                2018-09-08 17:47:28.495Z

                Fwiw this has been going on most of today...

                1. C
                  In reply tochrscheuer:
                  Christian Scheuer @chrscheuer
                    2018-09-08 23:49:57.464Z2018-09-09 01:38:32.070Z

                    This is now 8+ hours with sustained 50% downtime. It's hit or miss if we are connected. Are anyone monitoring this at all? @KajMagnus?

                    1. C
                      In reply tochrscheuer:
                      Christian Scheuer @chrscheuer
                        2018-09-08 23:57:34.795Z

                        Fwiw drafts feature seems to really be messing up everything when the server is slow. My text input field randomly resets to earlier versions of drafts of other posts while in the middle of typing. Please please please roll back these changes, it is completely devastating to have your post deleted or overwritten in the middle of typing an important response to someone.

                        1. C
                          In reply tochrscheuer:
                          Christian Scheuer @chrscheuer
                            2018-09-09 01:54:47.819Z

                            I've created separate threads for the various issues we're experiencing, hopefully in a more constructive tone. This thread I'm leaving here to discuss the (still existing) downtime and what can be done to mitigate it and make sure it doesn't happen in the future.

                            1. In reply tochrscheuer:

                              I think that with drafts disabled (as mentioned in another topic) this will happen no longer.

                              And I should try to reproduce this so I can verify it won't happen, some time later when enabling drafts again. (Maybe drafts could be a per site feature flag, for a start, so can be disabled easily if needed)

                              Thanks for posting separate topics about the other things.

                              1. CChristian Scheuer @chrscheuer
                                  2018-09-09 04:51:52.496Z

                                  Cool sounds good. The main forum here at talkyard (the one we're writing in now) didn't seem to be affected by the slowness, it was only happening on our own. But that might have been luck of course - we did have much more traffic to our own site while the issues were there.

                                • In reply tochrscheuer:

                                  Ok so, as per the chat, apparently the problem was too-many-open-connections. Seems this is a Chrome restriction, not the Nginx server config: Chrome allows at most sth like 10 open connections against a single host (whilst the Nginx server is configured to allow 60). And since each tab starts its own long polling request, and might do other requests in parallel (e.g. asking for the forum topic list) after 8 - 9 - 10 tabs, additional requests, block. Until one of the earlier long polling requests finishes, maybe after 10 seconds, maybe after 20 or 30. Which gives the impression that the server is slow.

                                  I believe I can fix this, by adding a service worker, and have it send just one long polling requests at a time, on behalf of all open tabs. And then it'll broadcast the long polling replies, to those tabs.

                                  1. CChristian Scheuer @chrscheuer
                                      2018-09-20 11:53:35.796Z

                                      This sounds like exactly what was happening. Nice research.
                                      I'm curious if any of the standard websocket libraries out there would have this built in already so you wouldn't have to invent your own solution?

                                    • Progress
                                      with handling this problem
                                    • @KajMagnus marked this topic as Planned 2018-09-20 11:22:37.615Z.
                                    • @KajMagnus marked this topic as Started 2018-09-20 11:22:39.879Z.
                                    • Adding a service worker that sends long polling requests, one at a time, on behalf of all open tabs, does fix this issue. However the changes I had to make, feels a bit risky (lots of changes, and message passing between browser tabs and the service worker), so this will have to wait for a while.

                                      When continuing with this, I'm thinking I'll also look into adding a custom PWA (progressive web app) manifest. ... The first steps towards creating a PWA mobile app. (Later steps: having the service worker offline-cache page JSON content.)

                                      1. @KajMagnus marked this topic as Done 2018-11-09 15:42:08.186Z.
                                      2. @KajMagnus marked this topic as Started 2019-05-02 08:33:04.948Z.
                                      3. Now I've stated with the "real" fix — namely a single service worker that keeps just one connection open, to the server (instead of one per browser tab).

                                        I think I will have deployed a version with a fix, to this "test" community (i.e. Talkyard .io), in about a week. Then I'll see how this works, for about a week, and if all fine, I'll deploy to "everyone", with a feature switch so can be toggled off.

                                        1. Just deployed a new version, which does long polling via a single service worker (instead of per tab). Now I could open 30+ tabs, without any issues. Will keep this new code running on this Talkyard .io server only, for about a week, to see how this works.

                                          1. @chrscheuer a few hours ago I enabled the service worker fix, on your forum. If you want to, you can open 20+ browser tabs and see if / verify-that "The server is slow" error is now gone (whilst being logged in).

                                            1. C
                                              Christian Scheuer @chrscheuer
                                                2019-05-26 11:09:25.897Zreplies toKajMagnus:

                                                FANTASTIC!!!!! <3 <3 <3 THIS WORKS. I'm SO happy :)

                                                1. Ok :- )   (be careful so you won't crash your browser if hereafter 100+ tabs open   o. O )

                                                  1. @KajMagnus marked this topic as Done 2019-05-26 11:19:37.550Z.