No internet connection
  1. Home
  2. Ideas

API request: Extend page creation to private threads + Notification API

By Christian Scheuer @chrscheuer
    2020-02-15 20:20:08.927Z

    Hi @KajMagnus

    We are strongly considering extending SoundFlow with some new features related to sharing of content on the platform.
    Today users can already share content publicly - this is when a forum category gets automatically created for the content they published.

    We now want to add a feature where a user can share some content directly to another user, privately.
    The ideal way to do this would be via the forum's private threads feature.

    So this feature request is to add the ability for the upsertSimple API to also support creating private threads on behalf of users. I'm not sure if it makes sense to have multiple threads privately between users, or if that's even a thing. So either it should support creating new private threads, or just appending to the existing private thread (or to a specific private thread that we control).

    Also - we would like to make this sharing even more socially integrated into SoundFlow, so that even if people don't have the Forum page open, they could still get notifications from the forum (for the private thread sharing, but also just for normal forum usage).
    Ideally we could open a websocket connection or another form of real-time connection where the user would get updates in real time. Then we could display this as push messages on the user's screen even if they don't have our app open (and it's just running in the background). This would make Slack-like functionality even more available for enterprise users.

    Edit:
    Real-time polling/websockets might be a version 2 of the feature. We could start by just having a regular API to fetch the last X number of notifications.
    They should each have links to where on the site clicking on the notification should take us.

    • 26 replies

    There are 26 replies. Estimated reading time: 21 minutes

    1. Ok, nice idea & use case for the chat & API. I'd like to implement this. (What's your time frame for adding this new feature to SoundFlow?)

      I'm not sure if it makes sense to have multiple threads privately between users, or if that's even a thing

      Well, yes it is a thing, and it doesn't make that much sense — makes me confused; I'd rather see the name of the other person in the chat or the private-message-tread.

      I think it's better with just one single chat, between two users, and appending to it. — But if you look at the JSON below, you'll notice that you can choose yourself how it'll work?

      B.t.w. an old post about appending to existing topics: https://www.talkyard.io/-288#post-7
      — we can try to make this new append-post API, fullfill both this private chat use case,
      and that other old use case?

      Maybe something like this: ?

      POST /-/v0/upsert-simple
      
      {
        pages: [{
          extId: ...,
          pageType: PageType.PrivateChat,
          pageMembers: [
            "extid:Your_User_Pelles_External_Id_1234",
            "extid:Your_User_Majas_External_Id_5678",
          ],
          categoryRef: "extid:private-chats",
          title: "Pelle's and Maja's chat",
          bodyHtmlUnsafe: "<p>Private chat, Pelle and Maja</p>",
        }],
        posts: [{
          extid: ...,
          pageRef:  (the ext id of the page above),
          authorRef: "extid:Your_User_Pelles_External_Id_1234",
          postHtmlUnsafe: "<p>Hello I am here are you there or here or elsewhere?</p>",
        }],
      }
      

      Note that the Post doesn't specify any parent-post-number. Then, Talkyard can just append the post at the bottom of the page — and the others (always just one person in SoundFlow's case?) can then get notified.

      even if people don't have the Forum page open, they could still get notifications from the forum [...] might be a version 2 of the feature
      start by just having a regular API to fetch the last X number of notifications

      Yes, maybe initially you could poll that API once per minute or half an hour? Would it be your server that does any such polling? Then maybe it could fetch recent events for all your online users, in just one API call?

      1. CChristian Scheuer @chrscheuer
          2020-02-17 09:20:38.784Z

          Nice, the API proposal / JSON looks great.

          Yes, maybe initially you could poll that API once per minute or half an hour? Would it be your server that does any such polling? Then maybe it could fetch recent events for all your online users, in just one API call?

          Yea we could do something like that. Eventually maybe setting up a web trigger would be best (so TY is the one sending calls outbound whenever something happens). This is how we sync with our other systems as well. Then TY is also in control of traffic wrt batching updates together if it prefers. For it to really work well I think users would expect notifications to come in fairly quickly. I get almost instant notifications on my phone already via email when a user responds to a forum request, so this would only have real value for users if was as quick.
          But anyway, the private thread creation is a much more important first step since it will allow us to implement sharing - so I'm happy to park the discussion about notifications until we've got that first one covered, since I suspect notifications can be quite a huge topic overall - I just wanted to give you the full context of how I envisioned the feature :)

          1. CChristian Scheuer @chrscheuer
              2020-02-21 23:22:07.274Z

              One thing I forgot about this proposal was that there's a 3rd API request that should go along with the first two, and this one is more important than the notification.
              It's an API to be used for the querying of active forum usernames. Since forum usernames are already public, and since users have a way to rename their usernames, the forum username would be a good mechanism for people choosing whom to share a piece of content with. So, instead of exposing their real full names as we have it registered in our system, we could use a system like the "mention" feature in the TY editor to let users pick from public forum usernames which person they want to share to.
              Maybe we could just use the same endpoint as TY is currently using, but I suspect it might be better to have as a separate "public" API call - one that doesn't require admin-authentication (as that would just cause an unnecessary route through our servers).

              1. CChristian Scheuer @chrscheuer
                  2020-02-21 23:24:18.182Z

                  I also forgot to mention - we are in beta right now of SoundFlow 3.5, so this won't make it into 3.5, but it definitely would be nice to include together with our other "sharing" features in 3.6 :)
                  We try to push out minor version updates every 4-6 weeks. Do you think that would be a realistic timeframe?

                  1. I think 4-6 weeks sounds realistic yes. So, there'd be 3 API requests: upsert post, poll for notifications, and list ... active? usernames. What does active mean — accounts that have visited the forum the last week or month maybe?

                    I cannot start with this this week, but the week after I would think (Mars 2).

                    1. CChristian Scheuer @chrscheuer
                        2020-02-23 21:47:22.763Z

                        Awesome! Thank you so much for doing this :)

                        Right, to sum up, this would be the requests (in order of importance):

                        1. Private thread extension of the upsertSimple API, as we discussed above.
                        2. Unauthorized/public querying of usernames based on a search query.
                        3. Notifications API, either polling or real realtime websocket based connection.

                        For #2:
                        I don't think we can filter on "active" users since there might be users who weren't (particularly) active on the forum yet that should still be valid targets for sharing.
                        Instead maybe it would make more sense to let a user type the first 2 characters of the username before it searches. The API could be restricted in number of users it returns, it just needs to display the first 5-10 users matching the query (for this purpose).

              2. Progress
                with doing this idea
              3. Hi @chrscheuer I hope to start with this at the end of this week — currently reviewing the drafts fixes I wrote some weeks ago.

                1. Now started with this

                  1. @KajMagnus marked this topic as Started 2020-03-11 08:02:10.202Z.
                  2. C
                    Christian Scheuer @chrscheuer
                      2020-03-11 08:28:26.506Zreplies toKajMagnus:

                      Amazing :) Quick question: I don't remember if we talked about this, but would this change include the ability for us to also append posts to existing public threads? (In addition to private threads).
                      Asking this since we could use that feature for when users want to add feedback to bugs (instead of having to create a new thread every time which we have to with the current upsert API for pages).

                      1. would this change include the ability for us to also append posts to existing public threads?

                        I think so yes. Maybe this'd almost work by default and I just need to write auto test cases, I'll have a look.

                        Should these posts 1) get appended at the bottom of the page,
                        or 2) become replies to the Original Post sorted by Like votes (best-first)?
                        (or 3) appear a replies to some other reply?)

                        1. C
                          Christian Scheuer @chrscheuer
                            2020-03-11 17:45:48.860Zreplies toKajMagnus:

                            It would probably be best if they got appended to the bottom of the page or became replies to the original post.

                            1. Ok. I think you'll be able to choose, by specifying Post Type.

                              I'm thinking when listing users by username prefix, then you want the users extId or ssoId included in the response? So you know what ext id to specify in subsequent API requests. So I'll add JSON fields for that ( + Talkyard id, username, full name).

                              The list-users requests, they'll be sent directly by the SoundFlow clients, without any API secret or Talkyard session cookie? Or they'll be sent via the SoundFlow server(s)?

                              1. C
                                Christian Scheuer @chrscheuer
                                  2020-03-17 12:11:31.774Zreplies toKajMagnus:

                                  That sounds great.
                                  Ideally we could send the list-users requests directly from SoundFlow clients, as the roundtrip via our server would not add any value, only make it slower.
                                  Usernames are already public in the sense that anybody can use the tag feature on the forum. We would of course do some caching on the clients so that multiple identical requests from the same client wouldn't re-fetch from the server every time.
                                  Generally it should probably work pretty similarly to how you've designed the @ tag API now (not that I've looked at it, but just from how it seems to work, being fast, and just returning a limited number of results).

                                  1. C
                                    Christian Scheuer @chrscheuer
                                      2020-03-17 12:12:00.112Zreplies tochrscheuer:

                                      I'm using the word "tag" here where I should maybe be calling it "mention"

                                      1. (Ok, thanks for the explanation. Yes I'm actually reusing the @mention internal API endpoint, — just added another /-/v0/... endpoint, + added comments in the source code about being part of the API, so won't accidentally break the API)

                                        This is mostly done now — here's an end-to-end test you might want to have a look at; it shows how you can use the API:
                                        https://github.com/debiki/talkyard/blob/w-km7e/tests/e2e/specs/api-private-chat-two-pps-impl.test.ts

                                        — maybe you'll notice something I should change?

                                        Actually now I think you don't need the extid or ssoid, but only the username, when upserting a chat message. Because you can use refs like username:someones_username to tell Talkyard which people are talking.

                                        Here's how to use the list-users endpoint: https://github.com/debiki/talkyard/blob/375fbede4a521be2d5e3fb8f98e25dd5b45aec32/tests/e2e/utils/server.ts#L491
                                        Here is the /-/v0/list-users API response: https://github.com/debiki/talkyard/blob/w-km7e/tests/e2e/pub-api.ts )

                                        (Some of the links above will break in some weeks, when I reuse the w-km7e work-in-progress branch for something else)

                                        1. C
                                          Christian Scheuer @chrscheuer
                                            2020-03-24 09:51:00.196Zreplies toKajMagnus:

                                            That sounds amazing Magnus :) These are not live yet anywhere right?
                                            Love the ref in username:blabla form, the response looks right too.
                                            I forgot what the difference between ssoid and extid is.. Can you remind me?
                                            The end to end test looks great as well :)

                                            Great work!!!

                                            1. C
                                              Christian Scheuer @chrscheuer
                                                2020-03-24 09:53:20.531Zreplies tochrscheuer:

                                                This would actually allow us to post messages to existing threads as well right? With threads I mean pages that are not private chat type, but normal "questions" etc.

                                                1. Yes this works for posting to existing threads too. There's another e2e test that API upserts replies to Question / Idea / Problem pages; here's the reply that gets upserted:

                                                  const michaelsReplyToMajasApiTopic = {
                                                    extId: 'michaelsReplyToMajasApiTopic extId',
                                                    postType: c.TestPostType.Normal,                 <—— instead of ChatMessage
                                                    parentNr: c.BodyNr,
                                                    pageRef: `extid:${majasApiTopicUpsData.extId}`,  <—— already existing Question/Idea/etc... topic
                                                    authorRef: `username:michael`,
                                                    body: 'michaelsReplyToMajasApiTopic hello Maja',
                                                  };
                                                  
                                                  https://github.com/debiki/talkyard/blob/375fbede4a521be2d5e3fb8f98e25dd5b45aec32/tests/e2e/specs/api-upsert-posts.2browsers.test.ts#L59

                                                  These are not live yet anywhere right?

                                                  That's right, this is not live yet. First I'll need to code review everything (will take a day at least).

                                                  the difference between ssoid and extid

                                                  In your case, for users, you have only ssoid, never mind extid :- )   (and sso-id it's the id you specify when calling sso-upsert-user-generate-login-secret).

                                                  And pages, posts, etc don't have any ssoid, they have only extids.

                                                  ( ExtId for users: One use case: Let's say an organization imports external users into Talkyard — then Talkyard can remember their ids in the extId field. Later, maybe these users get different Single Sign-On ids via a SSO integration (that uses different ids than from the import source) — and since the ssoid is kept in a different field, this won't overwrite the extId. So if the organization later on re-imports the external users (maybe because even more external users got created somehow and they want to re-import all of them), then the external users imported the very first time, will not get duplicated (because they remember their external ids from the original import, and know they're the same users). )

                                                  1. C
                                                    Christian Scheuer @chrscheuer
                                                      2020-03-24 10:49:09.547Zreplies toKajMagnus:

                                                      Brilliant - and thanks for the reminder about extid & ssoid - I remember now.
                                                      Can't wait to play with all this. We launched 3.5 yesterday, so now is the perfect timing to start the work on forum sharing :)

                                                      1. C
                                                        Christian Scheuer @chrscheuer
                                                          2020-03-27 18:34:20.194Zreplies tochrscheuer:

                                                          Please let me know when you bring this live so I can start playing :)

                                                          1. Yes I'll let you know. I'm done with the first code review — but not the next step: The code review of the code review.
                                                            (I'll do tomorrow ... plus some more minor things,
                                                            I think the new version will be out on Monday or Tuesday next week.)

                                                            B.t.w. someone submitted Talkayrd to HackerNews: https://news.ycombinator.com/item?id=22702311 (in case you're curious).
                                                            (After I mentioned it a topic about remote work tools.)

                                                            1. Hi @chrscheuer, Now this is live. I renamed the list users endpoint to /-/v0/list-users and it returns { users: ... }. (There'll be /-/v0/list-groups too, some time later.)

                                                              ( B.t.w., just FYI: I think some time later, this /-/v0/upsert-simple endpoint and the JSON format will need to change to something like /-/v0/do-action-batch and JSON like { upsertCategory: ..., editPost: ..., deletePage: ..., addChatMessage: ... } — it'll be more a "batch actions" API, than an upsert-things API. Because the endpoint name and JSON format we use right now, doesn't work so well for also deleting and editing things, plus, the notification generation code got a bit complicated. — But this can wait. )

                                                              1. C
                                                                Christian Scheuer @chrscheuer
                                                                  2020-03-31 09:03:21.108Zreplies toKajMagnus:

                                                                  Great to hear it's live!
                                                                  Yea makes sense to rename it a bit later. I would recommend that you add a new endpoint first, so that we can migrate to it and test it, before you delete the old endpoint. When that time comes.
                                                                  By the way, do the APIs log errors by any chance? I'm still seeing quite a few users who can't use the page creation (but since it fails, I also don't get the error, I just see that they have to manually send me the files).

                                                                  1. add a new endpoint first

                                                                    Yes

                                                                    APIs log errors by any chance?

                                                                    I had a quick look in the logs, didn't see anything that looks as if it's from the API or SoundFlow. I'm thinking it'd be good if I reviewed how error logging works and make log messages from your site available to you, so you can also see the logs.

                                                                    1. C
                                                                      Christian Scheuer @chrscheuer
                                                                        2020-03-31 12:28:32.875Zreplies toKajMagnus:

                                                                        Thanks, yea it would be wonderful to have some sort of log viewer for API errors :)
                                                                        I'll also see how we can better catch them on our end.