No internet connection

API request: Extend page creation to private threads + Notification API

By Christian Scheuer @chrscheuer2020-02-15 20:20:08.927Z

Hi @KajMagnus

We are strongly considering extending SoundFlow with some new features related to sharing of content on the platform.
Today users can already share content publicly - this is when a forum category gets automatically created for the content they published.

We now want to add a feature where a user can share some content directly to another user, privately.
The ideal way to do this would be via the forum's private threads feature.

So this feature request is to add the ability for the upsertSimple API to also support creating private threads on behalf of users. I'm not sure if it makes sense to have multiple threads privately between users, or if that's even a thing. So either it should support creating new private threads, or just appending to the existing private thread (or to a specific private thread that we control).

Also - we would like to make this sharing even more socially integrated into SoundFlow, so that even if people don't have the Forum page open, they could still get notifications from the forum (for the private thread sharing, but also just for normal forum usage).
Ideally we could open a websocket connection or another form of real-time connection where the user would get updates in real time. Then we could display this as push messages on the user's screen even if they don't have our app open (and it's just running in the background). This would make Slack-like functionality even more available for enterprise users.

Edit:
Real-time polling/websockets might be a version 2 of the feature. We could start by just having a regular API to fetch the last X number of notifications.
They should each have links to where on the site clicking on the notification should take us.

  • 27 replies

There are 27 replies. Estimated reading time: 21 minutes

  1. KajMagnus @KajMagnus2020-02-17 09:00:51.131Z

    Ok, nice idea & use case for the chat & API. I'd like to implement this. (What's your time frame for adding this new feature to SoundFlow?)

    I'm not sure if it makes sense to have multiple threads privately between users, or if that's even a thing

    Well, yes it is a thing, and it doesn't make that much sense — makes me confused; I'd rather see the name of the other person in the chat or the private-message-tread.

    I think it's better with just one single chat, between two users, and appending to it. — But if you look at the JSON below, you'll notice that you can choose yourself how it'll work?

    B.t.w. an old post about appending to existing topics: https://www.talkyard.io/-288#post-7
    — we can try to make this new append-post API, fullfill both this private chat use case,
    and that other old use case?

    Maybe something like this: ?

    POST /-/v0/upsert-simple
    
    {
      pages: [{
        extId: ...,
        pageType: PageType.PrivateChat,
        pageMembers: [
          "extid:Your_User_Pelles_External_Id_1234",
          "extid:Your_User_Majas_External_Id_5678",
        ],
        categoryRef: "extid:private-chats",
        title: "Pelle's and Maja's chat",
        bodyHtmlUnsafe: "<p>Private chat, Pelle and Maja</p>",
      }],
      posts: [{
        extid: ...,
        pageRef:  (the ext id of the page above),
        authorRef: "extid:Your_User_Pelles_External_Id_1234",
        postHtmlUnsafe: "<p>Hello I am here are you there or here or elsewhere?</p>",
      }],
    }
    

    Note that the Post doesn't specify any parent-post-number. Then, Talkyard can just append the post at the bottom of the page — and the others (always just one person in SoundFlow's case?) can then get notified.

    even if people don't have the Forum page open, they could still get notifications from the forum [...] might be a version 2 of the feature
    start by just having a regular API to fetch the last X number of notifications

    Yes, maybe initially you could poll that API once per minute or half an hour? Would it be your server that does any such polling? Then maybe it could fetch recent events for all your online users, in just one API call?

    1. CChristian Scheuer @chrscheuer2020-02-17 09:20:38.784Z

      Nice, the API proposal / JSON looks great.

      Yes, maybe initially you could poll that API once per minute or half an hour? Would it be your server that does any such polling? Then maybe it could fetch recent events for all your online users, in just one API call?

      Yea we could do something like that. Eventually maybe setting up a web trigger would be best (so TY is the one sending calls outbound whenever something happens). This is how we sync with our other systems as well. Then TY is also in control of traffic wrt batching updates together if it prefers. For it to really work well I think users would expect notifications to come in fairly quickly. I get almost instant notifications on my phone already via email when a user responds to a forum request, so this would only have real value for users if was as quick.
      But anyway, the private thread creation is a much more important first step since it will allow us to implement sharing - so I'm happy to park the discussion about notifications until we've got that first one covered, since I suspect notifications can be quite a huge topic overall - I just wanted to give you the full context of how I envisioned the feature :)

      1. CChristian Scheuer @chrscheuer2020-02-21 23:22:07.274Z

        One thing I forgot about this proposal was that there's a 3rd API request that should go along with the first two, and this one is more important than the notification.
        It's an API to be used for the querying of active forum usernames. Since forum usernames are already public, and since users have a way to rename their usernames, the forum username would be a good mechanism for people choosing whom to share a piece of content with. So, instead of exposing their real full names as we have it registered in our system, we could use a system like the "mention" feature in the TY editor to let users pick from public forum usernames which person they want to share to.
        Maybe we could just use the same endpoint as TY is currently using, but I suspect it might be better to have as a separate "public" API call - one that doesn't require admin-authentication (as that would just cause an unnecessary route through our servers).

        1. CChristian Scheuer @chrscheuer2020-02-21 23:24:18.182Z

          I also forgot to mention - we are in beta right now of SoundFlow 3.5, so this won't make it into 3.5, but it definitely would be nice to include together with our other "sharing" features in 3.6 :)
          We try to push out minor version updates every 4-6 weeks. Do you think that would be a realistic timeframe?

          1. KajMagnus @KajMagnus2020-02-23 20:51:44.644Z

            I think 4-6 weeks sounds realistic yes. So, there'd be 3 API requests: upsert post, poll for notifications, and list ... active? usernames. What does active mean — accounts that have visited the forum the last week or month maybe?

            I cannot start with this this week, but the week after I would think (Mars 2).

            1. CChristian Scheuer @chrscheuer2020-02-23 21:47:22.763Z

              Awesome! Thank you so much for doing this :)

              Right, to sum up, this would be the requests (in order of importance):

              1. Private thread extension of the upsertSimple API, as we discussed above.
              2. Unauthorized/public querying of usernames based on a search query.
              3. Notifications API, either polling or real realtime websocket based connection.

              For #2:
              I don't think we can filter on "active" users since there might be users who weren't (particularly) active on the forum yet that should still be valid targets for sharing.
              Instead maybe it would make more sense to let a user type the first 2 characters of the username before it searches. The API could be restricted in number of users it returns, it just needs to display the first 5-10 users matching the query (for this purpose).

    2. Progress
      with doing this idea
    3. KajMagnus @KajMagnus2020-03-03 15:19:46.559Z

      Hi @chrscheuer I hope to start with this at the end of this week — currently reviewing the drafts fixes I wrote some weeks ago.

      1. KajMagnus @KajMagnus2020-03-11 08:02:03.290Z

        Now started with this

        1. @KajMagnus marked this topic as Started 2020-03-11 08:02:10.202Z.
        2. C
          Christian Scheuer @chrscheuer2020-03-11 08:28:26.506Zreplies toKajMagnus:

          Amazing :) Quick question: I don't remember if we talked about this, but would this change include the ability for us to also append posts to existing public threads? (In addition to private threads).
          Asking this since we could use that feature for when users want to add feedback to bugs (instead of having to create a new thread every time which we have to with the current upsert API for pages).

          1. KajMagnus @KajMagnus2020-03-11 11:29:30.905Zreplies tochrscheuer:

            would this change include the ability for us to also append posts to existing public threads?

            I think so yes. Maybe this'd almost work by default and I just need to write auto test cases, I'll have a look.

            Should these posts 1) get appended at the bottom of the page,
            or 2) become replies to the Original Post sorted by Like votes (best-first)?
            (or 3) appear a replies to some other reply?)

            1. C
              Christian Scheuer @chrscheuer2020-03-11 17:45:48.860Zreplies toKajMagnus:

              It would probably be best if they got appended to the bottom of the page or became replies to the original post.

              1. KajMagnus @KajMagnus2020-03-17 12:04:34.505Zreplies tochrscheuer:

                Ok. I think you'll be able to choose, by specifying Post Type.

                I'm thinking when listing users by username prefix, then you want the users extId or ssoId included in the response? So you know what ext id to specify in subsequent API requests. So I'll add JSON fields for that ( + Talkyard id, username, full name).

                The list-users requests, they'll be sent directly by the SoundFlow clients, without any API secret or Talkyard session cookie? Or they'll be sent via the SoundFlow server(s)?

                1. C
                  Christian Scheuer @chrscheuer2020-03-17 12:11:31.774Zreplies toKajMagnus:

                  That sounds great.
                  Ideally we could send the list-users requests directly from SoundFlow clients, as the roundtrip via our server would not add any value, only make it slower.
                  Usernames are already public in the sense that anybody can use the tag feature on the forum. We would of course do some caching on the clients so that multiple identical requests from the same client wouldn't re-fetch from the server every time.
                  Generally it should probably work pretty similarly to how you've designed the @ tag API now (not that I've looked at it, but just from how it seems to work, being fast, and just returning a limited number of results).

                  1. C
                    Christian Scheuer @chrscheuer2020-03-17 12:12:00.112Zreplies tochrscheuer:

                    I'm using the word "tag" here where I should maybe be calling it "mention"

                    1. KajMagnus @KajMagnus2020-03-24 09:34:30.109Zreplies tochrscheuer:

                      (Ok, thanks for the explanation. Yes I'm actually reusing the @mention internal API endpoint, — just added another /-/v0/... endpoint, + added comments in the source code about being part of the API, so won't accidentally break the API)

                      This is mostly done now — here's an end-to-end test you might want to have a look at; it shows how you can use the API:
                      https://github.com/debiki/talkyard/blob/w-km7e/tests/e2e/specs/api-private-chat-two-pps-impl.test.ts

                      — maybe you'll notice something I should change?

                      Actually now I think you don't need the extid or ssoid, but only the username, when upserting a chat message. Because you can use refs like username:someones_username to tell Talkyard which people are talking.

                      Here's how to use the list-users endpoint: https://github.com/debiki/talkyard/blob/375fbede4a521be2d5e3fb8f98e25dd5b45aec32/tests/e2e/utils/server.ts#L491
                      Here is the /-/v0/list-users API response: https://github.com/debiki/talkyard/blob/w-km7e/tests/e2e/pub-api.ts )

                      (Some of the links above will break in some weeks, when I reuse the w-km7e work-in-progress branch for something else)

                      1. C
                        Christian Scheuer @chrscheuer2020-03-24 09:51:00.196Zreplies toKajMagnus:

                        That sounds amazing Magnus :) These are not live yet anywhere right?
                        Love the ref in username:blabla form, the response looks right too.
                        I forgot what the difference between ssoid and extid is.. Can you remind me?
                        The end to end test looks great as well :)

                        Great work!!!

                        1. C
                          Christian Scheuer @chrscheuer2020-03-24 09:53:20.531Zreplies tochrscheuer:

                          This would actually allow us to post messages to existing threads as well right? With threads I mean pages that are not private chat type, but normal "questions" etc.

                          1. KajMagnus @KajMagnus2020-03-24 10:35:53.095Zreplies tochrscheuer:

                            Yes this works for posting to existing threads too. There's another e2e test that API upserts replies to Question / Idea / Problem pages; here's the reply that gets upserted:

                            const michaelsReplyToMajasApiTopic = {
                              extId: 'michaelsReplyToMajasApiTopic extId',
                              postType: c.TestPostType.Normal,                 <—— instead of ChatMessage
                              parentNr: c.BodyNr,
                              pageRef: `extid:${majasApiTopicUpsData.extId}`,  <—— already existing Question/Idea/etc... topic
                              authorRef: `username:michael`,
                              body: 'michaelsReplyToMajasApiTopic hello Maja',
                            };
                            
                            https://github.com/debiki/talkyard/blob/375fbede4a521be2d5e3fb8f98e25dd5b45aec32/tests/e2e/specs/api-upsert-posts.2browsers.test.ts#L59

                            These are not live yet anywhere right?

                            That's right, this is not live yet. First I'll need to code review everything (will take a day at least).

                            the difference between ssoid and extid

                            In your case, for users, you have only ssoid, never mind extid :- )   (and sso-id it's the id you specify when calling sso-upsert-user-generate-login-secret).

                            And pages, posts, etc don't have any ssoid, they have only extids.

                            ( ExtId for users: One use case: Let's say an organization imports external users into Talkyard — then Talkyard can remember their ids in the extId field. Later, maybe these users get different Single Sign-On ids via a SSO integration (that uses different ids than from the import source) — and since the ssoid is kept in a different field, this won't overwrite the extId. So if the organization later on re-imports the external users (maybe because even more external users got created somehow and they want to re-import all of them), then the external users imported the very first time, will not get duplicated (because they remember their external ids from the original import, and know they're the same users). )

                            1. C
                              Christian Scheuer @chrscheuer2020-03-24 10:49:09.547Zreplies toKajMagnus:

                              Brilliant - and thanks for the reminder about extid & ssoid - I remember now.
                              Can't wait to play with all this. We launched 3.5 yesterday, so now is the perfect timing to start the work on forum sharing :)

                              1. C
                                Christian Scheuer @chrscheuer2020-03-27 18:34:20.194Zreplies tochrscheuer:

                                Please let me know when you bring this live so I can start playing :)

                                1. KajMagnus @KajMagnus2020-03-27 20:15:37.352Zreplies tochrscheuer:

                                  Yes I'll let you know. I'm done with the first code review — but not the next step: The code review of the code review.
                                  (I'll do tomorrow ... plus some more minor things,
                                  I think the new version will be out on Monday or Tuesday next week.)

                                  B.t.w. someone submitted Talkayrd to HackerNews: https://news.ycombinator.com/item?id=22702311 (in case you're curious).
                                  (After I mentioned it a topic about remote work tools.)

                                  1. KajMagnus @KajMagnus2020-03-31 08:34:18.353Z

                                    Hi @chrscheuer, Now this is live. I renamed the list users endpoint to /-/v0/list-users and it returns { users: ... }. (There'll be /-/v0/list-groups too, some time later.)

                                    ( B.t.w., just FYI: I think some time later, this /-/v0/upsert-simple endpoint and the JSON format will need to change to something like /-/v0/do-action-batch and JSON like { upsertCategory: ..., editPost: ..., deletePage: ..., addChatMessage: ... } — it'll be more a "batch actions" API, than an upsert-things API. Because the endpoint name and JSON format we use right now, doesn't work so well for also deleting and editing things, plus, the notification generation code got a bit complicated. — But this can wait. )

                                    1. C
                                      Christian Scheuer @chrscheuer2020-03-31 09:03:21.108Zreplies toKajMagnus:

                                      Great to hear it's live!
                                      Yea makes sense to rename it a bit later. I would recommend that you add a new endpoint first, so that we can migrate to it and test it, before you delete the old endpoint. When that time comes.
                                      By the way, do the APIs log errors by any chance? I'm still seeing quite a few users who can't use the page creation (but since it fails, I also don't get the error, I just see that they have to manually send me the files).

                                      1. KajMagnus @KajMagnus2020-03-31 09:20:25.459Zreplies tochrscheuer:

                                        add a new endpoint first

                                        Yes

                                        APIs log errors by any chance?

                                        I had a quick look in the logs, didn't see anything that looks as if it's from the API or SoundFlow. I'm thinking it'd be good if I reviewed how error logging works and make log messages from your site available to you, so you can also see the logs.

                                        1. C
                                          Christian Scheuer @chrscheuer2020-03-31 12:28:32.875Zreplies toKajMagnus:

                                          Thanks, yea it would be wonderful to have some sort of log viewer for API errors :)
                                          I'll also see how we can better catch them on our end.