No internet connection
  1. Home
  2. Ideas

Webhooks – last piece of the puzzle for full service integration

By Christian Scheuer @chrscheuer
    2021-10-04 21:46:22.316Z

    Hi @KajMagnus,

    I just wanted to sync up with you to see if getting webhooks support for certain events would be possible in the (near) future?

    Right now we can only download the entire json structure of the forum to allow support posts made in the forum to sync with our other services (ie. Linear, Intercom, Firebase, etc.)

    But because we have to download the entire json structure, this is very slow and can only be scheduled a few times per day at the max - otherwise I feel like we'd risk overloading your server by asking it to fetch everything – all the time.
    Beyond that, because of this delay syncing can't be made really effective - as our other systems would always be out of sync with the real data in the forum, because it would easily be up to 24 hours late in arriving.

    Because of this, we are currently not integrating the forum as deeply to our services as we'd like.

    Since your last great changes, we can now create posts in the forum impersonating a user. That's great :) So the last piece of the puzzle is the ability to react to things happening in Talkyard in real-time via webhooks.

    For info, Stripe has an excellent way of handling webhooks.

    All well run webhook integrations I've seen do it like this:

    • Have a place in the admin UI to add webhook endpoints. You typically just specify a URL, and sometimes you can opt-in to the types of events you want the webhook to receive. This is useful if the system can send many different event types and some webhooks may want to just subscribe to a few event types.
    • Record an event object in the database whenever something insteresting occurs.
    • Try sending this event to all registered webhooks matching that event and record if the events were processed correctly (webhook responded 200 OK within a timeout period)
    • If any of the webhooks failed for the event, retry sending it at a later time, typically with exponential rolloff.
    • Present a way to report in an admin UI which events were sent successfully and which had errors etc. Perhaps even the ability to show details about each event.
    • Event objects/payloads are typically sent via JSON, and often the event would contain info like:
      • Event type
      • Time
      • Idempotency Key (see Stripe docs for this – if for example the event is the result of an API call where you specified an idempotency key, this would be returned in the event created by the result of that API call, to let the external site know that the effect of the async API call was now received). Optional
      • The entity the event is about. For example, the customer, or the post, or the page, etc.
      • Other potential metadata

    The event types we'd be interested in would be:

    • Page gets created or updated
    • Post gets created or updated
    • Status of a page changes (closed, reopened)

    Obviously such a webhook implementation can be made quite complex. But I wonder if we could get something off the ground if we skipped some of these more thorough implementation details for now and opted for a simpler approach to get started.

    One that just as a bare minimum:

    • Sends a single HTTPS request to the webhook endpoint when one of the above mentioned events occurs.
    • It could be done with a simple, no retrying, no event persistence, no event admin UI implementation as a first step.

    Would you consider implementing something like this to make TY able to integrate into these other types of systems?

    Linked from:

    1. support-chat
    • 98 replies

    There are 98 replies. Estimated reading time: 62 minutes

    1. C
      Christian Scheuer @chrscheuer
        2021-10-04 21:50:06.498Z

        Here's an example of how Stripe displays recorded events.

        And for each event it will show us details about the contents of the event as well as the success/failure of attempts of sending that event to the webhooks registered in the account.

        1. In reply tochrscheuer⬆:

          I wonder if we could get something off the ground if we skipped some of these more thorough implementation details for now and opted for a simpler approach to get started.

          Yes: I'm thinking about implementing an /-/v0/events endpoint first. It's basically only the "Record an event object in the database" step (or not even that), plus an API endpoint. And, from what I've understood, it's needed in any case, as a complement to webhooks.

          Stripe has such an endpoint: https://stripe.com/docs/api/events

          A discussion at HackerNews: "Give me /events, not webhooks" https://news.ycombinator.com/item?id=27823109.
          Quoting the blog post: "In general, you can't rely on webhooks alone to keep two systems consistent".
          The first reply in the HackerNews discussion is from someone at Stripe.

          /-/v0/events is via polling. You could poll as often as you want (e.g. every second); the load would be negligible (proportional to number of Ty sites, rather than Ty sites * members). Later, we could add webhooks as well.

          (Thanks for the writeup about webhooks :- ) Β  A good list of things to think about. β€” There's also versioning of the webhook payload format, maybe that could be a per site and webhook setting. It's mentioned in the HackerNews discussion b.t.w.)

          1. CChristian Scheuer @chrscheuer
              2021-10-05 13:27:52.873Z

              Got it, yea if the endpoint supports:

              • Give me the events that happened since this last event that I have the ID of.

              Then that would work great. We'd then make a note of the last event we processed and always just ask for events newer than that.
              There's some complexity around corner cases which we'd end up having to implement on our end, so ideally, when you then do webhooks later, it would limit our exposure to that complexity and deal with it for us.

              1. CChristian Scheuer @chrscheuer
                  2021-10-05 13:28:24.031Z

                  Wrt versioning of webhook formats. Absolutely, should be controllable per site and/or per webhook. That's how Stripe does it.

                  1. CChristian Scheuer @chrscheuer
                      2021-10-05 13:30:08.208Z

                      I gotcha on /events and them being needed for consistency, but IMO webhooks are simpler to implement for those of us who don't want to mess with cron jobs if we don't have to :)

                      Since we're on GCP we already have some cronjobs running though, so it would be ok. But implementing polling will make the solution potentially quite a bit more expensive for us to run as a starting point.
                      The reason being that we use Cloud Run which bills per CPU milliseconds spent, so if the webhook would only send us a message, say, every 20 seconds, we'd be billed for those few milliseconds it takes to process the message, once every 20 seconds. With a real-time webhook implementation we'd need to run a cron job for example every 2 seconds (if we want real-time behavior). This would give us approx 10x the bill.

                      We can live with that though :) Just so you know why others might want to favor webhooks.

                      1. CChristian Scheuer @chrscheuer
                          2021-10-05 13:38:44.349Z

                          Wooop!! You tagged this! How are tags coming along? Soo excited for this :)

                  2. In reply tochrscheuer⬆:

                    a simple, no retrying, no event persistence, no event admin UI implementation as a first step.

                    That sounds like a good starting point for webhooks. Hmm, some persistence would actually be needed, since Ty .net is a SaaS β€” a pranks minded person could decide to send webhooks to the wrong server, causing unexpected weird things to happen. And then the Ty server would need to remember for example that "oops, that server doesn't want my webhooks".

                    1. CChristian Scheuer @chrscheuer
                        2022-02-03 11:36:23.933Z

                        Hey @KajMagnus

                        I wanted to check in with you to see if (simple) webhooks would be possible to implement some time soon?

                        Today, whenever a user posts in one of our package sub categories, we have to manually tag the user who authored the package. This is pretty time consuming (especially at scale)
                        https://forum.soundflow.org/-6636/future-suggestion#post-1

                        We'd like that to instead be our "robot" account that makes that CC (it would also mean I don't have to hush notifications from all those threads where all I do is CC somebody).
                        The robot could then be notified via the webhook that a new thread was created, and react accordingly.

                        1. Now seems like an ok / good time to get started with (the simple version of) this :- ) I'll have a look next week. I'm guessnig it'll take 2 – 3 weeks until basic "dumb" (maybe not yet any good retry mechanism) webhooks are available in Prod.

                          I can see how posting those "CC @ username" is distracting.

                          1. KajMagnus @KajMagnus2022-02-05 09:32:56.025Z2022-02-05 21:13:12.625Z

                            @chrscheuer what language is your backend written in? It's Javascript?

                            Talkyard might send the webhooks as PASETO tokens β€” then you'd know they're from the Talkyard server, and that they haven't been tampered with. There are PASETO libs for most languages: https://paseto.io

                            They would/could even be encrypted, so if some admin some day accidentally typed the wrong destination URL, the receiver couldn't understand anything.

                            Edit: Or maybe the webhooks will end up in different other software systems that's not yours, and which understand JSON only?

                            1. CChristian Scheuer @chrscheuer
                                2022-02-09 12:44:04.124Z

                                Hi @KajMagnus ,

                                click to show
                        2. In reply tochrscheuer⬆:

                          Here's the JSON in a webhook request: (improvement ideas are welcome)

                          {
                            "events": [{   // a list with exactly 1 event
                              "id": 1234,  // event id β€” currently sequential ids, maybe will change
                              "when": 1648625026090,   // Unix time, millis
                              "eventType": "PostCreated",
                              "eventData": {  ... },  // depends on the event type
                            }],
                            "origin": "http://e2e-test-cid-0-0-now-2498.localhost"
                          }
                          

                          Complete example β€” if someone posts a reply:

                          {
                            "events": [
                              {
                                "id": 5,
                                "when": 1648625026090,
                                "eventType": "PostCreated",
                                "eventData": {
                                  "post": {
                                    "id": 111,
                                    "nr": 4,
                                    "author": {
                                      "id": 102,
                                      "isGroup": false,
                                      "isGuest": false,
                                      "fullName": "Memah",
                                      "username": "memah",
                                      "tinyAvatarUrl": null
                                    },
                                    "pageId": "2",
                                    "urlPath": "/-2#post-4",
                                    "parentNr": 2,
                                    "pageTitle": "ideaTitle",
                                    "isPageBody": false,
                                    "isPageTitle": false,
                                    "approvedHtmlSanitized": "<p>memahsReplyTwo</p>\n"
                                  }
                                }
                              }
                            ],
                            "origin": "http://e2e-test-cid-0-0-now-2498.localhost"
                          }
                          

                          To construct the full URL to the new post, add the origin field to the urlPath field: http://e2e-test-cid-0-0-now-2498.localhost + /-2#post-4. (Or is having to do that, annoying?)

                          1. CChristian Scheuer @chrscheuer
                              2022-03-30 09:08:40.433Z

                              Concatenating is fine :) this looks great at first glance!

                              1. CChristian Scheuer @chrscheuer
                                  2022-03-30 09:15:27.055Z

                                  Do we have an existing API to retrieve pages (including posts) by ID? I think that could be needed for the webhook receiver to understand the context of a post (in some cases, in others, not).
                                  Not saying anything should be added to the JSON here, just thinking one step ahead.

                                  1. There's a Get API which you could use for this β€” it looks up & returns things, based on ids. However I think it only returns number of likes, currently, no other fields. I could fix that (will have to do sooner or later in any case).

                                    1. CChristian Scheuer @chrscheuer
                                        2022-03-30 09:39:54.932Z

                                        Thank you, that would be great :)

                                • In reply tochrscheuer⬆:
                                  KajMagnus @KajMagnus2022-03-30 08:55:44.406Z2022-03-30 13:25:00.975Z

                                  Another example: Someone created a new topic β€” a PageCreated event:

                                  {
                                    "events": [
                                      {
                                        "id": 2,
                                        "when": 1648625007053,
                                        "eventData": {
                                          "page": {
                                            "id": "2",
                                            "author": {
                                              "id": 102,
                                              "isGroup": false,
                                              "isGuest": false,
                                              "fullName": "Memah",
                                              "username": "memah",
                                              "tinyAvatarUrl": null
                                            },
                                            "posts": [   // the Original Post aka page body
                                              {
                                                "id": 108,
                                                "nr": 1,
                                                // "author" β€” included above already, in  page.author.
                                                "parentNr": null,
                                                "isPageBody": true,  // then, author is page author
                                                "isPageTitle": false,
                                                "approvedHtmlSanitized": "<p>ideaTextOrig</p>\n"
                                              }
                                            ],
                                            "title": "ideaTitle",
                                            "excerpt": "ideaTextOrig",
                                            "urlPath": "/-2/ideatitle",
                                            "pageType": "Idea",
                                            "doingStatus": null,
                                            "answerPostId": null,
                                            "closedStatus": null,
                                            "deletedStatus": null,
                                            "categoriesMainFirst": [   // currently only includes the parent
                                                                       // category, not any grandparent category
                                              {
                                                "id": 2,
                                                "name": "CategoryA",
                                                "urlPath": "/latest/category-a",
                                                "categoryId": 2
                                              }
                                            ]
                                          }
                                        },
                                        "eventType": "PageCreated"
                                      }
                                    ],
                                    "origin": "http://e2e-test-cid-0-0-now-2498.localhost"
                                  }
                                  

                                  Some time later, the null fields will be excluded completely instead (if they don't have any values).

                                  1. CChristian Scheuer @chrscheuer
                                      2022-03-30 09:09:59.114Z

                                      Looks great at first glance as well!

                                      Quick question, will the author objects and category objects contain SSO IDs as well?

                                      1. Not currently β€” sounds like a good idea though (so you won't need to keep a dictionary in your system, or send extra queries to look up the ids).

                                        I wonder if the sso ids and ext ids should be included by default. Hmm, probably there should be a config value for this. (So sso ids won't accidentally get sent to anywhere unexpected.)

                                        1. CChristian Scheuer @chrscheuer
                                            2022-03-30 09:42:46.336Z

                                            Oh ok, yea we will definitely need this. We do all calls to the TY API with sso IDs so we have no current way of tracking internal TY IDs of users etc., and the extra lookup would just cause unnecessary traffic to the TY servers, IMO.
                                            I wouldn't expect anybody to assume a SSO ID should be any more secret than any other content in the JSON you're sending, so not sure why that should be configurable. We certainly wouldn't need that to be hidden :)

                                            Would it be possible to do a quick change to this that included the SSO IDs of categories and authors?

                                            1. CChristian Scheuer @chrscheuer
                                                2022-03-30 09:45:01.079Z

                                                For example, if we program a "bot" to respond to certain posts, the bot should be able to know not to reply to itself ;)
                                                Do posts currently have SSO IDs as well btw / metadata?

                                                1. Yes, users, groups, categories, pages and posts β€” all these have ext-ids (and users have sso-ids too).

                                                  click to show
                                        2. In reply tochrscheuer⬆:

                                          A page got closed β€” a PageUpdated event:

                                          {
                                            "events": [
                                              {
                                                "id": 5,
                                                "when": 1648630983752,
                                                "eventData": {
                                                  "page": {
                                                    "id": "2",
                                                    "title": "webhBasicQuTitle",
                                                    "author": {
                                                      "id": 110,
                                                      "isGroup": false,
                                                      "isGuest": false,
                                                      "fullName": null,
                                                      "username": "mei",
                                                      "tinyAvatarUrl": null
                                                    },
                                                    "excerpt": "webhBasicQuBodyOrig",
                                                    "urlPath": "/-2/webhbasicqutitle",
                                                    "pageType": "Question",
                                                    "doingStatus": null,
                                                    "answerPostId": 109,
                                                    "closedStatus": "Closed",    <β€”β€”β€”
                                                    "deletedStatus": null,
                                                    "categoriesMainFirst": [
                                                      {
                                                        "id": 2,
                                                        "name": "CategoryA",
                                                        "urlPath": "/latest/category-a",
                                                        "categoryId": 2
                                                      }
                                                    ]
                                                  }
                                                },
                                                "eventType": "PageUpdated"   <β€”β€”β€”
                                              }
                                            ],
                                            "origin": "http://e2e-test-cid-0-0-now-3095.localhost"
                                          }
                                          
                                          1. Ooops there's a bug. These currently won't get sent. Fixed on localhost ... will be included in the next release.

                                          2. In reply tochrscheuer⬆:

                                            Security:

                                            Signing requests with a HMAC: Not yet implemented.

                                            In the next version, (in one or a few days), you'll be able to add custom headers
                                            β€” then you can include a Basic Auth: username-&-password so your servers will accept the incoming webhook requests. Or you could add a X-Secret-Key: .... header or something.

                                            But for now, you'll need to send the webhooks to an un-guessable URL, like /api/webhooks-in/2578892785076892415 where the last part is a long unguessable string.

                                            1. CChristian Scheuer @chrscheuer
                                                2022-03-30 09:43:36.636Z

                                                That's cool, thanks for the update!

                                              • In reply tochrscheuer⬆:

                                                Prod now upgraded to the webhooks version with the "eventType": "PageUpdated" missing bug, and that lacks extId and ssoId. ... I'll build t

                                                click to show
                                                1. In reply tochrscheuer⬆:

                                                  Now the next version, with extId and ssoId, is running here at Ty .io β€”Β will upgrade Prod this evening or tomorrow morning.

                                                  1. CChristian Scheuer @chrscheuer
                                                      2022-04-01 10:24:30.989Z

                                                      Wonderful, thank you!

                                                      1. CChristian Scheuer @chrscheuer
                                                          2022-04-01 11:09:42.461Z

                                                          Just testing with the current build without ssoIds and everything seems to be working perfectly :D

                                                          1. CChristian Scheuer @chrscheuer
                                                              2022-04-01 11:25:19.639Z

                                                              OMG this is gonna be so epic.

                                                              For the PageCreation, would it be possible to send along the "post nr 1" with it? That would save us a roundtrip.
                                                              I think very, very often, we would want to know the contents of the page being created to be able to react to it.

                                                              1. CChristian Scheuer @chrscheuer
                                                                  2022-04-01 12:43:49.229Z

                                                                  Conversational bot that can help us direct user support requests etc. :)

                                                                  1. CChristian Scheuer @chrscheuer
                                                                      2022-04-01 15:19:38.310Z

                                                                      I've also implemented it now so that if you post a question in one of our sub categories (under Packages), the bot will reply and auto-tag t

                                                                      click to show
                                                            • In reply tochrscheuer⬆:

                                                              The next version, with extId and ssoId, is in Prod now. And the PageUpdated event works now too (bug fixed).

                                                              1. CChristian Scheuer @chrscheuer
                                                                  2022-04-02 10:13:32.340Z

                                                                  Thanks @KajMagnus

                                                                  Unfortunately, it looks like the JSON structure sent doesn't fully meet the JSON standard so it's failing validation on our end.

                                                                  Sent JSON:
                                                                  {
                                                                    "events": [
                                                                      {
                                                                        "id": 54320,
                                                                        "when": 1648826049680,
                                                                        "eventData": {
                                                                          "post": {
                                                                            "id": 48012,
                                                                            "nr": 2,
                                                                            "extId": null,
                                                                            "pageId": "7196",
                                                                            "urlPath": "/-7196#post-2",
                                                                            "parentNr": 1,
                                                                            "pageTitle": "Open a session based on part of the filename",
                                                                            "approvedHtmlSanitized": "<ol>\n<li>\n<p>Navigate to directory/folder<br>\nSESSIONS</p>\n</li>\n<li>\n<p>Select file that contains β€œ.ptx” and β€œR1”</p>\n</li>\n<li>\n<p>Duplicate file</p>\n</li>\n<li>\n<p>Move duplicate to β€œold” folder</p>\n</li>\n<li>\n<p>Rename duplicate: deleting β€œ copy”</p>\n</li>\n<li>\n<p>Navigate to directory/folder<br>\nEG SESSIONS</p>\n</li>\n<li>\n<p>Select file that contains β€œ.ptx” and β€œR1”</p>\n</li>\n<li>\n<p>Rename file: replace old date with today’s date</p>\n</li>\n<li>\n<p>Open file</p>\n</li>\n</ol>\n<ul>\n<li>[ ]</li>\n</ul>\n"
                                                                          }
                                                                        },
                                                                        "eventType": "PostCreated"
                                                                      }
                                                                    ],
                                                                    "origin": "https://forum.soundflow.org"
                                                                  }
                                                                  Response headers: { "Date": [ "Sat, 02 Apr 2022 08:58:30 GMT" ], "Server": [ "Google Frontend" ], "content-type": [ "application/problem+json; charset=utf-8" ], "Content-Length": [ "354" ], "X-Cloud-Trace-Context": [ "a17febe17e6f926916b1a07d426c5988" ] }
                                                                  Response body: {"errors":{"events[0].eventData.post.pageTitle":["Unterminated string. Expected delimiter: \". Path 'events[0].eventData.post.pageTitle', line 1, position 1013."]},"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"00-a17febe17e6f926916b1a07d426c5988-a45e7a9e86c5b34a-00"}
                                                                  
                                                                  1. CChristian Scheuer @chrscheuer
                                                                      2022-04-02 10:22:11.313Z

                                                                      The weird thing is, when I copy/paste from the "event log" and send it manually, it works fine.
                                                                      Are you somehow buffering the output when you send it and somehow end up sending invalid characters like line feeds / carriage returns as part of the payload?

                                                                      1. CChristian Scheuer @chrscheuer
                                                                          2022-04-02 10:24:31.992Z

                                                                          We've not received any valid JSON input since 15:10 (UTC) yesterday. Before that, everything worked fine, after that, everything fails. It looks like when you upgraded the server to the newest version, the input stopped being valid when received on our end.
                                                                          Note the error reporting comes from validation on the Google servers we're using, so it's not in our code.

                                                                          1. That's odd. Yesterday, nothing was changed β€” I upgraded Prod this morning (only another server, Ty .io, yesterday).

                                                                            click to show
                                                                    • In reply tochrscheuer⬆:
                                                                      KajMagnus @KajMagnus2022-04-05 12:41:40.131Z2022-04-05 12:54:55.863Z

                                                                      A thought: What about including a field:

                                                                      click to show
                                                                      1. In reply tochrscheuer⬆:
                                                                        KajMagnus @KajMagnus2022-04-14 05:51:40.509Z2022-04-14 06:13:18.256Z

                                                                        I'm marking this topic as Doing, but not Done, because, although it's usable as-is, 4 things are missing:

                                                                        click to show
                                                                        1. Progress
                                                                          with doing this idea
                                                                        2. @KajMagnus marked this topic as Planned 2022-02-04 12:02:41.913Z.
                                                                        3. @KajMagnus marked this topic as Started 2022-04-14 05:51:47.943Z.