No internet connection
  1. Home
  2. Ideas

Talkyard API: Upserting categories

By KajMagnus @KajMagnus2019-06-14 10:38:22.764Z2019-06-14 10:48:35.808Z

So there's going to be an endpoint for upserting categories. This is what I have in mind: (feedback welcome :- ))

POST server-address/-/v0/upsert-simple

and that endpoint accepts a SimpleUpsertV0 object that looks like this, in Typescript: (and ? means optional )

interface SimpleUpsertV0 {
  categories?: SimpleCategoryV0[];

  // ... more things, later, if you want to upsert many things
  // in the same database transaction.
}


interface SimpleCategoryV0 {
  id: CategoryId;      // should be  2 000 000 001
  extImpId?: string;   // your own external unique id for the category
  parentId?: CategoryId;         // later, skip for now
  defaultSubCatId?: CategoryId;  // later, skip for now
  name: string;
  slug: string;
  description: string;
  position?: number;
  defaultTopicType?: PageType;
};

The extImpId is your own unique id for the category. When Talkyard (Ty) upserts the category, Ty first tries to look up any existing category with this "external import id", and, if found, Ty updates that category. If not found, Ty creates a new category, assigns it an id (less than 2e9) and remembers its extImpId.

The id field should be any number > 2e9, e.g. 2 000 000 001. Numbers > 2e9 are reserved for things you upsert into Ty and that you don't know if they're present in Ty's database (and you don't want to write extra code to find out).

You can set id to any 32 bit signed integer > 2e9. If you upsert more things, like pages and sub categories, in a single HTTP request — then, some of the other items you upsert, can refer to the category.id and then Talkyard knows you want to place them in that category.

(B.t.w. I'm including the word "simple" in the URL and Typescript interface, because under the hood, more things happen, that you don't need to know about: When you create a category, then, also, a category description page gets created, and a Post that stores the category description text and edit revisions, if you edit the description later. — Probably there'll also be a "complicated" endpoint /-/v0/upsert-dump that specifies exactly what things should be upserted and doesn't do anything for you automatically / implicitly)

@chrscheuer what do you think? Something that can be done in a better way, or maybe re-thinking the whole approach?

  • 75 replies

There are 75 replies. Estimated reading time: 91 minutes

  1. KajMagnus @KajMagnus2019-07-01 09:23:19.323Z

    @chrscheuer

    I think you'll need to be able to assign an "External Import IDs" of your choosing, to the parent Packages category, so you have an id to refer to, when upserting child categories for individual packages.

    Another approach would be that you refer to the Packages category, via Talkyard's internal category id, but not impossible that exposing such internal ids will cause problems in the future, if I need to change the ids somehow, or if they get remapped to different numbers during an export and import.

    I'm thinking it'd be more future proof, if you instead give the Packages category an extImpId, say, "soundflow_packages", to refer to when upserting child categories.

    To upsert a sub category, you'd then do this:

    POST server-address/-/v0/upsert-simple
    
    {
      categories: [{
        // This is the Packages category. It's in the database already; this entry is here only so you'll
        // have a "temporary import id" to refer to ---.
        extImpId: 'soundflow_packages',                |
        id: 2000000001,                   <------------+
      } {                                              |
        extImpId: 'someones_new_package_name',         |
        id: 2000000002,                                |
        parentId: 2000000001,          <---------------`
        name: "New Package Name",
        slug: "new_package_slug',
        description ....
      }]
    }
    

    Talkyard will then lookup extImpId = 'soundflow_packages' in the database to find the real internal id X of the Packages category, and then, when saving the New Package, Talkyard sets its parentId not to 2000000001 but to X.

    Another alternative could be:

    POST server-address/-/v0/upsert-simple
    
    {
      categories: [{
        extImpId: 'someones_new_package_name',
        id: 2000000001,
        parentExtImpId: 'soundflow_packages,    <———
        name: "New Package Name",
        slug: "new_package_slug',
        description ....
      }]
    }
    

    What do you think?

    1. CChristian Scheuer @chrscheuer2019-07-01 09:26:12.295Z

      This sounds great!

      1. CChristian Scheuer @chrscheuer2019-07-01 09:28:15.826Z

        As a general scheme, I don't know if you need to distinguish between "pure-reference entities" and "entities for upserting". Right now you seem to make the distinction based on the name/slug/description parameters being present, but imagine that all those parameters were optional (for something other than categories)..
        Idk. It's a hypothetical and not a problem for now. Problem / not a problem?

        1. CChristian Scheuer @chrscheuer2019-07-01 09:30:35.943Z

          A different way would be to have id and parentId be strings, that could be of the form:

          parentId: "talkyard:53"
          parentId: "ext:soundflow_packages"
          

          By using a scheme like this instead of magic numbers it's more manageable for any future additions.

          1. CChristian Scheuer @chrscheuer2019-07-01 09:32:08.046Z

            This doesn't change anything about the fact that the underlying id's are numerical. It's just making a more concise interface. Note this als

            click to show
    2. In reply toKajMagnus:
      KajMagnus @KajMagnus2019-07-14 02:15:14.024Z

      @chrscheuer What about permissions? The upserted Soundflow package categories, are they to be publicly visible, or visible only to community members?

      Maybe the JSON can be like: (note the permissions: array at the end, and the ...Ref: 'extid:...' references)

      POST server-address/-/v0/upsert-simple
      
      {
      categories: [{
         extId: 'a_package_name',
         parentRef: 'extid:soundflow_packages',
         name: 'Package Name',
         slug: 'package_slug',
         description: '...',
         permissions: [{
           forMemberRef: 'extid:all_members',
           maySee: true,
           mayCreatePages: true,
           mayPostReplies: true,
           mayEditOwn: true,
         }, {
           forMemberRef: 'extid:staff',   // admins and moderators
           maySee: true,
           mayCreatePages: true,
           mayPostReplies: true,
           mayEditAll: true,     // for staff only
           mayDeleteAll: true,   // for staff only
         }]
      }]
      }
      1. CChristian Scheuer @chrscheuer2019-07-14 16:43:23.681Z

        This looks great.

        Our auto-created packages should be visible to anybody for now, but I agree it makes sense to touch on the permissions in the upsert API schema. Maybe it can default to "normal public category" if no permissions are specified, just for ease of use?

        We're currently using "Trusted member" to give our alpha/beta team access to restricted categories that are only meant for them. Ideally whichever solution is made for the permissions schema it will be forwards compatible with if you will make access groups a real feature with more granular control some time in the future. I'm thinking the forMemberRef is referring to groups, so this sounds like you already have it covered.

        1. CChristian Scheuer @chrscheuer2019-07-14 16:45:43.980Z

          Oh wow. Now that I took a look, I see there's now a Custom Groups section!! That's epic!! When did this come? :)
          You should do a newsletter or something where we can follow the progress haha so we don't miss new awesome features

          1. KajMagnus @KajMagnus2019-07-22 11:06:52.746Z

            Ok yes a newsletter is a good idea :- ) Maybe I can create a Newsletter category here in the forum? I'll subscribe you to that Newsletter category then.

            The custom groups thing appeared about 2 months ago, and ... hmm I wasn't good at informing people about that. Maybe I should think a bit about who might want to know about that, and contact them and let them know. I can do shortly after the next release :- ) and then I can mention import-export one's site too.

            1. CChristian Scheuer @chrscheuer2019-07-22 11:12:37.896Z

              Newsletter category sounds good to me. Strictly with updates about new features or changes. That way we can see the progress on a timeline.

              click to show
            2. In reply tochrscheuer:
              CChristian Scheuer @chrscheuer2019-07-14 18:13:58.575Z

              One thing to note. If we go with the "extid:blabla" and "tyid:blabla" string ref thing. Make sure that it parses stuff like "extid:blabla:blabla:blabla" correctly, that is when the external ID itself has colons (since ours do). This essentially makes it a URN with a custom scheme name.

              1. KajMagnus @KajMagnus2019-07-22 11:15:41.331Z

                Ok. B.t.w. I'm thinking maybe it'd be good if the ext id can be a URL. So maybe I can allow everything that's ok in a URL.

                1. CChristian Scheuer @chrscheuer2019-07-25 00:23:37.861Z

                  Sure, sounds good. As long as it's not limited to urls. Ours would not be valid URLs in the http sense (we have two colons) and would feel a

                  click to show
                2. In reply tochrscheuer:
                  CChristian Scheuer @chrscheuer2019-07-14 18:45:13.058Z2019-07-14 19:18:52.841Z

                  Another thing I'm thinking could be good for the API implementation, maybe not in version 1, but down the road, is to address the very very

                  click to show
          2. C
            In reply toKajMagnus:
            Christian Scheuer @chrscheuer2019-06-14 12:44:34.620Z

            Wohoo! What a great Friday gift :)

            @chrscheuer what do you think? Something that can be done in a better way, or maybe re-thinking the whole approach?

            This definitely seems like a very good approach very close to what I was hoping for.
            I completely understand what you mean by "simple" and I agree with that concept. To me making the API surface smaller is only gonna help creating tests for it and ensuring that it works across all cases.

            Couple of things we should consider:

            • What does the endpoint return? Some would probably like the id, however we'd probably just want the url (value taken from the generated page's pagePath.. if I remember correctly). Maybe we need a SimpleCategoryUpsertResponseV0 interface that either responds with simple things like an url, and/or with all the generated entries from the various tables.

            • URLs matter. Right now all posts/threads/pages have id + slug based urls, where the slug can change without affecting the URL since the id stays the same. It appears categories don't have this (at least right now). In the SimpleCategoryV0 interface the slug is a required parameter. This leads me to think that by changing the slug we can (accidentally or because we want to) change the url of the category. We need to think about how this behavior should be. Our users can change the name of a package but it stays with the same internal ID (and thus internal url). We'd want the category url to stay the same too, across different slugs.
              Maybe you already have support for this, but it would make sense to me that these auto-generated categories would utilize the same url pattern as posts. Maybe a flag to set in the interface?

            • Now for the interesting part. In the future you'll want to support sub-categories. We'll need to think about how those urls should look. To the user, it would make sense that each subcategory appends itself to the url of its parent. However that doesn't seem consistent with the -uniqueid/slug pattern that I requested above. So this is something that needs to be considered.

            • Slug algorithm/restrictions. Which characters are allowed in the slug? Which length?

            • id magic numbers > 2e9. This sounds great and well thought out for more complex insertions.

            • Edit: added this: Are there any limitations for extId strings length wise? Our id's for packages are concatenated user id + package id + so ours are about 60 characters long...

            That's what I think for now. Thank you so much for looking into this :)

            1. CChristian Scheuer @chrscheuer2019-06-14 17:19:23.332Z

              Another way to handle category url changes would be to make sure to keep a "history" inside TY of previous urls, so that categories whose urls (slugs) change, would just have permanently occupied the old space that will then redirect to the new one.
              That would keep it possible to have easy-to-read nested category urls, like:
              http://forum.soundflow.org/latest/packages/my-package

              1. KajMagnus @KajMagnus2019-06-18 12:15:15.314Z
                • What does the endpoint return — yes seems like the category id and the slug should be included in the reply. Maybe a list of all things that got created, yes. Maybe you'd find the cateory and its id and slug like so responseJson.categories[0].slug and [0].id and .urlPath.

                • Category slugs and renaming categories: I like the approach with keeping a history, and having old (sub) category url paths rediect to the current up-to-date path, like /latest/packages/current-package-name. This is b.t.w. how discussion topic urls already work — if you change the url to a topic, the old url redirects to the new (should work also for pages for which the /-1234/ id number has been excluded in the url (there's an "advanced" setting for hiding the page id in the url))

                  I'm thinking people renaming one package to [the previous name of another package], is something you'll need to deal with in Soundflow? (mabe by not allowing such renames?) And it's okay if, in Talkyard, if a category Bbb gets renamed to [a previous name of another category Aaa], then Ty stops the old url path to Aaa from redirecting to the new path to Aaa, and instead that url path will be to Bbb (although it was previously used by Aaa).

                • "any limitations for extId strings length wise?" — I have in mind an external-import-id 100 char length restriction (i.e. > 60).

                • "Which characters are allowed in the slug" — I have in mind lowercase alpahumeric + hyphen - ok inside (but not starting or ending with -) and at least one letter (not only digits).

                1. CChristian Scheuer @chrscheuer2019-06-22 17:21:54.918Z

                  What does the endpoint return — yes seems like the category id and the slug should be included in the reply. Maybe a list of all things that got created, yes. Maybe you'd find the cateory and its id and slug like so responseJson.categories[0].slug and [0].id and .urlPath.

                  Sounds good to me - so basically close to a copy of the simple input with urlPath added...
                  It might make sense at one point to also have the complete raw entries present in the response, but for the current use cases that would just make the response larger and thus slow stuff down (slightly). Just thinking about how generic we can make the request/response object layout so that it's flexible enough for other use cases too.

                  I'm thinking people renaming one package to [the previous name of another package], is something you'll need to deal with in Soundflow? (mabe by not allowing such renames?) And it's okay if, in Talkyard, if a category Bbb gets renamed to [a previous name of another category Aaa], then Ty stops the old url path to Aaa from redirecting to the new path to Aaa, and instead that url path will be to Bbb (although it was previously used by Aaa).

                  Agree completely.
                  This also confirms that the API caller (SoundFlow in this case) should be in charge of creating slugs since that puts the responsibility on us for detecting identical URLs and providing a UI for the user or an algorithm to make sure that it doesn't happen.
                  The opposite approach would be for the slug algo to sit with Talkyard and then TY would just append numbers to slugs in case something already existed at the same address/path - but that would give the API caller less control.

                  "any limitations for extId strings length wise?" — I have in mind an external-import-id 100 char length restriction (i.e. > 60).

                  Sounds great!

                  "Which characters are allowed in the slug" — I have in mind lowercase alpahumeric + hyphen - ok inside (but not starting or ending with -) and at least one letter (not only digits).

                  Sounds great!

            2. C
              In reply toKajMagnus:
              Christian Scheuer @chrscheuer2019-07-29 01:33:14.312Z

              @KajMagnus let me know where you think we're at in the review process.
              I feel like we're pretty close to this being possible to implement in a v1 (v0), right?
              I'm asking since we still have a window to include this in our big next release :)

              But actually the reason I opened up this forum today was to let you know that using Talkyard for Soundflow has been a major shift for us in being able to handle support requests, building up our knowledge base, handling bugs etc. I'm so happy using it every day. If you want us to help promote it or you need some testimonials, I'd be very happy to help.

              1. KajMagnus @KajMagnus2019-07-29 08:29:57.319Z

                Hi Christian, I've written the upsert-categories code (just merged into the master branch), and the automatic tests; seems to work fine. I'm about to start code reviewing the API now — this will take about a week (I'll also code review the import-Disqus-comments code which I did together with this.) Probably I'll find things to fix, 1 more week. And after that, I will want to try out the new version, here at Talkyard .io, for maybe half a week, before deploying to Talkyard .net (your server). So about 2.5 weeks before the upsert-categories endpoint is live in your forum, I'd think.

                B.t.w. I also partly implemented sub categories, so you'll be able to upsert into sub categories in a parent "Packages" category with ext id "soundflow_packages", for example.

                Roughly when is that next release?

                Yes a testimonial would be helpful — I've been thinking about asking people about that actually :- ) How do we go about doing that? Maybe if you write something, then I create a Testimonials section on the website and insert?

                1. CChristian Scheuer @chrscheuer2019-07-29 08:50:04.576Z

                  That sounds great!
                  We plan to release around September 1st, but that also means to beta test it we'd need it pretty soon in order to get it in before we lock our branch. I think it sounds close to doable with your time frame, but it's gonna be a little tight. Would there be any way to test our integration before somehow? Maybe via a test forum somewhere?
                  Wrt testimonials - that's entirely up to you. The website thing would be a good place to start, but I'm thinking you should be pretty close to being able to make some public releases / statements about Talkyard (it feels pretty stable and feature complete to me). One of our partners did a nice graphic with logos of all their partners. You could do something similar. Anything that makes it look fancy is good IMO :)

                  1. KajMagnus @KajMagnus2019-07-29 09:32:32.694Z

                    Hmm I'll spend two days doing code review, and then I'll know better. Not impossible that there' won't be so many things to fix. Then I can get back to you.

                    Ok, a public realease / statement — that feels like a good idea to do, together with this new version with export & import. Mabye can combine with your / others logos or sth like that.

                    1. KajMagnus @KajMagnus2019-07-31 13:52:46.132Z

                      @chrscheuer Seems I won't run into any major unexpected things now during code review. Probably I'll have deployed the new version at the en

                      click to show
              2. Progress
                with doing this idea
              3. @KajMagnus marked this topic as Started 2019-06-23 00:09:08.286Z.
              4. KajMagnus @KajMagnus2019-08-11 08:16:07.707Z

                @chrscheuer Now I just deployed the new version (v0.6.41) with the upsert API. Feel free to try it out. B.t.w. there are rate limits:

                  object UpsertSimple extends RateLimits {
                    val key = "UpSm"
                    val what = "Upserted things too many times"
                    def maxPerFifteenSeconds = 10
                    def maxPerFifteenMinutes = 30
                    def maxPerDay = 90
                

                I you'd like to try out the API on a test site first (instead of your real live site), then go here: https://www.talkyard.io/choose-demo-forum, and click Create my own test forum. Then send me a link with the URL to the test forum, so I can enable the upsert API for that test forum too.

                1. C
                  Christian Scheuer @chrscheuer2019-08-11 11:21:11.408Zreplies toKajMagnus:

                  Awesome! Looking forward to trying it out. Thank you for pushing this!
                  Wow - the rate limits definitely will make sure that we can't use this as the primary method for users to get to the forum (that would limit us to 90 page views of a forum per day). So with this we'll need to implement our own caching for sure.
                  I'm a bit afraid to hit that daily or 15 min limit just in debugging when implementing the system. Since we'll need to do our own caching this also means I'll need to run a bulk operation once we have completed the integration, one that runs over all the packages in our system. I'm pretty sure we could hit those limits pretty easily once we want to run the bulk operation.
                  This essentially means we'll need to implement queueing on our end.

                  As I see it there are a couple of ways this can work:

                  1. Talkyard has a rate limited API. This forces us to have a database trigger (when user publishes a package, we register a slug, we talk to TY to get a forum url for that slug, this is stored)
                    When user wants to navigate to the forum, we'll know it already works. If the TY API call failed, we'll need to either have it add the request to a queue that manages to keep us below the rate limit (a lot of extra work), or build in workarounds to access the API when the user tries to go to a forum that didn't get properly created. This could in turn mean even more API access leading to cascading errors of the whole system.
                  2. Talkyard has a non-cached, non-rate-limited API (or the limit is high enough we won't see problems). We still have a database trigger - so we access the API only once per package publishing. Effectively the queueing concern will be gone so no need to build in that or any workarounds. The effective number of requests to the API would be the lowest with this approach.
                  3. Talkyard has a cached, non-rate-limited API. Since SoundFlow still needs to provide the slugs, that's a potentially slow operation, so we'd still like to do this not on a pageview level but as part of the database trigger.
                    So the Talkyard internal caching would here just be an implementation detail of the TY system and lead to potential better usage of server resources. However we'd have the exact number of API calls as in #2, just 1 per package creation.
                  4. (Rather 3b) SoundFlow splits up our system so that we make sure to define unique slugs as part of our database trigger (when user publishes a package), but then utilize a Talkyard caching endpoint to use it for every pageview/redirect operation. This would have TY API see the most traffic. We would send the same data as now but instead of for every package creation, we'd send for every page view / redirect from main site package to forum package category (same style as SSO).

                  If we did not need to provide slugs, the system could be simplified for the API consumer, and a potential cached solution from TY would have even more value than #4. Because this would allow us to use the API just like SSO and we wouldn't need to have our own separate algorithm for slugs. Essentially we could use it then more or less use it as part of a redirect for the user, whenever such a forum url would be accessed. We would in this case just send an extId and a name (and parentRef) and get back an url. All stuff that wouldn't require us to have any queueing or database triggers. This would be almost like embedded categories.


                  The issue here IMO is in putting a good distinction between what's the role of the API and the API consumer. With these pretty low limits it's making some scenarios that should be simple for us to implement much more complex. I'm a bit reluctant to go too far down that rabbithole if we can come up with a better solution.

                  1. C
                    Christian Scheuer @chrscheuer2019-08-11 11:23:07.592Zreplies tochrscheuer:

                    In essence what I'm afraid is that whereas this API would be called as often as SSO and have no bigger cost on the system than SSO when used as a Category upsert function, the API was designed to be able to do bulk operations, which of course you'd probably need to rate limit.
                    One idea: Would it make sense to lift the limit (or put it same as SSO) for simple queries that just contain a single category?

                    1. C
                      Christian Scheuer @chrscheuer2019-08-11 11:40:32.605Zreplies tochrscheuer:

                      Sorry if any of this sounded too critical :) I'm very happy we've gotten this far! And this should definitely be something we can work with. Via our Firebase setup we could do queueing etc if really needed. So we will get something great eventually, no matter what. This is good as a 1st API iteration.
                      But since what we're trying to do is essentially a specialized version of embedded categories - I think it doesn't hurt to talk about if the API can move towards something equally simple to that (which would be essentially completely URL based I suppose).

                      1. KajMagnus @KajMagnus2019-08-11 12:59:03.629Zreplies tochrscheuer:

                        whereas this API would be called as often as SSO and have no bigger cost on the system than SSO when used as a Category upsert function, the API was designed to be able to do bulk operations, which of course you'd probably need to rate limit.

                        One idea: Would it make sense to lift the limit (or put it same as SSO) for simple queries that just contain a single category?

                        Yes, this sounds like a good idea. Sometimes the /-/v0/upsert-simple requests wouldn't need to write to the database at all. Probably the requests can be optimized to access only the in-memory cache. — Maybe there could be different rate limits: 1) one for requests that upsert just a few items (e.g. one single category) and that actually didn't need to upsert anything, didn't write to the database. And 2) another rate limit, for requests that do write to the database, but just a few items (like, one category). And 3) a third rate limit, for bulk upserting lots of things.

                        What rate limits do you think would work for you, for the initial import of everything?

                        So you won't need to spend time writing your own queue to stay below the limits, and so you won't need to do a bulk operation, ... But instead can upsert everything, one request at a time (that's simplest for you?) as fast as you want?

                        (Maybe this would be approximately the number of packages you have? Per 15 seconds?) ...

                        ... Maybe I can just bump the limits a bit (or a lot) for now, since currently one needs to ask for permissons to use this API endpoint anyway. And I can restrict /-/v0/upsert-simple to only accept a single category — and reject bulk uploading lots of things. And then later on, I can add more restrictive rate limits, for API calls that bulk uploads lots of things, and start allowing that.

                        (I'll read everything you wrote in more detail a bit later)

                        1. C
                          Christian Scheuer @chrscheuer2019-08-11 13:08:55.461Zreplies toKajMagnus:

                          Thanks @KajMagnus for your quick reply! Great - sounds like we agree on it :)

                          click to show
                          1. C
                            Christian Scheuer @chrscheuer2019-08-11 13:24:57.331Z

                            I've created a demo site now here:
                            https://test--soundflowtest.talkyard.net

                            Would be great to test the integration here first :)

                            1. C
                              Christian Scheuer @chrscheuer2019-08-11 13:35:01.895Zreplies tochrscheuer:

                              Is this still the definition for slugs?

                              "Which characters are allowed in the slug" — I have in mind lowercase alpahumeric + hyphen - ok inside (but not starting or ending with -) and at least one letter (not only digits).

                              1. C
                                Christian Scheuer @chrscheuer2019-08-11 13:38:48.592Zreplies tochrscheuer:

                                Looking at this for a slugify function. Do you see any problems with it?

                                function slugify(string) {
                                  const a = 'àáäâãåăæąçćčđďèéěėëêęğǵḧìíïîįłḿǹńňñòóöôœøṕŕřßşśšșťțùúüûǘůűūųẃẍÿýźžż·/_,:;'
                                  const b = 'aaaaaaaaacccddeeeeeeegghiiiiilmnnnnooooooprrsssssttuuuuuuuuuwxyyzzz------'
                                  const p = new RegExp(a.split('').join('|'), 'g')
                                
                                  return string.toString().toLowerCase()
                                    .replace(/\s+/g, '-') // Replace spaces with -
                                    .replace(p, c => b.charAt(a.indexOf(c))) // Replace special characters
                                    .replace(/&/g, '-and-') // Replace & with 'and'
                                    .replace(/[^\w\-]+/g, '') // Remove all non-word characters
                                    .replace(/\-\-+/g, '-') // Replace multiple - with single -
                                    .replace(/^-+/, '') // Trim - from start of text
                                    .replace(/-+$/, '') // Trim - from end of text
                                    .replace(/^\d+$/, m => `category-${m}`) //Prepend "category-" if we had a number-only slug
                                }
                                
                                1. C
                                  Christian Scheuer @chrscheuer2019-08-11 15:14:10.682Z

                                  I appear to have it working now!

                                  https://forum.soundflow.org/latest/logic-pro-x

                                  Was just created via the upsert API :)
                                  Edit: and it was triggered by our full package publishing pipeline. Our cache layer and slug generator etc. seems to work as expected :) Yay!

                                  1. C
                                    Christian Scheuer @chrscheuer2019-08-11 15:22:01.432Z

                                    I decided to manually trigger the creation for now. Other users' packages' forums will be created next time they publish a new version of th

                                    click to show
                                    1. C
                                      Christian Scheuer @chrscheuer2019-08-11 15:25:30.645Z

                                      I'm inserting all these without setting the "position" property.
                                      Is there any chance automatic sorting could be implemented on the TY side, in case categories don't have positions?

                                      1. C
                                        Christian Scheuer @chrscheuer2019-08-11 21:03:58.516Z

                                        I have implemented the automatic categories across all of our packages now. I'm trying to embed the forum in an iframe here, but the SSO pro

                                        click to show
                                        1. C
                                          Christian Scheuer @chrscheuer2019-08-11 21:27:54.485Z

                                          And here it is now working in the app :)

                                          1. KajMagnus @KajMagnus2019-08-13 13:26:02.686Zreplies tochrscheuer:

                                            Hi @chrscheuer,

                                            automatic sorting could be implemented on the TY side, in case categories don't have positions?

                                            I think so — does that mean alphabetically? Or by most-active-first? (I'll look into the SSO-top-window-redirect issue first)

                                            And here it is now working in the app :)

                                            Thanks for the screenshot, I think it looks nicely integrated :- ) (Hmm possibly some minor things to maybe consider some time later, like: the title "soundflow" on the top nav bar with the blue background — maybe that title isn't needed? And the forum title a bit below, again. Maybe some URL parameters could hide the forum title & intro text, + top nav bar? Or a CSS class on the html elem, if is embedded. )

                                            If one navigates to a different category, then clicks a different app tab (say, the "Save Version" tab), and then clicks "Package Forum" again — then, does the forum view get recreated, and one starts in the Online/Offline package again? Or is the forum view remembered, and one is still in that different category?

                                            the SSO process seems to redirect the top window at some point in the process

                                            I think there're two problems: 1) The Package Forum tab link has a redirect url that sends the browser to Talkyard's forum, rather than back to the page at Soundflow. This page: https://soundflow.org/store/dialog-editing-izotope/forum somehow redirects the browser to: https://soundflow.org/forum?returnTo=https%3A%2F%2Fforum.soundflow.org%2Flatest%2Fdialog-editing-izotope — and look at the returnTo=... parameter. It tells SoundFlow to send the browser to Talkyard's domain.

                                            Something like this instead, could work: https://soundflow.org/forum?returnTo=https://soundflow.org/store/dialog-editing-izotope/forum

                                            Then there's problem 2: Because of maybe-security-issues, Talkyard strips the url origin from the_return_to_url in: http://localhost/-/v0/sso-login?oneTimeSecret=nnnnn&thenGoTo=the_return_to_url. So even if Talkyard gets an url with the correct origin (i.e. https://soundflow.org/store/dialog-editing-izotope/forum), Talkyard will refuse to use it, and only looks at the url path (/store/dialog-editing-izotope/forum) instead. — I previously didn't want to allow other url origins than the Talkyard server itself, because of maybe-pishing attacks.

                                            Here's Talkyard's code that runs when calling http://localhost/-/v0/sso-login?oneTimeSecret=nnnnn&thenGoTo=https://soundflow.org/store/dialog-editing-izotope/forum

                                                    // Remove server origin, so one cannot somehow get redirected to a phishing website
                                                    // (if one follows a link with an "evil" go-next url param to the SSO login page,
                                                    // which then redirects to this endpoint with that bad go-next url).
                                                    val thenGoToUnsafe = getOnly("thenGoTo")
                                                    val thenGoToHashEncoded = thenGoToUnsafe.flatMap(Prelude.stripOrigin) getOrElse "/"
                                            

                                            That protects against someone who crafts an evil link:

                                            https://soundflow.org/forum?returnTo=https%3A%2F%2Fwww.evil.com
                                            

                                            Then, the user would login over at soundflow.org — the real domain — and believe s/he is safe? But would then get sent to an evil server, possibly phishing. Well, unless one or both of SoundFlow and Talkyard detects and rejects unexpected redirect-to destination domains.

                                            Maybe there could be a whitelist of allowed go-to-after-SSO-login origins? There's a setting allowEmbeddingFrom, intended for embedded comments. Maybe Talkyard could allow redirecting to the allowEmbeddingFrom origins after SSO login?

                                            1. KajMagnus @KajMagnus2019-08-13 18:14:08.054Zreplies tochrscheuer:

                                              One little note - maybe there should be a limit UI wise on the number of sub categories shown (once we have 57 packages here it could become crowded):

                                              Hmm, yes that seems like a probably good idea. Maybe a max height, + a button "Click to show" ? I could look into this a bit later, after the iframe SSO redrect, and rate limiting issue, and returning the category url path.

                                              Another approach could be to place the Packages category last, so it'll be at the bottom of the page. Then maybe doesn't matter if there're a lot of sub packages and the Packages section gets really tall? Since there'd be nothing below

                                              1. KajMagnus @KajMagnus2019-08-13 18:36:02.708Zreplies tochrscheuer:

                                                Is this still the definition for slugs?

                                                Yes. I'm not sure about foreign langage chars, e.g. Chinese? But maybe for now, non-ASCII chars can be forbidden, ... until after a bit research. I'm thinking only [a-z0-9-] is safer, for a start.

                                                Looking at this for a slugify function. Do you see any problems with it?

                                                I had a not-thorough look ( + I'm tired :- )), and I think it looks fine, mainly because of this line: .replace(/[^\w\-]+/g, '') which sort of deletes all "weird" things ... since \w = [A-Za-z0-9_] says MDN. B.t.w I have a slugify function in Talkyard, which converts titles to slugs. Here it is:

                                                https://github.com/debiki/talkyard/blob/dd46d893a2d85dfaaf0d89fa0c599bf94aadb35c/client/third-party/non-angular-slugify.js

                                                Seems there's a bug in my slugify — it allows pure numeric category slugs. Hmm your slugify accepts _, Talkyard's doesn't. Maybe it's better if I change Talkyard and allow _ in slugs.

                                                (Talkyard's slugify is based on an AngularJS plugin or something. I was using AngularJS long ago before moving to React.)

                                                1. C
                                                  Christian Scheuer @chrscheuer2019-08-13 18:39:30.253Zreplies toKajMagnus:

                                                  Maybe a max height, + a button "Click to show" ? I could look into this a bit later, after the iframe SSO redrect, and rate limiting issue, and returning the category url path.

                                                  Yea that sounds great. Haha yea this did turn up quite a list of follow up items :)
                                                  For now we're completely good with the current solution, everything seems to work as expected.

                                                  I had to hard program the URLs around in our site and since we're caching them and prerendering the URLs to the site's static content (which means Google is also picking it up) we'll likely experience a little down time on forum links once you make the change. We may also have customers bookmarking or deep linking to the current URLs. So maybe before the URLs change it would make sense to look at redirects (history). We can live without it, but of course it would make the user experience a little better (and make sure we don't take SEO hits). Implementing redirects would be more important to us than returning URLs in the API.

                                                  Another approach could be to place the Packages category last, so it'll be at the bottom of the page. Then maybe doesn't matter if there're a lot of sub packages and the Packages section gets really tall? Since there'd be nothing below

                                                  Yea I'll look into that if we get an explosion of packages. But actually I prefer it to be up at top for the moment, since we primarily want users to use either the "How to" section or the "Packages" section, so hiding the content below is not a big issue for now - just something to think about in the future.
                                                  I think your max height + "click to show" sounds like a good solution and I agree with your prioritization to take this task last :)

                                                  1. C
                                                    Christian Scheuer @chrscheuer2019-08-14 03:37:44.772Zreplies tochrscheuer:

                                                    I just realized something.. Since you were talking about the slugs right now being global (not scoped).
                                                    Is there any potential for us to accidentally hit slugs that were created by talkyard in other categories, pages or posts (not synced from external)? We of course guarantee the slugs we use are unique, but if you just have a single index for everything, won't we risk accidentally using slugs that were already used by others?

                                                    1. KajMagnus @KajMagnus2019-08-14 05:36:38.618Zreplies tochrscheuer:

                                                      There're just a few default categories and slugs, like: staff, support, ideas and general or uncategorized. I'm thinking there're no SoundFl

                                                      click to show
                                                      1. C
                                                        Christian Scheuer @chrscheuer2019-08-14 06:05:29.941Zreplies toKajMagnus:

                                                        There're just a few default categories and slugs, like: staff, support, ideas and general or uncategorized. I'm thinking there're no SoundFl

                                                        click to show
                                                        1. C
                                                          Christian Scheuer @chrscheuer2019-08-14 06:09:04.900Zreplies tochrscheuer:

                                                          But yes just to be clear - this is more of a theoretical problem than a practical one at this point. If it's only the few default ones we ca

                                                          click to show
                                                          1. C
                                                            Christian Scheuer @chrscheuer2019-08-14 06:10:52.404Zreplies tochrscheuer:

                                                            I had a not-thorough look ( + I'm tired :- )), and I think it looks fine, mainly because of this line: .replace(/[^\w-]+/g, '') which sort o

                                                            click to show
                                                            1. KajMagnus @KajMagnus2019-08-16 04:30:43.095Z

                                                              @chrscheuer

                                                              click to show
                                                              1. C
                                                                Christian Scheuer @chrscheuer2019-08-16 18:53:58.677Zreplies toKajMagnus:

                                                                Ok, so I'll implement category redirects from old names to the current name, before moving sub category url paths to [below their parent category's url path].

                                                                Sounds great!

                                                                What if, when one upserts a category, if one specifies a title only, but no slug, then Talkyard can auto generate the slug?

                                                                This would be great. And I trust when Talkyard auto-generates it would always be valid, right? Meaning if there's a collision, it itself handles incrementing a counter at the end?

                                                                So, after having upserted a category, you can send the browser to:
                                                                serverOrigin + responseJson.categories[0].urlPaths.activeTopics.

                                                                Sounds good!

                                                                1. C
                                                                  Christian Scheuer @chrscheuer2019-08-25 09:11:01.219Zreplies toKajMagnus:

                                                                  Great stuff!
                                                                  I've been offline for a week, and will be for another, so won't get a look at this until early September. We're launching SoundFlow 3 on September 5 so right now we're very zoned in on the launch process.

                                                                  Thanks for asking my opinion about the order.

                                                                  I would say the order in your items right now makes sense (the first 3 ones are in the proper dependency order IMO), except for the iframe redirect problem which I would address earlier (it feels more like a bug where the rest is more feature work).

                                                                  One thing I'm not sure if you addressed yet is the TY slug generation if slug is null/undefined. I would probably place that somewhere around the top of the list (right after fixing the iframe problem).