No internet connection
  1. Home
  2. Issues

How to manage the posted attachments in my own talkyard server?

By liliming @liliming1976
    2021-07-30 02:08:27.441Z

    I have a talkyard environment in my own server.
    Now, the Disk usage is too high , so i need to delete some attachments to free the disk space, but I dont konw how to do that.
    Anyone can help me ?

    • 14 replies
    1. KajMagnus @KajMagnus2021-08-02 04:24:28.816Z2021-08-02 04:31:17.269Z

      How does the OS file system look? Is there one single partition, or is maybe /opt/ or /var/ on separate partitions?

      The attachments, called "uploaded files" or "uploads" in Talkyard, are stored in the file system.

      • The ones in use, are at /opt/talkyard/data/uploads/
      • Backups of uploads can take lots of space. They are here: /opt/talkyard-backups/archives/*-uploads-up-to-incl-*, you can see the upload backup directories if you do this:
        ls -1d /opt/talkyard-backups/archives/*-uploads-up-to-incl-*

      You can check the sizes of the uploads in use, and the backups, like so:

      # view upload dir sizes:
      du -hs /opt/talkyard/data/uploads/    # the ones in use
      du -hs /opt/talkyard-backups/archives/*-uploads-up-to-incl-*   # backups
      

      Here's a nice way to find out more exactly which files are large:

      apt install ncdu
      ncdu /opt/talkyard/data/uploads/public/
      

      You can click the arrows Up and Down on the keyboard, and Enter, to navigate between the directories. The one at the top with the most ##### is the directory with the largest size.

      The directories are numbers from 0 to 9 — and the smallest files are stored in 0, and the larger the file, the larger the directory number. If you want to delete all really large uploads, you could simply delete, say, directories 6, 7, 8, 9. But first you might want to have a look and see what's inside.

      1. In reply toliliming1976:

        Are you sure it's the uploads that use lots of disk space? Can I ask, what types of files do you upload? (maybe large images or videos?)

        In any case, what if you copy-paste the output from this:

        du -hs /opt/talkyard/data/*
        du -hs /var/log/
        du -hs /var/log/nginx/*
        du -hs /var/log/talkyard/*
        du -hs /opt/talkyard-backups/archives/*-uploads-up-to-incl-*
        du -hs /opt/talkyard-backups/archives/*-daily-* | tail -n15
        

        You could send it in a private message if you want, or maybe edit the output and remove the hostname from the backup file names (if you want to).

        1. Lliliming @liliming1976
            2021-08-02 06:26:07.921Z

            Thanks for ur helps.
            My OS is Centos7.0 , and Talkyard server installed by docker container.
            Actually, I just want to know if there is a web administration page for upload files?
            Or, I can only manage the upload files by command line?

            1. For now, you'll need to use the command line. The sample commands help you find the large files, although their names are in hex

          • In reply toliliming1976:

            Do you think that after a while, people will have uploaded lots of files again, so that the disk usage is too high again?

            Then, if it's not desirable to increase the disk space, maybe it could be helpful if Talkyard could automatically delete uploaded files larger than, say, 10 MB, after two weeks, or something like that. And to exclude such files from backups.
            And if there was an info message, if uploading such large files, telling the user that they're "too large" and will get auto deleted after X days to free disk space.

            Also, I suppose it'd be helpful with a list of uploaded files, in the Admin Area, where you could sort by file size, and see the original file names, and delete files.

            1. Lliliming @liliming1976
                2021-08-02 06:50:03.840Z

                I could not find any page about uploaded files list in Admin Area.

                1. Right now there is none (I was thinking about creating one)

                  1. Lliliming @liliming1976
                      2021-08-02 07:48:45.551Z

                      Got, thank you very much.
                      Look forward to see this function.

                      1. Ok, how short of disk are you? Using the Linux terminal, until I've implemented this, how complicated / frustrating would that be on a scale from 0 to 10?

                        1. Lliliming @liliming1976
                            2021-08-13 00:45:56.419Z

                            Problem resolved. Now the disk space is enough, you can do it in your plan.
                            Thanks.

                            1. Ok, thanks for the update

                    • In reply toKajMagnus:
                      Lliliming @liliming1976
                        2021-08-02 07:27:57.423Z

                        Sadly,I know nothing about coding in Talkyard.
                        I just need an Existing management page to help me do that.

                        1. In reply toKajMagnus:
                          @Mr.Nobody
                            2021-08-18 18:40:15.999Z

                            it could be helpful if Talkyard could automatically delete uploaded files larger than, say, 10 MB, after two weeks, or something like that. And to exclude such files from backups.

                            I really like this idea! Also would be nice to auto-delete attachments that are not being used in any post anymore (e.G. if someone uploads, then delete the topic, is the attachment deleted with it?)

                            Also, I suppose it'd be helpful with a list of uploaded files, in the Admin Area, where you could sort by file size, and see the original file names, and delete files.

                            If this is ever added, please add the topic/comment's age to that file browser so that it is easier to delete attachments from:

                            • content older than X days;
                            • content without activity (replies and/or likes) in more than X days;

                            I would not want to delete attachments from content that is being used in important content.

                            1. auto-delete attachments that are not being used in any post anymore

                              Yes, this is how it's going to work (some time later). There'll be some "timeout" — e.g. if one deletes a page with an image, then, if one doesn't undelete the page within, say, 1 month, the image gets deleted. (This "timeout" could be configurable, per site.)

                              And one will be able to delete uploaded files from disk, and Talkyard will notice (after a while).

                              please add the topic/comment's age

                              Good point. I had in mind to include the original name of the uploaded file, but it didn't occur to me that the page / reply where it occurs, is helpful as well. So, the page title, and the first words / first sentence in the post (Orig Post or reply) where it appears, too.

                              And number of likes in total, and the last X days, that's a good idea too. And number of page views maybe

                              I would not want to delete attachments from content that is being used

                              I think Imgur deletes images that no one looked at the last 6 months — there could be some admin-toggle-on-off functionality like that in Talkyard.

                          • Progress
                            with handling this problem
                          • @KajMagnus marked this topic as Planned 2021-08-27 08:44:35.878Z.