osmarks/gotosocial
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

[feature] add cleanup of old remote status threads (#4808)

Browse Source 
- adds a new cleaner package task to cleanup old remote status threads, i.e. those older than a given date that have zero local replies, zero local favourites and no boosts of any of the constituents.
- moves a bunch of status deletion logic of related models to the database
- improves status related model deletion performance by adding multi delete queries
- improves some query performance by removing step of converting ID string slices to accountIDValue slices by using some "unsafe" memory casting (with appropriate compile-time checks in place)
- adds support for deprecating + renaming configuration fields
- adds new "longer" duration and cron expression types for our job scheduling configuration (+ deprecates the configuration fields where replacing)

in a later PR i'll add support for calling it from the frontend as an admin panel action type dealio

Reviewed-on: https://codeberg.org/superseriousbusiness/gotosocial/pulls/4808
This commit is contained in:
kim
2026-05-15 14:52:28 +02:00
committed by kim
parent 12e6d0c76b
commit a2266f2818
80 changed files with 5611 additions and 1250 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TODO.md
+17 -17
View File
@@ -2,21 +2,21 @@
things noted down by the maintainers that could do with being done!
## database chores
- tidy up db.GetAccountStatuses() (separate functions perhaps?)
- update remaining database queries / API endpoints to use paging.Page{}
- add migrations to convert edits, accounts (, etc) flags to bitsets
- drop unnecessary 'updated_at' columns
- replace 'created_at' columns with time parsed from ULIDs (where possible)
- replace ULID columns with binary representation
- replace "upsert" queries with more performant alternatives (these do a lot of runtime logic which could be determined ahead of time)
## database
- [chore] tidy up db.GetAccountStatuses() (separate functions perhaps?)
- [chore] update remaining database queries / API endpoints to use paging.Page{}
- [space] add migrations to convert edits, accounts (, etc) flags to bitsets
- [space] drop unnecessary 'updated_at' columns
- [space] replace 'created_at' columns with time parsed from ULIDs (where possible)
- [performance] replace ULID columns with binary representation
- [performance] replace "upsert" queries with more performant alternatives (these do a lot of runtime logic which could be determined ahead of time)
- [performance] add multi-delete queries for mentions, statuses (+boosts), etc
## miscellaneous chores
- update to Go 1.25 (we're now being held back from dependency updates)
- finish code commenting where missing (search for '// (\w+\b)?...')
- move away from using Gin, they're all-in on "AI", blegh
## performance
- kim: update ffmpreg to use ncruces/wasm2go (?)
- kim: write alternative bundb serializing backend (?)
- kim: fork bundb to support not always going through "database/sql" (?)
## miscellaneous
- [chore] finish code commenting where missing (search for '// (\w+\b)?...')
- [chore] move away from using Gin, they're all-in on "AI", blegh
- [chore] deinterface the database somehow (where possible given dependency cycling 😭), have a single DB type so all bundb/*.go can access all other internal funcs
- kim: [chore/docs] update support matrix given go-sqlite3 dropped wazero usage, (potentially) remove modernc.org/sqlite dependency
- kim: [supported platforms] update ffmpreg to use ncruces/wasm2go (?)
- kim: [performance] write alternative bundb serializing backend (?)
- kim: [performance] fork bundb to support not always going through "database/sql" (?)
cmd/gotosocial/action/admin/media/prune/all.go → cmd/gotosocial/action/admin/media/prune_all.go
+13 -7
View File
@@ -15,10 +15,11 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package prune
package media
import (
"context"
"time"
"code.superseriousbusiness.org/gopkg/log"
"code.superseriousbusiness.org/gotosocial/cmd/gotosocial/action"
@@ -27,10 +28,11 @@ import (
)
// check function conformance.
var _ action.GTSAction = All
var _ action.GTSAction = PruneAll
// PruneAll performs all media clean actions
func PruneAll(ctx context.Context) error {
// All performs all media clean actions
func All(ctx context.Context) error {
// Setup pruning utilities.
prune, err := setupPrune(ctx)
if err != nil {
@@ -49,11 +51,15 @@ func All(ctx context.Context) error {
ctx = gtscontext.SetDryRun(ctx)
}
days := config.GetMediaRemoteCacheDays()
// Current time.
now := time.Now()
// Get media maximum remote cache duration.
dur := config.GetMediaRemoteCacheDuration()
// Perform the actual pruning with logging.
prune.cleaner.Media().AllAndFix(ctx, days)
prune.cleaner.Emoji().AllAndFix(ctx, days)
prune.cleaner.Media().AllAndFix(ctx, now, dur)
prune.cleaner.Emoji().AllAndFix(ctx, now, dur)
return nil
}
cmd/gotosocial/action/admin/media/prune/common.go → cmd/gotosocial/action/admin/media/prune_common.go
+22 -29
View File
@@ -15,56 +15,51 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package prune
package media
import (
"context"
"fmt"
"code.superseriousbusiness.org/gotosocial/internal/cleaner"
"code.superseriousbusiness.org/gotosocial/internal/db"
"code.superseriousbusiness.org/gotosocial/internal/db/bundb"
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
"code.superseriousbusiness.org/gotosocial/internal/media"
"code.superseriousbusiness.org/gotosocial/internal/state"
gtsstorage "code.superseriousbusiness.org/gotosocial/internal/storage"
)
type prune struct {
dbService db.DB
storage *gtsstorage.Driver
manager *media.Manager
cleaner *cleaner.Cleaner
state *state.State
manager *media.Manager
cleaner *cleaner.Cleaner
state *state.State
}
func setupPrune(ctx context.Context) (*prune, error) {
var state state.State
state.Caches.Init()
if err := state.Caches.Start(); err != nil {
err := state.Caches.Start()
if err != nil {
return nil, fmt.Errorf("error starting caches: %w", err)
}
// Set state DB connection.
// Don't need Actions for this.
state.DB, err = bundb.NewBunDBService(ctx, &state)
if err != nil {
return nil, fmt.Errorf("error starting database: %w", err)
}
// Scheduler is required for the
// cleaner, but no other workers
// are needed for this CLI action.
state.Workers.StartScheduler()
// Set state DB connection.
// Don't need Actions for this.
dbService, err := bundb.NewBunDBService(ctx, &state)
if err != nil {
return nil, fmt.Errorf("error creating dbservice: %w", err)
}
state.DB = dbService
//nolint:contextcheck
storage, err := gtsstorage.AutoConfig()
state.Storage, err = gtsstorage.AutoConfig()
if err != nil {
return nil, fmt.Errorf("error creating storage backend: %w", err)
}
state.Storage = storage
//nolint:contextcheck
manager := media.NewManager(&state)
@@ -73,23 +68,21 @@ func setupPrune(ctx context.Context) (*prune, error) {
cleaner := cleaner.New(&state)
return &prune{
dbService: dbService,
storage: storage,
manager: manager,
cleaner: cleaner,
state: &state,
manager: manager,
cleaner: cleaner,
state: &state,
}, nil
}
func (p *prune) shutdown() error {
errs := gtserror.NewMultiError(2)
var err error
if err := p.dbService.Close(); err != nil {
errs.Appendf("error stopping database: %w", err)
if err = p.state.DB.Close(); err != nil {
err = fmt.Errorf("error stopping database: %w", err)
}
p.state.Workers.Scheduler.Stop()
p.state.Caches.Stop()
return errs.Combine()
return err
}
cmd/gotosocial/action/admin/media/prune/orphaned.go → cmd/gotosocial/action/admin/media/prune_orphaned.go
+6 -5
View File
@@ -15,7 +15,7 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package prune
package media
import (
"context"
@@ -27,10 +27,11 @@ import (
)
// check function conformance.
var _ action.GTSAction = Orphaned
var _ action.GTSAction = PruneOrphaned
// PruneOrphaned prunes orphaned media from storage.
func PruneOrphaned(ctx context.Context) error {
// Orphaned prunes orphaned media from storage.
func Orphaned(ctx context.Context) error {
// Setup pruning utilities.
prune, err := setupPrune(ctx)
if err != nil {
@@ -53,7 +54,7 @@ func Orphaned(ctx context.Context) error {
prune.cleaner.Media().LogPruneOrphaned(ctx)
// Perform a cleanup of storage (for removed local dirs).
if err := prune.storage.Storage.Clean(ctx); err != nil {
if err := prune.state.Storage.Storage.Clean(ctx); err != nil {
log.Error(ctx, "error cleaning storage: %v", err)
}
cmd/gotosocial/action/admin/media/prune/remote.go → cmd/gotosocial/action/admin/media/prune_remote.go
+9 -7
View File
@@ -15,7 +15,7 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package prune
package media
import (
"context"
@@ -28,10 +28,11 @@ import (
)
// check function conformance.
var _ action.GTSAction = Remote
var _ action.GTSAction = PruneRemote
// PruneRemote prunes old and/or unused remote media.
func PruneRemote(ctx context.Context) error {
// Remote prunes old and/or unused remote media.
func Remote(ctx context.Context) error {
// Setup pruning utilities.
prune, err := setupPrune(ctx)
if err != nil {
@@ -50,14 +51,15 @@ func Remote(ctx context.Context) error {
ctx = gtscontext.SetDryRun(ctx)
}
t := time.Now().Add(-24 * time.Hour * time.Duration(config.GetMediaRemoteCacheDays()))
// Get media remote cache duration as an "olderThan" time.
olderThan := config.GetMediaRemoteCacheOlderThanTime(time.Now())
// Perform the actual pruning with logging.
prune.cleaner.Media().LogPruneUnused(ctx)
prune.cleaner.Media().LogUncacheRemote(ctx, t)
prune.cleaner.Media().LogUncacheRemote(ctx, olderThan)
// Perform a cleanup of storage (for removed local dirs).
if err := prune.storage.Storage.Clean(ctx); err != nil {
if err := prune.state.Storage.Storage.Clean(ctx); err != nil {
log.Error(ctx, "error cleaning storage: %v", err)
}
cmd/gotosocial/action/admin/statuses/prune.go
+103
View File
@@ -0,0 +1,103 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package statuses
import (
"context"
"fmt"
"time"
"code.superseriousbusiness.org/gopkg/log"
"code.superseriousbusiness.org/gotosocial/cmd/gotosocial/action"
"code.superseriousbusiness.org/gotosocial/internal/cleaner"
"code.superseriousbusiness.org/gotosocial/internal/config"
"code.superseriousbusiness.org/gotosocial/internal/db/bundb"
"code.superseriousbusiness.org/gotosocial/internal/state"
)
// check function conformance.
var _ action.GTSAction = PruneLeafStubs
var _ action.GTSAction = PruneOldRemote
func PruneLeafStubs(ctx context.Context) error {
return do(ctx, func(p *pruner) error {
olderThan := time.Now().Add(-7 * 24 * time.Hour)
p.cleaner.Status().LogPruneLeafStubs(ctx, olderThan, 0)
return nil
})
}
func PruneOldRemote(ctx context.Context) error {
return do(ctx, func(p *pruner) error {
_, dur := config.GetStatusesCleanupRemoteOlderThan().Duration()
if dur == 0 {
return fmt.Errorf("%s = 0, no statuses to cleanup", config.StatusesCleanupRemoteOlderThanFlag)
}
olderThan := time.Now().Add(-dur)
p.cleaner.Status().LogPruneOldRemote(ctx, olderThan, 0)
return nil
})
}
type pruner struct {
state *state.State
cleaner *cleaner.Cleaner
}
func do(ctx context.Context, do func(*pruner) error) error {
var state state.State
defer func() {
if state.DB != nil {
// Lastly, if database service was started,
// ensure it gets closed now all else stopped.
if err := state.DB.Close(); err != nil {
log.Errorf(ctx, "error stopping database: %v", err)
}
}
// Finally reached end of shutdown.
log.Info(ctx, "done! exiting...")
}()
// Initialize caches
state.Caches.Init()
err := state.Caches.Start()
if err != nil {
return fmt.Errorf("error starting caches: %w", err)
}
log.Info(ctx, "starting db service...")
// Open conn to database now caches are started.
state.DB, err = bundb.NewBunDBService(ctx, &state)
if err != nil {
return fmt.Errorf("error creating dbservice: %w", err)
}
// Create common receiver type.
cleaner := cleaner.New(&state)
pruner := &pruner{&state, cleaner}
// Perform provided CLI function.
if err := do(pruner); err != nil {
return fmt.Errorf("error performing action: %w", err)
}
return nil
}
cmd/gotosocial/action/server/server.go
+4 -6
View File
@@ -606,12 +606,10 @@ func Start(ctx context.Context) error {
return fmt.Errorf("error filling worker queues: %w", err)
}
// catch shutdown signals from the operating system
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
sig := <-sigs // block until signal received
log.Infof(ctx, "received signal %s, shutting down", sig)
// Block until
// signal given.
<-ctx.Done()
log.Info(ctx, "received signal, shutting down")
return nil
}
cmd/gotosocial/admin.go
+101 -80
View File
@@ -20,7 +20,7 @@ package main
import (
"code.superseriousbusiness.org/gotosocial/cmd/gotosocial/action/admin/account"
"code.superseriousbusiness.org/gotosocial/cmd/gotosocial/action/admin/media"
"code.superseriousbusiness.org/gotosocial/cmd/gotosocial/action/admin/media/prune"
"code.superseriousbusiness.org/gotosocial/cmd/gotosocial/action/admin/statuses"
"code.superseriousbusiness.org/gotosocial/cmd/gotosocial/action/admin/trans"
"code.superseriousbusiness.org/gotosocial/internal/config"
"github.com/spf13/cobra"
@@ -36,13 +36,12 @@ func adminCommands() *cobra.Command {
ADMIN ACCOUNT COMMANDS
*/
adminAccountCmd := &cobra.Command{
adminAccountCmd := add(adminCmd, &cobra.Command{
Use: "account",
Short: "admin commands related to local (this instance) accounts",
}
config.AddAdminAccount(adminAccountCmd)
})
adminAccountCreateCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "create",
Short: "create a new local account",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -51,11 +50,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Create)
},
}
config.AddAdminAccountCreate(adminAccountCreateCmd)
adminAccountCmd.AddCommand(adminAccountCreateCmd)
},
config.AddAdminAccountCreate,
)
adminAccountListCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "list",
Short: "list all existing local accounts",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -64,10 +63,9 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.List)
},
}
adminAccountCmd.AddCommand(adminAccountListCmd)
})
adminAccountConfirmCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "confirm",
Short: "confirm an existing local account manually, thereby skipping email confirmation",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -76,11 +74,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Confirm)
},
}
config.AddAdminAccount(adminAccountConfirmCmd)
adminAccountCmd.AddCommand(adminAccountConfirmCmd)
},
config.AddAdminAccount,
)
adminAccountPromoteCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "promote",
Short: "promote a local account to admin",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -89,11 +87,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Promote)
},
}
config.AddAdminAccount(adminAccountPromoteCmd)
adminAccountCmd.AddCommand(adminAccountPromoteCmd)
},
config.AddAdminAccount,
)
adminAccountDemoteCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "demote",
Short: "demote a local account from admin to normal user",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -102,11 +100,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Demote)
},
}
config.AddAdminAccount(adminAccountDemoteCmd)
adminAccountCmd.AddCommand(adminAccountDemoteCmd)
},
config.AddAdminAccount,
)
adminAccountDisableCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "disable",
Short: "set 'disabled' to true on a local account to prevent it from signing in or posting etc, but don't delete anything",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -115,11 +113,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Disable)
},
}
config.AddAdminAccount(adminAccountDisableCmd)
adminAccountCmd.AddCommand(adminAccountDisableCmd)
},
config.AddAdminAccount,
)
adminAccountEnableCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "enable",
Short: "undo a previous disable command by setting 'disabled' to false on a local account",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -128,11 +126,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Enable)
},
}
config.AddAdminAccount(adminAccountEnableCmd)
adminAccountCmd.AddCommand(adminAccountEnableCmd)
},
config.AddAdminAccount,
)
adminAccountPasswordCmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "password",
Short: "set a new password for the given local account",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -141,12 +139,12 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Password)
},
}
config.AddAdminAccount(adminAccountPasswordCmd)
config.AddAdminAccountPassword(adminAccountPasswordCmd)
adminAccountCmd.AddCommand(adminAccountPasswordCmd)
},
config.AddAdminAccount,
config.AddAdminAccountPassword,
)
adminAccountDisable2FACmd := &cobra.Command{
_ = add(adminAccountCmd, &cobra.Command{
Use: "disable-2fa",
Short: "disable 2fa for the given local account",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -155,17 +153,15 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), account.Disable2FA)
},
}
config.AddAdminAccount(adminAccountDisable2FACmd)
adminAccountCmd.AddCommand(adminAccountDisable2FACmd)
adminCmd.AddCommand(adminAccountCmd)
},
config.AddAdminAccount,
)
/*
ADMIN IMPORT/EXPORT COMMANDS
*/
adminExportCmd := &cobra.Command{
_ = add(adminCmd, &cobra.Command{
Use: "export",
Short: "export data from the database to file at the given path",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -174,11 +170,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), trans.Export)
},
}
config.AddAdminTrans(adminExportCmd)
adminCmd.AddCommand(adminExportCmd)
},
config.AddAdminTrans,
)
adminImportCmd := &cobra.Command{
_ = add(adminCmd, &cobra.Command{
Use: "import",
Short: "import data from a file into the database",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -187,24 +183,24 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), trans.Import)
},
}
config.AddAdminTrans(adminImportCmd)
adminCmd.AddCommand(adminImportCmd)
},
config.AddAdminTrans,
)
/*
ADMIN MEDIA COMMANDS
*/
adminMediaCmd := &cobra.Command{
adminMediaCmd := add(adminCmd, &cobra.Command{
Use: "media",
Short: "admin commands related to stored media / emojis",
}
})
/*
ADMIN MEDIA LIST COMMANDS
*/
adminMediaListAttachmentsCmd := &cobra.Command{
_ = add(adminMediaCmd, &cobra.Command{
Use: "list-attachments",
Short: "list local, remote, or all attachments",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -213,11 +209,11 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), media.ListAttachments)
},
}
config.AddAdminMediaList(adminMediaListAttachmentsCmd)
adminMediaCmd.AddCommand(adminMediaListAttachmentsCmd)
},
config.AddAdminMediaList,
)
adminMediaListEmojisLocalCmd := &cobra.Command{
_ = add(adminMediaCmd, &cobra.Command{
Use: "list-emojis",
Short: "list local, remote, or all emojis",
PreRunE: func(cmd *cobra.Command, args []string) error {
@@ -226,60 +222,85 @@ func adminCommands() *cobra.Command {
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), media.ListEmojis)
},
}
config.AddAdminMediaList(adminMediaListEmojisLocalCmd)
adminMediaCmd.AddCommand(adminMediaListEmojisLocalCmd)
},
config.AddAdminMediaList,
)
/*
ADMIN MEDIA PRUNE COMMANDS
*/
adminMediaPruneCmd := &cobra.Command{
adminMediaPruneCmd := add(adminMediaCmd, &cobra.Command{
Use: "prune",
Short: "admin commands for pruning media from storage",
}
})
adminMediaPruneOrphanedCmd := &cobra.Command{
_ = add(adminMediaPruneCmd, &cobra.Command{
Use: "orphaned",
Short: "prune orphaned media from storage",
PreRunE: func(cmd *cobra.Command, args []string) error {
return preRun(preRunArgs{cmd: cmd})
},
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), prune.Orphaned)
return run(cmd.Context(), media.PruneOrphaned)
},
}
config.AddAdminMediaPrune(adminMediaPruneOrphanedCmd)
adminMediaPruneCmd.AddCommand(adminMediaPruneOrphanedCmd)
},
config.AddAdminMediaPrune,
)
adminMediaPruneRemoteCmd := &cobra.Command{
_ = add(adminMediaPruneCmd, &cobra.Command{
Use: "remote",
Short: "prune unused / stale media from storage, older than given number of days",
Short: "prune unused / stale media from storage, older than given duration",
PreRunE: func(cmd *cobra.Command, args []string) error {
return preRun(preRunArgs{cmd: cmd})
},
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), prune.Remote)
return run(cmd.Context(), media.PruneRemote)
},
}
config.AddAdminMediaPrune(adminMediaPruneRemoteCmd)
adminMediaPruneCmd.AddCommand(adminMediaPruneRemoteCmd)
},
config.AddAdminMediaPrune,
)
adminMediaPruneAllCmd := &cobra.Command{
_ = add(adminMediaPruneCmd, &cobra.Command{
Use: "all",
Short: "perform all media and emoji prune / cleaning commands",
PreRunE: func(cmd *cobra.Command, args []string) error {
return preRun(preRunArgs{cmd: cmd})
},
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), prune.All)
return run(cmd.Context(), media.PruneAll)
},
}
config.AddAdminMediaPrune(adminMediaPruneAllCmd)
adminMediaPruneCmd.AddCommand(adminMediaPruneAllCmd)
},
config.AddAdminMediaPrune,
)
adminMediaCmd.AddCommand(adminMediaPruneCmd)
/*
ADMIN STATUS COMMANDS
*/
adminCmd.AddCommand(adminMediaCmd)
adminStatusesCmd := add(adminCmd, &cobra.Command{
Use: "statuses",
Short: "admin commands related to stored statuses",
})
/*
ADMIN STATUS PRUNE COMMANDS
*/
adminStatusesPruneCmd := add(adminStatusesCmd, &cobra.Command{
Use: "prune",
Short: "admin commands for pruning statuses from the database",
})
_ = add(adminStatusesPruneCmd, &cobra.Command{
Use: "remote",
Short: "prune old, locally-not-interacted-with remote status threads from the database",
PreRunE: func(cmd *cobra.Command, args []string) error {
return preRun(preRunArgs{cmd: cmd})
},
RunE: func(cmd *cobra.Command, args []string) error {
return run(cmd.Context(), statuses.PruneOldRemote)
},
})
return adminCmd
}
cmd/gotosocial/common.go
+9
View File
@@ -27,6 +27,15 @@ import (
"github.com/spf13/cobra"
)
// add will add 'next' to 'base' command, returning 'next' after applying any 'flagFns'.
func add(base, next *cobra.Command, flagFns ...func(*cobra.Command)) *cobra.Command {
base.AddCommand(next)
for _, flagFn := range flagFns {
flagFn(next)
}
return next
}
type preRunArgs struct {
cmd *cobra.Command
skipValidation bool
cmd/gotosocial/main.go
+23 -5
View File
@@ -18,10 +18,13 @@
package main
import (
"log"
"context"
"fmt"
"os"
"os/signal"
godebug "runtime/debug"
"strings"
"syscall"
_ "code.superseriousbusiness.org/gotosocial/docs"
"code.superseriousbusiness.org/gotosocial/internal/config"
@@ -62,18 +65,33 @@ func main() {
if testrigCmd := testrigCommands(); testrigCmd != nil {
rootCmd.AddCommand(testrigCmd)
} else if len(os.Args) > 1 && os.Args[1] == "testrig" {
log.Fatal("gotosocial must be built and run with the DEBUG enviroment variable set to enable and access testrig")
panic("gotosocial must be built and run with the DEBUG enviroment variable set to enable and access testrig")
}
// Run the prepared root command.
if err := rootCmd.Execute(); err != nil {
log.Fatalf("error executing command: %s", err)
// Start with background ctx.
ctx := context.Background()
// Setup context to cancel on signal.
ctx, cncl := signal.NotifyContext(ctx,
syscall.SIGTERM,
syscall.SIGKILL,
syscall.SIGINT,
)
// Cancel ctx
// on return.
defer cncl()
// Run the prepared root command with notify context.
if err := rootCmd.ExecuteContext(ctx); err != nil {
panic(fmt.Sprintf("error executing command: %v", err))
}
}
// version will build a version string from binary's stored build information.
// It is SemVer-compatible so long as Version is SemVer-compatible.
func version() string {
// Read build information from binary
build, ok := godebug.ReadBuildInfo()
if !ok {
docs/api/swagger.yaml
+3 -3
View File
@@ -9281,13 +9281,13 @@ paths:
operationId: mediaCleanup
parameters:
- description: |-
Number of days of remote media to keep.
Integer number of days, or duration string, of duration of remote media to keep.
If value is not specified, the value of media-remote-cache-days in the server config will be used.
format: int64
in: query
name: remote_cache_days
type: integer
type: string
x-go-name: RemoteCacheDays
x-go-type: code.superseriousbusiness.org/gotosocial/internal/api/model.DurationOrDays
produces:
- application/json
responses:
example/config.yaml
+27 -32
View File
@@ -524,15 +524,11 @@ instance-deliver-to-shared-inboxes: true
# Default: false
instance-inject-mastodon-version: false
# String. 24hr time of day formatted as hh:mm.
# Examples: ["14:30", "00:00", "04:00"]
# Default: "23:00" (11pm).
instance-subscriptions-process-from: "23:00"
# Duration. Period between subscription updates.
# Examples: ["24h", "72h", "12h"]
# Default: "24h" (once per day).
instance-subscriptions-process-every: "24h"
# Cron expression (see https://crontab.guru/ for help).
#
# Examples: ["0 0 * * *", "30 0 * * *", "0 0 * * 0"]
# Default: "0 23 * * *" (at 23:00pm, every day)
instance-subscriptions-process-cron: "0 23 * * *"
# String. Allows you to customize if and how stats are served to
# crawlers at the /api/v1|v2/instance and /nodeinfo endpoints.
@@ -782,30 +778,17 @@ media-ffmpeg-pool-size: 1
# what these settings do, with some customization examples, see the docs:
# https://docs.gotosocial.org/en/latest/admin/media_caching#cleanup
# Int. Number of days to cache media from remote instances before
# they are removed from the cache. When remote media is removed from
# the cache, it is deleted from storage but the database entries for
# the media are kept so that it can be fetched again if requested by a user.
#
# If this is set to 0, then media from remote instances will be cached indefinitely.
#
# Examples: [30, 60, 7, 0]
# Default: 7
media-remote-cache-days: 7
# Integer duration.
#
# Examples: ["7 days", "1 week", "1 month"]
# Default: "7 days"
media-remote-cache-duration: "7 days"
# String. 24hr time of day formatted as hh:mm.
# Examples: ["14:30", "00:00", "04:00"]
# Default: "00:00" (midnight).
media-cleanup-from: "00:00"
# Duration. Period between media cleanup runs.
# More than once per 24h is not recommended
# is likely overkill. Setting this to something
# very low like once every 10 minutes will probably
# cause lag and possibly other issues.
# Examples: ["24h", "72h", "12h"]
# Default: "24h" (once per day).
media-cleanup-every: "24h"
# Cron expression (see https://crontab.guru/ for help).
#
# Examples: ["0 0 * * *", "30 0 * * *", "0 0 * * 0"]
# Default: "0 0 * * *" (at 00:00am, every day)
media-cleanup-cron: "0 0 * * *"
##########################
##### STORAGE CONFIG #####
@@ -960,6 +943,18 @@ statuses-poll-option-max-chars: 50
# Default: 6
statuses-media-max-files: 6
# Cron expression (see https://crontab.guru/ for help).
#
# Examples: ["0 0 * * *", "30 0 * * *", "0 0 * * 0"]
# Default: "0 0 * * *" (at 00:00am, every day)
statuses-cleanup-cron: "0 0 * * *"
# Integer duration.
#
# Examples: ["7 days", "1 week", "1 month"]
# Default: "0" (i.e. disabled)
statuses-cleanup-remote-older-than: "0"
# Int. Maximum number of statuses a user can schedule at time.
# Examples: [300]
# Default: 300
go.mod
+3 -1
View File
@@ -11,7 +11,7 @@ replace github.com/gin-gonic/gin => codeberg.org/superseriousbusiness/gin v1.11.
require (
code.superseriousbusiness.org/activity v1.19.0
code.superseriousbusiness.org/exif-terminator v0.11.2
code.superseriousbusiness.org/gopkg v0.0.0-20260314203815-181df26aa86e
code.superseriousbusiness.org/gopkg v0.0.0-20260503204153-b9100c6c75dd
code.superseriousbusiness.org/httpsig v1.5.0
code.superseriousbusiness.org/oauth2/v4 v4.5.4-0.20250812115401-3961e46a7384
codeberg.org/gruf/go-bytesize v1.0.4
@@ -26,6 +26,7 @@ require (
codeberg.org/gruf/go-iotools v0.0.0-20240710125620-934ae9c654cf
codeberg.org/gruf/go-kv/v2 v2.0.10
codeberg.org/gruf/go-list v0.0.0-20240425093752-494db03d641f
codeberg.org/gruf/go-longdur v0.1.3
codeberg.org/gruf/go-mempool v0.0.0-20251205182607-a05549c9a993
codeberg.org/gruf/go-mmap v0.0.0-20251111184116-345a42dab154
codeberg.org/gruf/go-mutexes v1.5.9
@@ -49,6 +50,7 @@ require (
github.com/google/uuid v1.6.0
github.com/gorilla/feeds v1.2.0
github.com/gorilla/websocket v1.5.3
github.com/hashicorp/cronexpr v1.1.3
github.com/jackc/pgx/v5 v5.9.2
github.com/k3a/html2text v1.4.0
github.com/klauspost/compress v1.18.4
go.sum
Generated
+6 -2
View File
@@ -6,8 +6,8 @@ code.superseriousbusiness.org/go-jpeg-image-structure/v2 v2.3.0 h1:r9uq8StaSHYKJ
code.superseriousbusiness.org/go-jpeg-image-structure/v2 v2.3.0/go.mod h1:IK1OlR6APjVB3E9tuYGvf0qXMrwP+TrzcHS5rf4wffQ=
code.superseriousbusiness.org/go-png-image-structure/v2 v2.3.0 h1:I512jiIeXDC4//2BeSPrRM2ZS4wpBKUaPeTPxakMNGA=
code.superseriousbusiness.org/go-png-image-structure/v2 v2.3.0/go.mod h1:SNHomXNW88o1pFfLHpD4KsCZLfcr4z5dm+xcX5SV10A=
code.superseriousbusiness.org/gopkg v0.0.0-20260314203815-181df26aa86e h1:57ICwitbFxGPMJZG34srw20oou4RtqCGIrggy5/vazI=
code.superseriousbusiness.org/gopkg v0.0.0-20260314203815-181df26aa86e/go.mod h1:PNtiv80R4b+z7HvnZbX69dAPmgRSDGts3SNcSC6Ti30=
code.superseriousbusiness.org/gopkg v0.0.0-20260503204153-b9100c6c75dd h1:p6oya9zNDO8dNQeVdoUSYX3gS0dF41wxpkh7T73gKX8=
code.superseriousbusiness.org/gopkg v0.0.0-20260503204153-b9100c6c75dd/go.mod h1:PNtiv80R4b+z7HvnZbX69dAPmgRSDGts3SNcSC6Ti30=
code.superseriousbusiness.org/httpsig v1.5.0 h1:jw/qc//yYWSoOYytTZXHvW7yh8kceCipNIBfUeXQghA=
code.superseriousbusiness.org/httpsig v1.5.0/go.mod h1:i2AKpj/WbA/o/UTvia9TAREzt0jP1AH3T1Uxjyhdzlw=
code.superseriousbusiness.org/oauth2/v4 v4.5.4-0.20250812115401-3961e46a7384 h1:eJzULGUyhHGk2DdQxX/jbH9FKZOyoIF90p3dzukCfLA=
@@ -38,6 +38,8 @@ codeberg.org/gruf/go-kv/v2 v2.0.10 h1:aNIg4UzZhSorcGpSPAF2kSPlOzW4wWloNarTIoK9GE
codeberg.org/gruf/go-kv/v2 v2.0.10/go.mod h1:diLoh5ZMJyCy5cRQuOMeYKMCxb9n/0V/6ec4z6uqtBc=
codeberg.org/gruf/go-list v0.0.0-20240425093752-494db03d641f h1:Ss6Z+vygy+jOGhj96d/GwsYYDd22QmIcH74zM7/nQkw=
codeberg.org/gruf/go-list v0.0.0-20240425093752-494db03d641f/go.mod h1:F9pl4h34iuVN7kucKam9fLwsItTc+9mmaKt7pNXRd/4=
codeberg.org/gruf/go-longdur v0.1.3 h1:DLKFdNKeZKitPMu05QDGN0o/AGUx6pKpaN5imifaVro=
codeberg.org/gruf/go-longdur v0.1.3/go.mod h1:YeKA3ci+KlRB7keMqQ26jO1XLiVMqzAGi31kx+WMOOM=
codeberg.org/gruf/go-loosy v0.0.0-20231007123304-bb910d1ab5c4 h1:IXwfoU7f2whT6+JKIKskNl/hBlmWmnF1vZd84Eb3cyA=
codeberg.org/gruf/go-loosy v0.0.0-20231007123304-bb910d1ab5c4/go.mod h1:fiO8HE1wjZCephcYmRRsVnNI/i0+mhy44Z5dQalS0rM=
codeberg.org/gruf/go-mangler/v2 v2.0.9 h1:Zb4YCVQxM48bhV3bPBd4DWk3/7VwWnO79EXyMHSw3eI=
@@ -294,6 +296,8 @@ github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aN
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0/go.mod h1:JfhWUomR1baixubs02l85lZYYOm7LV6om4ceouMv45c=
github.com/hashicorp/cronexpr v1.1.3 h1:rl5IkxXN2m681EfivTlccqIryzYJSXRGRNa0xeG7NA4=
github.com/hashicorp/cronexpr v1.1.3/go.mod h1:P4wA0KBl9C5q2hABiMO7cp6jcIg96CDh1Efb3g1PWA4=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/huandu/xstrings v1.5.0 h1:2ag3IFq9ZDANvthTwTiqSSZLjDc+BedvHPAp5tJy2TI=
internal/api/client/admin/mediacleanup.go
+4 -11
View File
@@ -18,7 +18,6 @@
package admin
import (
"errors"
"fmt"
"net/http"
@@ -107,23 +106,17 @@ func (m *Module) MediaCleanupPOSTHandler(c *gin.Context) {
return
}
// Normalize remoteCacheDays.
var remoteCacheDays int
if form.RemoteCacheDays == nil {
remoteCacheDays = config.GetMediaRemoteCacheDays()
} else if remoteCacheDays = *form.RemoteCacheDays; remoteCacheDays < 0 {
text := fmt.Sprintf("invalid value for remote_cache_days; value was %d, cannot be less than 0", remoteCacheDays)
apiutil.ErrorHandler(c, gtserror.NewErrorBadRequest(errors.New(text), text), m.processor.InstanceGetV1)
return
if form.RemoteCacheDays.Duration == 0 {
form.RemoteCacheDays.Duration = config.GetMediaRemoteCacheDuration()
}
if errWithCode := m.processor.Admin().MediaPrune(
c.Request.Context(),
remoteCacheDays,
form.RemoteCacheDays.Duration,
); errWithCode != nil {
apiutil.ErrorHandler(c, errWithCode, m.processor.InstanceGetV1)
return
}
apiutil.JSON(c, http.StatusOK, remoteCacheDays)
apiutil.JSON(c, http.StatusOK, form.RemoteCacheDays)
}
internal/api/client/admin/mediacleanup_test.go
+1 -1
View File
@@ -132,7 +132,7 @@ func (suite *MediaCleanupTestSuite) TestMediaCleanupNegative() {
if err != nil {
suite.FailNow(err.Error())
}
suite.Equal(`{"error":"Bad Request: invalid value for remote_cache_days; value was -10, cannot be less than 0"}`, string(b))
suite.Equal(`{"error":"Bad Request: invalid unit"}`, string(b))
}
func TestMediaCleanupTestSuite(t *testing.T) {
internal/api/model/admin.go
+2 -2
View File
@@ -193,9 +193,9 @@ type AdminActionResponse struct {
//
// swagger:parameters mediaCleanup
type MediaCleanupRequest struct {
// Number of days of remote media to keep.
// Integer number of days, or duration string, of duration of remote media to keep.
// If value is not specified, the value of media-remote-cache-days in the server config will be used.
RemoteCacheDays *int `form:"remote_cache_days" json:"remote_cache_days" xml:"remote_cache_days"`
RemoteCacheDays DurationOrDays `form:"remote_cache_days" json:"remote_cache_days" xml:"remote_cache_days"`
}
// MediaPurgeRequest models admin media purge parameters
internal/api/model/common.go
+96
View File
@@ -0,0 +1,96 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package model
import (
"errors"
"strconv"
"unsafe"
"codeberg.org/gruf/go-longdur"
)
// DurationOrDays wraps a longdur.Duration type to also
// support unmarshaling direct integers as day counts.
// The zero value is (un)marshaled as JSON null or empty.
//
// NOTE: this type largely exists as a transitionary type
// to handle our older API calls requiring integer number
// of days, while a "longdur" is much more flexible.
type DurationOrDays struct{ longdur.Duration }
// MarshalJSON: implements json.Marshaler{}.
func (d DurationOrDays) MarshalJSON() ([]byte, error) {
if d.Duration == 0 {
return []byte("null"), nil
}
return []byte("\"" + d.String() + "\""), nil
}
// UnmarshalJSON: implements json.Unmarshaler{}.
func (d *DurationOrDays) UnmarshalJSON(data []byte) error {
if len(data) >= 2 && data[0] == '"' && data[len(data)-1] == '"' {
data = data[1 : len(data)-1]
}
return d.unmarshalb(data)
}
// MarshalText: implements encoding.TextMarshaler{}.
func (d DurationOrDays) MarshalText() ([]byte, error) {
if d.Duration == 0 {
return []byte(""), nil
}
return d.Duration.MarshalText()
}
// UnmarshalText: implements encoding.TextUmarshaler{}.
func (d *DurationOrDays) UnmarshalText(text []byte) error {
return d.unmarshalb(text)
}
// UnmarshalParam: implements binding.BindUnmarshaler{}.
func (d *DurationOrDays) UnmarshalParam(param string) error {
return d.unmarshal(param)
}
// unmarshalb converts byte slice to string (with length check) and calls d.unmarshal().
func (d *DurationOrDays) unmarshalb(b []byte) error {
if len(b) == 0 {
return errors.New("invalid duration")
} else if string(b) == "null" {
d.Duration = 0
return nil
}
return d.unmarshal(unsafe.String(&b[0], len(b)))
}
// unmarshal attempts to unmarshal string as either integer, or string encoded duration.
func (d *DurationOrDays) unmarshal(str string) error {
// Initially, try to parse as an integer.
i, err := strconv.ParseUint(str, 10, 64)
if err == nil {
// Set number of days from integer provided.
d.Duration = longdur.Duration(i) * longdur.Day
return nil
}
// Parse this as a duration.
return d.Duration.Set(str)
}
internal/cache/timeline/status_map.go
Vendored
+1 -1
View File
@@ -320,7 +320,7 @@ func (s *lazyset) Add(key string) {
}
// clone is functionally similar to maps.Clone(),
// except a nil input with return initialized out.
// except a nil input will return initialized output.
func clone[T any](m map[string]T) map[string]T {
m2 := make(map[string]T, len(m))
for key, val := range m {
internal/cleaner/cleaner.go
+18 -58
View File
@@ -28,9 +28,11 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
"code.superseriousbusiness.org/gotosocial/internal/state"
"code.superseriousbusiness.org/gotosocial/internal/storage"
"codeberg.org/gruf/go-longdur"
)
const selectLimit = 50
const stamp = "Jan _2 2006 15:04:05"
type Cleaner struct{ state *state.State }
@@ -115,55 +117,29 @@ func (c *Cleaner) removeFiles(ctx context.Context, files ...string) {
// ScheduleJobs schedules cleaning
// jobs using configured parameters.
//
// Returns an error if `MediaCleanupFrom`
// is not a valid format (hh:mm:ss).
func (c *Cleaner) ScheduleJobs() error {
const hourMinute = "15:04"
var expr config.CronExpression
var (
now = time.Now()
cleanupEvery = config.GetMediaCleanupEvery()
cleanupFromStr = config.GetMediaCleanupFrom()
)
expr = config.GetMediaCleanupCron()
log.Infof(nil, "scheduling media cleanup: %s", expr.Expr)
// Parse cleanupFromStr as hh:mm.
// Resulting time will be on 1 Jan year zero.
//
// TODO: make ConfigCleanupFrom() a wrapped time.Time{}
// type to move parsing as a stage into the config package.
cleanupFrom, err := time.Parse(hourMinute, cleanupFromStr)
if err != nil {
return gtserror.Newf(
"error parsing '%s' in time format 'hh:mm': %w",
cleanupFromStr, err,
)
}
// Determine first cleanup date from now, that's in future.
firstCleanupAt := firstAt(now, cleanupFrom, cleanupEvery)
log.Infof(nil,
"scheduling media clean to run every %s, starting from %s; next clean will run at %s",
cleanupEvery, cleanupFromStr, firstCleanupAt,
)
// Schedule media cleaning at parsed schedule.
if !c.state.Workers.Scheduler.AddRecurring(
// Schedule media cleaning by expr.
if !c.state.Workers.Scheduler.Add(
"@mediacleanup",
firstCleanupAt,
cleanupEvery,
c.cleanMedia,
expr,
) {
panic("failed to schedule @mediacleanup")
}
// Schedule status cleaning at fixed schedule.
if !c.state.Workers.Scheduler.AddRecurring(
expr = config.GetStatusesCleanupCron()
log.Infof(nil, "scheduling statuses cleanup: %s", expr.Expr)
// Schedule status cleaning by expr.
if !c.state.Workers.Scheduler.Add(
"@statuscleanup",
firstAt(now, time.Time{}, 24*time.Hour),
24*time.Hour,
c.cleanStatuses,
expr,
) {
panic("failed to schedule @statuscleanup")
}
@@ -173,30 +149,14 @@ func (c *Cleaner) ScheduleJobs() error {
func (c *Cleaner) cleanMedia(ctx context.Context, start time.Time) {
log.Info(ctx, "starting")
c.Media().All(ctx, config.GetMediaRemoteCacheDays())
c.Emoji().All(ctx, config.GetMediaRemoteCacheDays())
c.Media().All(ctx, start, config.GetMediaRemoteCacheDuration())
c.Emoji().All(ctx, start, config.GetMediaRemoteCacheDuration())
log.Infof(ctx, "finished after %s", time.Since(start))
}
func (c *Cleaner) cleanStatuses(ctx context.Context, start time.Time) {
log.Info(ctx, "starting")
c.Status().All(ctx, 7)
maxRemoteAge := config.GetStatusesCleanupRemoteOlderThan()
c.Status().All(ctx, start, 7*longdur.Day, maxRemoteAge)
log.Infof(ctx, "finished after %s", time.Since(start))
}
func firstAt(now, atHourMin time.Time, every time.Duration) time.Time {
firstAt := time.Date(
now.Year(),
now.Month(),
now.Day(),
atHourMin.Hour(),
atHourMin.Minute(),
0,
0,
now.Location(),
)
for firstAt.Before(now) {
firstAt = firstAt.Add(every)
}
return firstAt
}
internal/cleaner/emoji.go
+8 -6
View File
@@ -29,6 +29,7 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"code.superseriousbusiness.org/gotosocial/internal/id"
"code.superseriousbusiness.org/gotosocial/internal/paging"
"codeberg.org/gruf/go-longdur"
)
// Emoji encompasses a set of
@@ -37,9 +38,10 @@ type Emoji struct{ Cleaner }
// All will execute all cleaner.Emoji utilities synchronously, including output logging.
// Context will be checked for `gtscontext.DryRun()` in order to actually perform the action.
func (e *Emoji) All(ctx context.Context, maxRemoteDays int) {
t := time.Now().Add(-24 * time.Hour * time.Duration(maxRemoteDays))
e.LogUncacheRemote(ctx, t)
func (e *Emoji) All(ctx context.Context, now time.Time, maxRemoteAge longdur.Duration) {
if _, dur := maxRemoteAge.Duration(); dur > 0 {
e.LogUncacheRemote(ctx, now.Add(-dur))
}
e.LogFixBroken(ctx)
e.LogPruneUnused(ctx)
_ = e.state.Storage.Storage.Clean(ctx)
@@ -47,14 +49,14 @@ func (e *Emoji) All(ctx context.Context, maxRemoteDays int) {
// AllAndFix calls LogFixCacheStates(), followed by All(), it
// is done this way round so Storage.Clean() is performed last.
func (e *Emoji) AllAndFix(ctx context.Context, maxRemoteDays int) {
func (e *Emoji) AllAndFix(ctx context.Context, now time.Time, maxRemoteAge longdur.Duration) {
e.LogFixCacheStates(ctx)
e.All(ctx, maxRemoteDays)
e.All(ctx, now, maxRemoteAge)
}
// LogUncacheRemote performs Emoji.UncacheRemote(...), logging the start and outcome.
func (e *Emoji) LogUncacheRemote(ctx context.Context, olderThan time.Time) {
log.Infof(ctx, "start older than: %s", olderThan.Format(time.Stamp))
log.Infof(ctx, "start older than: %s", olderThan.Format(stamp))
if n, err := e.UncacheRemote(ctx, olderThan); err != nil {
log.Error(ctx, err)
} else {
internal/cleaner/emoji_test.go
+1 -2
View File
@@ -102,8 +102,7 @@ func (suite *CleanerTestSuite) testEmojiUncacheRemote(ctx context.Context, emoji
t := suite.T()
// Get max remote cache days to keep.
days := config.GetMediaRemoteCacheDays()
olderThan := time.Now().Add(-24 * time.Hour * time.Duration(days))
olderThan := config.GetMediaRemoteCacheOlderThanTime(time.Now())
for _, emoji := range emojis {
// Check whether this emoji should be uncached.
internal/cleaner/media.go
+12 -7
View File
@@ -33,6 +33,7 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/paging"
"code.superseriousbusiness.org/gotosocial/internal/regexes"
"code.superseriousbusiness.org/gotosocial/internal/uris"
"codeberg.org/gruf/go-longdur"
)
// Media encompasses a set of
@@ -41,9 +42,10 @@ type Media struct{ Cleaner }
// All will execute all cleaner.Media utilities synchronously, including output logging.
// Context will be checked for `gtscontext.DryRun()` in order to actually perform the action.
func (m *Media) All(ctx context.Context, maxRemoteDays int) {
t := time.Now().Add(-24 * time.Hour * time.Duration(maxRemoteDays))
m.LogUncacheRemote(ctx, t)
func (m *Media) All(ctx context.Context, now time.Time, maxRemoteAge longdur.Duration) {
if _, dur := maxRemoteAge.Duration(); dur > 0 {
m.LogUncacheRemote(ctx, now.Add(-dur))
}
m.LogPruneOrphaned(ctx)
m.LogPruneUnused(ctx)
_ = m.state.Storage.Storage.Clean(ctx)
@@ -51,14 +53,14 @@ func (m *Media) All(ctx context.Context, maxRemoteDays int) {
// AllAndFix calls LogFixCacheStates(), followed by All(), it
// is done this way round so Storage.Clean() is performed last.
func (m *Media) AllAndFix(ctx context.Context, maxRemoteDays int) {
func (m *Media) AllAndFix(ctx context.Context, now time.Time, maxRemoteAge longdur.Duration) {
m.LogFixCacheStates(ctx)
m.All(ctx, maxRemoteDays)
m.All(ctx, now, maxRemoteAge)
}
// LogUncacheRemote performs Media.UncacheRemote(...), logging the start and outcome.
func (m *Media) LogUncacheRemote(ctx context.Context, olderThan time.Time) {
log.Infof(ctx, "start older than: %s", olderThan.Format(time.Stamp))
log.Infof(ctx, "start older than: %s", olderThan.Format(stamp))
if n, err := m.UncacheRemote(ctx, olderThan); err != nil {
log.Error(ctx, err)
} else {
@@ -501,6 +503,7 @@ func (m *Media) pruneUnused(ctx context.Context, media *gtsmodel.MediaAttachment
account, missing, err := m.getOwningAccount(ctx, media)
if err != nil {
return false, err
} else if missing {
l.Debug("deleting due to missing account")
return true, m.delete(ctx, media)
@@ -520,6 +523,7 @@ func (m *Media) pruneUnused(ctx context.Context, media *gtsmodel.MediaAttachment
status, missing, err := m.getRelatedStatus(ctx, media)
if err != nil {
return false, err
} else if missing {
l.Debug("deleting due to missing status")
return true, m.delete(ctx, media)
@@ -539,6 +543,7 @@ func (m *Media) pruneUnused(ctx context.Context, media *gtsmodel.MediaAttachment
scheduledStatus, missing, err := m.getRelatedScheduledStatus(ctx, media)
if err != nil {
return false, err
} else if missing {
l.Debug("deleting due to missing scheduled status")
return true, m.delete(ctx, media)
@@ -776,7 +781,7 @@ func (m *Media) delete(ctx context.Context, media *gtsmodel.MediaAttachment) err
// Delete media attachment entirely from the database.
log.Debugf(ctx, "deleting media attachment: %s", media.ID)
if err := m.state.DB.DeleteAttachment(ctx, media.ID); err != nil {
if err := m.state.DB.DeleteAttachment(ctx, media); err != nil {
return gtserror.Newf("error deleting media: %w", err)
}
internal/cleaner/media_test.go
+18 -23
View File
@@ -27,7 +27,6 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/admin"
"code.superseriousbusiness.org/gotosocial/internal/cleaner"
"code.superseriousbusiness.org/gotosocial/internal/db"
"code.superseriousbusiness.org/gotosocial/internal/gtscontext"
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"code.superseriousbusiness.org/gotosocial/internal/media"
@@ -41,8 +40,6 @@ import (
type MediaTestSuite struct {
suite.Suite
db db.DB
storage *storage.Driver
state state.State
manager *media.Manager
cleaner *cleaner.Cleaner
@@ -63,14 +60,12 @@ func (suite *MediaTestSuite) SetupTest() {
suite.state.Caches.Init()
testrig.StartNoopWorkers(&suite.state)
suite.db = testrig.NewTestDB(&suite.state)
suite.storage = testrig.NewInMemoryStorage()
suite.state.DB = suite.db
_ = testrig.NewTestDB(&suite.state)
suite.state.Storage = testrig.NewInMemoryStorage()
suite.state.AdminActions = admin.New(suite.state.DB, &suite.state.Workers)
suite.state.Storage = suite.storage
testrig.StandardStorageSetup(suite.storage, "../../testrig/media")
testrig.StandardDBSetup(suite.db, nil)
testrig.StandardStorageSetup(suite.state.Storage, "../../testrig/media")
testrig.StandardDBSetup(suite.state.DB, nil)
suite.testAttachments = testrig.NewTestAttachments()
suite.testAccounts = testrig.NewTestAccounts()
@@ -81,8 +76,8 @@ func (suite *MediaTestSuite) SetupTest() {
}
func (suite *MediaTestSuite) TearDownTest() {
testrig.StandardDBTeardown(suite.db)
testrig.StandardStorageTeardown(suite.storage)
testrig.StandardDBTeardown(suite.state.DB)
testrig.StandardStorageTeardown(suite.state.Storage)
testrig.StopWorkers(&suite.state)
}
@@ -100,11 +95,11 @@ func (suite *MediaTestSuite) TestUncacheRemote() {
suite.NoError(err)
suite.Equal(3, totalUncached)
uncachedAttachment, err := suite.db.GetAttachmentByID(ctx, testStatusAttachment.ID)
uncachedAttachment, err := suite.state.DB.GetAttachmentByID(ctx, testStatusAttachment.ID)
suite.NoError(err)
suite.False(uncachedAttachment.Cached())
uncachedAttachment, err = suite.db.GetAttachmentByID(ctx, testHeader.ID)
uncachedAttachment, err = suite.state.DB.GetAttachmentByID(ctx, testHeader.ID)
suite.NoError(err)
suite.False(uncachedAttachment.Cached())
}
@@ -158,11 +153,11 @@ func (suite *MediaTestSuite) TestUncacheRemoteDry() {
suite.NoError(err)
suite.Equal(3, totalUncached)
uncachedAttachment, err := suite.db.GetAttachmentByID(ctx, testStatusAttachment.ID)
uncachedAttachment, err := suite.state.DB.GetAttachmentByID(ctx, testStatusAttachment.ID)
suite.NoError(err)
suite.True(uncachedAttachment.Cached())
uncachedAttachment, err = suite.db.GetAttachmentByID(ctx, testHeader.ID)
uncachedAttachment, err = suite.state.DB.GetAttachmentByID(ctx, testHeader.ID)
suite.NoError(err)
suite.True(uncachedAttachment.Cached())
}
@@ -192,13 +187,13 @@ func (suite *MediaTestSuite) TestUncacheAndRecache() {
suite.Equal(3, totalUncached)
// media should no longer be stored
_, err = suite.storage.Get(ctx, testStatusAttachment.File.Path)
_, err = suite.state.Storage.Get(ctx, testStatusAttachment.File.Path)
suite.True(storage.IsNotFound(err))
_, err = suite.storage.Get(ctx, testStatusAttachment.Thumbnail.Path)
_, err = suite.state.Storage.Get(ctx, testStatusAttachment.Thumbnail.Path)
suite.True(storage.IsNotFound(err))
_, err = suite.storage.Get(ctx, testHeader.File.Path)
_, err = suite.state.Storage.Get(ctx, testHeader.File.Path)
suite.True(storage.IsNotFound(err))
_, err = suite.storage.Get(ctx, testHeader.Thumbnail.Path)
_, err = suite.state.Storage.Get(ctx, testHeader.Thumbnail.Path)
suite.True(storage.IsNotFound(err))
// now recache the image....
@@ -230,9 +225,9 @@ func (suite *MediaTestSuite) TestUncacheAndRecache() {
suite.EqualValues(original.FileMeta, recachedAttachment.FileMeta) // and the filemeta should be the same
// recached files should be back in storage
_, err = suite.storage.Get(ctx, recachedAttachment.File.Path)
_, err = suite.state.Storage.Get(ctx, recachedAttachment.File.Path)
suite.NoError(err)
_, err = suite.storage.Get(ctx, recachedAttachment.Thumbnail.Path)
_, err = suite.state.Storage.Get(ctx, recachedAttachment.Thumbnail.Path)
suite.NoError(err)
}
}
@@ -242,10 +237,10 @@ func (suite *MediaTestSuite) TestUncacheOneNonExistent() {
testStatusAttachment := suite.testAttachments["remote_account_1_status_1_attachment_1"]
// Delete this attachment cached on disk
media, err := suite.db.GetAttachmentByID(ctx, testStatusAttachment.ID)
media, err := suite.state.DB.GetAttachmentByID(ctx, testStatusAttachment.ID)
suite.NoError(err)
suite.True(media.Cached())
err = suite.storage.Delete(ctx, media.File.Path)
err = suite.state.Storage.Delete(ctx, media.File.Path)
suite.NoError(err)
// Now attempt to uncache remote for item with db entry no file
internal/cleaner/status.go
+96 -33
View File
@@ -27,23 +27,41 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
"code.superseriousbusiness.org/gotosocial/internal/id"
"code.superseriousbusiness.org/gotosocial/internal/paging"
"codeberg.org/gruf/go-kv/v2"
"codeberg.org/gruf/go-longdur"
)
// Status encompasses a set of
// status cleanup / admin utils.
type Status struct{ Cleaner }
// All ...
func (s *Status) All(ctx context.Context, maxStubAge int) {
d := time.Duration(min(1, maxStubAge))
olderThan := time.Now().Add(24 * time.Hour * d)
s.LogPruneLeafStubs(ctx, olderThan)
// All will execute all cleaner.Status utilities synchronously, including output logging.
// NOTE: unlike other cleaner types, `gtscontext.DryRun()` is not checked or respected here.
func (s *Status) All(ctx context.Context, now time.Time, maxStubAge, maxRemoteAge longdur.Duration) {
var dur time.Duration
if _, dur = maxStubAge.Duration(); dur > 0 {
const rateLimit = 500 * time.Millisecond
s.LogPruneLeafStubs(ctx, now.Add(-dur), rateLimit)
}
if _, dur = maxRemoteAge.Duration(); dur > 0 {
const rateLimit = 500 * time.Millisecond
s.LogPruneOldRemote(ctx, now.Add(-dur), rateLimit)
}
}
// LogPruneLeafStubs ...
func (s *Status) LogPruneLeafStubs(ctx context.Context, olderThan time.Time) {
const rateLimit = 500 * time.Microsecond // TODO make configurable when accessible via CLI
log.Infof(ctx, "start older than: %s", olderThan.Format(time.Stamp))
// LogPruneOldRemote performs PruneOldRemote(...), logging the start and outcome.
func (s *Status) LogPruneOldRemote(ctx context.Context, olderThan time.Time, rateLimit time.Duration) {
log.Infof(ctx, "start older than: %s", olderThan.Format(stamp))
if n, err := s.PruneOldRemote(ctx, olderThan, rateLimit); err != nil {
log.Error(ctx, err)
} else {
log.Infof(ctx, "pruned: %d", n)
}
}
// LogPruneLeafStubs performs PruneLeafStubs(...), logging the start and outcome.
func (s *Status) LogPruneLeafStubs(ctx context.Context, olderThan time.Time, rateLimit time.Duration) {
log.Infof(ctx, "start older than: %s", olderThan.Format(stamp))
if n, err := s.PruneLeafStubs(ctx, olderThan, rateLimit); err != nil {
log.Error(ctx, err)
} else {
@@ -51,10 +69,59 @@ func (s *Status) LogPruneLeafStubs(ctx context.Context, olderThan time.Time) {
}
}
// PruneLeafStubs ...
// PruneOldRemote will delete old status threads without any boosts, local favourites or local replies, older than given time.
// Rate limit is an optional (i.e. when > 0) parameter to limit DB load by sleeping between subsequent delete calls.
func (s *Status) PruneOldRemote(ctx context.Context, olderThan time.Time, rateLimit time.Duration) (int, error) {
var total int
page := new(paging.Page)
// Setup page w/ select limit.
page.Max = paging.MaxID("")
page.Limit = selectLimit
// Drop time by a minute to improve search,
// (i.e. make it olderThan inclusive search).
olderThan = olderThan.Add(+time.Minute)
// Get binary ULID for 'olderThan' to use as maxID.
olderThanID := id.ZeroBinaryULIDForTime(olderThan)
page.Max.Value = olderThanID.String()
for page != nil {
if rateLimit > 0 {
// Rate limiting was requested, this is very
// heavy on the db and doesn't do anything but
// loop on db queries, so give the db a break.
time.Sleep(rateLimit)
}
// Delete given page of old remote status threads, returning deleted count.
count, next, err := s.state.DB.DeleteOldRemoteStatuses(ctx, olderThanID, page)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
return total, gtserror.Newf("error deleting statuses: %w", err)
}
log.DebugKVs(ctx, kv.Fields{
{K: "count", V: count},
{K: "page", V: page.Max.Value},
}...)
// Update count.
total += count
// Set next.
page = next
}
return total, nil
}
// PruneLeafStubs will delete orphaned leaf status stubs older than given time, i.e. those marked as deleted and
// not helping to maintain status threading by being the middle connector between otherwise undeleted statuses.
// Rate limit is an optional (i.e. when > 0) parameter to limit DB load by sleeping between subsequent delete calls.
func (s *Status) PruneLeafStubs(ctx context.Context, olderThan time.Time, rateLimit time.Duration) (int, error) {
var total int
var page paging.Page
page := new(paging.Page)
// Setup page w/ select limit.
page.Max = paging.MaxID("")
@@ -68,34 +135,30 @@ func (s *Status) PruneLeafStubs(ctx context.Context, olderThan time.Time, rateLi
olderThanID := id.ZeroULIDForTime(olderThan)
page.Max.Value = olderThanID
for {
// Delete given page of status leaf stubs, returning deleted.
statuses, err := s.state.DB.DeleteStatusLeafStubs(ctx, &page)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
return total, gtserror.Newf("error deleting statuses: %w", err)
}
// Get current max ID.
maxID := page.Max.Value
// If no statuses or same group is returned, we reached the end.
if len(statuses) == 0 || maxID == statuses[len(statuses)-1].ID {
break
}
// Use last ID as the next 'maxID' value.
maxID = statuses[len(statuses)-1].ID
page.Max.Value = maxID
// Update deleted count.
total += len(statuses)
for page != nil {
if rateLimit > 0 {
// Rate limiting was requested, this is very
// heavy on the db and doesn't do anything but
// loop on db queries, so give the db a break.
time.Sleep(rateLimit)
}
// Delete given page of status leaf stubs, return delete count.
count, next, err := s.state.DB.DeleteLeafStubStatuses(ctx, page)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
return total, gtserror.Newf("error deleting statuses: %w", err)
}
log.DebugKVs(ctx, kv.Fields{
{K: "count", V: count},
{K: "page", V: page.Max.Value},
}...)
// Update count.
total += count
// Set next.
page = next
}
return total, nil
internal/cleaner/status_test.go
+267
View File
@@ -0,0 +1,267 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package cleaner_test
import (
"testing"
"time"
"code.superseriousbusiness.org/gotosocial/internal/ap"
"code.superseriousbusiness.org/gotosocial/internal/cleaner"
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"code.superseriousbusiness.org/gotosocial/internal/id"
"code.superseriousbusiness.org/gotosocial/internal/state"
"code.superseriousbusiness.org/gotosocial/testrig"
"codeberg.org/gruf/go-longdur"
"github.com/stretchr/testify/suite"
)
type StatusTestSuite struct {
suite.Suite
state state.State
cleaner *cleaner.Cleaner
testStatuses map[string]*gtsmodel.Status
}
func TestStatusTestSuite(t *testing.T) {
suite.Run(t, &StatusTestSuite{})
}
func (suite *StatusTestSuite) SetupTest() {
testrig.InitTestConfig()
testrig.InitTestLog()
suite.state.Caches.Init()
testrig.StartNoopWorkers(&suite.state)
_ = testrig.NewTestDB(&suite.state)
testrig.StandardDBSetup(suite.state.DB, nil)
suite.cleaner = cleaner.New(&suite.state)
suite.testStatuses = testrig.NewTestStatuses()
}
func (suite *StatusTestSuite) TearDownTest() {
testrig.StandardDBTeardown(suite.state.DB)
testrig.StopWorkers(&suite.state)
}
func (suite *StatusTestSuite) TestPruneOldRemote() {
suite.pruneOldRemote(4 * longdur.YearApprox)
suite.pruneOldRemote(5 * longdur.YearApprox)
suite.pruneOldRemote(6 * longdur.YearApprox)
}
func (suite *StatusTestSuite) pruneOldRemote(olderThan longdur.Duration) {
ctx := suite.T().Context()
now := time.Now()
// Get older than as concrete time.
_, dur := olderThan.Duration()
olderThanTime := now.Add(-dur)
var threadTimes []time.Time
var statuses []*gtsmodel.Status
// Generate a number of thread creation times
// based on duration before olderThanTime.
for _, d := range []longdur.Duration{
longdur.Hour,
longdur.Day,
longdur.Week,
longdur.MonthApprox,
longdur.YearApprox,
} {
_, dur := d.Duration()
time := olderThanTime.Add(-dur)
threadTimes = append(threadTimes, time)
}
// Generate a number of remote statuses based on thread
// times, which we actually expect to be later purged.
statuses = suite.generateRemoteStatuses(olderThanTime, threadTimes)
expect := len(statuses)
// Generate a new group of remote statuses based on thread
// times, which are also within age range to be later purged.
statuses = suite.generateRemoteStatuses(olderThanTime, threadTimes)
// Insert a boost at current date
// for this second batch of statuses,
// this should prevent being purged.
for _, status := range statuses {
// Generate status ID for current time.
statusID := id.NewULIDFromTime(now)
statusURI := "https://google.com/s/" + statusID
// Use just some random account ID.
accountID := "some_boostin_dude"
accountURI := "https://google.com/u/" + accountID
// Set boost of details from status.
boostOfAccountID := status.AccountID
boostOfID := status.ID
// Create the status model.
boost := &gtsmodel.Status{
ID: statusID,
URI: statusURI,
AccountID: accountID,
AccountURI: accountURI,
BoostOfID: boostOfID,
BoostOfAccountID: boostOfAccountID,
ThreadID: status.ThreadID,
// This needs a proper fetched_at
// setting so it gets appropriately
// selected for its age.
FetchedAt: now,
ActivityStreamsType: ap.ObjectNote,
Visibility: status.Visibility,
}
// Insert the new status in database,
// specifically bypassing our typical
// status threading logic confusing it.
err := suite.state.DB.Put(ctx, boost)
suite.NoError(err)
}
// Generate a new group of remote statuses based on thread
// times, which are also within age range to be later purged.
statuses = suite.generateRemoteStatuses(olderThanTime, threadTimes)
// Insert a favourite by a local account for each status
// into the database, this should prevent their purging.
localAccountID := testrig.NewTestAccounts()["admin_account"].ID
for _, status := range statuses {
// Generate a new favourite ID.
faveID := id.NewRandomULID()
faveURI := "https://google.com/f/" + faveID
// Create the favourite model.
fave := &gtsmodel.StatusFave{
ID: faveID,
URI: faveURI,
AccountID: localAccountID,
TargetAccountID: status.AccountID,
StatusID: status.ID,
}
// Insert the new favourite into the database.
err := suite.state.DB.PutStatusFave(ctx, fave)
suite.NoError(err)
}
// Generate a new group of remote statuses based on thread
// times, which are also within age range to be later purged.
statuses = suite.generateRemoteStatuses(olderThanTime, threadTimes)
// Insert a bookmark by a local account for each status
// into the database, this should prevent their purging.
for _, status := range statuses {
// Generate a new bookmark ID.
bookmarkID := id.NewRandomULID()
// Create the status bookmark model.
bookmark := &gtsmodel.StatusBookmark{
ID: bookmarkID,
AccountID: localAccountID,
TargetAccountID: status.AccountID,
StatusID: status.ID,
}
// Insert the new status bookmark into the database.
err := suite.state.DB.PutStatusBookmark(ctx, bookmark)
suite.NoError(err)
}
// Perform the status pruning for those older than time.
count, err := suite.cleaner.Status().PruneOldRemote(ctx,
olderThanTime, 0)
suite.NoError(err)
suite.Equal(expect, count)
}
func (suite *StatusTestSuite) generateRemoteStatuses(olderThan time.Time, threadTimes []time.Time) []*gtsmodel.Status {
var statuses []*gtsmodel.Status
ctx := suite.T().Context()
// Generate a number of thread IDs
// based on generated thread times.
for _, t := range threadTimes {
threadID := id.NewULIDFromTime(t)
// Determine difference between
// thread time and 'olderThan'.
diff := olderThan.Sub(t)
// Iterate between thread time and older than, creating
// statuses of differing ages between these two points.
for i := time.Duration(0); i < diff; i += (diff / 10) {
// Generate new status ID from this time.
statusID := id.NewULIDFromTime(t.Add(i))
statusURI := "https://google.com/s/" + statusID
// Use some random account ID.
accountID := "some_random_dude"
accountURI := "https://google.com/u/" + accountID
// Create the status model.
status := &gtsmodel.Status{
ID: statusID,
URI: statusURI,
AccountID: accountID,
AccountURI: accountURI,
ThreadID: threadID,
ActivityStreamsType: ap.ObjectNote,
Visibility: gtsmodel.VisibilityDefault,
}
// Insert the new status in database,
// specifically bypassing our typical
// status threading logic confusing it.
err := suite.state.DB.Put(ctx, status)
suite.NoError(err)
// Append status to the return slice.
statuses = append(statuses, status)
}
}
return statuses
}
internal/config/config.go
+29 -21
View File
@@ -23,7 +23,9 @@ import (
"time"
"code.superseriousbusiness.org/gotosocial/internal/language"
"codeberg.org/gruf/go-bytesize"
"codeberg.org/gruf/go-longdur"
)
// cfgtype is the reflected type information of Configuration{}.
@@ -51,8 +53,8 @@ func fieldtag(field, tag string) string {
// will need to regenerate the global Getter/Setter helpers by running:
// `go run ./internal/config/gen/ -out ./internal/config/helpers.gen.go`.
//
// You will need to have gofumpt installed in order for this to work:
// https://github.com/mvdan/gofumpt.
// You will need to have goimports installed in order for this to work:
// golang.org/x/tools/cmd/goimports.
type Configuration struct {
LogLevel string `name:"log-level" usage:"Log level to run at: [trace, debug, info, warn, fatal]"`
LogFormat string `name:"log-format" usage:"Log output format: [logfmt, json]"`
@@ -102,8 +104,9 @@ type Configuration struct {
InstanceDeliverToSharedInboxes bool `name:"instance-deliver-to-shared-inboxes" usage:"Deliver federated messages to shared inboxes, if they're available."`
InstanceInjectMastodonVersion bool `name:"instance-inject-mastodon-version" usage:"This injects a Mastodon compatible version in /api/v1/instance to help Mastodon clients that use that version for feature detection"`
InstanceLanguages language.Languages `name:"instance-languages" usage:"BCP47 language tags for the instance. Used to indicate the preferred languages of instance residents (in order from most-preferred to least-preferred)."`
InstanceSubscriptionsProcessFrom string `name:"instance-subscriptions-process-from" usage:"Time of day from which to start running instance subscriptions processing jobs. Should be in the format 'hh:mm:ss', eg., '15:04:05'."`
InstanceSubscriptionsProcessEvery time.Duration `name:"instance-subscriptions-process-every" usage:"Period to elapse between instance subscriptions processing jobs, starting from instance-subscriptions-process-from."`
InstanceSubscriptionsProcessFrom Deprecated `name:"instance-subscriptions-process-from" deprecated-by:"instance-subscriptions-process-cron"`
InstanceSubscriptionsProcessEvery Deprecated `name:"instance-subscriptions-process-every" deprecated-by:"instance-subscriptions-process-cron"`
InstanceSubscriptionsProcessCron CronExpression `name:"instance-subscriptions-process-cron" usage:"Cron expression defining instance subscription processing job scheduling"`
InstanceStatsMode string `name:"instance-stats-mode" usage:"Allows you to customize the way stats are served to crawlers: one of '', 'serve', 'zero', 'baffle'. Home page stats remain unchanged."`
InstanceAllowBackdatingStatuses bool `name:"instance-allow-backdating-statuses" usage:"Allow local accounts to backdate statuses using the scheduled_at param to /api/v1/statuses"`
InstanceRobotsAllowIndexing bool `name:"instance-robots-allow-indexing" usage:"Return robots headers and meta tags that allow search engine indexing of instance home page, directory (if enabled), and accounts that have opted in to being discoverable."`
@@ -129,10 +132,12 @@ type Configuration struct {
StorageS3KeyPrefix string `name:"storage-s3-key-prefix" usage:"Prefix to use for S3 keys. This is useful for separating multiple instances sharing the same S3 bucket."`
StorageS3Region string `name:"storage-s3-region" usage:"Region to use for S3."`
StatusesMaxChars int `name:"statuses-max-chars" usage:"Max permitted characters for posted statuses, including content warning"`
StatusesPollMaxOptions int `name:"statuses-poll-max-options" usage:"Max amount of options permitted on a poll"`
StatusesPollOptionMaxChars int `name:"statuses-poll-option-max-chars" usage:"Max amount of characters for a poll option"`
StatusesMediaMaxFiles int `name:"statuses-media-max-files" usage:"Maximum number of media files/attachments per status"`
StatusesMaxChars int `name:"statuses-max-chars" usage:"Max permitted characters for posted statuses, including content warning"`
StatusesPollMaxOptions int `name:"statuses-poll-max-options" usage:"Max amount of options permitted on a poll"`
StatusesPollOptionMaxChars int `name:"statuses-poll-option-max-chars" usage:"Max amount of characters for a poll option"`
StatusesMediaMaxFiles int `name:"statuses-media-max-files" usage:"Maximum number of media files/attachments per status"`
StatusesCleanupCron CronExpression `name:"statuses-cleanup-cron" usage:"Cron expression defining statuses cleanup task scheduling"`
StatusesCleanupRemoteOlderThan longdur.Duration `name:"statuses-cleanup-remote-older-than" usage:"Duration defining status age beyond which to clean"`
ScheduledStatusesMaxTotal int `name:"scheduled-statuses-max-total" usage:"Maximum number of scheduled statuses per user"`
ScheduledStatusesMaxDaily int `name:"scheduled-statuses-max-daily" usage:"Maximum number of scheduled statuses per user for a single day"`
@@ -242,19 +247,22 @@ type HTTPClientConfiguration struct {
}
type MediaConfiguration struct {
DescriptionMinChars int `name:"description-min-chars" usage:"Min required chars for an image description"`
DescriptionMaxChars int `name:"description-max-chars" usage:"Max permitted chars for an image description"`
RemoteCacheDays int `name:"remote-cache-days" usage:"Number of days to locally cache media from remote instances. If set to 0, remote media will be kept indefinitely."`
EmojiLocalMaxSize bytesize.Size `name:"emoji-local-max-size" usage:"Max size in bytes of emojis uploaded to this instance via the admin API."`
EmojiRemoteMaxSize bytesize.Size `name:"emoji-remote-max-size" usage:"Max size in bytes of emojis to download from other instances."`
ImageSizeHint bytesize.Size `name:"image-size-hint" usage:"Size in bytes of max image size referred to on /api/v_/instance endpoints (else, local max size)"`
VideoSizeHint bytesize.Size `name:"video-size-hint" usage:"Size in bytes of max video size referred to on /api/v_/instance endpoints (else, local max size)"`
LocalMaxSize bytesize.Size `name:"local-max-size" usage:"Max size in bytes of media uploaded to this instance via API"`
RemoteMaxSize bytesize.Size `name:"remote-max-size" usage:"Max size in bytes of media to download from other instances"`
CleanupFrom string `name:"cleanup-from" usage:"Time of day from which to start running media cleanup/prune jobs. Should be in the format 'hh:mm:ss', eg., '15:04:05'."`
CleanupEvery time.Duration `name:"cleanup-every" usage:"Period to elapse between cleanups, starting from media-cleanup-at."`
FfmpegPoolSize int `name:"ffmpeg-pool-size" usage:"Number of concurrent running instances of ffmpeg to permit. 0 or less uses GOMAXPROCS."`
ThumbMaxPixels int `name:"thumb-max-pixels" usage:"Max size in pixels of any one dimension of a thumbnail (as input media ratio is preserved)."`
DescriptionMinChars int `name:"description-min-chars" usage:"Min required chars for an image description"`
DescriptionMaxChars int `name:"description-max-chars" usage:"Max permitted chars for an image description"`
EmojiLocalMaxSize bytesize.Size `name:"emoji-local-max-size" usage:"Max size in bytes of emojis uploaded to this instance via the admin API."`
EmojiRemoteMaxSize bytesize.Size `name:"emoji-remote-max-size" usage:"Max size in bytes of emojis to download from other instances."`
ImageSizeHint bytesize.Size `name:"image-size-hint" usage:"Size in bytes of max image size referred to on /api/v_/instance endpoints (else, local max size)"`
VideoSizeHint bytesize.Size `name:"video-size-hint" usage:"Size in bytes of max video size referred to on /api/v_/instance endpoints (else, local max size)"`
LocalMaxSize bytesize.Size `name:"local-max-size" usage:"Max size in bytes of media uploaded to this instance via API"`
RemoteMaxSize bytesize.Size `name:"remote-max-size" usage:"Max size in bytes of media to download from other instances"`
FfmpegPoolSize int `name:"ffmpeg-pool-size" usage:"Number of instances of the embedded ffmpeg WASM binary to add to the media processing pool. 0 or less uses GOMAXPROCS."`
ThumbMaxPixels int `name:"thumb-max-pixels" usage:"Max size in pixels of any one dimension of a thumbnail (as input media ratio is preserved)."`
RemoteCacheDuration longdur.Duration `name:"remote-cache-duration" usage:"Duration defining how long to locally cache media from remote instances. (zero keeps indefinitely)"`
CleanupCron CronExpression `name:"cleanup-cron" usage:"Cron expression defining media cleanup task scheduling"`
RemoteCacheDays Deprecated `name:"remote-cache-days" deprecated-by:"media-remote-cache-duration"`
CleanupFrom Deprecated `name:"cleanup-from" deprecated-by:"media-cleanup-cron"`
CleanupEvery Deprecated `name:"cleanup-every" deprecated-by:"media-cleanup-cron"`
}
type CacheConfiguration struct {
internal/config/defaults.go
+21 -20
View File
@@ -22,6 +22,7 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/language"
"codeberg.org/gruf/go-bytesize"
"codeberg.org/gruf/go-longdur"
)
// Defaults contains a populated Configuration with reasonable defaults. Note that
@@ -58,19 +59,18 @@ var Defaults = Configuration{
WebTemplateBaseDir: "./web/template/",
WebAssetBaseDir: "./web/assets/",
InstanceFederationMode: InstanceFederationModeDefault,
InstanceFederationSpamFilter: false,
InstanceExposePeers: false,
InstanceExposeBlocklist: false,
InstanceExposeBlocklistWeb: false,
InstanceExposeCustomEmojis: false,
InstanceDeliverToSharedInboxes: true,
InstanceLanguages: make(language.Languages, 0),
InstanceSubscriptionsProcessFrom: "23:00", // 11pm,
InstanceSubscriptionsProcessEvery: 24 * time.Hour, // 1/day.
InstanceAllowBackdatingStatuses: true,
InstanceDirectoryMode: InstanceDirectoryModeWebOnly,
InstanceRobotsAllowIndexing: false,
InstanceFederationMode: InstanceFederationModeDefault,
InstanceFederationSpamFilter: false,
InstanceExposePeers: false,
InstanceExposeBlocklist: false,
InstanceExposeBlocklistWeb: false,
InstanceExposeCustomEmojis: false,
InstanceDeliverToSharedInboxes: true,
InstanceLanguages: make(language.Languages, 0),
InstanceSubscriptionsProcessCron: MustParseCron("0 23 * * *"), // daily at 11pm
InstanceAllowBackdatingStatuses: true,
InstanceDirectoryMode: InstanceDirectoryModeWebOnly,
InstanceRobotsAllowIndexing: false,
AccountsRegistrationOpen: false,
AccountsReasonRequired: true,
@@ -83,15 +83,14 @@ var Defaults = Configuration{
Media: MediaConfiguration{
DescriptionMinChars: 0,
DescriptionMaxChars: 1500,
RemoteCacheDays: 7,
LocalMaxSize: 40 * bytesize.MiB,
RemoteMaxSize: 40 * bytesize.MiB,
EmojiLocalMaxSize: 50 * bytesize.KiB,
EmojiRemoteMaxSize: 100 * bytesize.KiB,
CleanupFrom: "00:00", // Midnight.
CleanupEvery: 24 * time.Hour, // 1/day.
FfmpegPoolSize: 1,
ThumbMaxPixels: 512,
RemoteCacheDuration: 7 * longdur.Day,
CleanupCron: MustParseCron("0 0 * * *"), // daily at 0am
},
StorageBackend: "local",
@@ -101,10 +100,12 @@ var Defaults = Configuration{
StorageS3RedirectURL: "",
StorageS3BucketLookup: "auto",
StatusesMaxChars: 5000,
StatusesPollMaxOptions: 6,
StatusesPollOptionMaxChars: 50,
StatusesMediaMaxFiles: 6,
StatusesMaxChars: 5000,
StatusesPollMaxOptions: 6,
StatusesPollOptionMaxChars: 50,
StatusesMediaMaxFiles: 6,
StatusesCleanupCron: MustParseCron("0 1 * * 0"), // weekly at 1am
StatusesCleanupRemoteOlderThan: 0,
ScheduledStatusesMaxTotal: 300,
ScheduledStatusesMaxDaily: 25,
internal/config/gen/gen.go
+65 -24
View File
@@ -92,7 +92,7 @@ func main() {
generateGetSetters(output, fields)
generateMapFlattener(output, fields)
must(output.Close())
must(exec.Command("gofumpt", "-w", out).Run())
must(exec.Command("goimports", "-w", out).Run())
}
type ConfigField struct {
@@ -118,6 +118,10 @@ type ConfigField struct {
// Whether to generate
// CLI flag registering.
RegisterCLI bool
// i.e. is this a deprecated field we don't
// want being used, point to this field instead.
DeprecatedBy string
}
// Flag returns the combined "prefixes-name" CLI flag for config field.
@@ -176,7 +180,14 @@ func loadConfigFields(pathPrefixes, flagPrefixes []string, t reflect.Type) []Con
continue
}
if ft := field.Type; ft.Kind() == reflect.Struct {
// Get field's tagged usage.
usage := field.Tag.Get("usage")
// Look for name that deprecates this one.
depBy := field.Tag.Get("deprecated-by")
if ft := field.Type; ft.Kind() == reflect.Struct &&
depBy == "" && usage == "" {
// This is a nested struct, load nested fields.
pathPrefixes := append(pathPrefixes, field.Name)
flagPrefixes := append(flagPrefixes, name)
@@ -189,12 +200,13 @@ func loadConfigFields(pathPrefixes, flagPrefixes []string, t reflect.Type) []Con
// Append prepared ConfigField.
out = append(out, ConfigField{
Prefixes: flagPrefixes,
Name: name,
Path: fieldPath,
Type: field.Type,
Usage: field.Tag.Get("usage"),
RegisterCLI: field.Tag.Get("nocli") != "yes",
Prefixes: flagPrefixes,
Name: name,
Path: fieldPath,
Type: field.Type,
Usage: usage,
RegisterCLI: field.Tag.Get("nocli") != "yes",
DeprecatedBy: depBy,
})
}
return out
@@ -218,6 +230,12 @@ func generateFlagRegistering(out io.Writer, fields []ConfigField) {
continue
}
if field.DeprecatedBy != "" {
// Skip registering
// deprecated flags.
continue
}
// Check for easy cases of just regular primitive types.
if field.Type.Kind().String() == field.Type.String() {
typeName := field.Type.String()
@@ -283,6 +301,12 @@ func generateMapMarshaler(out io.Writer, fields []ConfigField) {
fprintf(out, "func (cfg *Configuration) MarshalMap() map[string]any {\n")
fprintf(out, "\tcfgmap := make(map[string]any, %d)\n", len(fields))
for _, field := range fields {
// Deprecated fields don't need
// including in marshaled map.
if field.DeprecatedBy != "" {
continue
}
// Check for easy cases of just regular primitive types.
if field.Type.Kind().String() == field.Type.String() {
fprintf(out, "\tcfgmap[\"%s\"] = cfg.%s\n", field.Flag(), field.Path)
@@ -343,12 +367,18 @@ func generateMapMarshaler(out io.Writer, fields []ConfigField) {
func generateMapUnmarshaler(out io.Writer, fields []ConfigField) {
fprintf(out, "func (cfg *Configuration) UnmarshalMap(cfgmap map[string]any) error {\n")
fprintf(out, "// VERY IMPORTANT FIRST STEP!\n")
fprintf(out, "// flatten to normalize map to\n")
fprintf(out, "// entirely un-nested key values\n")
fprintf(out, "flattenConfigMap(cfgmap)\n")
fprintf(out, "\t// VERY IMPORTANT FIRST STEP!\n")
fprintf(out, "\t// flatten to normalize map to\n")
fprintf(out, "\t// entirely un-nested key values\n")
fprintf(out, "\tflattenConfigMap(cfgmap)\n")
fprintf(out, "\n")
for _, field := range fields {
// Check for case of deprecated.
if field.DeprecatedBy != "" {
generateUnmarshalerDeprecated(out, field)
continue
}
// Check for easy cases of just regular primitive types.
if field.Type.Kind().String() == field.Type.String() {
generateUnmarshalerPrimitive(out, field)
@@ -391,8 +421,15 @@ func generateMapUnmarshaler(out io.Writer, fields []ConfigField) {
fprintf(out, "}\n\n")
}
func generateUnmarshalerDeprecated(out io.Writer, field ConfigField) {
fprintf(out, "\tif ival, ok := cfgmap[\"%s\"]; ok && ival != \"\" {\n", field.Flag())
fprintf(out, "\t\treturn errors.New(\"value received for deprecated field '%s', please use '%s' instead\")\n", field.Flag(), field.DeprecatedBy)
fprintf(out, "\t}\n")
fprintf(out, "\n")
}
func generateUnmarshalerPrimitive(out io.Writer, field ConfigField) {
fprintf(out, "\t\tif ival, ok := cfgmap[\"%s\"]; ok {\n", field.Flag())
fprintf(out, "\tif ival, ok := cfgmap[\"%s\"]; ok {\n", field.Flag())
if field.Type.Kind() == reflect.Slice {
elem := field.Type.Elem()
typeName := elem.String()
@@ -400,23 +437,23 @@ func generateUnmarshalerPrimitive(out io.Writer, field ConfigField) {
typeName = typeName[i+1:]
}
typeName = strings.ToUpper(typeName[:1]) + typeName[1:]
fprintf(out, "\t\t\tvar err error\n")
fprintf(out, "\t\tvar err error\n")
// note we specifically handle slice types ourselves to split by comma
fprintf(out, "\t\t\tcfg.%s, err = to%sSlice(ival)\n", field.Path, typeName)
fprintf(out, "\t\t\tif err != nil {\n")
fprintf(out, "\t\t\t\treturn fmt.Errorf(\"error casting %%#v -> []%s for '%s': %%w\", ival, err)\n", elem.String(), field.Flag())
fprintf(out, "\t\t\t}\n")
fprintf(out, "\t\tcfg.%s, err = to%sSlice(ival)\n", field.Path, typeName)
fprintf(out, "\t\tif err != nil {\n")
fprintf(out, "\t\t\treturn fmt.Errorf(\"error casting %%#v -> []%s for '%s': %%w\", ival, err)\n", elem.String(), field.Flag())
fprintf(out, "\t\t}\n")
} else {
typeName := field.Type.String()
if i := strings.IndexRune(typeName, '.'); i >= 0 {
typeName = typeName[i+1:]
}
typeName = strings.ToUpper(typeName[:1]) + typeName[1:]
fprintf(out, "\t\t\tvar err error\n")
fprintf(out, "\t\t\tcfg.%s, err = cast.To%sE(ival)\n", field.Path, typeName)
fprintf(out, "\t\t\tif err != nil {\n")
fprintf(out, "\t\t\t\treturn fmt.Errorf(\"error casting %%#v -> %s for '%s': %%w\", ival, err)\n", field.Type.String(), field.Flag())
fprintf(out, "\t\t\t}\n")
fprintf(out, "\t\tvar err error\n")
fprintf(out, "\t\tcfg.%s, err = cast.To%sE(ival)\n", field.Path, typeName)
fprintf(out, "\t\tif err != nil {\n")
fprintf(out, "\t\t\treturn fmt.Errorf(\"error casting %%#v -> %s for '%s': %%w\", ival, err)\n", field.Type.String(), field.Flag())
fprintf(out, "\t\t}\n")
}
fprintf(out, "\t}\n")
fprintf(out, "\n")
@@ -441,7 +478,7 @@ func generateUnmarshalerFlagType(out io.Writer, field ConfigField) {
fprintf(out, "\t\tif err != nil {\n")
fprintf(out, "\t\t\treturn fmt.Errorf(\"error casting %%#v -> string for '%s': %%w\", ival, err)\n", field.Flag())
fprintf(out, "\t\t}\n")
fprintf(out, "\t\tcfg.%s = %#v\n", field.Path, reflect.New(field.Type).Elem().Interface())
fprintf(out, "\t\tcfg.%s = %s\n", field.Path, strings.TrimPrefix(zeroValueStr(field.Type), "config."))
fprintf(out, "\t\tif err := cfg.%s.Set(t); err != nil {\n", field.Path)
fprintf(out, "\t\t\treturn fmt.Errorf(\"error parsing %%#v for '%s': %%w\", ival, err)\n", field.Flag())
fprintf(out, "\t\t}\n")
@@ -450,6 +487,10 @@ func generateUnmarshalerFlagType(out io.Writer, field ConfigField) {
fprintf(out, "\n")
}
func zeroValueStr(t reflect.Type) string {
return fmt.Sprintf("%#v", reflect.New(t).Elem().Interface())
}
func generateGetSetters(out io.Writer, fields []ConfigField) {
for _, field := range fields {
// Get name from struct path, without periods.
internal/config/helpers.gen.go
+299 -141
View File
@@ -19,11 +19,14 @@
package config
import (
"errors"
"fmt"
"time"
"code.superseriousbusiness.org/gotosocial/internal/language"
"codeberg.org/gruf/go-bytesize"
"codeberg.org/gruf/go-longdur"
"github.com/hashicorp/cronexpr"
"github.com/spf13/cast"
"github.com/spf13/pflag"
)
@@ -76,6 +79,7 @@ const (
InstanceLanguagesFlag = "instance-languages"
InstanceSubscriptionsProcessFromFlag = "instance-subscriptions-process-from"
InstanceSubscriptionsProcessEveryFlag = "instance-subscriptions-process-every"
InstanceSubscriptionsProcessCronFlag = "instance-subscriptions-process-cron"
InstanceStatsModeFlag = "instance-stats-mode"
InstanceAllowBackdatingStatusesFlag = "instance-allow-backdating-statuses"
InstanceRobotsAllowIndexingFlag = "instance-robots-allow-indexing"
@@ -102,6 +106,8 @@ const (
StatusesPollMaxOptionsFlag = "statuses-poll-max-options"
StatusesPollOptionMaxCharsFlag = "statuses-poll-option-max-chars"
StatusesMediaMaxFilesFlag = "statuses-media-max-files"
StatusesCleanupCronFlag = "statuses-cleanup-cron"
StatusesCleanupRemoteOlderThanFlag = "statuses-cleanup-remote-older-than"
ScheduledStatusesMaxTotalFlag = "scheduled-statuses-max-total"
ScheduledStatusesMaxDailyFlag = "scheduled-statuses-max-daily"
LetsEncryptEnabledFlag = "letsencrypt-enabled"
@@ -173,17 +179,19 @@ const (
HTTPClientWriteBufferSizeFlag = "http-client-write-buffer-size"
MediaDescriptionMinCharsFlag = "media-description-min-chars"
MediaDescriptionMaxCharsFlag = "media-description-max-chars"
MediaRemoteCacheDaysFlag = "media-remote-cache-days"
MediaEmojiLocalMaxSizeFlag = "media-emoji-local-max-size"
MediaEmojiRemoteMaxSizeFlag = "media-emoji-remote-max-size"
MediaImageSizeHintFlag = "media-image-size-hint"
MediaVideoSizeHintFlag = "media-video-size-hint"
MediaLocalMaxSizeFlag = "media-local-max-size"
MediaRemoteMaxSizeFlag = "media-remote-max-size"
MediaCleanupFromFlag = "media-cleanup-from"
MediaCleanupEveryFlag = "media-cleanup-every"
MediaFfmpegPoolSizeFlag = "media-ffmpeg-pool-size"
MediaThumbMaxPixelsFlag = "media-thumb-max-pixels"
MediaRemoteCacheDurationFlag = "media-remote-cache-duration"
MediaCleanupCronFlag = "media-cleanup-cron"
MediaRemoteCacheDaysFlag = "media-remote-cache-days"
MediaCleanupFromFlag = "media-cleanup-from"
MediaCleanupEveryFlag = "media-cleanup-every"
CacheS3ObjectInfoFlag = "cache-s3-object-info"
CacheHomeTimelineSizeFlag = "cache-home-timeline-size"
CacheListTimelineSizeFlag = "cache-list-timeline-size"
@@ -317,8 +325,7 @@ func (cfg *Configuration) RegisterFlags(flags *pflag.FlagSet) {
flags.Bool("instance-deliver-to-shared-inboxes", cfg.InstanceDeliverToSharedInboxes, "Deliver federated messages to shared inboxes, if they're available.")
flags.Bool("instance-inject-mastodon-version", cfg.InstanceInjectMastodonVersion, "This injects a Mastodon compatible version in /api/v1/instance to help Mastodon clients that use that version for feature detection")
flags.StringSlice("instance-languages", cfg.InstanceLanguages.Strings(), "BCP47 language tags for the instance. Used to indicate the preferred languages of instance residents (in order from most-preferred to least-preferred).")
flags.String("instance-subscriptions-process-from", cfg.InstanceSubscriptionsProcessFrom, "Time of day from which to start running instance subscriptions processing jobs. Should be in the format 'hh:mm:ss', eg., '15:04:05'.")
flags.Duration("instance-subscriptions-process-every", cfg.InstanceSubscriptionsProcessEvery, "Period to elapse between instance subscriptions processing jobs, starting from instance-subscriptions-process-from.")
flags.String("instance-subscriptions-process-cron", cfg.InstanceSubscriptionsProcessCron.String(), "Cron expression defining instance subscription processing job scheduling")
flags.String("instance-stats-mode", cfg.InstanceStatsMode, "Allows you to customize the way stats are served to crawlers: one of '', 'serve', 'zero', 'baffle'. Home page stats remain unchanged.")
flags.Bool("instance-allow-backdating-statuses", cfg.InstanceAllowBackdatingStatuses, "Allow local accounts to backdate statuses using the scheduled_at param to /api/v1/statuses")
flags.Bool("instance-robots-allow-indexing", cfg.InstanceRobotsAllowIndexing, "Return robots headers and meta tags that allow search engine indexing of instance home page, directory (if enabled), and accounts that have opted in to being discoverable.")
@@ -345,6 +352,8 @@ func (cfg *Configuration) RegisterFlags(flags *pflag.FlagSet) {
flags.Int("statuses-poll-max-options", cfg.StatusesPollMaxOptions, "Max amount of options permitted on a poll")
flags.Int("statuses-poll-option-max-chars", cfg.StatusesPollOptionMaxChars, "Max amount of characters for a poll option")
flags.Int("statuses-media-max-files", cfg.StatusesMediaMaxFiles, "Maximum number of media files/attachments per status")
flags.String("statuses-cleanup-cron", cfg.StatusesCleanupCron.String(), "Cron expression defining statuses cleanup task scheduling")
flags.String("statuses-cleanup-remote-older-than", cfg.StatusesCleanupRemoteOlderThan.String(), "Duration defining status age beyond which to clean")
flags.Int("scheduled-statuses-max-total", cfg.ScheduledStatusesMaxTotal, "Maximum number of scheduled statuses per user")
flags.Int("scheduled-statuses-max-daily", cfg.ScheduledStatusesMaxDaily, "Maximum number of scheduled statuses per user for a single day")
flags.Bool("letsencrypt-enabled", cfg.LetsEncryptEnabled, "Enable letsencrypt TLS certs for this server. If set to true, then cert dir also needs to be set (or take the default).")
@@ -416,17 +425,16 @@ func (cfg *Configuration) RegisterFlags(flags *pflag.FlagSet) {
flags.String("http-client-write-buffer-size", cfg.HTTPClient.WriteBufferSize.String(), "")
flags.Int("media-description-min-chars", cfg.Media.DescriptionMinChars, "Min required chars for an image description")
flags.Int("media-description-max-chars", cfg.Media.DescriptionMaxChars, "Max permitted chars for an image description")
flags.Int("media-remote-cache-days", cfg.Media.RemoteCacheDays, "Number of days to locally cache media from remote instances. If set to 0, remote media will be kept indefinitely.")
flags.String("media-emoji-local-max-size", cfg.Media.EmojiLocalMaxSize.String(), "Max size in bytes of emojis uploaded to this instance via the admin API.")
flags.String("media-emoji-remote-max-size", cfg.Media.EmojiRemoteMaxSize.String(), "Max size in bytes of emojis to download from other instances.")
flags.String("media-image-size-hint", cfg.Media.ImageSizeHint.String(), "Size in bytes of max image size referred to on /api/v_/instance endpoints (else, local max size)")
flags.String("media-video-size-hint", cfg.Media.VideoSizeHint.String(), "Size in bytes of max video size referred to on /api/v_/instance endpoints (else, local max size)")
flags.String("media-local-max-size", cfg.Media.LocalMaxSize.String(), "Max size in bytes of media uploaded to this instance via API")
flags.String("media-remote-max-size", cfg.Media.RemoteMaxSize.String(), "Max size in bytes of media to download from other instances")
flags.String("media-cleanup-from", cfg.Media.CleanupFrom, "Time of day from which to start running media cleanup/prune jobs. Should be in the format 'hh:mm:ss', eg., '15:04:05'.")
flags.Duration("media-cleanup-every", cfg.Media.CleanupEvery, "Period to elapse between cleanups, starting from media-cleanup-at.")
flags.Int("media-ffmpeg-pool-size", cfg.Media.FfmpegPoolSize, "Number of concurrent running instances of ffmpeg to permit. 0 or less uses GOMAXPROCS.")
flags.Int("media-ffmpeg-pool-size", cfg.Media.FfmpegPoolSize, "Number of instances of the embedded ffmpeg WASM binary to add to the media processing pool. 0 or less uses GOMAXPROCS.")
flags.Int("media-thumb-max-pixels", cfg.Media.ThumbMaxPixels, "Max size in pixels of any one dimension of a thumbnail (as input media ratio is preserved).")
flags.String("media-remote-cache-duration", cfg.Media.RemoteCacheDuration.String(), "Duration defining how long to locally cache media from remote instances. (zero keeps indefinitely)")
flags.String("media-cleanup-cron", cfg.Media.CleanupCron.String(), "Cron expression defining media cleanup task scheduling")
flags.Uint32("cache-s3-object-info", cfg.Cache.S3ObjectInfo, "Enables caching of S3 object information in the storage driver to reduce S3 calls, value is cache capacity.")
flags.Uint32("cache-home-timeline-size", cfg.Cache.HomeTimelineSize, "Per-user home timeline cache length, in number of posts. (minimum = 100)")
flags.Uint32("cache-list-timeline-size", cfg.Cache.ListTimelineSize, "Per-list timeline cache length, in number of posts. (minimum = 100)")
@@ -506,7 +514,7 @@ func (cfg *Configuration) RegisterFlags(flags *pflag.FlagSet) {
}
func (cfg *Configuration) MarshalMap() map[string]any {
cfgmap := make(map[string]any, 240)
cfgmap := make(map[string]any, 245)
cfgmap["log-level"] = cfg.LogLevel
cfgmap["log-format"] = cfg.LogFormat
cfgmap["log-timestamp-format"] = cfg.LogTimestampFormat
@@ -552,8 +560,7 @@ func (cfg *Configuration) MarshalMap() map[string]any {
cfgmap["instance-deliver-to-shared-inboxes"] = cfg.InstanceDeliverToSharedInboxes
cfgmap["instance-inject-mastodon-version"] = cfg.InstanceInjectMastodonVersion
cfgmap["instance-languages"] = cfg.InstanceLanguages.Strings()
cfgmap["instance-subscriptions-process-from"] = cfg.InstanceSubscriptionsProcessFrom
cfgmap["instance-subscriptions-process-every"] = cfg.InstanceSubscriptionsProcessEvery
cfgmap["instance-subscriptions-process-cron"] = cfg.InstanceSubscriptionsProcessCron.String()
cfgmap["instance-stats-mode"] = cfg.InstanceStatsMode
cfgmap["instance-allow-backdating-statuses"] = cfg.InstanceAllowBackdatingStatuses
cfgmap["instance-robots-allow-indexing"] = cfg.InstanceRobotsAllowIndexing
@@ -580,6 +587,8 @@ func (cfg *Configuration) MarshalMap() map[string]any {
cfgmap["statuses-poll-max-options"] = cfg.StatusesPollMaxOptions
cfgmap["statuses-poll-option-max-chars"] = cfg.StatusesPollOptionMaxChars
cfgmap["statuses-media-max-files"] = cfg.StatusesMediaMaxFiles
cfgmap["statuses-cleanup-cron"] = cfg.StatusesCleanupCron.String()
cfgmap["statuses-cleanup-remote-older-than"] = cfg.StatusesCleanupRemoteOlderThan.String()
cfgmap["scheduled-statuses-max-total"] = cfg.ScheduledStatusesMaxTotal
cfgmap["scheduled-statuses-max-daily"] = cfg.ScheduledStatusesMaxDaily
cfgmap["letsencrypt-enabled"] = cfg.LetsEncryptEnabled
@@ -651,17 +660,16 @@ func (cfg *Configuration) MarshalMap() map[string]any {
cfgmap["http-client-write-buffer-size"] = cfg.HTTPClient.WriteBufferSize.String()
cfgmap["media-description-min-chars"] = cfg.Media.DescriptionMinChars
cfgmap["media-description-max-chars"] = cfg.Media.DescriptionMaxChars
cfgmap["media-remote-cache-days"] = cfg.Media.RemoteCacheDays
cfgmap["media-emoji-local-max-size"] = cfg.Media.EmojiLocalMaxSize.String()
cfgmap["media-emoji-remote-max-size"] = cfg.Media.EmojiRemoteMaxSize.String()
cfgmap["media-image-size-hint"] = cfg.Media.ImageSizeHint.String()
cfgmap["media-video-size-hint"] = cfg.Media.VideoSizeHint.String()
cfgmap["media-local-max-size"] = cfg.Media.LocalMaxSize.String()
cfgmap["media-remote-max-size"] = cfg.Media.RemoteMaxSize.String()
cfgmap["media-cleanup-from"] = cfg.Media.CleanupFrom
cfgmap["media-cleanup-every"] = cfg.Media.CleanupEvery
cfgmap["media-ffmpeg-pool-size"] = cfg.Media.FfmpegPoolSize
cfgmap["media-thumb-max-pixels"] = cfg.Media.ThumbMaxPixels
cfgmap["media-remote-cache-duration"] = cfg.Media.RemoteCacheDuration.String()
cfgmap["media-cleanup-cron"] = cfg.Media.CleanupCron.String()
cfgmap["cache-s3-object-info"] = cfg.Cache.S3ObjectInfo
cfgmap["cache-home-timeline-size"] = cfg.Cache.HomeTimelineSize
cfgmap["cache-list-timeline-size"] = cfg.Cache.ListTimelineSize
@@ -1127,19 +1135,22 @@ func (cfg *Configuration) UnmarshalMap(cfgmap map[string]any) error {
}
}
if ival, ok := cfgmap["instance-subscriptions-process-from"]; ok {
var err error
cfg.InstanceSubscriptionsProcessFrom, err = cast.ToStringE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> string for 'instance-subscriptions-process-from': %w", ival, err)
}
if ival, ok := cfgmap["instance-subscriptions-process-from"]; ok && ival != "" {
return errors.New("value received for deprecated field 'instance-subscriptions-process-from', please use 'instance-subscriptions-process-cron' instead")
}
if ival, ok := cfgmap["instance-subscriptions-process-every"]; ok {
var err error
cfg.InstanceSubscriptionsProcessEvery, err = cast.ToDurationE(ival)
if ival, ok := cfgmap["instance-subscriptions-process-every"]; ok && ival != "" {
return errors.New("value received for deprecated field 'instance-subscriptions-process-every', please use 'instance-subscriptions-process-cron' instead")
}
if ival, ok := cfgmap["instance-subscriptions-process-cron"]; ok {
t, err := cast.ToStringE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> time.Duration for 'instance-subscriptions-process-every': %w", ival, err)
return fmt.Errorf("error casting %#v -> string for 'instance-subscriptions-process-cron': %w", ival, err)
}
cfg.InstanceSubscriptionsProcessCron = CronExpression{Expression: (*cronexpr.Expression)(nil), Expr: ""}
if err := cfg.InstanceSubscriptionsProcessCron.Set(t); err != nil {
return fmt.Errorf("error parsing %#v for 'instance-subscriptions-process-cron': %w", ival, err)
}
}
@@ -1351,6 +1362,28 @@ func (cfg *Configuration) UnmarshalMap(cfgmap map[string]any) error {
}
}
if ival, ok := cfgmap["statuses-cleanup-cron"]; ok {
t, err := cast.ToStringE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> string for 'statuses-cleanup-cron': %w", ival, err)
}
cfg.StatusesCleanupCron = CronExpression{Expression: (*cronexpr.Expression)(nil), Expr: ""}
if err := cfg.StatusesCleanupCron.Set(t); err != nil {
return fmt.Errorf("error parsing %#v for 'statuses-cleanup-cron': %w", ival, err)
}
}
if ival, ok := cfgmap["statuses-cleanup-remote-older-than"]; ok {
t, err := cast.ToStringE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> string for 'statuses-cleanup-remote-older-than': %w", ival, err)
}
cfg.StatusesCleanupRemoteOlderThan = 0x0
if err := cfg.StatusesCleanupRemoteOlderThan.Set(t); err != nil {
return fmt.Errorf("error parsing %#v for 'statuses-cleanup-remote-older-than': %w", ival, err)
}
}
if ival, ok := cfgmap["scheduled-statuses-max-total"]; ok {
var err error
cfg.ScheduledStatusesMaxTotal, err = cast.ToIntE(ival)
@@ -1951,14 +1984,6 @@ func (cfg *Configuration) UnmarshalMap(cfgmap map[string]any) error {
}
}
if ival, ok := cfgmap["media-remote-cache-days"]; ok {
var err error
cfg.Media.RemoteCacheDays, err = cast.ToIntE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> int for 'media-remote-cache-days': %w", ival, err)
}
}
if ival, ok := cfgmap["media-emoji-local-max-size"]; ok {
t, err := cast.ToStringE(ival)
if err != nil {
@@ -2025,22 +2050,6 @@ func (cfg *Configuration) UnmarshalMap(cfgmap map[string]any) error {
}
}
if ival, ok := cfgmap["media-cleanup-from"]; ok {
var err error
cfg.Media.CleanupFrom, err = cast.ToStringE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> string for 'media-cleanup-from': %w", ival, err)
}
}
if ival, ok := cfgmap["media-cleanup-every"]; ok {
var err error
cfg.Media.CleanupEvery, err = cast.ToDurationE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> time.Duration for 'media-cleanup-every': %w", ival, err)
}
}
if ival, ok := cfgmap["media-ffmpeg-pool-size"]; ok {
var err error
cfg.Media.FfmpegPoolSize, err = cast.ToIntE(ival)
@@ -2057,6 +2066,40 @@ func (cfg *Configuration) UnmarshalMap(cfgmap map[string]any) error {
}
}
if ival, ok := cfgmap["media-remote-cache-duration"]; ok {
t, err := cast.ToStringE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> string for 'media-remote-cache-duration': %w", ival, err)
}
cfg.Media.RemoteCacheDuration = 0x0
if err := cfg.Media.RemoteCacheDuration.Set(t); err != nil {
return fmt.Errorf("error parsing %#v for 'media-remote-cache-duration': %w", ival, err)
}
}
if ival, ok := cfgmap["media-cleanup-cron"]; ok {
t, err := cast.ToStringE(ival)
if err != nil {
return fmt.Errorf("error casting %#v -> string for 'media-cleanup-cron': %w", ival, err)
}
cfg.Media.CleanupCron = CronExpression{Expression: (*cronexpr.Expression)(nil), Expr: ""}
if err := cfg.Media.CleanupCron.Set(t); err != nil {
return fmt.Errorf("error parsing %#v for 'media-cleanup-cron': %w", ival, err)
}
}
if ival, ok := cfgmap["media-remote-cache-days"]; ok && ival != "" {
return errors.New("value received for deprecated field 'media-remote-cache-days', please use 'media-remote-cache-duration' instead")
}
if ival, ok := cfgmap["media-cleanup-from"]; ok && ival != "" {
return errors.New("value received for deprecated field 'media-cleanup-from', please use 'media-cleanup-cron' instead")
}
if ival, ok := cfgmap["media-cleanup-every"]; ok && ival != "" {
return errors.New("value received for deprecated field 'media-cleanup-every', please use 'media-cleanup-cron' instead")
}
if ival, ok := cfgmap["cache-s3-object-info"]; ok {
var err error
cfg.Cache.S3ObjectInfo, err = cast.ToUint32E(ival)
@@ -3509,45 +3552,66 @@ func GetInstanceLanguages() language.Languages { return global.GetInstanceLangua
func SetInstanceLanguages(v language.Languages) { global.SetInstanceLanguages(v) }
// GetInstanceSubscriptionsProcessFrom safely fetches the Configuration value for state's 'InstanceSubscriptionsProcessFrom' field
func (st *ConfigState) GetInstanceSubscriptionsProcessFrom() (v string) {
func (st *ConfigState) GetInstanceSubscriptionsProcessFrom() (v Deprecated) {
return st.config.InstanceSubscriptionsProcessFrom
}
// SetInstanceSubscriptionsProcessFrom safely sets the Configuration value for state's 'InstanceSubscriptionsProcessFrom' field
func (st *ConfigState) SetInstanceSubscriptionsProcessFrom(v string) {
func (st *ConfigState) SetInstanceSubscriptionsProcessFrom(v Deprecated) {
st.config.InstanceSubscriptionsProcessFrom = v
st.reloadToViper()
}
// GetInstanceSubscriptionsProcessFrom safely fetches the value for global configuration 'InstanceSubscriptionsProcessFrom' field
func GetInstanceSubscriptionsProcessFrom() string {
func GetInstanceSubscriptionsProcessFrom() Deprecated {
return global.GetInstanceSubscriptionsProcessFrom()
}
// SetInstanceSubscriptionsProcessFrom safely sets the value for global configuration 'InstanceSubscriptionsProcessFrom' field
func SetInstanceSubscriptionsProcessFrom(v string) { global.SetInstanceSubscriptionsProcessFrom(v) }
func SetInstanceSubscriptionsProcessFrom(v Deprecated) { global.SetInstanceSubscriptionsProcessFrom(v) }
// GetInstanceSubscriptionsProcessEvery safely fetches the Configuration value for state's 'InstanceSubscriptionsProcessEvery' field
func (st *ConfigState) GetInstanceSubscriptionsProcessEvery() (v time.Duration) {
func (st *ConfigState) GetInstanceSubscriptionsProcessEvery() (v Deprecated) {
return st.config.InstanceSubscriptionsProcessEvery
}
// SetInstanceSubscriptionsProcessEvery safely sets the Configuration value for state's 'InstanceSubscriptionsProcessEvery' field
func (st *ConfigState) SetInstanceSubscriptionsProcessEvery(v time.Duration) {
func (st *ConfigState) SetInstanceSubscriptionsProcessEvery(v Deprecated) {
st.config.InstanceSubscriptionsProcessEvery = v
st.reloadToViper()
}
// GetInstanceSubscriptionsProcessEvery safely fetches the value for global configuration 'InstanceSubscriptionsProcessEvery' field
func GetInstanceSubscriptionsProcessEvery() time.Duration {
func GetInstanceSubscriptionsProcessEvery() Deprecated {
return global.GetInstanceSubscriptionsProcessEvery()
}
// SetInstanceSubscriptionsProcessEvery safely sets the value for global configuration 'InstanceSubscriptionsProcessEvery' field
func SetInstanceSubscriptionsProcessEvery(v time.Duration) {
func SetInstanceSubscriptionsProcessEvery(v Deprecated) {
global.SetInstanceSubscriptionsProcessEvery(v)
}
// GetInstanceSubscriptionsProcessCron safely fetches the Configuration value for state's 'InstanceSubscriptionsProcessCron' field
func (st *ConfigState) GetInstanceSubscriptionsProcessCron() (v CronExpression) {
return st.config.InstanceSubscriptionsProcessCron
}
// SetInstanceSubscriptionsProcessCron safely sets the Configuration value for state's 'InstanceSubscriptionsProcessCron' field
func (st *ConfigState) SetInstanceSubscriptionsProcessCron(v CronExpression) {
st.config.InstanceSubscriptionsProcessCron = v
st.reloadToViper()
}
// GetInstanceSubscriptionsProcessCron safely fetches the value for global configuration 'InstanceSubscriptionsProcessCron' field
func GetInstanceSubscriptionsProcessCron() CronExpression {
return global.GetInstanceSubscriptionsProcessCron()
}
// SetInstanceSubscriptionsProcessCron safely sets the value for global configuration 'InstanceSubscriptionsProcessCron' field
func SetInstanceSubscriptionsProcessCron(v CronExpression) {
global.SetInstanceSubscriptionsProcessCron(v)
}
// GetInstanceStatsMode safely fetches the Configuration value for state's 'InstanceStatsMode' field
func (st *ConfigState) GetInstanceStatsMode() (v string) {
return st.config.InstanceStatsMode
@@ -3990,6 +4054,44 @@ func GetStatusesMediaMaxFiles() int { return global.GetStatusesMediaMaxFiles() }
// SetStatusesMediaMaxFiles safely sets the value for global configuration 'StatusesMediaMaxFiles' field
func SetStatusesMediaMaxFiles(v int) { global.SetStatusesMediaMaxFiles(v) }
// GetStatusesCleanupCron safely fetches the Configuration value for state's 'StatusesCleanupCron' field
func (st *ConfigState) GetStatusesCleanupCron() (v CronExpression) {
return st.config.StatusesCleanupCron
}
// SetStatusesCleanupCron safely sets the Configuration value for state's 'StatusesCleanupCron' field
func (st *ConfigState) SetStatusesCleanupCron(v CronExpression) {
st.config.StatusesCleanupCron = v
st.reloadToViper()
}
// GetStatusesCleanupCron safely fetches the value for global configuration 'StatusesCleanupCron' field
func GetStatusesCleanupCron() CronExpression { return global.GetStatusesCleanupCron() }
// SetStatusesCleanupCron safely sets the value for global configuration 'StatusesCleanupCron' field
func SetStatusesCleanupCron(v CronExpression) { global.SetStatusesCleanupCron(v) }
// GetStatusesCleanupRemoteOlderThan safely fetches the Configuration value for state's 'StatusesCleanupRemoteOlderThan' field
func (st *ConfigState) GetStatusesCleanupRemoteOlderThan() (v longdur.Duration) {
return st.config.StatusesCleanupRemoteOlderThan
}
// SetStatusesCleanupRemoteOlderThan safely sets the Configuration value for state's 'StatusesCleanupRemoteOlderThan' field
func (st *ConfigState) SetStatusesCleanupRemoteOlderThan(v longdur.Duration) {
st.config.StatusesCleanupRemoteOlderThan = v
st.reloadToViper()
}
// GetStatusesCleanupRemoteOlderThan safely fetches the value for global configuration 'StatusesCleanupRemoteOlderThan' field
func GetStatusesCleanupRemoteOlderThan() longdur.Duration {
return global.GetStatusesCleanupRemoteOlderThan()
}
// SetStatusesCleanupRemoteOlderThan safely sets the value for global configuration 'StatusesCleanupRemoteOlderThan' field
func SetStatusesCleanupRemoteOlderThan(v longdur.Duration) {
global.SetStatusesCleanupRemoteOlderThan(v)
}
// GetScheduledStatusesMaxTotal safely fetches the Configuration value for state's 'ScheduledStatusesMaxTotal' field
func (st *ConfigState) GetScheduledStatusesMaxTotal() (v int) {
return st.config.ScheduledStatusesMaxTotal
@@ -5219,23 +5321,6 @@ func GetMediaDescriptionMaxChars() int { return global.GetMediaDescriptionMaxCha
// SetMediaDescriptionMaxChars safely sets the value for global configuration 'Media.DescriptionMaxChars' field
func SetMediaDescriptionMaxChars(v int) { global.SetMediaDescriptionMaxChars(v) }
// GetMediaRemoteCacheDays safely fetches the Configuration value for state's 'Media.RemoteCacheDays' field
func (st *ConfigState) GetMediaRemoteCacheDays() (v int) {
return st.config.Media.RemoteCacheDays
}
// SetMediaRemoteCacheDays safely sets the Configuration value for state's 'Media.RemoteCacheDays' field
func (st *ConfigState) SetMediaRemoteCacheDays(v int) {
st.config.Media.RemoteCacheDays = v
st.reloadToViper()
}
// GetMediaRemoteCacheDays safely fetches the value for global configuration 'Media.RemoteCacheDays' field
func GetMediaRemoteCacheDays() int { return global.GetMediaRemoteCacheDays() }
// SetMediaRemoteCacheDays safely sets the value for global configuration 'Media.RemoteCacheDays' field
func SetMediaRemoteCacheDays(v int) { global.SetMediaRemoteCacheDays(v) }
// GetMediaEmojiLocalMaxSize safely fetches the Configuration value for state's 'Media.EmojiLocalMaxSize' field
func (st *ConfigState) GetMediaEmojiLocalMaxSize() (v bytesize.Size) {
return st.config.Media.EmojiLocalMaxSize
@@ -5338,40 +5423,6 @@ func GetMediaRemoteMaxSize() bytesize.Size { return global.GetMediaRemoteMaxSize
// SetMediaRemoteMaxSize safely sets the value for global configuration 'Media.RemoteMaxSize' field
func SetMediaRemoteMaxSize(v bytesize.Size) { global.SetMediaRemoteMaxSize(v) }
// GetMediaCleanupFrom safely fetches the Configuration value for state's 'Media.CleanupFrom' field
func (st *ConfigState) GetMediaCleanupFrom() (v string) {
return st.config.Media.CleanupFrom
}
// SetMediaCleanupFrom safely sets the Configuration value for state's 'Media.CleanupFrom' field
func (st *ConfigState) SetMediaCleanupFrom(v string) {
st.config.Media.CleanupFrom = v
st.reloadToViper()
}
// GetMediaCleanupFrom safely fetches the value for global configuration 'Media.CleanupFrom' field
func GetMediaCleanupFrom() string { return global.GetMediaCleanupFrom() }
// SetMediaCleanupFrom safely sets the value for global configuration 'Media.CleanupFrom' field
func SetMediaCleanupFrom(v string) { global.SetMediaCleanupFrom(v) }
// GetMediaCleanupEvery safely fetches the Configuration value for state's 'Media.CleanupEvery' field
func (st *ConfigState) GetMediaCleanupEvery() (v time.Duration) {
return st.config.Media.CleanupEvery
}
// SetMediaCleanupEvery safely sets the Configuration value for state's 'Media.CleanupEvery' field
func (st *ConfigState) SetMediaCleanupEvery(v time.Duration) {
st.config.Media.CleanupEvery = v
st.reloadToViper()
}
// GetMediaCleanupEvery safely fetches the value for global configuration 'Media.CleanupEvery' field
func GetMediaCleanupEvery() time.Duration { return global.GetMediaCleanupEvery() }
// SetMediaCleanupEvery safely sets the value for global configuration 'Media.CleanupEvery' field
func SetMediaCleanupEvery(v time.Duration) { global.SetMediaCleanupEvery(v) }
// GetMediaFfmpegPoolSize safely fetches the Configuration value for state's 'Media.FfmpegPoolSize' field
func (st *ConfigState) GetMediaFfmpegPoolSize() (v int) {
return st.config.Media.FfmpegPoolSize
@@ -5406,6 +5457,91 @@ func GetMediaThumbMaxPixels() int { return global.GetMediaThumbMaxPixels() }
// SetMediaThumbMaxPixels safely sets the value for global configuration 'Media.ThumbMaxPixels' field
func SetMediaThumbMaxPixels(v int) { global.SetMediaThumbMaxPixels(v) }
// GetMediaRemoteCacheDuration safely fetches the Configuration value for state's 'Media.RemoteCacheDuration' field
func (st *ConfigState) GetMediaRemoteCacheDuration() (v longdur.Duration) {
return st.config.Media.RemoteCacheDuration
}
// SetMediaRemoteCacheDuration safely sets the Configuration value for state's 'Media.RemoteCacheDuration' field
func (st *ConfigState) SetMediaRemoteCacheDuration(v longdur.Duration) {
st.config.Media.RemoteCacheDuration = v
st.reloadToViper()
}
// GetMediaRemoteCacheDuration safely fetches the value for global configuration 'Media.RemoteCacheDuration' field
func GetMediaRemoteCacheDuration() longdur.Duration { return global.GetMediaRemoteCacheDuration() }
// SetMediaRemoteCacheDuration safely sets the value for global configuration 'Media.RemoteCacheDuration' field
func SetMediaRemoteCacheDuration(v longdur.Duration) { global.SetMediaRemoteCacheDuration(v) }
// GetMediaCleanupCron safely fetches the Configuration value for state's 'Media.CleanupCron' field
func (st *ConfigState) GetMediaCleanupCron() (v CronExpression) {
return st.config.Media.CleanupCron
}
// SetMediaCleanupCron safely sets the Configuration value for state's 'Media.CleanupCron' field
func (st *ConfigState) SetMediaCleanupCron(v CronExpression) {
st.config.Media.CleanupCron = v
st.reloadToViper()
}
// GetMediaCleanupCron safely fetches the value for global configuration 'Media.CleanupCron' field
func GetMediaCleanupCron() CronExpression { return global.GetMediaCleanupCron() }
// SetMediaCleanupCron safely sets the value for global configuration 'Media.CleanupCron' field
func SetMediaCleanupCron(v CronExpression) { global.SetMediaCleanupCron(v) }
// GetMediaRemoteCacheDays safely fetches the Configuration value for state's 'Media.RemoteCacheDays' field
func (st *ConfigState) GetMediaRemoteCacheDays() (v Deprecated) {
return st.config.Media.RemoteCacheDays
}
// SetMediaRemoteCacheDays safely sets the Configuration value for state's 'Media.RemoteCacheDays' field
func (st *ConfigState) SetMediaRemoteCacheDays(v Deprecated) {
st.config.Media.RemoteCacheDays = v
st.reloadToViper()
}
// GetMediaRemoteCacheDays safely fetches the value for global configuration 'Media.RemoteCacheDays' field
func GetMediaRemoteCacheDays() Deprecated { return global.GetMediaRemoteCacheDays() }
// SetMediaRemoteCacheDays safely sets the value for global configuration 'Media.RemoteCacheDays' field
func SetMediaRemoteCacheDays(v Deprecated) { global.SetMediaRemoteCacheDays(v) }
// GetMediaCleanupFrom safely fetches the Configuration value for state's 'Media.CleanupFrom' field
func (st *ConfigState) GetMediaCleanupFrom() (v Deprecated) {
return st.config.Media.CleanupFrom
}
// SetMediaCleanupFrom safely sets the Configuration value for state's 'Media.CleanupFrom' field
func (st *ConfigState) SetMediaCleanupFrom(v Deprecated) {
st.config.Media.CleanupFrom = v
st.reloadToViper()
}
// GetMediaCleanupFrom safely fetches the value for global configuration 'Media.CleanupFrom' field
func GetMediaCleanupFrom() Deprecated { return global.GetMediaCleanupFrom() }
// SetMediaCleanupFrom safely sets the value for global configuration 'Media.CleanupFrom' field
func SetMediaCleanupFrom(v Deprecated) { global.SetMediaCleanupFrom(v) }
// GetMediaCleanupEvery safely fetches the Configuration value for state's 'Media.CleanupEvery' field
func (st *ConfigState) GetMediaCleanupEvery() (v Deprecated) {
return st.config.Media.CleanupEvery
}
// SetMediaCleanupEvery safely sets the Configuration value for state's 'Media.CleanupEvery' field
func (st *ConfigState) SetMediaCleanupEvery(v Deprecated) {
st.config.Media.CleanupEvery = v
st.reloadToViper()
}
// GetMediaCleanupEvery safely fetches the value for global configuration 'Media.CleanupEvery' field
func GetMediaCleanupEvery() Deprecated { return global.GetMediaCleanupEvery() }
// SetMediaCleanupEvery safely sets the value for global configuration 'Media.CleanupEvery' field
func SetMediaCleanupEvery(v Deprecated) { global.SetMediaCleanupEvery(v) }
// GetCacheS3ObjectInfo safely fetches the Configuration value for state's 'Cache.S3ObjectInfo' field
func (st *ConfigState) GetCacheS3ObjectInfo() (v uint32) {
return st.config.Cache.S3ObjectInfo
@@ -7402,17 +7538,6 @@ func flattenConfigMap(cfgmap map[string]any) {
}
}
for _, key := range [][]string{
{"media", "remote-cache-days"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-remote-cache-days"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"media", "emoji-local-max-size"},
} {
@@ -7479,28 +7604,6 @@ func flattenConfigMap(cfgmap map[string]any) {
}
}
for _, key := range [][]string{
{"media", "cleanup-from"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-cleanup-from"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"media", "cleanup-every"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-cleanup-every"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"media", "ffmpeg-pool-size"},
} {
@@ -7523,6 +7626,61 @@ func flattenConfigMap(cfgmap map[string]any) {
}
}
for _, key := range [][]string{
{"media", "remote-cache-duration"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-remote-cache-duration"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"media", "cleanup-cron"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-cleanup-cron"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"media", "remote-cache-days"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-remote-cache-days"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"media", "cleanup-from"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-cleanup-from"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"media", "cleanup-every"},
} {
ival, ok := mapGet(cfgmap, key...)
if ok {
cfgmap["media-cleanup-every"] = ival
nestedKeys[key[0]] = struct{}{}
break
}
}
for _, key := range [][]string{
{"cache", "s3-object-info"},
} {
internal/config/helpers.go
+37
View File
@@ -0,0 +1,37 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package config
import "time"
func GetHTTPClientOutgoingScheme() (schema string) {
if GetHTTPClientInsecureOutgoing() {
return "http://"
}
return "https://"
}
func GetMediaRemoteCacheOlderThanTime(now time.Time) time.Time {
_, dur := GetMediaRemoteCacheDuration().Duration()
return now.Add(-dur)
}
func GetStatusesCleanupRemoteOlderThanTime(now time.Time) time.Time {
_, dur := GetStatusesCleanupRemoteOlderThan().Duration()
return now.Add(-dur)
}
internal/config/types.go
+59 -30
View File
@@ -20,16 +20,56 @@ package config
import (
"errors"
"net/netip"
"strings"
"github.com/hashicorp/cronexpr"
)
// Deprecated is a placeholder type
// for use with config fields that have
// the "deprecated-by" field tag set.
type Deprecated string
// CronExpression is a wrapper for cronexpr.Expression
// to allow parsing by CLI "flag"-like utilities.
type CronExpression struct {
*cronexpr.Expression
Expr string
}
func MustParseCron(expr string) (cron CronExpression) {
if err := cron.Set(expr); err != nil {
panic(err)
}
return
}
func (expr *CronExpression) Set(in string) (err error) {
if in == "" {
return
}
expr.Expr = in // set the raw expression string
expr.Expression, err = cronexpr.Parse(in)
return
}
func (expr *CronExpression) MarshalText() ([]byte, error) {
return []byte(expr.Expr), nil
}
func (expr *CronExpression) UnmarshalText(text []byte) error {
return expr.Set(string(text))
}
func (expr *CronExpression) String() string {
return expr.Expr
}
// IPPrefixes is a type-alias for []netip.Prefix
// to allow parsing by CLI "flag"-like utilities.
type IPPrefixes []netip.Prefix
func (p *IPPrefixes) Set(in string) error {
if p == nil {
return errors.New("nil receiver")
}
prefix, err := netip.ParsePrefix(in)
if err != nil {
return err
@@ -39,7 +79,7 @@ func (p *IPPrefixes) Set(in string) error {
}
func (p *IPPrefixes) Strings() []string {
if p == nil || len(*p) == 0 {
if len(*p) == 0 {
return nil
}
strs := make([]string, len(*p))
@@ -49,14 +89,6 @@ func (p *IPPrefixes) Strings() []string {
return strs
}
func GetHTTPClientOutgoingScheme() (schema string) {
if GetHTTPClientInsecureOutgoing() {
return "http://"
}
return "https://"
}
type InstanceDirectoryMode int16
const (
@@ -67,7 +99,7 @@ const (
)
// MarshalText implements encoding.TextMarshaler{}.
func (i *InstanceDirectoryMode) MarshalText() ([]byte, error) {
func (i InstanceDirectoryMode) MarshalText() ([]byte, error) {
return []byte(i.String()), nil
}
@@ -76,24 +108,8 @@ func (i *InstanceDirectoryMode) UnmarshalText(text []byte) error {
return i.Set(string(text))
}
func (i *InstanceDirectoryMode) String() string {
switch *i {
case InstanceDirectoryModeOff:
return "off"
case InstanceDirectoryModeWebOnly:
return "webonly"
case InstanceDirectoryModeOpen:
return "open"
default:
return "unknown"
}
}
func (i *InstanceDirectoryMode) Set(in string) error {
if i == nil {
return errors.New("nil receiver")
}
switch in {
switch strings.ToLower(in) {
case "off":
*i = InstanceDirectoryModeOff
return nil
@@ -107,3 +123,16 @@ func (i *InstanceDirectoryMode) Set(in string) error {
return errors.New("unrecognized instance directory mode '" + in + "'")
}
}
func (i InstanceDirectoryMode) String() string {
switch i {
case InstanceDirectoryModeOff:
return "off"
case InstanceDirectoryModeWebOnly:
return "webonly"
case InstanceDirectoryModeOpen:
return "open"
default:
return "unknown"
}
}
internal/db/bundb/media.go
+48 -102
View File
@@ -20,11 +20,9 @@ package bundb
import (
"context"
"errors"
"slices"
"code.superseriousbusiness.org/gopkg/xslices"
"code.superseriousbusiness.org/gotosocial/internal/db"
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"code.superseriousbusiness.org/gotosocial/internal/paging"
"code.superseriousbusiness.org/gotosocial/internal/state"
@@ -106,121 +104,69 @@ func (m *mediaDB) UpdateAttachment(ctx context.Context, media *gtsmodel.MediaAtt
return m.state.Caches.DB.Media.Store(media, func() error {
_, err := m.db.NewUpdate().
Model(media).
Where("? = ?", bun.Ident("media_attachment.id"), media.ID).
Where("? = ?", bun.Ident("id"), media.ID).
Column(columns...).
Exec(ctx)
return err
})
}
func (m *mediaDB) DeleteAttachment(ctx context.Context, id string) error {
// Gather necessary fields from
// deleted for cache invaliation.
var deleted gtsmodel.MediaAttachment
deleted.ID = id
func (m *mediaDB) UnattachAttachments(ctx context.Context, ids ...string) error {
// Update media attachments with
// given IDs in the database,
// clearing their `status_id` col.
if _, err := m.db.NewUpdate().
Table("media_attachments").
Where("? IN (?)", bun.Ident("id"), bun.List(ids)).
Set("? = ?", bun.Ident("status_id"), "").
Exec(ctx); err != nil && !errors.Is(err, db.ErrNoEntries) {
return err
}
// Delete media attachment and update related models in new transaction.
err := m.db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
// Invalidate all updated models with given IDs.
m.state.Caches.DB.Media.InvalidateIDs("ID", ids)
// Initially, delete the media model,
// returning the required fields we need.
if _, err := tx.NewDelete().
Model(&deleted).
Where("? = ?", bun.Ident("id"), id).
Returning("?, ?, ?, ?",
bun.Ident("account_id"),
bun.Ident("status_id"),
bun.Ident("avatar"),
bun.Ident("header"),
).
Exec(ctx); err != nil {
return gtserror.Newf("error deleting media: %w", err)
}
return nil
}
// If media was attached to account,
// we need to remove link from account.
if deleted.AccountID != "" {
var account gtsmodel.Account
func (m *mediaDB) DeleteAttachment(ctx context.Context, media *gtsmodel.MediaAttachment) error {
// Delete media attachments with
// given IDs from the database.
if _, err := m.db.NewDelete().
Table("media_attachments").
Where("? = ?", bun.Ident("id"), media.ID).
Exec(ctx); err != nil && !errors.Is(err, db.ErrNoEntries) {
return err
}
// Get related account model.
if _, err := tx.NewSelect().
Model(&account).
Where("? = ?", bun.Ident("id"), deleted.AccountID).
Exec(ctx); err != nil && !errors.Is(err, db.ErrNoEntries) {
return gtserror.Newf("error selecting account: %w", err)
}
// Invalidate deleted media model with its ID.
m.state.Caches.DB.Media.Invalidate("ID", media.ID)
m.state.Caches.OnInvalidateMedia(media)
var set func(*bun.UpdateQuery) *bun.UpdateQuery
return nil
}
switch {
case *deleted.Avatar && account.AvatarMediaAttachmentID == id:
set = func(q *bun.UpdateQuery) *bun.UpdateQuery {
return q.Set("? = NULL", bun.Ident("avatar_media_attachment_id"))
}
case *deleted.Header && account.HeaderMediaAttachmentID == id:
set = func(q *bun.UpdateQuery) *bun.UpdateQuery {
return q.Set("? = NULL", bun.Ident("header_media_attachment_id"))
}
}
func (m *mediaDB) DeleteAttachments(ctx context.Context, ids ...string) error {
deleted := make([]*gtsmodel.MediaAttachment, 0, len(ids))
if set != nil {
// Note: this handles not found.
//
// Update the account model.
q := tx.NewUpdate().
Table("accounts").
Where("? = ?", bun.Ident("id"), account.ID)
if _, err := set(q).Exec(ctx); err != nil {
return gtserror.Newf("error updating account: %w", err)
}
}
}
// Delete media attachments with
// given IDs from the database.
if _, err := m.db.NewDelete().
Model(&deleted).
Where("? IN (?)", bun.Ident("id"), bun.List(ids)).
Returning("?, ?, ?", bun.Ident("id"), bun.Ident("account_id"), bun.Ident("status_id")).
Exec(ctx); err != nil && !errors.Is(err, db.ErrNoEntries) {
return err
}
// If media was attached to a status,
// we need to remove link from status.
if deleted.StatusID != "" {
var status gtsmodel.Status
// Invalidate all deleted models with given IDs,
// calling hooks manually in case not in cache.
m.state.Caches.DB.Media.InvalidateIDs("ID", ids)
for _, deleted := range deleted {
m.state.Caches.OnInvalidateMedia(deleted)
}
// Get related status model.
if _, err := tx.NewSelect().
Model(&status).
Where("? = ?", bun.Ident("id"), deleted.StatusID).
Exec(ctx); err != nil && !errors.Is(err, db.ErrNoEntries) {
return gtserror.Newf("error selecting status: %w", err)
}
// Delete all instances of this deleted media ID from status attachments.
updatedIDs := slices.DeleteFunc(status.AttachmentIDs, func(s string) bool {
return s == id
})
if len(updatedIDs) != len(status.AttachmentIDs) {
// Convert to bun array for serialization.
arrIDs := bunArrayType(tx, updatedIDs)
// Note: this handles not found.
//
// Attachments changed, update the status.
if _, err := tx.NewUpdate().
Table("statuses").
Where("? = ?", bun.Ident("id"), status.ID).
Set("? = ?", bun.Ident("attachment_ids"), arrIDs).
Exec(ctx); err != nil {
return gtserror.Newf("error updating status: %w", err)
}
}
}
return nil
})
// Invalidate cached media with ID, manually
// call invalidate hook in case not in cache.
m.state.Caches.DB.Media.Invalidate("ID", id)
m.state.Caches.OnInvalidateMedia(&deleted)
return err
return nil
}
func (m *mediaDB) GetAttachments(ctx context.Context, page *paging.Page) ([]*gtsmodel.MediaAttachment, error) {
internal/db/bundb/mention.go
+5 -6
View File
@@ -191,19 +191,18 @@ func (m *mentionDB) PutMention(ctx context.Context, mention *gtsmodel.Mention) e
})
}
func (m *mentionDB) DeleteMentionByID(ctx context.Context, id string) error {
// Delete mention with given ID,
// returning the deleted models.
func (m *mentionDB) DeleteMentions(ctx context.Context, ids ...string) error {
// Delete mentions with IDs.
if _, err := m.db.NewDelete().
Table("mentions").
Where("? = ?", bun.Ident("id"), id).
Where("? IN (?)", bun.Ident("id"), bun.List(ids)).
Exec(ctx); err != nil &&
!errors.Is(err, db.ErrNoEntries) {
return err
}
// Invalidate the cached mention with ID.
m.state.Caches.DB.Mention.Invalidate("ID", id)
// Invalidate the cached mentions with given IDs.
m.state.Caches.DB.Mention.InvalidateIDs("ID", ids)
return nil
}
internal/db/bundb/migrations/20260427171503_old_post_cleanup.go
+51
View File
@@ -0,0 +1,51 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package migrations
import (
"context"
dbpkg "code.superseriousbusiness.org/gotosocial/internal/db"
"github.com/uptrace/bun"
// we haven't changed anything on the status model in regards to the
// database since the last migration, but we still need a snapshot so
// just use the status model used in the previous migtration here.
gtsmodel "code.superseriousbusiness.org/gotosocial/internal/db/bundb/migrations/20260221171254_add_flags_column/new"
)
func init() {
up := func(ctx context.Context, db *bun.DB) error {
// Create new statuses index that
// tracks threads with flags.local=true.
return createIndex(ctx, db,
"statuses_local_threads_idx",
"statuses",
dbpkg.BunExpr{"?", dbpkg.Idents("thread_id")},
dbpkg.BitSetExpr("flags", gtsmodel.StatusFlagLocal),
)
}
down := func(ctx context.Context, db *bun.DB) error {
return nil
}
if err := Migrations.Register(up, down); err != nil {
panic(err)
}
}
internal/db/bundb/migrations/util.go
+5 -1
View File
@@ -279,14 +279,18 @@ func dropColumn(ctx context.Context, db bun.IDB, model any, fieldName string) er
func createIndex(ctx context.Context, db bun.IDB, indexName, tableName string, cols db.BunExpr, where ...db.BunExpr) error {
log.Infof(ctx, "creating index '%s' on '%s'", indexName, tableName)
// Attempt to create index.
// Start index create query.
q := db.NewCreateIndex().
Table(tableName).
Index(indexName).
ColumnExpr(cols.Fmt, cols.Arg...)
// Apply any provided clauses.
for _, where := range where {
q = q.Where(where.Fmt, where.Arg...)
}
// Execute the calculated index query.
if _, err := q.Exec(ctx); err != nil {
return gtserror.Newf("error creating index '%s': %w", indexName, err)
}
internal/db/bundb/status.go
+461 -63
View File
@@ -652,7 +652,10 @@ func (s *statusDB) UpdateStatus(ctx context.Context, status *gtsmodel.Status, co
})
}
func (s *statusDB) StubStatus(ctx context.Context, status *gtsmodel.Status) error {
func (s *statusDB) StubStatus(ctx context.Context, status *gtsmodel.Status, includeMedia bool) error {
// Delete status related models before anything.
s.preStatusDelete(ctx, status, includeMedia)
// Take pointer to original
// status before changes, used
// for later cache invalidation.
@@ -696,9 +699,21 @@ func (s *statusDB) StubStatus(ctx context.Context, status *gtsmodel.Status) erro
return nil
}
func (s *statusDB) DeleteStatus(ctx context.Context, status *gtsmodel.Status) error {
if err := s.db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
func (s *statusDB) DeleteStatus(ctx context.Context, status *gtsmodel.Status, includeMedia bool) error {
s.preStatusDelete(ctx, status, includeMedia)
return s.deleteStatus(ctx, status)
}
func (s *statusDB) DeleteStatusBoost(ctx context.Context, boost *gtsmodel.Status) error {
if boost.BoostOfID == "" {
return gtserror.New("not a status boost")
}
// note: there are no related models to delete
return s.deleteStatus(ctx, boost)
}
func (s *statusDB) deleteStatus(ctx context.Context, status *gtsmodel.Status) error {
if err := s.db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
// Actually delete the status.
if res, err := tx.NewDelete().
Model(status).
@@ -727,14 +742,132 @@ func (s *statusDB) DeleteStatus(ctx context.Context, status *gtsmodel.Status) er
return nil
}
func (s *statusDB) onStatusDelete(ctx context.Context, tx bun.Tx, status *gtsmodel.Status) error {
// preStatusDelete performs common status deletion
// logic, deleting related models before the actual
// status is deleted. no error is returned as this
// all is expected to happen in a non-error-blocking
// manner, to ensure status itself is deleted.
func (s *statusDB) preStatusDelete(
ctx context.Context,
status *gtsmodel.Status,
includeMedia bool,
) {
log := log.New().
WithContext(ctx).
WithField("uri", status.URI)
// Delete all notifications referencing this status.
if err := s.state.DB.DeleteNotificationsForStatus(ctx,
status.ID); err != nil {
log.Errorf("db error deleting notifications: %v", err)
}
// Before handling media, ensure
// historic edits are populated.
if !status.EditsPopulated() {
var err error
// Fetch all historic edits of status from database.
status.Edits, err = s.state.DB.GetStatusEditsByIDs(
gtscontext.SetBarebones(ctx),
status.EditIDs,
)
if err != nil {
log.Errorf("db error getting status edits: %v", err)
}
}
if includeMedia {
// Media included, delete all attachments for this status (including edits).
err := s.state.DB.DeleteAttachments(ctx, status.AllAttachmentIDs()...)
if err != nil {
log.Errorf("db error deleting media: %v", err)
}
} else {
// Media not included, simply unattach all attachments from this status.
err := s.state.DB.UnattachAttachments(ctx, status.AllAttachmentIDs()...)
if err != nil {
log.Errorf("db error unattaching media: %v", err)
}
}
// Delete all historical edits of status.
if ids := status.EditIDs; len(ids) > 0 {
if err := s.state.DB.DeleteStatusEdits(ctx, ids); err != nil {
log.Errorf("db error deleting edits: %v", err)
}
}
// Delete any mentions attached to status.
if ids := status.MentionIDs; len(ids) > 0 {
if err := s.state.DB.DeleteMentions(ctx, ids...); err != nil {
log.Errorf("db error deleting mentions: %v", err)
}
}
// Delete all local bookmarks targetting this status.
if err := s.state.DB.DeleteStatusBookmarksForStatus(ctx,
status.ID); err != nil {
log.Errorf("db error deleting bookmarks: %v", err)
}
// Delete any status pin targetting this status.
if err := s.state.DB.DeleteStatusPin(ctx, status.ID); //
err != nil {
log.Errorf("db error deleting pin: %v", err)
}
// Delete all stored favourites targetting status.
if err := s.state.DB.DeleteStatusFavesForStatus(ctx,
status.ID); err != nil {
log.Errorf("db error deleting faves: %v", err)
}
if id := status.PollID; id != "" {
// Delete stored poll attached to this status.
if err := s.state.DB.DeletePollByID(ctx, id); //
err != nil {
log.Errorf("db error deleting poll %s: %v", id, err)
}
}
// Get IDs of any boosts referencing this status.
boostIDs, err := s.state.DB.GetStatusBoostIDs(ctx,
status.ID)
if err != nil {
log.Errorf("db error getting boosts: %v", err)
}
if len(boostIDs) > 0 {
// Delete all boosts with the given selected IDs.
if err := s.DeleteStatusBoosts(ctx, boostIDs...); //
err != nil {
log.Errorf("db error deleting boosts: %v", err)
}
}
// Delete this status from direct message conversations it's part of.
if err := s.state.DB.DeleteStatusFromConversations(ctx, status.ID); //
err != nil {
log.Errorf("db error deleting status from conversations: %v", err)
}
}
// onStatusDelete handles shared side-effects
// on deletion (or stubbing) of a status model.
func (s *statusDB) onStatusDelete(
ctx context.Context,
tx bun.Tx,
status *gtsmodel.Status,
) error {
// delete links between this
// status and any emojis it uses
if _, err := tx.NewDelete().
Table("status_to_emojis").
Where("? = ?", bun.Ident("status_id"), status.ID).
Exec(ctx); err != nil {
return err
return gtserror.Newf("error deleting emoji links: %w", err)
}
// delete links between this
@@ -743,7 +876,7 @@ func (s *statusDB) onStatusDelete(ctx context.Context, tx bun.Tx, status *gtsmod
Table("status_to_tags").
Where("? = ?", bun.Ident("status_id"), status.ID).
Exec(ctx); err != nil {
return err
return gtserror.Newf("error deleting tag links: %w", err)
}
// decrement status author statistics.
@@ -753,7 +886,245 @@ func (s *statusDB) onStatusDelete(ctx context.Context, tx bun.Tx, status *gtsmod
)
}
func (s *statusDB) DeleteStatusLeafStubs(ctx context.Context, page *paging.Page) ([]*gtsmodel.Status, error) {
func (s *statusDB) DeleteStatusBoosts(ctx context.Context, ids ...string) error {
deleted := make([]*gtsmodel.Status, 0, len(ids))
// Delete boosts by their ID.
if _, err := s.db.NewDelete().
Model(&deleted).
Where("? IN (?)", bun.Ident("id"), bun.List(ids)).
Where("? IS NOT NULL", bun.Ident("boost_of_id")). // to double check
Returning("?, ?, ?",
bun.Ident("id"),
bun.Ident("account_id"),
bun.Ident("thread_id"),
).Exec(ctx); err != nil && !errors.Is(err, db.ErrNoEntries) {
return err
}
// Invalidate related DB caches.
for _, status := range deleted {
s.state.Caches.OnInvalidateStatus(status)
}
return nil
}
func (s *statusDB) DeleteOldRemoteStatuses(
ctx context.Context,
olderThan id.ULID,
page *paging.Page,
) (
int, // count
*paging.Page, // next page
error,
) {
if page == nil || page.Limit < 1 {
panic("paging is required")
}
// An example of this query:
// SELECT DISTINCT "statuses"."thread_id" FROM "statuses"
// WHERE (NOT EXISTS (SELECT 1 FROM "statuses" AS "inner"
// WHERE ("inner"."thread_id" = "statuses"."thread_id") AND
// (("inner"."flags" & 8 != 0) OR ("inner"."fetched_at" > '2020-05-16 11:06:18.023+00:00'))))
// AND ("statuses"."thread_id" < '01DB2QFDE7N8KMTSCX1XYQM260')
// ORDER BY "statuses"."thread_id" DESC LIMIT 50
// A quick type alias
// to make things a
// bit more concise.
type ID = bun.Ident
// Extract page params.
maxID := page.Max.Value
limit := page.Limit
if page.Order() != paging.OrderDescending {
panic("must be descending page order")
} else if limit <= 0 {
panic("a limit must be supplied")
}
// Extract timestamp from 'olderThan'.
olderThanTs := olderThan.Timestamp()
// Preallocate slice to store threads.
threadIDs := make([]string, 0, limit)
// Start the main query, SELECT
// distinct `thread_id` entries.
q := s.db.NewSelect()
q = q.Distinct()
q = q.Table("statuses")
q = q.Column("statuses.thread_id")
// Start the sub query, SELECTing
// a 1 return where the local flag
// is set OR fetched_at is recent.
//
// i.e. in the grand scheme of the
// query this lets us filter threads
// by those not fetched recently and
// not containing any local statuses.
sub := q.NewSelect()
sub = sub.TableExpr("? AS ?", ID("statuses"), ID("inner"))
sub = sub.ColumnExpr("1")
sub = sub.Where("? = ?", ID("inner.thread_id"), ID("statuses.thread_id"))
sub = sub.WhereGroup("AND", func(sub *bun.SelectQuery) *bun.SelectQuery {
sub = sub.Where(db.BitIsSet("inner.flags", gtsmodel.StatusFlagLocal))
sub = sub.WhereOr("? > ?", ID("inner.fetched_at"), olderThanTs)
return sub
})
// Apply the sub-query as a WHERE
// NOT EXISTS to the main query,
// and apply paging parameters to
// thread_id to given age window.
q = q.Where("NOT EXISTS (?)", sub)
q = q.Where("? < ?", ID("statuses.thread_id"), maxID)
q = q.OrderExpr("? DESC", ID("statuses.thread_id"))
q = q.Limit(limit)
// Scan query results into slice.
err := q.Scan(ctx, &threadIDs)
if err != nil {
return 0, nil, gtserror.Newf("error selecting remote threads: %w", err)
}
// Check for a return.
if len(threadIDs) == 0 {
return 0, nil, nil
}
// Return a copy of page
// with updated maxID value.
next := new(paging.Page)
(*next) = (*page)
next.Max.Value = threadIDs[0]
// Create an ID lookup map to store returned
// thread IDs that we're going to filter out.
withInteractions := make(idmap, len(threadIDs))
// Convert our list of threadIDs to
// a scannable struct with CTE tags.
values := toThreadIDValues(threadIDs)
// Scan database for statuses contained
// in each thread ID, looking for threads
// that contain statuses with favourites
// by local accounts. This query returns
// only threads with local favourites,
// i.e. the inverse of what we want.
q = s.db.NewSelect()
q = q.With("threads", q.NewValues(&values))
q = q.Table("threads")
q = q.Column("threads.thread_id")
q = q.Join("JOIN ? ON ? = ?", ID("statuses"), ID("statuses.thread_id"), ID("threads.thread_id"))
q = q.Join("JOIN ? ON ? = ?", ID("status_faves"), ID("status_faves.status_id"), ID("statuses.id"))
q = q.Join("JOIN ? ON ? = ?", ID("accounts"), ID("accounts.id"), ID("status_faves.account_id"))
q = q.Where("? IS NULL", ID("accounts.domain"))
if err := q.Scan(ctx, &withInteractions); err != nil {
return 0, nil, gtserror.Newf("error selecting threads with local faves: %w", err)
}
// Filter thread IDs to delete those that have been locally faved.
if threadIDs = slices.DeleteFunc(threadIDs, func(id string) bool {
_, ok := withInteractions[id]
return ok
}); len(threadIDs) == 0 {
// i.e. nothing to delete,
// just return next page.
return 0, next, nil
}
// Convert our remaining list of threadIDs
// to scannable struct type with CTE tags.
values = toThreadIDValues(threadIDs)
// Reset interaction map
// for use again below.
clear(withInteractions)
// Scan database for statuses contained
// in each thread ID, looking for threads
// that contain statuses with bookmarks
// by local accounts. This query returns
// only threads with local bookmarks,
// i.e. the inverse of what we want.
q = s.db.NewSelect()
q = q.With("threads", q.NewValues(&values))
q = q.Table("threads")
q = q.Column("threads.thread_id")
q = q.Join("JOIN ? ON ? = ?", ID("statuses"), ID("statuses.thread_id"), ID("threads.thread_id"))
q = q.Join("JOIN ? ON ? = ?", ID("status_bookmarks"), ID("status_bookmarks.status_id"), ID("statuses.id"))
if err := q.Scan(ctx, &withInteractions); err != nil {
return 0, nil, gtserror.Newf("error selecting threads with bookmarks: %w", err)
}
// Filter our thread IDs to delete those that have been bookmarked.
if threadIDs = slices.DeleteFunc(threadIDs, func(id string) bool {
_, ok := withInteractions[id]
return ok
}); len(threadIDs) == 0 {
// i.e. nothing to delete,
// just return next page.
return 0, next, nil
}
// Prepare slice to store returned deleted statuses.
statuses := make([]*gtsmodel.Status, 0, len(threadIDs))
// Delete statuses contained in determined threads.
statuses, err = s.deleteStatuses(ctx, statuses,
func(q *bun.SelectQuery) *bun.SelectQuery {
return q.Where("? IN (?)", bun.Ident("thread_id"), bun.List(threadIDs))
})
if err != nil {
return 0, nil, err
}
return len(statuses), next, err
}
func (s *statusDB) DeleteLeafStubStatuses(ctx context.Context, page *paging.Page) (int, *paging.Page, error) {
return s.deleteStatusPage(ctx, "id", func(q *bun.SelectQuery) *bun.SelectQuery {
// Only select stubbed statuses.
q = q.Where(db.BitIsSet("flags",
gtsmodel.StatusFlagDeleted))
// Append WHERE clause selecting
// only stubbed statuses that have
// zero replies replies to them.
q = q.Where("(?) = 0",
q.NewRaw("SELECT COUNT(1) FROM ? AS ? WHERE ? = ?",
bun.Ident("statuses"),
bun.Ident("inner"),
bun.Ident("status.id"),
bun.Ident("inner.in_reply_to_id"),
))
return q
}, page)
}
// deleteStatusPage is a wrapper around deleteStatuses()
// that adds paging on the given column name, returning
// the number of deleted statuses and page for the next
// determined page of statuses based on just deleted.
func (s *statusDB) deleteStatusPage(
ctx context.Context,
col string,
where func(*bun.SelectQuery) *bun.SelectQuery,
page *paging.Page,
) (
int, // deleted count
*paging.Page, // next page
error,
) {
if page == nil || page.Limit < 1 {
panic("paging is required")
}
@@ -764,68 +1135,100 @@ func (s *statusDB) DeleteStatusLeafStubs(ctx context.Context, page *paging.Page)
limit := page.Limit
order := page.Order()
// Pprepare status slice to store stubbed.
// Prepare slice to store returned statuses.
statuses := make([]*gtsmodel.Status, 0, limit)
// Start preparing the SELECT query selecting
// only stubbed statuses, and returning enough
// details performing any cache invalidation.
q := s.db.NewSelect().Model(&statuses).
Where(db.BitIsSet("flags", gtsmodel.StatusFlagDeleted)).
ColumnExpr("?, ?, ?, ?, ?",
bun.Ident("status.in_reply_to_id"),
bun.Ident("status.account_id"),
bun.Ident("status.thread_id"),
bun.Ident("status.flags"),
bun.Ident("status.id"),
)
// Finally, delete the statuses, within page.
statuses, err := s.deleteStatuses(ctx, statuses,
func(q *bun.SelectQuery) *bun.SelectQuery {
// Append WHERE clause selecting
// only stubbed statuses that have
// zero replies replies to them.
q = q.Where("(?) = 0",
q.NewRaw("SELECT COUNT(1) FROM ? AS ? WHERE ? = ?",
bun.Ident("statuses"),
bun.Ident("sub"),
bun.Ident("status.id"),
bun.Ident("sub.in_reply_to_id"),
))
// Apply caller specific
// select query filtering.
q = where(q)
if maxID != "" {
// Set a maximum ID boundary if was given.
q = q.Where("? < ?", bun.Ident("id"), maxID)
if maxID != "" {
// Set a maximum ID boundary if was given.
q = q.Where("? < ?", bun.Ident(col), maxID)
}
if minID != "" {
// Set a minimum ID boundary if was given.
q = q.Where("? > ?", bun.Ident(col), minID)
}
// Set query ordering.
if order.Ascending() {
q = q.OrderExpr("? ASC", bun.Ident(col))
} else /* i.e. descending */ {
q = q.OrderExpr("? DESC", bun.Ident(col))
}
// A limit should always
// be supplied for this.
q = q.Limit(limit)
return q
})
if err != nil || len(statuses) == 0 {
return 0, nil, err
}
if minID != "" {
// Set a minimum ID boundary if was given.
q = q.Where("? > ?", bun.Ident("id"), minID)
}
// Set query ordering.
// Ensure statuses are
// in expected order.
if order.Ascending() {
q = q.OrderExpr("? ASC", bun.Ident("id"))
} else /* i.e. descending */ {
q = q.OrderExpr("? DESC", bun.Ident("id"))
slices.Reverse(statuses)
}
// A limit should always
// be supplied for this.
q = q.Limit(limit)
// Calculate and return the
// next page up for deletion.
lo := statuses[len(statuses)-1].ID
return len(statuses), page.Next(lo, ""), err
}
// deleteStatuses encapsulates common logic for
// deleting batches of statuses and handling any
// necessary cache eviction, returning the mostly
// bare (except for core fields) deleted statuses.
func (s *statusDB) deleteStatuses(
ctx context.Context,
dst []*gtsmodel.Status, // preallocated delete destination slice
where func(*bun.SelectQuery) *bun.SelectQuery,
) (
[]*gtsmodel.Status,
error,
) {
// Empty the
// input slice.
dst = dst[:0]
// Start preparing the SELECT query.
q := s.db.NewSelect().Model(&dst)
// Apply caller specific
// select query filtering.
q = where(q)
// Perform the actual database query.
if err := q.Scan(ctx); err != nil {
return nil, err
return dst, err
}
// Check for no values.
if len(statuses) == 0 {
return nil, nil
// Check for values.
if len(dst) == 0 {
return dst, nil
}
// Perform "pre-delete" hooks for status,
// cleaning up any associated models.
for _, status := range dst {
const media = true // include media
s.preStatusDelete(ctx, status, media)
}
// Get status IDs for the actual delete query
// and to later minimize cache mutex unlocks.
statusIDs := make([]string, 0, len(statuses))
statusIDs = xslices.Gather(statusIDs, statuses,
statusIDs := make([]string, 0, len(dst))
statusIDs = xslices.Gather(statusIDs, dst,
func(s *gtsmodel.Status) string { return s.ID })
// Now actually DELETE the statuses by their IDs! This has
@@ -835,7 +1238,7 @@ func (s *statusDB) DeleteStatusLeafStubs(ctx context.Context, page *paging.Page)
Table("statuses").
Where("? IN (?)", bun.Ident("id"), bun.List(statusIDs)).
Exec(ctx); err != nil && !errors.Is(err, db.ErrNoEntries) {
return nil, err
return dst, err
}
// Invalidate all status IDs from cache in one call.
@@ -843,17 +1246,11 @@ func (s *statusDB) DeleteStatusLeafStubs(ctx context.Context, page *paging.Page)
// Manually call invalidate hooks
// for statuses in case not cached.
for _, status := range statuses {
for _, status := range dst {
s.state.Caches.OnInvalidateStatus(status)
}
// We always want returned
// statuses to be DESC order.
if order.Ascending() {
slices.Reverse(statuses)
}
return statuses, nil
return dst, nil
}
func (s *statusDB) GetStatusesUsingEmoji(ctx context.Context, emojiID string) ([]*gtsmodel.Status, error) {
@@ -883,7 +1280,8 @@ func (s *statusDB) GetStatusParents(ctx context.Context, status *gtsmodel.Status
}
if parent == nil {
// Parent status not found (e.g. deleted)
// Parent status not
// found (e.g. deleted)
break
}
@@ -959,7 +1357,7 @@ func getStatusReplyIDs(ctx context.Context, bundb *bun.DB, statusID string) ([]s
}
func (s *statusDB) GetStatusBoosts(ctx context.Context, statusID string) ([]*gtsmodel.Status, error) {
statusIDs, err := s.getStatusBoostIDs(ctx, statusID)
statusIDs, err := s.GetStatusBoostIDs(ctx, statusID)
if err != nil {
return nil, err
}
@@ -984,7 +1382,7 @@ func (s *statusDB) CountStatusBoosts(ctx context.Context, statusID string) (int,
})
}
func (s *statusDB) getStatusBoostIDs(ctx context.Context, statusID string) ([]string, error) {
func (s *statusDB) GetStatusBoostIDs(ctx context.Context, statusID string) ([]string, error) {
return s.state.Caches.DB.BoostOfIDs.Load(statusID, func() ([]string, error) {
return getStatusBoostIDs(ctx, s.db, statusID)
})
internal/db/bundb/status_test.go
+4 -4
View File
@@ -179,7 +179,7 @@ func (suite *StatusTestSuite) TestDeleteStatus() {
targetStatus := &gtsmodel.Status{}
*targetStatus = *suite.testStatuses["admin_account_status_1"]
err := suite.db.DeleteStatus(suite.T().Context(), targetStatus)
err := suite.db.DeleteStatus(suite.T().Context(), targetStatus, true)
suite.NoError(err)
_, err = suite.db.GetStatusByID(suite.T().Context(), targetStatus.ID)
@@ -198,7 +198,7 @@ func (suite *StatusTestSuite) TestPutPopulatedStatus() {
}
// Delete it from the database.
if err := suite.db.DeleteStatus(ctx, targetStatus); err != nil {
if err := suite.db.DeleteStatus(ctx, targetStatus, true); err != nil {
suite.FailNow(err.Error())
}
@@ -562,9 +562,9 @@ func (suite *StatusTestSuite) TestDeleteStatusLeafStubsNowNotALeaf() {
func (suite *StatusTestSuite) testDeleteStatusLeafStubs(expect int) {
ctx := suite.T().Context()
deleted, err := suite.db.DeleteStatusLeafStubs(ctx, &paging.Page{Limit: 100})
n, _, err := suite.db.DeleteLeafStubStatuses(ctx, &paging.Page{Limit: 100})
suite.NoError(err)
suite.Len(deleted, expect)
suite.Equal(n, expect)
}
// hasReply returns whether status ID has a reply (child) in given map of test statuses.
internal/db/bundb/thread_test.go
-11
View File
@@ -28,17 +28,6 @@ type ThreadTestSuite struct {
BunDBStandardTestSuite
}
func (suite *ThreadTestSuite) TestPutThread() {
suite.NoError(
suite.db.PutThread(
suite.T().Context(),
&gtsmodel.Thread{
ID: "01HCWK4HVQ4VGSS1G4VQP3AXZF",
},
),
)
}
func (suite *ThreadTestSuite) TestMuteUnmuteThread() {
var (
threadID = suite.testThreads["local_account_1_status_1"].ID
internal/db/bundb/timeline.go
+4 -16
View File
@@ -53,14 +53,8 @@ func (t *timelineDB) GetHomeTimeline(ctx context.Context, accountID string, page
return nil, gtserror.Newf("error getting home account ids: %w", err)
}
// Provide IDs as common table expression values.
values := make([]accountIDValue, len(accountIDs))
if len(values) != len(accountIDs) {
panic(gtserror.New("bound check elimination"))
}
for i, id := range accountIDs {
values[i] = accountIDValue{id}
}
// Provide IDs as a bun CTE value type.
values := toAccountIDValues(accountIDs)
// "Join" on the CTE values to select only
// statuses belonging to those account IDs.
@@ -221,14 +215,8 @@ func (t *timelineDB) GetListTimeline(ctx context.Context, listID string, page *p
return nil, gtserror.Newf("error getting account IDs in list: %w", err)
}
// Provide IDs as common table expression values.
values := make([]accountIDValue, len(accountIDs))
if len(values) != len(accountIDs) {
panic(gtserror.New("bound check elimination"))
}
for i, id := range accountIDs {
values[i] = accountIDValue{id}
}
// Provide IDs as a bun CTE value type.
values := toAccountIDValues(accountIDs)
// "Join" on the CTE values to select only
// statuses belonging to those account IDs.
internal/db/bundb/util.go
+60 -15
View File
@@ -22,6 +22,7 @@ import (
"database/sql"
"slices"
"strings"
"unsafe"
"code.superseriousbusiness.org/gopkg/log"
"code.superseriousbusiness.org/gotosocial/internal/cache"
@@ -31,7 +32,6 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/paging"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect"
"github.com/uptrace/bun/dialect/pgdialect"
)
// likeEscaper is a thread-safe string replacer which escapes
@@ -63,19 +63,6 @@ func likeOperator(query *bun.SelectQuery) string {
return ""
}
// bunArrayType wraps the given type in a pgdialect.Array
// if needed, which postgres wants for serializing arrays.
func bunArrayType(db bun.IDB, arr any) any {
switch db.Dialect().Name() {
case dialect.SQLite:
return arr // return as-is
case dialect.PG:
return pgdialect.Array(arr)
default:
panic("unreachable")
}
}
// whereLike appends a WHERE clause to the
// given SelectQuery, which searches for
// matches of `search` in the given subQuery
@@ -264,7 +251,65 @@ func parseWhere(w db.Where) (query string, args []interface{}) {
}
// accountIDValue is a convenience struct for using
// CTE's to provide accountIDs to select statuses of.
// CTE's to provide account IDs to a database query.
type accountIDValue struct {
AccountID string `bun:"type:CHAR(26)"`
}
// toAccountIDValues converts a slice of string IDs to []accountIDValue.
func toAccountIDValues(ids []string) []accountIDValue {
if unsafe.Sizeof(accountIDValue{}) != unsafe.Sizeof("") ||
unsafe.Offsetof(accountIDValue{}.AccountID) != 0 {
panic(gtserror.New("compile time assertion"))
}
ptr := (*accountIDValue)(unsafe.Pointer(&ids[0]))
return unsafe.Slice(ptr, len(ids))
}
// threadIDValue is a convenience struct for using
// CTE's to provide thread IDs to a database query.
type threadIDValue struct {
ThreadID string `bun:"type:CHAR(26)"`
}
// toThreadIDValues converts a slice of string IDs to []threadIDValue.
func toThreadIDValues(ids []string) []threadIDValue {
if unsafe.Sizeof(threadIDValue{}) != unsafe.Sizeof("") ||
unsafe.Offsetof(threadIDValue{}.ThreadID) != 0 {
panic(gtserror.New("compile time assertion"))
}
ptr := (*threadIDValue)(unsafe.Pointer(&ids[0]))
return unsafe.Slice(ptr, len(ids))
}
// idmap is a useful scan target for database
// operations that stores a selection of ID
// results directly into a map type for lookups.
type idmap map[string]struct{}
func (m idmap) ScanRow(ctx context.Context, rows *sql.Rows) (err error) {
var id string
err = rows.Scan(&id)
if err != nil {
return
}
m[id] = struct{}{}
return
}
func (m idmap) ScanRows(ctx context.Context, rows *sql.Rows) (n int, err error) {
for rows.Next() {
var id string
err = rows.Scan(&id)
if err != nil {
return
}
m[id] = struct{}{}
n++
}
return
}
func (m idmap) Value() any {
panic("unimplemented")
}
internal/db/media.go
+8 -2
View File
@@ -38,8 +38,14 @@ type Media interface {
// UpdateAttachment will update the given attachment in the database.
UpdateAttachment(ctx context.Context, media *gtsmodel.MediaAttachment, columns ...string) error
// DeleteAttachment deletes the attachment with given ID from the database.
DeleteAttachment(ctx context.Context, id string) error
// UnattachAttachments will unattach given media attachments from any status.
UnattachAttachments(ctx context.Context, ids ...string) error
// DeleteAttachment will delete the single given media attachment from the database.
DeleteAttachment(ctx context.Context, media *gtsmodel.MediaAttachment) error
// DeleteAttachments will delete the given media attachments from the database.
DeleteAttachments(ctx context.Context, ids ...string) error
// GetAttachments fetches media attachments, with given paging parameters.
GetAttachments(ctx context.Context, page *paging.Page) ([]*gtsmodel.MediaAttachment, error)
internal/db/mention.go
+2 -2
View File
@@ -40,6 +40,6 @@ type Mention interface {
// PutMention will insert the given mention into the database.
PutMention(ctx context.Context, mention *gtsmodel.Mention) error
// DeleteMentionByID will delete mention with given ID from the database.
DeleteMentionByID(ctx context.Context, id string) error
// DeleteMentions deletes all given mentions from the database.
DeleteMentions(ctx context.Context, ids ...string) error
}
internal/db/status.go
+32 -6
View File
@@ -21,6 +21,7 @@ import (
"context"
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"code.superseriousbusiness.org/gotosocial/internal/id"
"code.superseriousbusiness.org/gotosocial/internal/paging"
)
@@ -41,6 +42,15 @@ type Status interface {
// GetStatusBoost fetches the status whose boost_of_id column refers to boostOfID, authored by given account ID.
GetStatusBoost(ctx context.Context, boostOfID string, byAccountID string) (*gtsmodel.Status, error)
// GetStatusBoostIDs returns IDs of any boosts of the given status ID.
GetStatusBoostIDs(ctx context.Context, statusID string) ([]string, error)
// DeleteStatusBoost deletes the given boost, unlike DeleteStatus() this will not perform other side-effects.
DeleteStatusBoost(ctx context.Context, boost *gtsmodel.Status) error
// DeleteStatusBoosts deletes boosts with the given IDs, unlike DeleteStatus() this will not perform other side-effects.
DeleteStatusBoosts(ctx context.Context, ids ...string) error
// PopulateStatus ensures that all sub-models of a status are populated (e.g. mentions, attachments, etc).
// Except for edits, to fetch these please call PopulateStatusEdits() .
PopulateStatus(ctx context.Context, status *gtsmodel.Status) error
@@ -54,14 +64,30 @@ type Status interface {
// UpdateStatus updates one status in the database, limited to specific columns if provided.
UpdateStatus(ctx context.Context, status *gtsmodel.Status, columns ...string) error
// StubStatus ...
StubStatus(ctx context.Context, status *gtsmodel.Status) error
// StubStatus stubs out the model (i.e. marks as deleted) given status in the database.
//
// NOTE: unlike other database functions, this handles deletion of related models.
// 'includeMedia' determines whether to also delete vs just unattach any related media.
StubStatus(ctx context.Context, status *gtsmodel.Status, includeMedia bool) error
// DeleteStatus ...
DeleteStatus(ctx context.Context, status *gtsmodel.Status) error
// DeleteStatus removes the given status from the database.
//
// NOTE: unlike other database functions, this handles deletion of related models.
// 'includeMedia' determines whether to also delete vs just unattach any related media.
DeleteStatus(ctx context.Context, status *gtsmodel.Status, includeMedia bool) error
// DeleteLeafStubs ...
DeleteStatusLeafStubs(ctx context.Context, page *paging.Page) ([]*gtsmodel.Status, error)
// DeleteLeafStubStatuses deletes 'leaf' stub statuses from the database according to paging parameters,
// where a 'leaf' status is refers to a status at the end of a thread tree-branch with zero replies.
//
// This returns the number of statuses deleted, and the next page to use for next query, if any remain.
DeleteLeafStubStatuses(ctx context.Context, page *paging.Page) (count int, next *paging.Page, err error)
// DeleteOldRemoteStatuses deletes remote status from the database according to paging parameters,
// specifically limited to statuses that have zero interactions with local users, where no statuses
// have have recent boosts or replies, and all statuses have not been fetched recently.
//
// This returns the number of statuses deleted, and the next page to use for next query, if any remain.
DeleteOldRemoteStatuses(ctx context.Context, olderThan id.ULID, threadPage *paging.Page) (count int, next *paging.Page, err error)
// GetStatuses gets a slice of statuses corresponding to the given status IDs.
GetStatusesByIDs(ctx context.Context, ids []string) ([]*gtsmodel.Status, error)
internal/db/thread.go
-2
View File
@@ -26,8 +26,6 @@ import (
// Thread contains functions for getting/creating
// status threads and thread mutes in the database.
type Thread interface {
// PutThread inserts a new thread.
PutThread(ctx context.Context, thread *gtsmodel.Thread) error
// GetThreadMute gets a single threadMute by its ID.
GetThreadMute(ctx context.Context, id string) (*gtsmodel.ThreadMute, error)
internal/db/util.go
+105 -2
View File
@@ -18,11 +18,14 @@
package db
import (
"context"
"database/sql"
"database/sql/driver"
"github.com/uptrace/bun"
"github.com/uptrace/bun/dialect"
"github.com/uptrace/bun/dialect/pgdialect"
"github.com/uptrace/bun/schema"
)
// BunExpr encompasses the arguments
@@ -33,6 +36,32 @@ type BunExpr struct {
Arg []any
}
// BunQueryable defines a bun type that
// permits starting a new database query.
type BunQueryable interface {
Dialect() schema.Dialect
NewValues(model any) *bun.ValuesQuery
NewSelect() *bun.SelectQuery
NewInsert() *bun.InsertQuery
NewUpdate() *bun.UpdateQuery
NewDelete() *bun.DeleteQuery
NewRaw(query string, args ...any) *bun.RawQuery
}
// BunQueryBuilder defines a bun query builder type.
type BunQueryBuilder[QueryType any] interface {
BunQueryable
Table(tables ...string) QueryType
TableExpr(query string, args ...any) QueryType
Column(columns ...string) QueryType
ColumnExpr(query string, args ...any) QueryType
Where(query string, args ...any) QueryType
Limit(n int) QueryType
Order(orders ...string) QueryType
OrderExpr(query string, args ...any) QueryType
Scan(ctx context.Context, args ...any) error
}
// ToNamedValues converts older driver.Value types to driver.NamedValue types.
func ToNamedValues(args []driver.Value) []driver.NamedValue {
if args == nil {
@@ -68,12 +97,12 @@ func BitNotSet[Type ~int16](col string, value Type) (
return "? & ? = 0", bun.Ident(col), value
}
// BitSetExpr ...
// BitSetExpr returns the results of BitSet() as a BunExpr{}.
func BitSetExpr[Type ~int16](col string, value Type) BunExpr {
return BunExpr{"? & ? != 0", []any{bun.Ident(col), value}}
}
// BitNotSetExpr ...
// BitNotSetExpr returns the results of BitNotSet() as a BunExpr{}.
func BitNotSetExpr[Type ~int16](col string, value Type) BunExpr {
return BunExpr{"? & ? = 0", []any{bun.Ident(col), value}}
}
@@ -107,6 +136,24 @@ func ArrayType(db bun.IDB, arr any) any {
}
}
// GroupArrayExpr returns the appropriate expression for database
// type for grouping an array of column values into a single array.
func GroupArrayExpr(db bun.IDB) string {
switch db.Dialect().Name() {
case dialect.SQLite:
return "json_group_array(?)"
case dialect.PG:
return "array_ag(?)"
default:
panic("unreachable")
}
}
// GroupArray returns a grouped selection of column values as an array type parseable by bun.
func GroupArray(db bun.IDB, col string) (string, bun.Ident) {
return GroupArrayExpr(db), bun.Ident(col)
}
// WhereArrayIsNullOrEmpty returns a BunExpr checking whether value contained in
// 'col' is NULL or is an empty JSON array, depending on current database type.
func WhereArrayIsNullOrEmpty(db bun.IDB, col string) (string, bun.Ident, bun.Ident) {
@@ -136,3 +183,59 @@ func WhereArrayIsNullOrEmptyExpr(db bun.IDB, col string) (expr BunExpr) {
expr.Arg = []any{bun.Ident(col), bun.Ident(col)}
return
}
// Scannable defines a type that
// provides separate SQLite and
// PostgreSQL scanning functions.
type Scannable interface {
ScanRowPG(context.Context, *sql.Rows) error
ScanRowSQLite(context.Context, *sql.Rows) error
ScanRowsPG(context.Context, *sql.Rows) (int, error)
ScanRowsSQLite(context.Context, *sql.Rows) (int, error)
}
// Scan calls Scan() by wrapping type T to use the appropriate scanning function for current db.
// This should only be required for types that bun otherwise gets confused about trying to scan.
func Scan[Q any, T Scannable](ctx context.Context, q BunQueryBuilder[Q], dst T) error {
switch q.Dialect().Name() {
case dialect.PG:
return q.Scan(ctx, &asPGScanner[T]{dst})
case dialect.SQLite:
return q.Scan(ctx, &asSQLiteScanner[T]{dst})
default:
panic("unreachable")
}
}
// asPGScanner is a type-alias for
// ScannableValue that calls ScanRow(s)?PG().
type asPGScanner[T Scannable] struct{ T T }
func (v *asPGScanner[T]) ScanRow(ctx context.Context, rows *sql.Rows) error {
return v.T.ScanRowPG(ctx, rows)
}
func (v *asPGScanner[T]) ScanRows(ctx context.Context, rows *sql.Rows) (int, error) {
return v.T.ScanRowsPG(ctx, rows)
}
func (v *asPGScanner[T]) Value() any {
return v.T
}
// asSQLiteScanner is a type-alias for
// ScannableValue that calls ScanRow(s)?SQLite().
type asSQLiteScanner[T Scannable] struct{ T T }
func (v *asSQLiteScanner[T]) ScanRow(ctx context.Context, rows *sql.Rows) error {
return v.T.ScanRowSQLite(ctx, rows)
}
func (v *asSQLiteScanner[T]) ScanRows(ctx context.Context, rows *sql.Rows) (int, error) {
return v.T.ScanRowsSQLite(ctx, rows)
}
func (v *asSQLiteScanner[T]) Value() any {
return v.T
}
internal/federation/dereferencing/status.go
+1 -1
View File
@@ -299,7 +299,7 @@ func (d *Dereferencer) enrichAndStoreStatusSafely(
// Gone (410) definitely indicates deletion.
// Remove status if it was an existing one.
case code == http.StatusGone && !isNew:
if err := d.state.DB.StubStatus(ctx, status); err != nil {
if err := d.state.DB.StubStatus(ctx, status, true); err != nil {
log.Error(ctx, "error deleting gone status %s: %v", uriStr, err)
}
internal/federation/dereferencing/status_permitted.go
+2 -2
View File
@@ -102,8 +102,8 @@ func (d *Dereferencer) isPermittedStatus(
if !permitted && !isNew {
log.Infof(ctx, "deleting unpermitted: %s", existing.URI)
// Delete existing status from database as no longer permitted.
if err := d.state.DB.DeleteStatus(ctx, existing); err != nil {
// Delete existing status from database as is no longer permitted.
if err := d.state.DB.DeleteStatus(ctx, existing, true); err != nil {
log.Errorf(ctx, "error deleting %s after permissivity fail: %v", existing.URI, err)
}
}
internal/gtsmodel/statusflags.go
-2
View File
@@ -35,8 +35,6 @@ const (
// StatusFlagDeleted indices whether status is marked as deleted,
// (and thus should be stubbed-out and replaced with placeholders).
//
// TODO: NOT YET IMPLEMENTED
StatusFlagDeleted StatusFlag = 1 << 1
// StatusFlagSensitive indicates whether status is marked as sensitive.
internal/id/ulid.go
+15 -2
View File
@@ -42,8 +42,10 @@ const (
// bigRandomRange contains randomRange as big.Int.
var bigRandomRange = big.NewInt(randomRange)
// ULID represents a Universally Unique Lexicographically Sortable Identifier of 26 characters. See https://github.com/oklog/ulid
type ULID string
// ULID represents the actual ULID binary type, which
// itself can easily be converted to string or have its
// timestamp value extracted from it.
type ULID = ulid.ULID
// newAt returns a new ulid.ULID from timestamp,
// else panics with caller's caller information.
@@ -117,3 +119,14 @@ func ZeroULIDForTime(t time.Time) string {
}
return ulid.String()
}
// ZeroBinaryULIDForTime is the same as ZeroULIDForTime()
// except that it returns the binary ULID representation.
func ZeroBinaryULIDForTime(t time.Time) ULID {
ts := ulid.Timestamp(t)
var ulid ulid.ULID
if err := ulid.SetTime(ts); err != nil {
panic(err)
}
return ulid
}
internal/processing/account/rss_test.go
+1 -1
View File
@@ -332,7 +332,7 @@ func (suite *GetRSSTestSuite) TestGetAccountRSSZorkNoPosts() {
// Now delete them! Hahaha!
for _, status := range statuses {
if err := suite.db.DeleteStatus(ctx, status); err != nil {
if err := suite.db.DeleteStatus(ctx, status, true); err != nil {
suite.FailNow(err.Error())
}
}
internal/processing/admin/media.go
+7 -8
View File
@@ -20,11 +20,13 @@ package admin
import (
"context"
"fmt"
"time"
"code.superseriousbusiness.org/gopkg/log"
"code.superseriousbusiness.org/gotosocial/internal/gtscontext"
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"codeberg.org/gruf/go-longdur"
)
// MediaRefetch forces a refetch of remote emojis.
@@ -49,19 +51,16 @@ func (p *Processor) MediaRefetch(ctx context.Context, requestingAccount *gtsmode
return nil
}
// MediaPrune triggers a non-blocking prune of unused
// media, orphaned, uncaching remote and fixing cache states.
func (p *Processor) MediaPrune(
ctx context.Context,
remoteCacheDays int,
) gtserror.WithCode {
// MediaPrune triggers a non-blocking prune of unused media, orphaned, uncaching remote and fixing cache states.
func (p *Processor) MediaPrune(ctx context.Context, remoteCacheAge longdur.Duration) gtserror.WithCode {
// Start background task
// performing media cleanup.
go func() {
now := time.Now()
ctx := gtscontext.WithValues(context.Background(), ctx)
p.cleaner.Media().AllAndFix(ctx, remoteCacheDays)
p.cleaner.Emoji().AllAndFix(ctx, remoteCacheDays)
p.cleaner.Media().AllAndFix(ctx, now, remoteCacheAge)
p.cleaner.Emoji().AllAndFix(ctx, now, remoteCacheAge)
}()
return nil
internal/processing/media/delete.go
+2 -2
View File
@@ -57,11 +57,11 @@ func (p *Processor) Delete(ctx context.Context, mediaAttachmentID string) gtserr
}
// delete the attachment
if err := p.state.DB.DeleteAttachment(ctx, mediaAttachmentID); err != nil && !errors.Is(err, db.ErrNoEntries) {
if err := p.state.DB.DeleteAttachment(ctx, attachment); err != nil && !errors.Is(err, db.ErrNoEntries) {
errs = append(errs, fmt.Sprintf("remove attachment: %s", err))
}
if len(errs) != 0 {
if len(errs) > 0 {
return gtserror.NewErrorInternalError(fmt.Errorf("Delete: one or more errors removing attachment with id %s: %s", mediaAttachmentID, strings.Join(errs, "; ")))
}
internal/processing/status/boost.go
+12 -8
View File
@@ -25,6 +25,7 @@ import (
"code.superseriousbusiness.org/gotosocial/internal/ap"
apimodel "code.superseriousbusiness.org/gotosocial/internal/api/model"
"code.superseriousbusiness.org/gotosocial/internal/db"
"code.superseriousbusiness.org/gotosocial/internal/gtscontext"
"code.superseriousbusiness.org/gotosocial/internal/gtserror"
"code.superseriousbusiness.org/gotosocial/internal/gtsmodel"
"code.superseriousbusiness.org/gotosocial/internal/messages"
@@ -181,8 +182,7 @@ func (p *Processor) BoostRemove(
targetID string,
) (*apimodel.Status, gtserror.WithCode) {
// Get target status and ensure it's not a boost.
target, errWithCode := p.c.GetVisibleTargetStatus(
ctx,
target, errWithCode := p.c.GetVisibleTargetStatus(ctx,
requester,
targetID,
nil, // default freshness
@@ -191,8 +191,7 @@ func (p *Processor) BoostRemove(
return nil, errWithCode
}
target, errWithCode = p.c.UnwrapIfBoost(
ctx,
target, errWithCode = p.c.UnwrapIfBoost(ctx,
requester,
target,
)
@@ -227,7 +226,7 @@ func (p *Processor) BoostRemove(
}
// Delete boost wrapper from the database.
err = p.state.DB.DeleteStatus(ctx, boost)
err = p.state.DB.DeleteStatusBoost(ctx, boost)
if err != nil && !errors.Is(err, db.ErrNoEntries) {
err := gtserror.Newf("db error deleting status: %w", err)
return nil, gtserror.NewErrorInternalError(err)
@@ -277,25 +276,30 @@ func (p *Processor) StatusBoostedBy(ctx context.Context, requestingAccount *gtsm
err = fmt.Errorf("BoostedBy: error seeing if status %s is visible: %s", targetStatus.ID, err)
return nil, gtserror.NewErrorNotFound(err)
}
if !visible {
err = errors.New("BoostedBy: status is not visible")
return nil, gtserror.NewErrorNotFound(err)
}
statusBoosts, err := p.state.DB.GetStatusBoosts(ctx, targetStatus.ID)
boosts, err := p.state.DB.GetStatusBoosts(
gtscontext.SetBarebones(ctx),
targetStatus.ID,
)
if err != nil {
err = fmt.Errorf("BoostedBy: error seeing who boosted status: %s", err)
return nil, gtserror.NewErrorNotFound(err)
}
// filter account IDs so the user doesn't see accounts they blocked or which blocked them
accountIDs := make([]string, 0, len(statusBoosts))
for _, s := range statusBoosts {
accountIDs := make([]string, 0, len(boosts))
for _, s := range boosts {
blocked, err := p.state.DB.IsEitherBlocked(ctx, requestingAccount.ID, s.AccountID)
if err != nil {
err = fmt.Errorf("BoostedBy: error checking blocks: %s", err)
return nil, gtserror.NewErrorNotFound(err)
}
if !blocked {
accountIDs = append(accountIDs, s.AccountID)
}
internal/processing/workers/fromclientapi_test.go
+1 -15
View File
@@ -128,15 +128,7 @@ func (suite *FromClientAPITestSuite) newStatus(
}
if createThread {
newThread := &gtsmodel.Thread{
ID: id.NewULID(),
}
newStatus.ThreadID = newThread.ID
if err := state.DB.PutThread(ctx, newThread); err != nil {
suite.FailNow(err.Error())
}
newStatus.ThreadID = id.NewULID()
}
// Put the status in the db, to mimic what would
@@ -2180,12 +2172,6 @@ func (suite *FromClientAPITestSuite) TestProcessStatusDelete() {
homeStream = streams[stream.TimelineHome]
)
// Delete the status from the db first, to mimic what
// would have already happened earlier up the flow
if err := testStructs.State.DB.DeleteStatus(ctx, deletedStatus); err != nil {
suite.FailNow(err.Error())
}
// Process the status delete.
if err := testStructs.Processor.Workers().ProcessFromClientAPI(
ctx,
internal/processing/workers/util.go
+12 -119
View File
@@ -55,7 +55,7 @@ func (u *utils) deleteBoost(
u.surfacer.DeleteStatusFromTimelines(ctx, boost.ID)
// Finally, delete boost wrapper status itself.
if err := u.state.DB.DeleteStatus(ctx, boost); //
if err := u.state.DB.DeleteStatusBoost(ctx, boost); //
err != nil {
return gtserror.Newf("db error deleting boost %s: %w", boost.URI, err)
}
@@ -99,139 +99,32 @@ func (u *utils) deleteStatus(
}
}
// Delete all notifications referencing this status.
if err := u.state.DB.DeleteNotificationsForStatus(ctx,
status.ID); err != nil {
log.Errorf("db error deleting notifications: %v", err)
}
// Before handling media, ensure
// historic edits are populated.
if !status.EditsPopulated() {
var err error
// Fetch all historic edits of status from database.
status.Edits, err = u.state.DB.GetStatusEditsByIDs(
gtscontext.SetBarebones(ctx),
status.EditIDs,
)
if err != nil {
log.Errorf("db error getting status edits: %v", err)
}
}
// Either delete all attachments for this status,
// or simply detach + clean them separately later.
//
// Reason to detach rather than delete is that
// the author might want to reattach them to another
// status immediately (in case of delete + redraft).
if attachments {
// todo:u.state.DB.DeleteAttachmentsForStatus
for _, id := range status.AllAttachmentIDs() {
if err := u.media.Delete(ctx, id); err != nil {
log.Errorf("db error deleting media %s: %v", id, err)
}
}
} else {
// todo:u.state.DB.UnattachAttachmentsForStatus
for _, id := range status.AllAttachmentIDs() {
if _, err := u.media.Unattach(ctx, status.Account, id); err != nil {
log.Errorf("error unattaching media %s: %v", id, err)
}
}
}
// Delete all historical edits of status.
if ids := status.EditIDs; len(ids) > 0 {
if err := u.state.DB.DeleteStatusEdits(ctx, ids); err != nil {
log.Errorf("db error deleting edits: %v", err)
}
}
// Delete mentions attached to status.
// todo:u.state.DB.DeleteMentionsForStatus
for _, id := range status.MentionIDs {
if err := u.state.DB.DeleteMentionByID(ctx, id); err != nil {
log.Errorf("db error deleting mention %s: %v", id, err)
}
}
// Delete all local bookmarks targetting this status.
if err := u.state.DB.DeleteStatusBookmarksForStatus(ctx,
status.ID); err != nil {
log.Errorf("db error deleting bookmarks: %v", err)
}
// Delete any status pin targetting this status.
if err := u.state.DB.DeleteStatusPin(ctx, status.ID); //
err != nil {
log.Errorf("db error deleting pin: %v", err)
}
// Delete all stored favourites targetting status.
if err := u.state.DB.DeleteStatusFavesForStatus(ctx,
status.ID); err != nil {
log.Errorf("db error deleting faves: %v", err)
}
if id := status.PollID; id != "" {
// Delete stored poll attached to this status.
if err := u.state.DB.DeletePollByID(ctx, id); //
err != nil {
log.Errorf("db error deleting poll %s: %v", id, err)
}
// Cancel scheduled expiry task for poll.
_ = u.state.Workers.Scheduler.Cancel(id)
}
// Get all boost of this status so that we can
// delete those boosts + remove from timelines.
//
// TODO: page this to prevent memory issues.
boosts, err := u.state.DB.GetStatusBoosts(
// We MUST set a barebones context here,
// as depending on where it came from the
// original BoostOf may already be gone.
gtscontext.SetBarebones(ctx),
// Get IDs of any boosts referencing this status.
boostIDs, err := u.state.DB.GetStatusBoostIDs(ctx,
status.ID)
if err != nil {
log.Errorf("db error getting boosts: %v", err)
}
for _, boost := range boosts {
// Delete boost wrapper targetting main status.
if err := u.state.DB.DeleteStatus(ctx, boost); //
err != nil {
log.Errorf("db error deleting boost %s: %v", boost.URI, err)
}
// Remove the status boost from any and all timelines.
u.surfacer.DeleteStatusFromTimelines(ctx, boost.ID)
}
// Delete this status from direct message conversations it's part of.
if err := u.state.DB.DeleteStatusFromConversations(ctx, status.ID); //
err != nil {
log.Errorf("db error deleting status from conversations: %v", err)
}
if wipe {
// Fully delete status model from database.
err := u.state.DB.DeleteStatus(ctx, status)
// Fully delete status and related models from database.
err := u.state.DB.DeleteStatus(ctx, status, attachments)
if err != nil {
return gtserror.Newf("db error deleting status %s: %w", status.URI, err)
}
} else {
// Stub out the status model to delete it.
err := u.state.DB.StubStatus(ctx, status)
// Stub out the status and related model to delete it.
err := u.state.DB.StubStatus(ctx, status, attachments)
if err != nil {
return gtserror.Newf("db error stubbing status %s: %w", status.URI, err)
}
}
for _, id := range boostIDs {
// Remove the boost from any and all timelines.
u.surfacer.DeleteStatusFromTimelines(ctx, id)
}
// Remove the status from timeline caches / streams.
u.surfacer.DeleteStatusFromTimelines(ctx, status.ID)
internal/scheduler/scheduler.go
+46 -31
View File
@@ -22,6 +22,7 @@ import (
"sync"
"time"
"code.superseriousbusiness.org/gotosocial/internal/config"
"codeberg.org/gruf/go-runners"
"codeberg.org/gruf/go-sched"
)
@@ -54,41 +55,18 @@ func (sch *Scheduler) Stop() bool {
return false
}
// AddOnce schedules the given task to run at time, registered under the given ID. Returns false if task already exists for id.
func (sch *Scheduler) AddOnce(id string, start time.Time, fn func(context.Context, time.Time)) bool {
return sch.schedule(id, fn, (*sched.Once)(&start))
}
// AddRecurring schedules the given task to return at given period, starting at given time, registered under given id. Returns false if task already exists for id.
func (sch *Scheduler) AddRecurring(id string, start time.Time, freq time.Duration, fn func(context.Context, time.Time)) bool {
return sch.schedule(id, fn, &sched.PeriodicAt{Once: sched.Once(start), Period: sched.Periodic(freq)})
}
// Cancel attempts to cancel a scheduled task with id, returns false if no task found.
func (sch *Scheduler) Cancel(id string) bool {
// Attempt to acquire and
// delete task with iD.
sch.mu.Lock()
task, ok := sch.ts[id]
delete(sch.ts, id)
sch.mu.Unlock()
if !ok {
// none found.
return false
}
// Cancel the queued
// job from Scheduler.
task.cncl()
return true
}
func (sch *Scheduler) schedule(id string, fn func(context.Context, time.Time), t sched.Timing) bool {
// Add schedules the given task to run with timing 't', registered under the given 'id'. Returns false if task already exists for 'id'.
func (sch *Scheduler) Add(id string, fn func(context.Context, time.Time), t sched.Timing) bool {
if fn == nil {
panic("nil function")
}
if isEmptyCron(t) {
// nothing
// to schedule
return true
}
// Acquire lock.
sch.mu.Lock()
defer sch.mu.Unlock()
@@ -119,9 +97,46 @@ func (sch *Scheduler) schedule(id string, fn func(context.Context, time.Time), t
return true
}
// AddOnce schedules the given task to run at time, registered under the given ID. Returns false if task already exists for id.
func (sch *Scheduler) AddOnce(id string, start time.Time, fn func(context.Context, time.Time)) bool {
return sch.Add(id, fn, (*sched.Once)(&start))
}
// AddRecurring schedules the given task to return at given period, starting at given time, registered under given id. Returns false if task already exists for id.
func (sch *Scheduler) AddRecurring(id string, start time.Time, freq time.Duration, fn func(context.Context, time.Time)) bool {
return sch.Add(id, fn, &sched.PeriodicAt{Once: sched.Once(start), Period: sched.Periodic(freq)})
}
// Cancel attempts to cancel a scheduled task with id, returns false if no task found.
func (sch *Scheduler) Cancel(id string) bool {
// Attempt to acquire and
// delete task with iD.
sch.mu.Lock()
task, ok := sch.ts[id]
delete(sch.ts, id)
sch.mu.Unlock()
if !ok {
// none found.
return false
}
// Cancel the queued
// job from Scheduler.
task.cncl()
return true
}
// task simply wraps together a scheduled
// job, and the matching cancel function.
type task struct {
job *sched.Job
cncl func()
}
// isEmptyCron checks whether timing is an
// empty CronExpression{} value, i.e. unset.
func isEmptyCron(t sched.Timing) bool {
expr, ok := t.(config.CronExpression)
return ok && expr.Expr == ""
}
internal/subscriptions/domainperms.go
+8 -49
View File
@@ -46,46 +46,7 @@ import (
// ScheduleJobs schedules domain permission subscription
// fetching + updating using configured parameters.
//
// Returns an error if `MediaCleanupFrom`
// is not a valid format (hh:mm:ss).
func (s *Subscriptions) ScheduleJobs() error {
const hourMinute = "15:04"
var (
now = time.Now()
processEvery = config.GetInstanceSubscriptionsProcessEvery()
processFromStr = config.GetInstanceSubscriptionsProcessFrom()
)
// Parse processFromStr as hh:mm.
// Resulting time will be on 1 Jan year zero.
processFrom, err := time.Parse(hourMinute, processFromStr)
if err != nil {
return gtserror.Newf(
"error parsing '%s' in time format 'hh:mm': %w",
processFromStr, err,
)
}
// Time travel from
// year zero, groovy.
firstProcessAt := time.Date(
now.Year(),
now.Month(),
now.Day(),
processFrom.Hour(),
processFrom.Minute(),
0,
0,
now.Location(),
)
// Ensure first processing is in the future.
for firstProcessAt.Before(now) {
firstProcessAt = firstProcessAt.Add(processEvery)
}
fn := func(ctx context.Context, start time.Time) {
log.Info(ctx, "starting instance subscriptions processing")
@@ -115,19 +76,17 @@ func (s *Subscriptions) ScheduleJobs() error {
log.Infof(ctx, "finished instance subscriptions processing after %s", time.Since(start))
}
log.Infof(nil,
"scheduling instance subscriptions processing to run every %s, starting from %s; next processing will run at %s",
processEvery, processFromStr, firstProcessAt,
)
expr := config.GetInstanceSubscriptionsProcessCron()
log.Infof(nil, "scheduling instance subscriptions processing: %s", expr.Expr)
// Schedule processing to execute according to schedule.
if !s.state.Workers.Scheduler.AddRecurring(
"@subsprocessing",
firstProcessAt,
processEvery,
// Schedule processing to
// execute according to schedule.
if !s.state.Workers.Scheduler.Add(
"@instancesubsprocessing",
fn,
expr,
) {
panic("failed to schedule @subsprocessing")
panic("failed to schedule @instancesubsprocessing")
}
return nil
internal/typeutils/internaltoas_test.go
+1 -1
View File
@@ -1167,7 +1167,7 @@ func (suite *InternalToASTestSuite) TestStatusToASDeletePublicReplyOriginalDelet
ctx := suite.T().Context()
// Delete the status this replies to.
if err := suite.db.DeleteStatus(ctx, testStatus); err != nil {
if err := suite.db.DeleteStatus(ctx, testStatus, true); err != nil {
suite.FailNow(err.Error())
}
test/envparsing.sh
+5 -5
View File
@@ -169,8 +169,7 @@ EXPECT=$(cat << "EOF"
],
"instance-robots-allow-indexing": true,
"instance-stats-mode": "baffle",
"instance-subscriptions-process-every": 86400000000000,
"instance-subscriptions-process-from": "23:00",
"instance-subscriptions-process-cron": "0 23 * * *",
"landing-page-user": "admin",
"letsencrypt-cert-dir": "/gotosocial/storage/certs",
"letsencrypt-email-address": "",
@@ -182,8 +181,7 @@ EXPECT=$(cat << "EOF"
"log-format": "json",
"log-level": "info",
"log-timestamp-format": "banana",
"media-cleanup-every": 86400000000000,
"media-cleanup-from": "00:00",
"media-cleanup-cron": "0 0 * * *",
"media-description-max-chars": 5000,
"media-description-min-chars": 69,
"media-emoji-local-max-size": "420B",
@@ -191,7 +189,7 @@ EXPECT=$(cat << "EOF"
"media-ffmpeg-pool-size": 8,
"media-image-size-hint": "5.00MiB",
"media-local-max-size": "420B",
"media-remote-cache-days": 30,
"media-remote-cache-duration": "1 week",
"media-remote-max-size": "420B",
"media-thumb-max-pixels": 42069,
"media-video-size-hint": "40.0MiB",
@@ -231,6 +229,8 @@ EXPECT=$(cat << "EOF"
"smtp-port": 4269,
"smtp-username": "sex-haver",
"software-version": "",
"statuses-cleanup-cron": "0 1 * * 0",
"statuses-cleanup-remote-older-than": "0 sec",
"statuses-max-chars": 69,
"statuses-media-max-files": 1,
"statuses-poll-max-options": 1,
testrig/config.go
+14 -14
View File
@@ -103,11 +103,10 @@ func testDefaults() config.Configuration {
TagStr: "en-gb",
},
},
InstanceSubscriptionsProcessFrom: "23:00", // 11pm,
InstanceSubscriptionsProcessEvery: 24 * time.Hour, // 1/day.
InstanceAllowBackdatingStatuses: true,
InstanceDirectoryMode: config.InstanceDirectoryModeOpen,
InstanceRobotsAllowIndexing: true,
InstanceSubscriptionsProcessCron: config.Defaults.InstanceSubscriptionsProcessCron, // daily at 11pm
InstanceAllowBackdatingStatuses: true,
InstanceDirectoryMode: config.InstanceDirectoryModeOpen,
InstanceRobotsAllowIndexing: true,
AccountsRegistrationOpen: true,
AccountsReasonRequired: true,
@@ -120,14 +119,13 @@ func testDefaults() config.Configuration {
Media: config.MediaConfiguration{
DescriptionMinChars: 0,
DescriptionMaxChars: 500,
RemoteCacheDays: 7,
LocalMaxSize: 40 * bytesize.MiB,
RemoteMaxSize: 40 * bytesize.MiB,
EmojiLocalMaxSize: 51200, // 50KiB
EmojiRemoteMaxSize: 102400, // 100KiB
CleanupFrom: "00:00", // midnight.
CleanupEvery: 24 * time.Hour, // 1/day.
EmojiLocalMaxSize: 51200, // 50KiB
EmojiRemoteMaxSize: 102400, // 100KiB
ThumbMaxPixels: 512,
RemoteCacheDuration: config.Defaults.Media.RemoteCacheDuration, // i.e. 7 days
CleanupCron: config.Defaults.Media.CleanupCron, // i.e. daily at 0am
},
// the testrig uses in-memory storage by default, so we can
@@ -146,10 +144,12 @@ func testDefaults() config.Configuration {
StorageS3Proxy: envBool("GTS_STORAGE_S3_PROXY", false),
StorageS3RedirectURL: envStr("GTS_STORAGE_S3_REDIRECT_URL", ""),
StatusesMaxChars: 5000,
StatusesPollMaxOptions: 6,
StatusesPollOptionMaxChars: 50,
StatusesMediaMaxFiles: 6,
StatusesMaxChars: 5000,
StatusesPollMaxOptions: 6,
StatusesPollOptionMaxChars: 50,
StatusesMediaMaxFiles: 6,
StatusesCleanupCron: config.Defaults.StatusesCleanupCron, // i.e. daily at 1am
StatusesCleanupRemoteOlderThan: config.Defaults.StatusesCleanupRemoteOlderThan, // i.e. disabled
ScheduledStatusesMaxTotal: 300,
ScheduledStatusesMaxDaily: 25,
vendor/code.superseriousbusiness.org/gopkg/log/entry.go
Generated Vendored
+138
View File
@@ -79,6 +79,26 @@ func (e Entry) Debugf(s string, a ...any) {
logf(e.ctx, DEBUG, e.kvs, s, a...)
}
// DebugKV will log the one key-value field to the log at DEBUG level.
func (e Entry) DebugKV(key string, value any) {
if DEBUG < state.level {
return
}
e.kvs = append(e.kvs, kv.Field{K: key, V: value})
logf(e.ctx, DEBUG, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-1]
}
// DebugKVs will log key-value fields to the log at DEBUG level.
func (e Entry) DebugKVs(kvs ...kv.Field) {
if DEBUG < state.level {
return
}
e.kvs = append(e.kvs, kvs...)
logf(e.ctx, DEBUG, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-len(kvs)]
}
// Info will log formatted args as 'msg' field to the log at INFO level.
func (e Entry) Info(a ...any) {
if INFO < state.level {
@@ -95,6 +115,26 @@ func (e Entry) Infof(s string, a ...any) {
logf(e.ctx, INFO, e.kvs, s, a...)
}
// InfoKV will log the one key-value field to the log at INFO level.
func (e Entry) InfoKV(key string, value any) {
if INFO < state.level {
return
}
e.kvs = append(e.kvs, kv.Field{K: key, V: value})
logf(e.ctx, INFO, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-1]
}
// InfoKVs will log key-value fields to the log at INFO level.
func (e Entry) InfoKVs(kvs ...kv.Field) {
if INFO < state.level {
return
}
e.kvs = append(e.kvs, kvs...)
logf(e.ctx, INFO, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-len(kvs)]
}
// Warn will log formatted args as 'msg' field to the log at WARN level.
func (e Entry) Warn(a ...any) {
if WARN < state.level {
@@ -111,6 +151,26 @@ func (e Entry) Warnf(s string, a ...any) {
logf(e.ctx, WARN, e.kvs, s, a...)
}
// WarnKV will log the one key-value field to the log at WARN level.
func (e Entry) WarnKV(key string, value any) {
if WARN < state.level {
return
}
e.kvs = append(e.kvs, kv.Field{K: key, V: value})
logf(e.ctx, WARN, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-1]
}
// WarnKVs will log key-value fields to the log at WARN level.
func (e Entry) WarnKVs(kvs ...kv.Field) {
if WARN < state.level {
return
}
e.kvs = append(e.kvs, kvs...)
logf(e.ctx, WARN, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-len(kvs)]
}
// Error will log formatted args as 'msg' field to the log at ERROR level.
func (e Entry) Error(a ...any) {
if ERROR < state.level {
@@ -127,6 +187,26 @@ func (e Entry) Errorf(s string, a ...any) {
logf(e.ctx, ERROR, e.kvs, s, a...)
}
// ErrorKV will log the one key-value field to the log at ERROR level.
func (e Entry) ErrorKV(key string, value any) {
if ERROR < state.level {
return
}
e.kvs = append(e.kvs, kv.Field{K: key, V: value})
logf(e.ctx, ERROR, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-1]
}
// ErrorKVs will log key-value fields to the log at ERROR level.
func (e Entry) ErrorKVs(kvs ...kv.Field) {
if ERROR < state.level {
return
}
e.kvs = append(e.kvs, kvs...)
logf(e.ctx, ERROR, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-len(kvs)]
}
// Panic will log formatted args as 'msg' field to the log at PANIC level.
// This will then call panic causing the application to crash.
func (e Entry) Panic(a ...any) {
@@ -147,6 +227,30 @@ func (e Entry) Panicf(s string, a ...any) {
logf(e.ctx, PANIC, e.kvs, s, a...)
}
// PanicKV will log the one key-value field to the log at PANIC level.
// This will then call panic causing the application to crash.
func (e Entry) PanicKV(key string, value any) {
defer panic(kv.Field{K: key, V: value}.String())
if PANIC < state.level {
return
}
e.kvs = append(e.kvs, kv.Field{K: key, V: value})
logf(e.ctx, PANIC, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-1]
}
// PanicKVs will log key-value fields to the log at PANIC level.
// This will then call panic causing the application to crash.
func (e Entry) PanicKVs(kvs ...kv.Field) {
defer panic(kv.Fields(kvs).String())
if PANIC < state.level {
return
}
e.kvs = append(e.kvs, kvs...)
logf(e.ctx, PANIC, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-len(kvs)]
}
// Log will log formatted args as 'msg' field to the log at given level.
func (e Entry) Log(lvl LEVEL, a ...any) {
if lvl < state.level {
@@ -163,6 +267,26 @@ func (e Entry) Logf(lvl LEVEL, s string, a ...any) {
logf(e.ctx, lvl, e.kvs, s, a...)
}
// LogKV will log the one key-value field to the log at given level.
func (e Entry) LogKV(lvl LEVEL, key string, value any) { //nolint:revive
if lvl < state.level {
return
}
e.kvs = append(e.kvs, kv.Field{K: key, V: value})
logf(e.ctx, lvl, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-1]
}
// LogKVs will log key-value fields to the log at given level.
func (e Entry) LogKVs(lvl LEVEL, kvs ...kv.Field) { //nolint:revive
if lvl < state.level {
return
}
e.kvs = append(e.kvs, kvs...)
logf(e.ctx, lvl, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-len(kvs)]
}
// Print will log formatted args to the log output.
func (e Entry) Print(a ...any) {
logf(e.ctx, UNSET, e.kvs, "", a...)
@@ -172,3 +296,17 @@ func (e Entry) Print(a ...any) {
func (e Entry) Printf(s string, a ...any) {
logf(e.ctx, UNSET, e.kvs, s, a...)
}
// PrintKV will log the one key-value field to the log.
func (e Entry) PrintKV(key string, value any) {
e.kvs = append(e.kvs, kv.Field{K: key, V: value})
logf(e.ctx, UNSET, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-1]
}
// PrintKVs will log key-value fields to the log.
func (e Entry) PrintKVs(kvs ...kv.Field) {
e.kvs = append(e.kvs, kvs...)
logf(e.ctx, UNSET, e.kvs, "")
e.kvs = e.kvs[:len(e.kvs)-len(kvs)]
}
vendor/code.superseriousbusiness.org/gopkg/log/format/json.go
Generated Vendored
+3 -168
View File
@@ -20,9 +20,9 @@ package format
import (
"encoding/json"
"time"
"unicode/utf8"
"code.superseriousbusiness.org/gopkg/log/level"
"code.superseriousbusiness.org/gopkg/xjson"
"codeberg.org/gruf/go-byteutil"
"codeberg.org/gruf/go-caller"
@@ -65,7 +65,7 @@ func (fmt *JSON) Format(buf *byteutil.Buffer, stamp time.Time, pc uintptr, lvl l
// Append JSON formatted fields.
for _, field := range kvs {
appendStringJSON(buf, field.K)
buf.B = xjson.Quote(buf.B, field.K)
buf.B = append(buf.B, `:`...)
b, _ := json.Marshal(field.V)
buf.B = append(buf.B, b...)
@@ -75,7 +75,7 @@ func (fmt *JSON) Format(buf *byteutil.Buffer, stamp time.Time, pc uintptr, lvl l
if msg != "" {
// Append JSON formatted msg string.
buf.B = append(buf.B, `"msg":`...)
appendStringJSON(buf, msg)
buf.B = xjson.Quote(buf.B, msg)
} else if string(buf.B[len(buf.B)-2:]) == ", " {
// Drop the trailing ", ".
buf.B = buf.B[:len(buf.B)-2]
@@ -84,168 +84,3 @@ func (fmt *JSON) Format(buf *byteutil.Buffer, stamp time.Time, pc uintptr, lvl l
// Append closing JSON brace.
buf.B = append(buf.B, `}`...)
}
// appendStringJSON is modified from the encoding/json.appendString()
// function, copied in here such that we can use it for key appending.
func appendStringJSON(buf *byteutil.Buffer, src string) {
const hex = "0123456789abcdef"
buf.B = append(buf.B, '"')
start := 0
for i := 0; i < len(src); {
if b := src[i]; b < utf8.RuneSelf {
if jsonSafeSet[b] {
i++
continue
}
buf.B = append(buf.B, src[start:i]...)
switch b {
case '\\', '"':
buf.B = append(buf.B, '\\', b)
case '\b':
buf.B = append(buf.B, '\\', 'b')
case '\f':
buf.B = append(buf.B, '\\', 'f')
case '\n':
buf.B = append(buf.B, '\\', 'n')
case '\r':
buf.B = append(buf.B, '\\', 'r')
case '\t':
buf.B = append(buf.B, '\\', 't')
default:
// This encodes bytes < 0x20 except for \b, \f, \n, \r and \t.
buf.B = append(buf.B, '\\', 'u', '0', '0', hex[b>>4], hex[b&0xF])
}
i++
start = i
continue
}
n := len(src) - i
if n > utf8.UTFMax {
n = utf8.UTFMax
}
c, size := utf8.DecodeRuneInString(src[i : i+n])
if c == utf8.RuneError && size == 1 {
buf.B = append(buf.B, src[start:i]...)
buf.B = append(buf.B, `\ufffd`...)
i += size
start = i
continue
}
// U+2028 is LINE SEPARATOR.
// U+2029 is PARAGRAPH SEPARATOR.
// They are both technically valid characters in JSON strings,
// but don't work in JSONP, which has to be evaluated as JavaScript,
// and can lead to security holes there. It is valid JSON to
// escape them, so we do so unconditionally.
// See https://en.wikipedia.org/wiki/JSON#Safety.
if c == '\u2028' || c == '\u2029' {
buf.B = append(buf.B, src[start:i]...)
buf.B = append(buf.B, '\\', 'u', '2', '0', '2', hex[c&0xF])
i += size
start = i
continue
}
i += size
}
buf.B = append(buf.B, src[start:]...)
buf.B = append(buf.B, '"')
}
var jsonSafeSet = [utf8.RuneSelf]bool{
' ': true,
'!': true,
'"': false,
'#': true,
'$': true,
'%': true,
'&': true,
'\'': true,
'(': true,
')': true,
'*': true,
'+': true,
',': true,
'-': true,
'.': true,
'/': true,
'0': true,
'1': true,
'2': true,
'3': true,
'4': true,
'5': true,
'6': true,
'7': true,
'8': true,
'9': true,
':': true,
';': true,
'<': true,
'=': true,
'>': true,
'?': true,
'@': true,
'A': true,
'B': true,
'C': true,
'D': true,
'E': true,
'F': true,
'G': true,
'H': true,
'I': true,
'J': true,
'K': true,
'L': true,
'M': true,
'N': true,
'O': true,
'P': true,
'Q': true,
'R': true,
'S': true,
'T': true,
'U': true,
'V': true,
'W': true,
'X': true,
'Y': true,
'Z': true,
'[': true,
'\\': false,
']': true,
'^': true,
'_': true,
'`': true,
'a': true,
'b': true,
'c': true,
'd': true,
'e': true,
'f': true,
'g': true,
'h': true,
'i': true,
'j': true,
'k': true,
'l': true,
'm': true,
'n': true,
'o': true,
'p': true,
'q': true,
'r': true,
's': true,
't': true,
'u': true,
'v': true,
'w': true,
'x': true,
'y': true,
'z': true,
'{': true,
'|': true,
'}': true,
'~': true,
'\u007f': true,
}
vendor/code.superseriousbusiness.org/gopkg/xjson/GO-LICENSE
Generated Vendored
+27
View File
@@ -0,0 +1,27 @@
Copyright 2009 The Go Authors.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google LLC nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
vendor/code.superseriousbusiness.org/gopkg/xjson/array.go
Generated Vendored
+195
View File
@@ -0,0 +1,195 @@
// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package xjson
import (
"bytes"
"database/sql/driver"
"errors"
"reflect"
"codeberg.org/gruf/go-byteutil"
)
// StringArrayNullable as a wrapper around StringArray
// to support marshaling / unmarshaling as "null".
type StringArrayNullable struct{ StringArray }
// MarshalJSON: implements json.Marshaler{}.
func (arr StringArrayNullable) MarshalJSON() ([]byte, error) {
if arr.StringArray == nil {
return []byte{'n', 'u', 'l', 'l'}, nil
}
return arr.StringArray.MarshalJSON()
}
// UnmarshalJSON: implements json.Unmarshaler{}.
func (arr *StringArrayNullable) UnmarshalJSON(data []byte) error {
if string(data) == "null" {
arr.StringArray = nil
return nil
}
return arr.StringArray.UnmarshalJSON(data)
}
// Scan: implements sql.Scanner{}.
func (arr *StringArrayNullable) Scan(src any) error {
switch v := src.(type) {
case nil:
arr.StringArray = nil
return nil
case string:
b := byteutil.S2B(v)
return arr.UnmarshalJSON(b)
case []byte:
return arr.UnmarshalJSON(v)
default:
return errors.New("cannot scan from: " + reflect.TypeOf(v).String())
}
}
// Value: implements driver.Valuer{}.
func (arr StringArrayNullable) Value() (driver.Value, error) {
return arr.MarshalJSON()
}
// StringArray is a []string type-alias
// for marshaling / unmarshaling string
// arrays. It additionally comes with
// database/sql scanner and valuer methods.
type StringArray []string
// MarshalJSON: implements json.Marshaler{}.
func (arr StringArray) MarshalJSON() ([]byte, error) {
if len(arr) == 0 {
return []byte{'[', ']'}, nil
}
// Determine slice
// size to allocate.
var l uint
// array
// braces
l = 2
for _, str := range arr {
// elem + quotes + comma.
l += uint(len(str)) + 2
}
// Start with array brace.
b := make([]byte, 0, l)
b = append(b, '[')
// Append each quoted elem.
for _, str := range arr {
b = Quote(b, str)
b = append(b, ',')
}
// Set final brace.
b[len(b)-1] = ']'
return b, nil
}
// UnmarshalJSON: implements json.Unmarshaler{}.
func (arr *StringArray) UnmarshalJSON(data []byte) error {
if len(data) < 2 || data[0] != '[' || data[len(data)-1] != ']' {
return errors.New("json value was not array")
}
// Trim array square braces.
data = data[1 : len(data)-1]
for len(data) > 0 {
var elem []byte
// Look for next elem separator.
i := bytes.IndexByte(data, ',')
if i >= 0 {
elem = data[:i]
data = data[i+1:]
} else {
elem = data
data = nil
}
// Trim space around elem.
elem = trimJsonSpace(elem)
// Attempt to unquote elem.
elem, ok := Unquote(elem)
if !ok {
return errors.New("invalid json array string elem")
}
// Append a COPY of string to array.
(*arr) = append((*arr), string(elem))
}
return nil
}
// Scan: implements sql.Scanner{}.
func (arr *StringArray) Scan(src any) error {
switch v := src.(type) {
case nil:
return errors.New("nil source")
case string:
b := byteutil.S2B(v)
return arr.UnmarshalJSON(b)
case []byte:
return arr.UnmarshalJSON(v)
default:
return errors.New("cannot scan from: " + reflect.TypeOf(v).String())
}
}
// Value: implements driver.Valuer{}.
func (arr StringArray) Value() (driver.Value, error) {
return arr.MarshalJSON()
}
// trimjsonspace is an optimized ASCII space char
// trimmer according to JSON whitespace specification,
// optimized for our specific case of JSON that does
// not contain any unicode characters.
func trimJsonSpace(b []byte) []byte {
var i, j int
// Skip space chars at start.
for i = 0; i < len(b); i++ {
switch b[i] {
case ' ', '\t', '\r', '\n':
continue
}
break
}
// Skip space characters from end.
for j = len(b) - 1; j >= 0; i-- {
switch b[j] {
case ' ', '\t', '\r', '\n':
continue
}
break
}
return b[i : j+1]
}
vendor/code.superseriousbusiness.org/gopkg/xjson/std.go
Generated Vendored
+318
View File
@@ -0,0 +1,318 @@
// Copyright 2010 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the GO-LICENSE file.
package xjson
import (
"unicode"
"unicode/utf16"
"unicode/utf8"
)
// Quote will JSON quote the provided JSON string.
//
// NOTE: copied from go/src/encoding/json/encode.go
func Quote(buf []byte, src string) []byte {
const hex = "0123456789abcdef"
buf = append(buf, '"')
start := 0
for i := 0; i < len(src); {
if b := src[i]; b < utf8.RuneSelf {
if jsonSafeSet[b] {
i++
continue
}
buf = append(buf, src[start:i]...)
switch b {
case '\\', '"':
buf = append(buf, '\\', b)
case '\b':
buf = append(buf, '\\', 'b')
case '\f':
buf = append(buf, '\\', 'f')
case '\n':
buf = append(buf, '\\', 'n')
case '\r':
buf = append(buf, '\\', 'r')
case '\t':
buf = append(buf, '\\', 't')
default:
// This encodes bytes < 0x20 except for \b, \f, \n, \r and \t.
buf = append(buf, '\\', 'u', '0', '0', hex[b>>4], hex[b&0xF])
}
i++
start = i
continue
}
n := len(src) - i
if n > utf8.UTFMax {
n = utf8.UTFMax
}
c, size := utf8.DecodeRuneInString(src[i : i+n])
if c == utf8.RuneError && size == 1 {
buf = append(buf, src[start:i]...)
buf = append(buf, `\ufffd`...)
i += size
start = i
continue
}
// U+2028 is LINE SEPARATOR.
// U+2029 is PARAGRAPH SEPARATOR.
// They are both technically valid characters in JSON strings,
// but don't work in JSONP, which has to be evaluated as JavaScript,
// and can lead to security holes there. It is valid JSON to
// escape them, so we do so unconditionally.
// See https://en.wikipedia.org/wiki/JSON#Safety.
if c == '\u2028' || c == '\u2029' {
buf = append(buf, src[start:i]...)
buf = append(buf, '\\', 'u', '2', '0', '2', hex[c&0xF])
i += size
start = i
continue
}
i += size
}
buf = append(buf, src[start:]...)
buf = append(buf, '"')
return buf
}
// Unquote will JSON unquote the provided JSON string,
// returning false if invalid encoding is encountered.
//
// NOTE: copied from go/src/encoding/json/decode.go
func Unquote(s []byte) (t []byte, ok bool) {
if len(s) < 2 || s[0] != '"' || s[len(s)-1] != '"' {
return
}
s = s[1 : len(s)-1]
// Check for unusual characters. If there are none,
// then no unquoting is needed, so return a slice of the
// original bytes.
r := 0
for r < len(s) {
c := s[r]
if c == '\\' || c == '"' || c < ' ' {
break
}
rr, size := utf8.DecodeRune(s[r:])
if rr == utf8.RuneError && size == 1 {
break
}
r += size
}
if r == len(s) {
return s, true
}
b := make([]byte, len(s)+2*utf8.UTFMax)
w := copy(b, s[0:r])
for r < len(s) {
// Out of room? Can only happen if s is full of
// malformed UTF-8 and we're replacing each
// byte with RuneError.
if w >= len(b)-2*utf8.UTFMax {
nb := make([]byte, (len(b)+utf8.UTFMax)*2)
copy(nb, b[0:w])
b = nb
}
switch c := s[r]; {
case c == '\\':
r++
if r >= len(s) {
return
}
switch s[r] {
default:
return
case '"', '\\', '/', '\'':
b[w] = s[r]
r++
w++
case 'b':
b[w] = '\b'
r++
w++
case 'f':
b[w] = '\f'
r++
w++
case 'n':
b[w] = '\n'
r++
w++
case 'r':
b[w] = '\r'
r++
w++
case 't':
b[w] = '\t'
r++
w++
case 'u':
r--
rr := getu4(s[r:])
if rr < 0 {
return
}
r += 6
if utf16.IsSurrogate(rr) {
rr1 := getu4(s[r:])
if dec := utf16.DecodeRune(rr, rr1); dec != unicode.ReplacementChar {
// A valid pair; consume.
r += 6
w += utf8.EncodeRune(b[w:], dec)
break
}
// Invalid surrogate; fall back to replacement rune.
rr = unicode.ReplacementChar
}
w += utf8.EncodeRune(b[w:], rr)
}
// Quote, control characters are invalid.
case c == '"', c < ' ':
return
// ASCII
case c < utf8.RuneSelf:
b[w] = c
r++
w++
// Coerce to well-formed UTF-8.
default:
rr, size := utf8.DecodeRune(s[r:])
r += size
w += utf8.EncodeRune(b[w:], rr)
}
}
return b[0:w], true
}
var jsonSafeSet = [utf8.RuneSelf]bool{
' ': true,
'!': true,
'"': false,
'#': true,
'$': true,
'%': true,
'&': true,
'\'': true,
'(': true,
')': true,
'*': true,
'+': true,
',': true,
'-': true,
'.': true,
'/': true,
'0': true,
'1': true,
'2': true,
'3': true,
'4': true,
'5': true,
'6': true,
'7': true,
'8': true,
'9': true,
':': true,
';': true,
'<': true,
'=': true,
'>': true,
'?': true,
'@': true,
'A': true,
'B': true,
'C': true,
'D': true,
'E': true,
'F': true,
'G': true,
'H': true,
'I': true,
'J': true,
'K': true,
'L': true,
'M': true,
'N': true,
'O': true,
'P': true,
'Q': true,
'R': true,
'S': true,
'T': true,
'U': true,
'V': true,
'W': true,
'X': true,
'Y': true,
'Z': true,
'[': true,
'\\': false,
']': true,
'^': true,
'_': true,
'`': true,
'a': true,
'b': true,
'c': true,
'd': true,
'e': true,
'f': true,
'g': true,
'h': true,
'i': true,
'j': true,
'k': true,
'l': true,
'm': true,
'n': true,
'o': true,
'p': true,
'q': true,
'r': true,
's': true,
't': true,
'u': true,
'v': true,
'w': true,
'x': true,
'y': true,
'z': true,
'{': true,
'|': true,
'}': true,
'~': true,
'\u007f': true,
}
// getu4 decodes \uXXXX from the beginning of s,
// returning the hex value, or it returns -1.
//
// NOTE: copied from go/src/encoding/json/decode.go
func getu4(s []byte) rune {
if len(s) < 6 || s[0] != '\\' || s[1] != 'u' {
return -1
}
var r rune
for _, c := range s[2:6] {
switch {
case '0' <= c && c <= '9':
c = c - '0'
case 'a' <= c && c <= 'f':
c = c - 'a' + 10
case 'A' <= c && c <= 'F':
c = c - 'A' + 10
default:
return -1
}
r = r*16 + rune(c)
}
return r
}
vendor/code.superseriousbusiness.org/gopkg/xslices/slices.go
Generated Vendored
+34 -15
View File
@@ -95,24 +95,23 @@ func Deduplicate[T comparable](in []T) []T {
// DeduplicateFunc deduplicates entries in the given
// slice, using the result of key() to gauge uniqueness.
func DeduplicateFunc[T any, C comparable](in []T, key func(v T) C) []T {
var (
inL = len(in)
unique = make(map[C]struct{}, inL)
deduped = make([]T, 0, inL)
)
if key == nil {
panic("nil func")
}
unique := make(map[C]struct{}, len(in))
deduped := make([]T, 0, len(in))
// Iterate input slice.
for _, v := range in {
k := key(v)
// Check if already exists.
if _, ok := unique[k]; ok {
// Already have this.
continue
}
// Append unique value.
unique[k] = struct{}{}
deduped = append(deduped, v)
}
@@ -180,6 +179,24 @@ func GatherIf[T, V any](out []V, in []T, get func(T) (V, bool)) []V {
return out
}
// Count returns the combined count of
// calling incr on all elements in input slice.
func Count[T any](in []T, incr func(T) int) int {
if incr == nil {
panic("nil func")
}
var i int
// Count getting incr from
// each elem in slice.
for _, v := range in {
i += incr(v)
}
return i
}
// Collate will collect the values of type K from input type []T,
// passing each item to 'get' and deduplicating the end result.
// This is equivalent to calling Gather() followed by Deduplicate().
@@ -191,16 +208,18 @@ func Collate[T any, K comparable](in []T, get func(T) K) []K {
ks := make([]K, 0, len(in))
km := make(map[K]struct{}, len(in))
for i := 0; i < len(in); i++ {
// Get next k.
k := get(in[i])
// Iterate input slice.
for _, v := range in {
k := get(v)
if _, ok := km[k]; !ok {
// New value, add
// to map + slice.
ks = append(ks, k)
km[k] = struct{}{}
// Check if already exists.
if _, ok := km[k]; ok {
continue
}
// Append unique k.
km[k] = struct{}{}
ks = append(ks, k)
}
return ks
vendor/codeberg.org/gruf/go-longdur/LICENSE
Generated Vendored
+9
View File
@@ -0,0 +1,9 @@
MIT License
Copyright (c) 2021 gruf
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
vendor/codeberg.org/gruf/go-longdur/README.md
Generated Vendored
+3
View File
@@ -0,0 +1,3 @@
Longer (than "time") duration parsing and formatting.
Also faster, and without float duration support.
vendor/codeberg.org/gruf/go-longdur/const.go
Generated Vendored
+30
View File
@@ -0,0 +1,30 @@
package longdur
import (
"time"
)
const (
YearApprox = Duration(year)
MonthApprox = Duration(month)
Week = Duration(week)
Day = Duration(day)
Hour = Duration(hour)
Minute = Duration(minute)
Second = Duration(second)
Millisecond = Duration(millisecond)
Microsecond = Duration(microsecond)
Nanosecond = Duration(nanosecond)
)
const year = 365 * day
const month = 30 * day
const week = 7 * day
const day = uint64(24 * time.Hour)
const hour = uint64(time.Hour)
const minute = uint64(time.Minute)
const second = uint64(time.Second)
const millisecond = uint64(time.Millisecond)
const microsecond = uint64(time.Microsecond)
const nanosecond = uint64(time.Nanosecond)
const maxTimeDuration = 1<<63 - 1
vendor/codeberg.org/gruf/go-longdur/duration.go
Generated Vendored
+365
View File
@@ -0,0 +1,365 @@
package longdur
import (
"errors"
"time"
"unsafe"
)
// ErrInvalidNumber is returned when a required number cannot be parsed from duration string.
var ErrInvalidNumber = errors.New("invalid number")
// ErrInvalidUnit is returned when a required unit cannot parsed from duration string.
var ErrInvalidUnit = errors.New("invalid unit")
// ErrOverflow is returned on math causing Duration integer overflow.
var ErrOverflow = errors.New("integer overflow")
// ErrUnderflow is returned on math causing Duration integer underflow.
var ErrUnderflow = errors.New("integer underflow")
// Duration define a duration stored in nanoseconds,
// much like the time.Duration type, with the exception
// that this is an unsigned integer and the helper methods
// are aware of days, weeks, months and years.
type Duration uint64
// Parse will attempt to parse the given string as a duration.
// Where a string formatted duration may contain ASCII space
// separated numbers with (again, optionally space separated)
// units, of which the following are supported:
// - y, yr, yrs, year, years =~ 365 days
// - mo, month, months =~ 30 days
// - w, wk, wks, week, weeks =~ 7 days
// - d, day, days =~ 24 hours
// - h, hr, hrs, hour, hours = 60 minutes
// - m, min, mins, minute, minutes = 60 seconds
// - s, sec, secs, second, seconds = 1000 milliseconds
// - ms, milli, millis, millisecond, milliseconds = 1000 microseconds
// - us, micro, micros, microsecond, microseconds = 1000 nanoseconds
// - ns, nano, nanos, nanosecond, nanoseconds = 1
//
// NOTE: unlike the time.ParseDuration() function, this
// does not accept floating point values or fractions.
func Parse(in string) (l Duration, err error) {
var d, u uint64
for len(in) > 0 {
in, d, u, err = parse(in)
if err != nil {
return
}
old := l
l += Duration(d * u)
if l < old {
err = ErrOverflow
return
}
}
return
}