* feat: Relax URL matching
Instead of only linkifying things with an explicit http or https scheme,
the xurls.Relaxed also matches links with known TLDs. This means that
text like 'banana.com' will also be matched, despite the missing
http/https scheme. This also works to linkify email addresses, which is
handy.
This should also ensure we catch links without a scheme for the purpose
of spam checking.
This will be either an mp1, mp2 or mp3 file. In practice it'll probably
be mp3, but this handles mp1 too for good measure. We don't advertise
audio/mp1 as a supported media type since best I can tell that was never
a MIME type that's been used.
This also changes the returned MIME-type for mp2 and mp3 to audio/mpeg,
to match what's expected and supported by most things nowadays.
Fixes: #3531
* [chore] Stub /api/v1/announcements implementation
This implements the /api/v1/announcements endpoint by simply returning
an empty array. This indicates there are no instance announcements.
Some clients retrieve this endpoint and get surprised by a 404. It tends
to be harmless, but results in some unnecessary logging when trying to
debug other things.
* [fix] Forgot some swagger
* [fix] Fix swagger some more
The maintainers messed with the v1.5.2 tag which causes Go checksum
validation problems as the Go module proxy saw and recorded the original
hash.
This updates to 1.5.3 which doesn't have the issue.
* .go files are now tagged with diff=go for better diffs
* go.sum is tagged with generated which suppresses the diff
* go.sum is tagged with merge=ours to reduce merge conflicts
go.mod isn't tagged with generated since it's useful to see it on
dependency bump PRs.
* [docs] Don't run SQLite on networked storage
This updates our documentation to include warnings against running
SQLite on networked storage.
It also slightly reorganises and consolidates some admonitions on the
Getting Started page because it was getting a bit too busy with repeated
information.
* [docs] Reword a few things
* [chore] Synchronise our robots.txt with upstream
* [feature] Add headers to escape AI crawlers
This adds 2 headers that a number of AI crawlers respect to signal that
content should not be included in their datasets.
This fixes some linkname shenanigans previous versions of the library
were using. It's now safe to upgrade to Go 1.23 and beyond once they
become available.
* Set frame-ancestors in the CSP
This ensures we can't be loaded/embedded in an iframe. It also sets the
older X-Frame-Options for fallback.
* Disable MIME type sniffing
* Set Referrer-Policy
This sets the policy such that browsers will never send the Referer
header along with a request, unless it's a request to the same protocol,
host/domain and port. Basically, only send it when navigating through
our own UI, but not anything external.
The default is strict-origin-when-cross-origin when unset, which sends
the Referer header for requests unless it's going from HTTPS to HTTP
(i.e a security downgrade, hence the 'strict').
Since 1.25.1 the newer 'http2 on;' syntax should be used. The previous
syntax still works, but throws warnings when testing the configuration
with 'nginx -t'.
This also updates the certbot template to match what's currently
generated. It removes ipv6only=on as that's the default on a listen
directive binding on a wildcard IPv6 address.
* [feature] Default to WASM-based SQLite driver
With 0.16 out this switches our default SQLite driver to the WASM-based
solution instead. So far the driver seems to perform just as well.
Switching our default should result in it getting a bit more testing
during the 0.17 development cycle.
* add the ol' john hancock
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
This syncs our copy with the current state of the ai.robots.txt
repository. Upstream has tightened their scope to be AI-only, whereas
before it included a bunch of SEO and "web intelligence" marketing
stuff. I've kept those but moved them into their own section.
The memlimit package started to log any error returned by automemlimit.
This updates our implementation to call SetGoMemLimitWithOpts() instead
which uses the same defaults as automemlimit except for being
initialised with a noop logger.
We check the returned error for a particular substring, as when cgroups
isn't available even when running on a Linux system that's not a
problem. If it's anything but that error, we log it at the warning
level so that admins can still diagnose other cgroup related issues.
Fixes#2983
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
On startup and shutdown of a worker, we log a message of the worker
being started together with a textual representation of a memory
address. Though this can be handy for developers to debug
startup/shutdown sequencing issues of the workers, it's typically not
very useful or informative for an admin. We can also output a lot of
these (on my system I get 265 lines of these during startup).
This changes the messages from Info to Debug, to not print them under
normal circumstances.
This updates the documentation to remove the last stray references to
the copaganda Pi. It now uses the the term single-board computer. GtS
can run fine on all kinds of SBCs and isn't limited to that one
particular fruit version.
* [chore] Update the flags passed to goreleaser
--rm-dist has been deprecated, the new flag is called --clean.
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
cmp.Or was introduced in Go 1.22 and picks the first value that's not
the zero value for the type. For a string, the zero value is the empty
string, which is what os.Getenv will return if the environment variable
is not set. That then results in "error" being returned instead.
This allows loading an environment variable with a default without
having to do the check and write out the conditional.
* [chore] Upgrade our Go version to 1.22
With Go 1.22 having been released at the start of February, it's now
been a few months. No major issues have shown up, and the two point
release since then have primarily been security fixes plus some general
bug fixing.
This sets the required Go version to 1.22, as there's nothing in 1.22.1
or 1.22.2 that we would explicitly require. It sets the toolchain to the
latest point release, to ensure we pick up any fixes from there when
building releases etc.
* [chore] Update CI to Go 1.22
* [chore] Update golangci-lint to 1.25.7
Newer version should know about Go 1.22 and run fine.
* [chore] Update Docker container to Go 1.22
* [chore] Update Dockerfile to newer Alpine version
* sign drone.yml
* add missing license header
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
This updates the robots.txt based on the list of the ai.robots.txt
repository. We can look at automating that at some point.
It's worth pointing out that some robots, namely the ones by Bytedance,
are known to ignore robots.txt entirely.
* [chore] Log less output on failed test
This changes the testrig log level to be error by default instead of
info. This makes test failures a lot easier to read, as we don't have
the parade of info logs for each failure to scroll through.
It speeds up the test suite by a couple of seconds since we need
to buffer and flush a lot less messages. On a clean run, so no test
failures, it's about a 3s difference on my machine. Depending on the
amount of test failures, total time saved can vary.
This also introduces a GTS_TESTRIG_LOG_LEVEL environment variable that
we explicitly check for, making it easy to override the log level should
we have a need for it. This would be primarily for running locally, and
not so much as part of go test.
Lastly, it updates the syslog tests to use log.Error because if the log
level is set to error but we call log.Info no message is emitted and we
hang indefinitely on the channel read.
* [chore] Rename the testrig log level env var
The id on the follows table is not a ULID, but a random ID. Sorting on
them results in a completely random order. Instead, sort on created_at,
which sould result in a stable and intended sort order.
Fixes: #2769
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
This switches the documentation to use human sizes, so 50MiB instead of
an integer number of bytes. This makes it much easier to understand what
values are set, and less likely to accidentally set the wrong value.
* [chore] Simplify the User-Agent string
RFC 9110[1] includes a definition for the format of a user-agent header:
User-Agent = product *( RWS ( product / comment ) )
product = token ["/" product-version]
product-version = token
comment = "(" *( ctext / quoted-pair / comment ) ")"
ctext = HTAB / SP / %x21-27 / %x2A-5B / %x5D-7E / obs-text
An example given in the RFC: User-Agent: CERN-LineMode/2.15 libwww/2.17b3
The idea is typically start with the most important product/version, add
a (comment) if necessary and then include any auxilliary products.
However, the RFC warns against including too many auxiliary products as
those can be unnecessarily revealing.
For automated systems (i.e not a browser), the common and recommended
format is <product></version> (+uri-for-contact), followed with any
additional <product>/<version> pairs that are relevant.
This changes our UA to match that convention more closely. This makes it
easier for administrators who do user-agent parsing for statistics or
other purposes to correctly identify the version of GoToSocial.
Currently tools tend to get confused by the lack of a /<version> on the
start of our string.
[1]: https://www.rfc-editor.org/rfc/rfc9110.html#name-user-agents
* [chore] Don't use app name in UA
From all the GtS UAs I've collected, nobody seems to set/change this,
so we might as well use the static string.
The main usefulness for this is when you have multilpe GtS instances
connecting to the same DB, so they can identify as different instances
by changing the application name (though it should already be obvious
from having different usernames).
* [docs] Mounting config file in container
This adds a small section clarifying how to mount a config.yaml in a
container.
* [docs] Work in GTS_CONFIG_PATH
* Restructure the sections
* Mention how to clear out remote avatars and headers, related to #2328
* Add more links to authoritative documentation on the AWS side
* Show how to use the official AWS CLI
* Make a separate section for migration to S3 compatible object backends
For some reason httpconv seems to have disappeared from 1.21, which
results in a 1.21 runtime with 1.20 semconv/httpconv which seems to
break some things.
For now, this rolls the OTEL dependencies back to 1.20 which should fix
the observability issues. We'll need to take a look at how to upgrade
safely and correctly in the future.
Relates to #2503.
* [docs] Describe how to block IPs using a firewall
This adds some documentation on how to block one, multiple or whole IP
ranges using the firewall. This can be helpful to protect from DDoS
attacks or block certain parties from being able to communicate with
your instance at all.
* [docs] Fix indenting of nft rule
* [docs] Split iptables and nftables into sections
* [feature] Run ANALYZE after migrations on SQLite
This ensures that at the end of migrations, we run ANALYZE if we're
using SQLite. This should be relatively quick and guarantees that the
table and index statistics have been updated. This helps to ensure the
query planner makes better choices when it comes to picking which
indexes are used when running queries.
* [chore] use ExecContext
Uses ExecContext so we pass the context through, this is helpful for
anyone running with tracing enabled
* [docs] Change configuration creation instructions
This changes the wording to push people towards creating their own
configuration, without copying the whole example configuration. This
makes it much easier to reconcile necessary configruation changes on
upgrades.
* [docs] Reword container version section
This changes the wording in the Version section to make it more clear
what the risks of a moving tag are. It pushes people to use an explicit
release tag instead.
* [feature] Block Google Bard/AI crawlers
* [feature] Block the other OpenAI crawler
* [feature] Block Common Crawl crawler
This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.
Fixes: #2240
* [feature] Block Omgilikebot
Used by some shady big web data engine company.
* [feature] Block Meta's language model crawler
* [feature] Block well-known.dev crawler
* [docs] Callout no cluster/multi-node support
Though we have a GitHub issue for this, we don't mention it anywhere in
the docs. This adds it to the deployment considerations so hopefully
folks will see it when standing up their own instance.
* [docs] Add daenney as a developer
* [feature] Support Actor URIs for webfinger queries
It's now possible to pass an Actor URI as the resource to query for when
doing a webfinger query. The code now extracts the username and domain
from the URI. The URI needs to be fully qualified, including having a
scheme of http or https to be recognised as such.
The acct scheme is handled as we used to, including dealing with an
erroneous leading @ on the username. We retain the ability to handle
resources without a scheme by parsing them again with the acct scheme if
the original parse failed. This can happen due to parsing ambiguities
when dealing with a string like user@domain.tld:port.
* [bugfix] Remove debugging changes
* [chore] Make TestExtractNamestring table-driven
* [chore] Unnest Trim and Split for readability
* [feature] Add http trace exporter, drop Jaeger
Jaeger supports ingesting traces using the OpenTelemetry gRPC or HTTP
methods. The Jaeger project has deprecated the old jaeger transport.
* Add support for submitting traces over HTTP
* Drop support for the old Jaeger protocol
* Upgrade the trace libraries to v1.17
Fixes: #2176Fixes: #2179
c.FullPath() is the empty string if a request doesn't match any route on
our mux. In those cases, there's no value in emitting a trace. The trace
will be empty, containing no other information beyond the fact that we
didn't match a route. Since Gin breaks off the processing early we don't
need to trace this request as it won't do anything and consumes no
further resources.
The 404 will still be emitted by our logs and will be visible from a
reverse proxy too.
* [feature] Don't emit timestamp in log lines
When running gotosocial with a service manager like systemd, or a
container runtime, the associated log driver usually emits timestamps
itself. In those cases, having the extra timestamp from our own log
lines ends up being a bit noisy and when centrally ingesting logs is
duplicate information.
This introduces a configuration flag that allows disabling emitting the
timestamp. It's only wired up for "daemonised" processes, meaning server
and testrig.
* [chore] Add docs for log-timestamp
* [feature] Simplify timestamp handling
Co-Authored-By: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
* [chore] Less escaped double-quotes
* [chore] Fix help string
---------
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
This adds the CSP header with a policy of only loading from the same
domain. We don't make use of external media, CSS, JS, fonts, so we don't
ever need external data loaded in our context.
When building a DEBUG build, the policy gets extended to include
localhost:*, i.e localhost on any port. This keeps the live-reloading
flow for JS development working. localhost and 127.0.0.1 are considered
to be the same so mixing and matching those doesn't result in a CSP
violation.
With Feditext now accepting beta users, this adds it as the third
client to recommend so we have web and the dominant mobile platforms
covered.
This also removes the screenshots from the README, because it became a
mess trying to add a third one. Either the cells become very narrow, or
the table doubles in height. As the UI may also change over time, it
might be better to point folks at the apps instead who'll hopefully have
up to date screenshots in their storefronts.
* [chore] Remove go-playground/validator
It turns out we're not actually using the validator code. This is a
remnant from when we intended to use it, but the presence of it and its
struct tags creates the illusion we're validating a lot of things we're
not. It resulted in some confusion when we were trying to figure out
language valdiation.
Remove all this code, so that only the validation functions from the
validate package we actually use remain. I'm not touching the struct
tags in the migrations in order to avoid things potentially thinking
migrations need to be re-run.
* [chore] Bring back a struct tag on api
The validate on internal/api is Gin doing form validation, not the
validator from go-playground/validator.
The old default of 30d can lead to a lot of media getting cached and
significant disk usage, even on small or single person instances. A lot
of deployments decrease this value, to 15 or even less. This is less of
an issue when using object storage, but for local storage which is the
more popular deployment option running out of disk space is unpleasant.
With GoToSocial's aim to fit in small places, this changes the default
to a much more conservative 7 days. In all likelihood people aren't
scrolling that far back in their timeline so this change shouldn't
result in any issue. Existing deployments will only be affected by
this change if the admin hasn't already configured this value, or didn't
bootstrap from the example configuration.
We already set edit_uri, but we forgot to enable the feature flag. By
turning it on each page now gets an edit button on the right hand side
of the page header. This'll open the edit view in GitHub.
* Add instructions on how to make a new page show up in the sidebar / navigation
* Explain how to create a virtual env to install the dependencies in if you're not using Conda
If someone makes a post with a long, uninterrupted piece of text in a
code snippet, we would stretch the column to fit it, resulting in the
UI going a bit whacky.
By setting min-width: 0% this fixes it, and we now automatically get a
scrollbar on overflow instead.
Fixes: #1952
* [docs] Rework backups a bit
This changes the existing backup documentation to:
* Push a bit harder on people to perform backups, it's not really just a
nice to have
* Removes the language about migrating to/from GoToSocial and a
different ActivityPub server since that's really not supported
* Adds a section about using backup software and provides an example on
how to do this using Borgmatic
* [docs] Remove too much info in db section
* [docs] Add docs on how to backup local media
This adds documentation pointing people at the media list-local command
in order to determine what media files they need to include as part of
their backups.
Provides a Python script that people can use to transform the media
listing from the CLI into Borg patterns. It also includes a Borgmatic
config.yaml in the repository so people can easily fetch it instead of
copy-pasting from the docs.
* [bugfix] Ensure we emit an absolute path prefix
It works either way, as a pattern like data/files/<ID> would match a
file on /data/files/<ID>. But it would potentially also match any path
that happens to include data/files/<ID> but not rooted at the
storage-local-base-path.
* [docs] Add more links to media list CLI reference
* [feature] Add media list command
This is an attempt to help alleviate #1776. Using admin media list
--local the full path to each local media file will be printed, with a
newline. The output of this should be feadable into backup tools in
order to allow to backup local media too. Together with the database
this should allow to fully recover from the loss of an instance.
The list command also gets a --remote flag for symmetry. In the case
of --remote we print the RemoteURL instead, the location the asset can
be retrieved from.
To get all media, you can run with --local and --remote.
* [bugfix] Fix the test failures
* [feature] Reimplement list media as top commands
This changes the implementation of admin media list --<variant> to two
separate top-level commands, list-local and list-remote.
The implementation now iterates over over the database in batches of 200
in order to avoid loading all media metadata into memory.
* [feature] Implement ListMedia with filter callback
This does away with the somewhat odd iterator-like structure we had
before and does away with most of the loop duplication in list-local and
list-remote. Instead they call GetAllMediaPaths with a filter func to
select the media they want. That's accumulated into a slice and
eventually returned.
* [bugfix] Simplify remote filter
Since we don't append the empty string anywhere, the remote filter can
be limited to returning RemoteURL, as that'll be an empty string for
local media.
* [docs] Add media list commands to CLI reference
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
When you configure the landing-page-user redirect, you lose access to
the one page that displays server stats. This adds the same stats as we
have on / to /about to bring those back.
For some reason we hit the case in CI where the
TestFingerWithHostMetaCacheStrategy seems to experience some time
dilation. It's possible this is a genuine bug, but I can't for the life
of me reproduce it locally, even after having run this test thousands of
times (-count=1000 when invoking go test etc.)
This changes the test to explicitly stop the webfinger cache, set TTL
and Sweep frequency to something well beyond the lifetime of the cache
during the test and then starts the cache again. Hopefully that does it,
because the other option that remains is that for some reason
timekeeping in CI/Docker is not as precise as when running the test on a
host.
* [docs] Add a certificates guide in Advanced
This adds some documentation about the process of getting certificates
through ACME in general. It also provides a number of links to
alternative clients and certbot deployment guides that are up to date.
Slightly restructure the NGINX and Apache reverse proxy documentation
and insert mentions to the Provisioning TLS certificates advanced
documentation in them.
* [docs] Add firewall section in Advanced
* [docs] Add new guides to section indexes
* [docs] Fix spelling issue
* [docs] Fix a few typos
* [docs] Made Advanced its own section
This splits the Advanced page off from the Getting Started guide and
makes it its own thing. It now has some additional sub-sections for
bigger topics like caching and enhanced security. This also moves
tracing from Getting Started to Advanced as that feels like a more
appropriate location for it.
The enhanced security looks a little silly with a single section, but I
have guides pending for firewall configurations and I'd also like to
consolidate our how to provision TLS certificates in there as we repeat
this information multiple times.
* [docs] Fix all my spelling errors
* [docs] Inline the links in sandboxing
This tries to revamp/restructure the installation guide. It's renamed to
"Getting Started" as it now contains a few more things than just the
installation, especially the deployment considerations which we didn't
use to spell out as much ahead of time.
Installation is now a section with the guides on their own. I've removed
a bit of redundancy like the user creation. I also removed the rogue
reverse proxy section in the Docker guide and lifted that into the
reverse proxy section.
This adds an example on how to get Grafana Tempo up to receive spans as
well as Grafana itself to view them.
I've added this as a separate Tracing doc in the installation guide as
the Advanced one was starting to get rather full.
Fixes: #1791
Our default configuration places the SQLite DB in /gotosocial/, but the
AppArmor profile doesn't allow us to write there. Instead of making the
whole directory writable, add a writable area in /gotosocial/db/ instead
and advise in the docs to move the DB there.
Instead of a manually curated list of distribution packages, this adds
the Repology[1] widget instead.
This also separates out the distribution packages from the
self-hosting/deployment options into their own sections.
[1]: https://repology.org/project/gotosocial/versions
Every now and then the TestFingerWithHostMetaCacheStrategy would fail on
a time related error. I suspect suite.Equal doesn't quite work as
expected when given two time.Time's, so instead explicitly check with
the time.Equal.
* [bugfix] Fix NegotiateAccept with multi accept
There's a bug in Gin's NegotiateFormat that doesn't handle the presence
of multilpe accept headers. This lifts the code from the PR @tsmethurst
sent a year ago to Gin into our codebase to fix the issue.
* [bugfix] Concat accept header in webfinger
Some implementations bug out when there's multiple accept headers,
including Gin (see 7050112af1ccc935ec542cb41fa8b07f7357539d). But things
seem to work reliably with a single accept header with multiple parts.
Fixes: #1793
It turns out that in Masto v2.3.0 the languages key was added to the V1
Instance and that it's effectively mandatory. Though in GtS we don't
really have this concept yet, some apps will explode if the languages
key is missing altogether. So at least return the empty array on V1 too
in the hopes that it makes things work well enough.
For history's sake, you can see the attributes that will get serialised
in
https://github.com/mastodon/mastodon/blob/f877aa9d70d0d600961989b8e97c0e0ce3ac1db6/app/serializers/rest/v1/instance_serializer.rb#L6-L9.
Because the attribute does not have a conditional defined for it,
there isn't a filter that optionally omits it, or a def languages to
modify the behaviour the attribute is effectively always included and
serialised.
Fixes: #1662
This adds the preferences endpoint to our Mastodon Client API
implementation. It's a read-only endpoint that returns a number of
user preferences. Applications can query these settings when logging in
a user (for the first time) to configure themselves.
Linguist already classifies vendor/ as vendored files, meaning they
don't count towards things like a repo language stat, but are still
shows in diffs.
By classifying them as generated instead they're not only excluded from
langauge stats but also hidden in diffs.
This adds a section to the docs instructing how to ensure apps will be
able to successfully login to an instance when host- and account-domain
differ.
Resolves#1609
Daenney