Introduction

Grapevine is a Matrix homeserver that was originally forked from Conduit 0.7.0.

Chat with us

Currently, the Matrix room at #grapevine:computer.surgery serves multiple purposes:

• General discussion about the project, such as answering questions about it
• Reporting issues with Grapevine, if getting GitLab access is too much trouble for you
• Providing support to users running Grapevine
• Discussing the development of Grapevine

If you'd like to engage in or observe any of those things, please join!

Can I use it?

Theoretically yes, but it's not ready for general use yet, because:

• There aren't any releases.
• There isn't very much user-facing documentation.
• There have been several additions and breaking changes to the configuration file format that haven't been documented in detail. This means you'll need to read the source code to figure out what all the options are and what they do.

If these issues don't scare you away, go for it! (And if you use NixOS, here's an example.)

Expectations management

This project is run and maintained entirely by volunteers who are doing their best. Additionally, due to our goals, the development of new features may be slower than alternatives. We find this to be an acceptable tradeoff considering the importance of the reliability of a project like this.

Goals

Our goal is to provide a robust and reliable Matrix homeserver implementation. In order to accomplish this, we aim to do the following:

• Optimize for maintainability
• Implement automated testing to ensure correctness
• Improve instrumentation to provide real-world data to aid decision-making

Non-goals

We also have some things we specifically want to avoid as we feel they inhibit our ability to accomplish our goals:

• macOS or Windows support
  • These operating systems are very uncommon in the hobbyist server space, and we feel our effort is better spent elsewhere.
• Docker support
  • Docker tends to generate a high volume of support requests that are solely due to Docker itself or how users are using Docker. In attempt to mitigate this, we will not provide first-party Docker images. Instead, we'd recommend avoiding Docker and either using our pre-built statically-linked binaries or building from source. However, if your deployment mechanism requires Docker, it should be straightforward to build your own Docker image.
• Configuration via environment variables
  • Environment variables restrict the options for structuring configuration and support for them would increase the maintenance burden. If your deployment mechanism requires this, consider using an external tool like envsubst.
• Configuration compatibility with Conduit
  • To provide a secure and ergonomic configuration experience, breaking changes are required. However, we do intend to provide a migration tool to ease migration; this feature is tracked here.
• Perfect database compatibility with Conduit
  • The current database compatibility status can be tracked here. In the long run, it's inevitable that changes will be made to Conduit that we won't want to pull in, or that we need to make changes that Conduit won't want to pull in.

Project management

The project's current maintainers¹ are:

We would like to expand this list in the future as social trust is built and technical competence is demonstrated by other contributors.

We require at least 1 approving code review from a maintainer² before changes can be merged. This number may increase in the future as the list of maintainers grows.

Code of conduct

We follow the Rust Code of Conduct with some extra points:

• In the absence of evidence to suggest otherwise, assume good faith when engaging with others
• Moderation actions may be taken for behavior observed outside of project-specific spaces
• We have limited patience, so violations may skip the warning and directly result in a ban

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

This will be the first release of Grapevine since it was forked from Conduit 0.7.0.

Security

1. Prevent XSS via user-uploaded media. (!8)
2. Switch from incorrect, hand-rolled X-Matrix Authorization parser to the much better implementation provided by Ruma. (!31)
   • This is not practically exploitable to our knowledge, but this change does reduce risk.
3. Switch to a more trustworthy password hashing library. (!29)
   • This is not practically exploitable to our knowledge, but this change does reduce risk.
4. Don't return redacted events from the search endpoint. (!41 (f74043d), !41 (83cdc9c))
5. Prevent impersonation in EDUs. (!41 (da99b07))
   • m.signing_key_update was not affected by this bug.
6. Verify PDUs and transactions against the temporally-correct signing keys. (!41 (9087da9))
7. Only allow the admin bot to change the room ID that the admin room alias points to. (!42)

Removed

1. Remove update checker. (17a0b34)
2. Remove optional automatic display name emoji for newly registered users. (cddf699)
3. Remove admin room welcome message on first startup. (c9945f6)
4. Remove incomplete presence implementation. (f27941d)
5. Remove Debian packaging. (d41f0fb)
6. Remove Docker packaging. (!48)
7. BREAKING: Remove unstable room versions. (!59)
8. Remove memory-usage, clear-database-caches, and clear-service-caches admin commands. (!123)
   • The memory-usage command wasn't particularly useful since it can't actually give you an accurate value in bytes and isn't supported on all database backends.
   • The latter two commands had poor UX and didn't have any noticable effect on memory consumption.
9. BREAKING: Remove the global.conduit_cache_capacity_modifier and global.pdu_cache_capacity configuration options. (!124)
   • Instead, it is now possible to configure each cache capacity individually.

Changed

1. BREAKING: Rename Conduit to Grapevine. (360e020)
   • The CONDUIT_VERSION_EXTRA build-time environment variable has been renamed to GRAPEVINE_VERSION_EXTRA. This change only affects distribution packagers or non-Nix users who are building from source. If you fall into one of those categories and were explicitly setting this environment variable, make sure to change its name before building Grapevine.
2. BREAKING: Change the default port from 8000 to 6167. (f205280)
   • If you relied on the default port being 8000, either update your other configuration to use the new port, or explicitly configure Grapevine's port to 8000.
3. Improve tracing spans and events. (!11 (a275db3) (merged as 5172f66), !11 (a275db3) (merged as 5172f66), !11 (f556fce) (merged as ac42e0b), !18, !26, !50, !52, !54, !56, !69, !102, !127, !141)
4. Stop returning unnecessary member counts from /_matrix/client/{r0,v3}/sync. (!12)
5. BREAKING: Allow federation by default. (!24)
   • If you relied on federation being disabled by default, make sure to explicitly disable it before upgrading.
6. BREAKING: Remove the [global] section from the configuration file. (!38)
   • Details on how to migrate can be found in the merge request's description.
7. BREAKING: Allow specifying multiple transport listeners in the configuration file. (!39)
   • Details on how to migrate can be found in the merge request's description.
8. Increase default log level so that span information is included. (!50)
9. BREAKING: Reorganize config into sections. (!49)
   • Details on how to migrate can be found in the merge request's description.
10. Try to generate thumbnails for remote media ourselves if the federation thumbnail request fails. (!58)
11. BREAKING: Disable unauthenticated access to media by default. Use media.allow_unauthenticated_access to configure this behavior. (!103, !140)
12. BREAKING: Split CLI into multiple subcommands. The CLI invocation to run the server is now behind the serve command, so grapevine --config ... becomes grapevine serve --config .... (!108)
13. BREAKING: The path to media files is now specified separately from the database path. (!140, !170)
14. Use trust-dns for all DNS queries, instead of only for SRV records and SRV record targets in server discovery. (!156)

Fixed

1. Fix questionable numeric conversions. (71c48f6)
2. Stop sending no-longer-valid cached responses from the /_matrix/client/{r0,v3}/sync endpoints. (!7)
3. Stop returning extra E2EE device updates from /_matrix/client/{r0,v3}/sync as that violates the specification. (!12)
4. Make certain membership state transitions work correctly again. (!16)
   • For example, it was previously impossible to unban users from rooms.
5. Ensure that tracing-flame flushes all its data before the process exits. (!20 (263edcc))
6. Reduce the likelihood of locking up the async runtime. (!19)
7. Fix dynamically linked jemalloc builds. (!23)
8. Fix search results not including subsequent pages in certain situations. (!35 (0cdf032))
9. Fix search results missing events in subsequent pages in certain situations. (!35 (3551a6e))
10. Only process admin commands if the admin bot is in the admin room. (!43)
11. Fix bug where invalid account data from a client could prevent a user from joining any upgraded rooms and brick rooms that affected users attempted to upgrade. (!53)
12. Fix bug where unexpected keys were deleted from m.direct account data events when joining an upgraded room. (!53)
13. Fixed appservice users not receiving federated invites if the local server isn't already resident in the room (!80)
14. Fix bug where, if a server has multiple public keys, only one would be fetched. (!78)
15. Fix bug where expired keys may not be re-fetched in some scenarios. (!78)
16. Fix bug where signing keys would not be fetched when joining a room if we hadn't previously seen any signing keys from that server. (!87)
17. Fixed bug (#48) that caused us to attempt to fetch our own signing keys from ourselves over federation, and fail ("Won't send federation request to ourselves"). (!96)
18. Fixed incoming HTTP/2 requests failing federation signature check. (!104)
19. Return 403 instead of 500 when joins to a local-only room are denied. Consequently fixes Heisenbridge being unable to join puppeted users to its rooms (#85). (!127)
20. Fix handling of v11 rooms with m.room.create event content that passes the authorization rules but doesn't match other parts of the spec. (!139)
21. Fix tiebreaking comparisons between events during state resolution. This will reduce the rate at which servers disagree about the state of rooms. (!141)
22. Fix bug where the backoff state for remote device key queries was not reset after a successful request, causing an increasing rate of key query failures over time until a server restart. (!149)
23. Fix bug where remote key queries that were skipped because the target server was in backoff would increment the backoff delay further, leading to a positive feedback loop. (!149)
24. Return 504 M_NOT_YET_UPLOADED instead of 500 M_UNKNOWN when a media file is present in the database but the contents are missing in the filesystem. Removing media from the filesystem was the only way to delete media before !99, so this situation is common. (!55) (!153)
25. Return 400 M_BAD_ALIAS from PUT /_matrix/client/v3/rooms/{roomId}/state/{eventType}/{stateKey} instead of 400 M_FORBIDDEN when trying to set a canonical alias that does not exist. (!158)
26. Validate schema of new m.room.canonical_alias event sent by clients, rather than silently allowing any contents if the event can't be parsed. (!158)
27. Only validate canonical aliases that are new, rather than rather than revalidating every alias. This makes it possible to add/remove aliases when some of the existing aliases cannot be validated. (!158)
28. Fix read receipts not being sent over federation (or only arbitrarily late) (!162)

Added

1. Add various conveniences for users of the Nix package. (51f9650, bbb1a6f)
2. Add a NixOS module. (33e7a46)
3. Add a Conduit compat mode. (a25f2ec)
   • BREAKING: If you're migrating from Conduit, this option must be enabled or else your homeserver will refuse to start.
4. Include GRAPEVINE_VERSION_EXTRA information in the /_matrix/federation/v1/version endpoint. (509b70b)
5. Allow multiple tracing subscribers to be active at once. (!20 (7a154f74))
6. Allow configuring the filter for tracing-flame. (!20 (507de06))
7. Collect HTTP response time metrics via OpenTelemetry and optionally expose them as Prometheus metrics. This functionality is disabled by default. (!22)
8. Collect metrics for lookup results (e.g. cache hits/misses). (!15, !36)
9. Add configuration options for controlling the log format and colors. (!46)
10. Recognize the !admin prefix to invoke admin commands. (!45)
11. Add the tracing-filter admin command to view and change log/metrics/flame filters dynamically at runtime. (!49, !164)
12. Add more configuration options. (!49)
    • observability.traces.filter: The tracing filter to use for OpenTelemetry traces.
    • observability.traces.endpoint: Where OpenTelemetry should send traces.
    • observability.flame.filter: The tracing filter for tracing-flame.
    • observability.flame.filename: Where tracing-flame will write its output.
    • observability.logs.timestamp: Whether timestamps should be included in the logs.
13. Support building nix packages without IFD (!73)
14. Report local users getting banned in the server logs and admin room. (!65, !84)
15. Added support for Authenticated Media (MSC3916). (!58, !111)
16. BREAKING: Added support for configuring and serving /.well-known/matrix/... data. (!90, !94)
    • The server_discovery.client.base_url option is now required.
17. Added support for configuring old verify/signing keys in config (federation.old_verify_keys) (!96)
18. Added admin commands to delete media (!99, !102, !148)
19. Allow configuring the served API components per listener. (!109)
20. Include the traceresponse header if OpenTelemetry Tracing is in use. (!112)
21. Sending SIGHUP to the grapevine process now reloads TLS certificates from disk. (!97)
22. Added a federation self-test, perfomed automatically on startup. (!106)
23. Added support for HAProxy proxy protocol listeners. (!97)
24. Add a check-config CLI subcommand to check whether the configuration file is valid. (!121)
25. Add configuration options to tune the value of each cache individually. (!124)
26. Allow adding canonical aliases from remote servers. (!158)

Installing

This chapter will explain how to start running a Grapevine instance for the first time.

Note: Pre-built binaries can be found in the Supported targets subchapter.

Supported targets

Target tiers

The "Tier" column for each target indicates the level of support that target has. Below is an explanation of what each tier means.

Tier 1

Tier 1 targets are guaranteed to:

• Build
• Pass the test suite

Tier 2

Tier 2 targets are guaranteed to:

• Build

Unsupported targets

Targets that don't appear in the table at the top of this page are unsupported. At any given time, such targets may or may not build, and may or may not pass the test suite.

Migrating to/from Conduit

Before migrating a Conduit instance to Grapevine, make sure to read through all of the breaking changes listed in the changelog.

In order to migrate an existing Conduit instance to/from Grapevine, the Grapevine config must include conduit_compat = true. This parameter cannot currently be modified after creating the database for the first time, so make sure to set it when creating a fresh Grapevine instance that you may want to migrate to a different implementation in the future.

Config

Grapevine includes several breaking changes to the config schema. We don't currently have docs on how to migrate an existing config. All breaking config changes are mentioned in the changelog, so the best current option is to read through those. Feel free to ask for config migration help in #grapevine:computer.surgery if something is unclear.

We plan to add a config migration tool to support automatically migrating existing configs to the new schema.

Filesystem

Grapevine requires database data and media data to live in separate, non-nested directories, which are configurable. Here is a typical example, starting with the filesystem structure:

/var/lib/grapevine
+ database/
| + database-file-1
| + ...
| + database-file-n
+ media/
  + media-file-1
  + ...
  + media-file-n

And here is the matching configuration:

[database]
path = "/var/lib/grapevine/database"

[media.backend]
type = "filesystem"
path = "/var/lib/grapevine/media"

On the other hand, Conduit's filesystem layout looks like this:

/var/lib/conduit
+ media/
| + media-file-1
| + ...
| + media-file-n
+ database-file-1
+ ...
+ database-file-n

Which nests the media directory inside the database directory. Grapevine will reject this layout, so the filesystem layout must be changed before starting Grapevine. It is important to migrate the filesystem layout before starting Grapevine, because otherwise it will create a fresh database instead of using the existing one.

Database

Grapevine is currently compatible with the Conduit 0.7.0 database format. It is still possible to migrate to or from some newer Conduit versions, but it may require manual intervention or break some functionality.

We plan to add a migration tool to support cleanly migrating to or from Conduit versions we are not internally compatible with.

Conduit 0.9.0 to Grapevine

Conduit 0.9.0 includes a database migration that modifies data that Grapevine doesn't read. Grapevine does not currently recognize the new database schema version, and will fail to start against a Conduit 0.9.0 database. Grapevine can start and run without issues if the version recorded in the databse is rolled back from 16 to 13. It is possible to do this by editing the database manually, or by modifying Grapevine to change the version. This patch is an example of the latter approach.

Grapevine to Conduit 0.8.0 or 0.9.0

Conduit 0.8.0 added a new database table to track which users created each room alias. Grapevine does not write to this table, so it is not possible to delete aliases created in Grapevine through the normal client-server API after migrating to Conduit 0.8.0. It is possible to delete aliases with the remove-alias admin command. Note that this issue also applies to migrations from Conduit <0.8.0 to Conduit 0.8.0.

There are no additional known issues when migrating to Conduit 0.9.0.

Migrating to/from Conduwuit

Current Conduwuit is explicitly incompatible with the Conduit/Grapevine database format. Some older versions have been migrated to Grapevine successfully, but we haven't evaluated which versions it is safe to migrate from yet. Try this at your own risk, and definitely take a backup first.

Contributing

Instructions for getting GitLab access can be found on the sign-in page.

GitLab access is primarily useful if you'd like to open issues, engage in discussions on issues or merge requests, or submit your own merge requests.

Note that if the sign-up process is too much trouble and you'd just like to report an issue, feel free to report it in the Matrix room at #grapevine:computer.surgery; someone with GitLab access can open an issue on your behalf.

Coordinated vulnerability disclosure

If you find a security vulnerability in Grapevine, please privately report it to the Grapevine maintainers in one of the following ways:

• Open a GitLab issue that's marked as confidential
• Create a private, invite-only, E2EE Matrix room and invite the following users:
  • @charles:computer.surgery
  • @olivia:computer.surgery
  • @xiretza:xiretza.xyz

If the maintainers determine that the vulnerability is shared with Conduit or other forks, we'll work with their teams to ensure that all affected projects can release a fix at the same time.

Style guide

It is recommended that contributors follow this guide to minimize nitpicking on their merge requests.

However, this guide is not set in stone. It is open to changes as new patterns emerge, requirements change, compelling arguments are made, and so on. The goal is to document the existing style so it can be applied consistently, not to ensure the style never changes.

Merge requests

When updating a merge request branch, use git rebase; do not create merge commits to merge other branches into a merge request branch.

Why? This keeps the history simpler, and lacking merge commits makes it easier to revert any individual commit.

Commit messages

Here's a good article on how to write good commit messages in general.

Specifically for this project:

• Capitalizing the first letter is not required.
• It is recommended to avoid "conventional commits", as they take away from the very limited subject line length, and will not be processed in an automated fashion anyway.

Why? The linked article explains why this is good practice.

Structuring commits

Try to structure each commit so that it falls into one of the following categories:

• Refactoring, with no behavior change.
• Changing behavior, with no refactoring.
• Removing behavior, with no refactoring.
• Adding behavior, with no refactoring.
• Rewriting something completely. It is rare that these kinds of commits are warranted.

If you find yourself wanting to use the word "and" in the commit's subject line, it should probably be broken into multiple commits.

During code review, it's common to get feedback requesting changes to your commits. To apply this feedback, do not make and push a new commit containing the requested change. Instead, include the requested change in the commit of yours that gave rise to the suggestion. If you are unfamiliar with rewriting history in git, this website is a great tutorial.

Why? Small, targeted, and well-explained commits make it easier for reviewers to verify that a change has its intended effect. Or, for someone running git bisect to find a more granular answer to why their test began failing.

mod/use order

mod and use statements should appear in the following order, separated by a blank line:

1. use statements referring to std, alloc, or core, if any.
2. use statements referring to other crates, if any.
3. use statements referring to super or crate, if any.
4. Macro definitions that need to be accessible from child modules, if any.
5. mod statements, if any.
6. use statements referring to modules declared by the above mod statements, if any.

rust-analyzer and rustfmt automate most of this except points 4 and 5.

Why? Consistency is good.

Testing

When writing tests, be sure to keep the contents of this article in mind. Especially, keeping Cargo unit tests in a dedicated tests file (mentioned towards the end of the article).

Why? The linked article explains why this is good practice.

Tracing

tracing events should:

1. Start with a capital letter (when applicable).

tracing events should not:

1. End with punctuation.
2. Interpolate values into the event's message.
   • Instead, add those values as structured fields.

Why? Consistency is good. Also, interpolating values into the event message essentially defeats the point of structured logging.

Examples

1

// This does not conform because it does not start with a capital letter.
info!("started pentametric fan");

// Do this instead:
info!("Started pentametric fan");

2

// This does not conform because it ends with punctuation.
info!("Started pentametric fan.");

// Do this instead:
info!("Started pentametric fan");

3

// This does not conform because it interpolates values into the event's
// message.
warn!("Noticed {} discombobulated waneshafts", count);

// Do this instead:
warn!(count, "Noticed discombobulated waneshafts");

Services

Services are abstraction units that live inside the src/service directory.

Calling service constructors must not cause side effects, with a few exceptions:

• Database reads.
• Local filesystem reads.

Why? This restriction makes it possible to implement subcommands that run "offline" that reuse service code.