diff options
62 files changed, 1970 insertions, 931 deletions
diff --git a/.gitignore b/.gitignore index 4e71a7df0..3b0c7d361 100644 --- a/.gitignore +++ b/.gitignore @@ -43,3 +43,7 @@ docs/generated_config.md # Code test coverage /cover /Elixir.*.coverdata + +.idea +pleroma.iml + diff --git a/CHANGELOG.md b/CHANGELOG.md index 4eb72c002..6a49bc4dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,14 +4,20 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] +### Added +- Refreshing poll results for remote polls +### Changed +- **Breaking:** Elixir >=1.8 is now required (was >= 1.7) +- Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) and `Pleroma.Web.Federator.RetryQueue` with [Oban](https://github.com/sorentwo/oban) (see [`docs/config.md`](docs/config.md) on migrating customized worker / retry settings) +- Introduced [quantum](https://github.com/quantum-elixir/quantum-core) job scheduler +- Admin API: Return `total` when querying for reports + +## [1.1.0] - 2019-??-?? ### Security -- OStatus: eliminate the possibility of a protocol downgrade attack. -- OStatus: prevent following locked accounts, bypassing the approval process. - Mastodon API: respect post privacy in `/api/v1/statuses/:id/{favourited,reblogged}_by` ### Removed - **Breaking:** GNU Social API with Qvitter extensions support -- **Breaking:** ActivityPub: The `accept_blocks` configuration setting. - Emoji: Remove longfox emojis. - Remove `Reply-To` header from report emails for admins. @@ -19,8 +25,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config - **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired - **Breaking:** `/api/pleroma/notifications/read` is moved to `/api/v1/pleroma/notifications/read` and now supports `max_id` and responds with Mastodon API entities. -- Configuration: OpenGraph and TwitterCard providers enabled by default -- Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text - Configuration: added `config/description.exs`, from which `docs/config.md` is generated - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option @@ -30,23 +34,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses) - Improve digest email template – Pagination: (optional) return `total` alongside with `items` when paginating -- Replaced [pleroma_job_queue](https://git.pleroma.social/pleroma/pleroma_job_queue) and `Pleroma.Web.Federator.RetryQueue` with [Oban](https://github.com/sorentwo/oban) (see [`docs/config.md`](docs/config.md) on migrating customized worker / retry settings) -- Introduced [quantum](https://github.com/quantum-elixir/quantum-core) job scheduler ### Fixed - Following from Osada -- Not being able to pin unlisted posts -- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised. - Favorites timeline doing database-intensive queries - Metadata rendering errors resulting in the entire page being inaccessible - `federation_incoming_replies_max_depth` option being ignored in certain cases -- Federation/MediaProxy not working with instances that have wrong certificate order - Mastodon API: Handling of search timeouts (`/api/v1/search` and `/api/v2/search`) - Mastodon API: Misskey's endless polls being unable to render - Mastodon API: Embedded relationships not being properly rendered in the Account entity of Status entity - Mastodon API: Notifications endpoint crashing if one notification failed to render -- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set -- Mastodon API: `muted` in the Status entity, using author's account to determine if the tread was muted - Mastodon API: Add `account_id`, `type`, `offset`, and `limit` to search API (`/api/v1/search` and `/api/v2/search`) - Mastodon API, streaming: Fix filtering of notifications based on blocks/mutes/thread mutes - ActivityPub C2S: follower/following collection pages being inaccessible even when authentifucated if `hide_followers`/ `hide_follows` was set @@ -54,15 +51,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - Rich Media: Parser failing when no TTL can be found by image TTL setters - Rich Media: The crawled URL is now spliced into the rich media data. - ActivityPub S2S: sharedInbox usage has been mostly aligned with the rules in the AP specification. -- ActivityPub S2S: remote user deletions now work the same as local user deletions. -- ActivityPub S2S: POST requests are now signed with `(request-target)` pseudo-header. -- Not being able to access the Mastodon FE login page on private instances -- Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag - Pleroma.Upload base_url was not automatically whitelisted by MediaProxy. Now your custom CDN or file hosting will be accessed directly as expected. - Report email not being sent to admins when the reporter is a remote user -- MRF: ensure that subdomain_match calls are case-insensitive - Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances -- MRF: fix use of unserializable keyword lists in describe() implementations - ActivityPub: Deactivated user deletion - ActivityPub: Fix `/users/:nickname/inbox` crashing without an authenticated user - MRF: fix ability to follow a relay when AntiFollowbotPolicy was enabled @@ -73,16 +64,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - Mastodon API: all status JSON responses contain a `pleroma.expires_at` item which states when an activity will expire. The value is only shown to the user who created the activity. To everyone else it's empty. - Configuration: `ActivityExpiration.enabled` controls whether expired activites will get deleted at the appropriate time. Enabled by default. - Conversations: Add Pleroma-specific conversation endpoints and status posting extensions. Run the `bump_all_conversations` task again to create the necessary data. -- **Breaking:** MRF describe API, which adds support for exposing configuration information about MRF policies to NodeInfo. - Custom modules will need to be updated by adding, at the very least, `def describe, do: {:ok, %{}}` to the MRF policy modules. - MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`) - MRF: Support for excluding specific domains from Transparency. - MRF: Support for filtering posts based on who they mention (`Pleroma.Web.ActivityPub.MRF.MentionPolicy`) -- MRF: Support for filtering posts based on ActivityStreams vocabulary (`Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`) -- MRF (Simple Policy): Support for wildcard domains. -- Support for wildcard domains in user domain blocks setting. -- Configuration: `quarantined_instances` support wildcard domains. -- Configuration: `federation_incoming_replies_max_depth` option - Mastodon API: Support for the [`tagged` filter](https://github.com/tootsuite/mastodon/pull/9755) in [`GET /api/v1/accounts/:id/statuses`](https://docs.joinmastodon.org/api/rest/accounts/#get-api-v1-accounts-id-statuses) - Mastodon API, streaming: Add support for passing the token in the `Sec-WebSocket-Protocol` header - Mastodon API, extension: Ability to reset avatar, profile banner, and background @@ -110,9 +94,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - Admin API: Endpoint for fetching latest user's statuses - Pleroma API: Add `/api/v1/pleroma/accounts/confirmation_resend?email=<email>` for resending account confirmation. - Pleroma API: Email change endpoint. -- Relays: Added a task to list relay subscriptions. -- Mix Tasks: `mix pleroma.database fix_likes_collections` -- Federation: Remove `likes` from objects. - Admin API: Added moderation log - Web response cache (currently, enabled for ActivityPub) - Mastodon API: Added an endpoint to get multiple statuses by IDs (`GET /api/v1/statuses/?ids[]=1&ids[]=2`) @@ -123,6 +104,61 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - RichMedia: parsers and their order are configured in `rich_media` config. - RichMedia: add the rich media ttl based on image expiration time. +## [1.0.6] - 2019-08-14 +### Fixed +- MRF: fix use of unserializable keyword lists in describe() implementations +- ActivityPub S2S: POST requests are now signed with `(request-target)` pseudo-header. + +## [1.0.5] - 2019-08-13 +### Fixed +- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set +- Mastodon API: `muted` in the Status entity, using author's account to determine if the thread was muted +- Mastodon API: return the actual profile URL in the Account entity's `url` property when appropriate +- Templates: properly style anchor tags +- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised. +- Not being able to access the Mastodon FE login page on private instances +- MRF: ensure that subdomain_match calls are case-insensitive +- Fix internal server error when using the healthcheck API. + +### Added +- **Breaking:** MRF describe API, which adds support for exposing configuration information about MRF policies to NodeInfo. + Custom modules will need to be updated by adding, at the very least, `def describe, do: {:ok, %{}}` to the MRF policy modules. +- Relays: Added a task to list relay subscriptions. +- MRF: Support for filtering posts based on ActivityStreams vocabulary (`Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`) +- MRF (Simple Policy): Support for wildcard domains. +- Support for wildcard domains in user domain blocks setting. +- Configuration: `quarantined_instances` support wildcard domains. +- Mix Tasks: `mix pleroma.database fix_likes_collections` +- Configuration: `federation_incoming_replies_max_depth` option + +### Removed +- Federation: Remove `likes` from objects. +- **Breaking:** ActivityPub: The `accept_blocks` configuration setting. + +## [1.0.4] - 2019-08-01 +### Fixed +- Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag + +## [1.0.3] - 2019-07-31 +### Security +- OStatus: eliminate the possibility of a protocol downgrade attack. +- OStatus: prevent following locked accounts, bypassing the approval process. +- TwitterAPI: use CommonAPI to handle remote follows instead of OStatus. + +## [1.0.2] - 2019-07-28 +### Fixed +- Not being able to pin unlisted posts +- Mastodon API: represent poll IDs as strings +- MediaProxy: fix matching filenames +- MediaProxy: fix filename encoding +- Migrations: fix a sporadic migration failure +- Metadata rendering errors resulting in the entire page being inaccessible +- Federation/MediaProxy not working with instances that have wrong certificate order +- ActivityPub S2S: remote user deletions now work the same as local user deletions. + +### Changed +- Configuration: OpenGraph and TwitterCard providers enabled by default +- Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text ## [1.0.1] - 2019-07-14 ### Security diff --git a/config/config.exs b/config/config.exs index ab6e00c98..c7e0cf09f 100644 --- a/config/config.exs +++ b/config/config.exs @@ -276,7 +276,7 @@ config :pleroma, :instance, max_account_fields: 10, max_remote_account_fields: 20, account_field_name_length: 512, - account_field_value_length: 512, + account_field_value_length: 2048, external_user_synchronization: true config :pleroma, :markup, @@ -331,6 +331,10 @@ config :pleroma, :activitypub, follow_handshake_timeout: 500, sign_object_fetches: true +config :pleroma, :streamer, + workers: 3, + overflow_workers: 2 + config :pleroma, :user, deny_follow_blocked: true config :pleroma, :mrf_normalize_markup, scrub_policy: Pleroma.HTML.Scrubber.Default diff --git a/config/description.exs b/config/description.exs index be5eb0cc3..65ea6bf01 100644 --- a/config/description.exs +++ b/config/description.exs @@ -39,11 +39,7 @@ config :pleroma, :config_description, [ key: :link_name, type: :boolean, description: - "If enabled, a name parameter will be added to the url of the upload. For example `https://instance.tld/media/imagehash.png?name=realname.png`", - suggestions: [ - true, - false - ] + "If enabled, a name parameter will be added to the url of the upload. For example `https://instance.tld/media/imagehash.png?name=realname.png`" }, %{ key: :base_url, @@ -57,11 +53,7 @@ config :pleroma, :config_description, [ key: :proxy_remote, type: :boolean, description: - "If enabled, requests to media stored using a remote uploader will be proxied instead of being redirected.", - suggestions: [ - true, - false - ] + "If enabled, requests to media stored using a remote uploader will be proxied instead of being redirected." }, %{ key: :proxy_opts, @@ -117,8 +109,7 @@ config :pleroma, :config_description, [ type: :string, description: "If you use S3 compatible service such as Digital Ocean Spaces or CDN, set folder name or \"\" etc." <> - " For example, when using CDN to S3 virtual host format, set \"\". At this time, write CNAME to CDN in public_endpoint.", - suggestions: [""] + " For example, when using CDN to S3 virtual host format, set \"\". At this time, write CNAME to CDN in public_endpoint." } ] }, @@ -190,11 +181,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "Allow/disallow send emails", - suggestions: [ - true, - false - ] + description: "Allow/disallow send emails" }, %{ group: {:subgroup, Swoosh.Adapters.SMTP}, @@ -221,8 +208,7 @@ config :pleroma, :config_description, [ group: {:subgroup, Swoosh.Adapters.SMTP}, key: :ssl, type: :boolean, - description: "`Swoosh.Adapters.SMTP` adapter specific setting", - suggestions: [true, false] + description: "`Swoosh.Adapters.SMTP` adapter specific setting" }, %{ group: {:subgroup, Swoosh.Adapters.SMTP}, @@ -256,8 +242,7 @@ config :pleroma, :config_description, [ group: {:subgroup, Swoosh.Adapters.SMTP}, key: :no_mx_lookups, type: :boolean, - description: "`Swoosh.Adapters.SMTP` adapter specific setting", - suggestions: [true, false] + description: "`Swoosh.Adapters.SMTP` adapter specific setting" }, %{ group: {:subgroup, Swoosh.Adapters.Sendgrid}, @@ -284,8 +269,7 @@ config :pleroma, :config_description, [ group: {:subgroup, Swoosh.Adapters.Sendmail}, key: :qmail, type: :boolean, - description: "`Swoosh.Adapters.Sendmail` adapter specific setting", - suggestions: [true, false] + description: "`Swoosh.Adapters.Sendmail` adapter specific setting" }, %{ group: {:subgroup, Swoosh.Adapters.Mandrill}, @@ -375,22 +359,19 @@ config :pleroma, :config_description, [ group: {:subgroup, Swoosh.Adapters.SocketLabs}, key: :server_id, type: :string, - description: "`Swoosh.Adapters.SocketLabs` adapter specific setting", - suggestions: [""] + description: "`Swoosh.Adapters.SocketLabs` adapter specific setting" }, %{ group: {:subgroup, Swoosh.Adapters.SocketLabs}, key: :api_key, type: :string, - description: "`Swoosh.Adapters.SocketLabs` adapter specific setting", - suggestions: [""] + description: "`Swoosh.Adapters.SocketLabs` adapter specific setting" }, %{ group: {:subgroup, Swoosh.Adapters.Gmail}, key: :access_token, type: :string, - description: "`Swoosh.Adapters.Gmail` adapter specific setting", - suggestions: [""] + description: "`Swoosh.Adapters.Gmail` adapter specific setting" } ] }, @@ -553,38 +534,22 @@ config :pleroma, :config_description, [ %{ key: :registrations_open, type: :boolean, - description: "Enable registrations for anyone, invitations can be enabled when false", - suggestions: [ - true, - false - ] + description: "Enable registrations for anyone, invitations can be enabled when false" }, %{ key: :invites_enabled, type: :boolean, - description: "Enable user invitations for admins (depends on registrations_open: false)", - suggestions: [ - true, - false - ] + description: "Enable user invitations for admins (depends on registrations_open: false)" }, %{ key: :account_activation_required, type: :boolean, - description: "Require users to confirm their emails before signing in", - suggestions: [ - true, - false - ] + description: "Require users to confirm their emails before signing in" }, %{ key: :federating, type: :boolean, - description: "Enable federation with other instances", - suggestions: [ - true, - false - ] + description: "Enable federation with other instances" }, %{ key: :federation_incoming_replies_max_depth, @@ -618,11 +583,7 @@ config :pleroma, :config_description, [ %{ key: :allow_relay, type: :boolean, - description: "Enable Pleroma's Relay, which makes it possible to follow a whole instance", - suggestions: [ - true, - false - ] + description: "Enable Pleroma's Relay, which makes it possible to follow a whole instance" }, %{ key: :rewrite_policy, @@ -638,11 +599,7 @@ config :pleroma, :config_description, [ type: :boolean, description: "Makes the client API in authentificated mode-only except for user-profiles." <> - " Useful for disabling the Local Timeline and The Whole Known Network", - suggestions: [ - true, - false - ] + " Useful for disabling the Local Timeline and The Whole Known Network" }, %{ key: :quarantined_instances, @@ -658,11 +615,7 @@ config :pleroma, :config_description, [ key: :managed_config, type: :boolean, description: - "Whenether the config for pleroma-fe is configured in this config or in static/config.json", - suggestions: [ - true, - false - ] + "Whenether the config for pleroma-fe is configured in this config or in static/config.json" }, %{ key: :static_dir, @@ -689,11 +642,7 @@ config :pleroma, :config_description, [ key: :mrf_transparency, type: :boolean, description: - "Make the content of your Message Rewrite Facility settings public (via nodeinfo)", - suggestions: [ - true, - false - ] + "Make the content of your Message Rewrite Facility settings public (via nodeinfo)" }, %{ key: :mrf_transparency_exclusions, @@ -709,11 +658,7 @@ config :pleroma, :config_description, [ type: :boolean, description: "Set to true to use extended local nicknames format (allows underscores/dashes)." <> - " This will break federation with older software for theses nicknames", - suggestions: [ - true, - false - ] + " This will break federation with older software for theses nicknames" }, %{ key: :max_pinned_statuses, @@ -741,11 +686,7 @@ config :pleroma, :config_description, [ key: :no_attachment_links, type: :boolean, description: - "Set to true to disable automatically adding attachment link text to statuses", - suggestions: [ - true, - false - ] + "Set to true to disable automatically adding attachment link text to statuses" }, %{ key: :welcome_message, @@ -780,20 +721,12 @@ config :pleroma, :config_description, [ description: "If set to true, only mentions at the beginning of a post will be used to address people in direct messages." <> " This is to prevent accidental mentioning of people when talking about them (e.g. \"@friend hey i really don't like @enemy\")." <> - " Default: false", - suggestions: [ - true, - false - ] + " Default: false" }, %{ key: :healthcheck, type: :boolean, - description: "If set to true, system data will be shown on /api/pleroma/healthcheck", - suggestions: [ - true, - false - ] + description: "If set to true, system data will be shown on /api/pleroma/healthcheck" }, %{ key: :remote_post_retention_days, @@ -823,11 +756,7 @@ config :pleroma, :config_description, [ %{ key: :skip_thread_containment, type: :boolean, - description: "Skip filter out broken threads. The default is true", - suggestions: [ - true, - false - ] + description: "Skip filter out broken threads. The default is true" }, %{ key: :limit_to_local_content, @@ -844,11 +773,7 @@ config :pleroma, :config_description, [ key: :dynamic_configuration, type: :boolean, description: - "Allow transferring configuration to DB with the subsequent customization from Admin api. Defaults to `false`", - suggestions: [ - true, - false - ] + "Allow transferring configuration to DB with the subsequent customization from Admin api. Defaults to `false`" }, %{ key: :max_account_fields, @@ -878,19 +803,15 @@ config :pleroma, :config_description, [ %{ key: :account_field_value_length, type: :integer, - description: "An account field value maximum length (default: 512)", + description: "An account field value maximum length (default: 2048)", suggestions: [ - 512 + 2048 ] }, %{ key: :external_user_synchronization, type: :boolean, - description: "Enabling following/followers counters synchronization for external users", - suggestions: [ - true, - false - ] + description: "Enabling following/followers counters synchronization for external users" } ] }, @@ -936,7 +857,6 @@ config :pleroma, :config_description, [ %{ key: :metadata, type: {:list, :atom}, - description: "", suggestions: [[:request_id]] } ] @@ -962,7 +882,6 @@ config :pleroma, :config_description, [ %{ key: :metadata, type: {:list, :atom}, - description: "", suggestions: [[:request_id]] } ] @@ -1069,48 +988,40 @@ config :pleroma, :config_description, [ %{ key: :showInstanceSpecificPanel, type: :boolean, - description: "Whenether to show the instance's specific panel", - suggestions: [true, false] + description: "Whenether to show the instance's specific panel" }, %{ key: :scopeOptionsEnabled, type: :boolean, - description: "Enable setting an notice visibility and subject/CW when posting", - suggestions: [true, false] + description: "Enable setting an notice visibility and subject/CW when posting" }, %{ key: :formattingOptionsEnabled, type: :boolean, description: - "Enable setting a formatting different than plain-text (ie. HTML, Markdown) when posting, relates to :instance, allowed_post_formats", - suggestions: [true, false] + "Enable setting a formatting different than plain-text (ie. HTML, Markdown) when posting, relates to :instance, allowed_post_formats" }, %{ key: :collapseMessageWithSubject, type: :boolean, description: - "When a message has a subject(aka Content Warning), collapse it by default", - suggestions: [true, false] + "When a message has a subject(aka Content Warning), collapse it by default" }, %{ key: :hidePostStats, type: :boolean, - description: "Hide notices statistics(repeats, favorites, ...)", - suggestions: [true, false] + description: "Hide notices statistics(repeats, favorites, ...)" }, %{ key: :hideUserStats, type: :boolean, description: - "Hide profile statistics(posts, posts per day, followers, followings, ...)", - suggestions: [true, false] + "Hide profile statistics(posts, posts per day, followers, followings, ...)" }, %{ key: :scopeCopy, type: :boolean, - description: - "Copy the scope (private/unlisted/public) in replies to posts by default", - suggestions: [true, false] + description: "Copy the scope (private/unlisted/public) in replies to posts by default" }, %{ key: :subjectLineBehavior, @@ -1124,8 +1035,7 @@ config :pleroma, :config_description, [ %{ key: :alwaysShowSubjectInput, type: :boolean, - description: "When set to false, auto-hide the subject field when it's empty", - suggestions: [true, false] + description: "When set to false, auto-hide the subject field when it's empty" } ] }, @@ -1142,8 +1052,7 @@ config :pleroma, :config_description, [ %{ key: :showInstanceSpecificPanel, type: :boolean, - description: "Whenether to show the instance's specific panel", - suggestions: [true, false] + description: "Whenether to show the instance's specific panel" } ] } @@ -1266,19 +1175,16 @@ config :pleroma, :config_description, [ group: :pleroma, key: :mrf_rejectnonpublic, type: :group, - description: "", children: [ %{ key: :allow_followersonly, type: :boolean, - description: "whether to allow followers-only posts", - suggestions: [true, false] + description: "whether to allow followers-only posts" }, %{ key: :allow_direct, type: :boolean, - description: "whether to allow direct messages", - suggestions: [true, false] + description: "whether to allow direct messages" } ] }, @@ -1393,8 +1299,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "Enables proxying of remote media to the instance's proxy", - suggestions: [true, false] + description: "Enables proxying of remote media to the instance's proxy" }, %{ key: :base_url, @@ -1426,8 +1331,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "Enables the gopher interface", - suggestions: [true, false] + description: "Enables the gopher interface" }, %{ key: :ip, @@ -1537,43 +1441,36 @@ config :pleroma, :config_description, [ %{ key: :instrumenters, type: {:list, :module}, - description: "", suggestions: [Pleroma.Web.Endpoint.Instrumenter] }, %{ key: :protocol, type: :string, - description: "", suggestions: ["https"] }, %{ key: :secret_key_base, type: :string, - description: "", suggestions: ["aK4Abxf29xU9TTDKre9coZPUgevcVCFQJe/5xP/7Lt4BEif6idBIbjupVbOrbKxl"] }, %{ key: :signing_salt, type: :string, - description: "", suggestions: ["CqaoopA2"] }, %{ key: :render_errors, type: :keyword, - description: "", suggestions: [[view: Pleroma.Web.ErrorView, accepts: ~w(json)]], children: [ %{ key: :view, type: :module, - description: "", suggestions: [Pleroma.Web.ErrorView] }, %{ key: :accepts, type: {:list, :string}, - description: "", suggestions: ["json"] } ] @@ -1581,33 +1478,27 @@ config :pleroma, :config_description, [ %{ key: :pubsub, type: :keyword, - description: "", suggestions: [[name: Pleroma.PubSub, adapter: Phoenix.PubSub.PG2]], children: [ %{ key: :name, type: :module, - description: "", suggestions: [Pleroma.PubSub] }, %{ key: :adapter, type: :module, - description: "", suggestions: [Phoenix.PubSub.PG2] } ] }, %{ key: :secure_cookie_flag, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean }, %{ key: :extra_cookie_attrs, type: {:list, :string}, - description: "", suggestions: ["SameSite=Lax"] } ] @@ -1621,20 +1512,17 @@ config :pleroma, :config_description, [ %{ key: :unfollow_blocked, type: :boolean, - description: "Whether blocks result in people getting unfollowed", - suggestions: [true, false] + description: "Whether blocks result in people getting unfollowed" }, %{ key: :outgoing_blocks, type: :boolean, - description: "Whether to federate blocks to other instances", - suggestions: [true, false] + description: "Whether to federate blocks to other instances" }, %{ key: :sign_object_fetches, type: :boolean, - description: "Sign object fetches with HTTP signatures", - suggestions: [true, false] + description: "Sign object fetches with HTTP signatures" }, %{ key: :follow_handshake_timeout, @@ -1653,14 +1541,12 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "Whether the managed content security policy is enabled", - suggestions: [true, false] + description: "Whether the managed content security policy is enabled" }, %{ key: :sts, type: :boolean, - description: "Whether to additionally send a Strict-Transport-Security header", - suggestions: [true, false] + description: "Whether to additionally send a Strict-Transport-Security header" }, %{ key: :sts_max_age, @@ -1727,8 +1613,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "Whether the captcha should be shown on registration", - suggestions: [true, false] + description: "Whether the captcha should be shown on registration" }, %{ key: :method, @@ -1778,8 +1663,7 @@ config :pleroma, :config_description, [ group: :pleroma_job_queue, key: :queues, type: :group, - description: "[Deprecated] Replaced with `Oban`/`:queues` (keeping the same format)", - children: [] + description: "[Deprecated] Replaced with `Oban`/`:queues` (keeping the same format)" }, %{ group: :pleroma, @@ -1790,8 +1674,7 @@ config :pleroma, :config_description, [ %{ key: :max_retries, type: :integer, - description: "[Deprecated] Replaced as `Oban`/`:queues`/`:outgoing_federation` value", - suggestions: [] + description: "[Deprecated] Replaced as `Oban`/`:queues`/`:outgoing_federation` value" } ] }, @@ -1817,8 +1700,7 @@ config :pleroma, :config_description, [ %{ key: :verbose, type: :boolean, - description: "Logs verbose mode", - suggestions: [false, true] + description: "Logs verbose mode" }, %{ key: :prune, @@ -1937,11 +1819,7 @@ config :pleroma, :config_description, [ %{ key: :unfurl_nsfw, type: :boolean, - description: "If set to true nsfw attachments will be shown in previews", - suggestions: [ - true, - false - ] + description: "If set to true nsfw attachments will be shown in previews" } ] }, @@ -1949,14 +1827,12 @@ config :pleroma, :config_description, [ group: :pleroma, key: :rich_media, type: :group, - description: "", children: [ %{ key: :enabled, type: :boolean, description: - "if enabled the instance will parse metadata from attached links to generate link previews", - suggestions: [true, false] + "if enabled the instance will parse metadata from attached links to generate link previews" }, %{ key: :ignore_hosts, @@ -1998,8 +1874,7 @@ config :pleroma, :config_description, [ key: :enabled, type: :boolean, description: - "if enabled, when a new user is federated with, fetch some of their latest posts", - suggestions: [true, false] + "if enabled, when a new user is federated with, fetch some of their latest posts" }, %{ key: :pages, @@ -2030,14 +1905,12 @@ config :pleroma, :config_description, [ %{ key: :new_window, type: :boolean, - description: "set to false to remove target='_blank' attribute", - suggestions: [true, false] + description: "set to false to remove target='_blank' attribute" }, %{ key: :scheme, type: :boolean, - description: "Set to true to link urls with schema http://google.com", - suggestions: [true, false] + description: "Set to true to link urls with schema http://google.com" }, %{ key: :truncate, @@ -2049,14 +1922,12 @@ config :pleroma, :config_description, [ %{ key: :strip_prefix, type: :boolean, - description: "Strip the scheme prefix", - suggestions: [true, false] + description: "Strip the scheme prefix" }, %{ key: :extra, type: :boolean, - description: "link urls with rarely used schemes (magnet, ipfs, irc, etc.)", - suggestions: [true, false] + description: "link urls with rarely used schemes (magnet, ipfs, irc, etc.)" } ] }, @@ -2083,8 +1954,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "whether scheduled activities are sent to the job queue to be executed", - suggestions: [true, false] + description: "whether scheduled activities are sent to the job queue to be executed" } ] }, @@ -2097,8 +1967,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "whether expired activities will be sent to the job queue to be deleted", - suggestions: [true, false] + description: "whether expired activities will be sent to the job queue to be deleted" } ] }, @@ -2110,7 +1979,6 @@ config :pleroma, :config_description, [ %{ key: Pleroma.Web.Auth.Authenticator, type: :module, - description: "", suggestions: [Pleroma.Web.Auth.PleromaAuthenticator, Pleroma.Web.Auth.LDAPAuthenticator] } ] @@ -2128,8 +1996,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "enables LDAP authentication", - suggestions: [true, false] + description: "enables LDAP authentication" }, %{ key: :host, @@ -2146,26 +2013,22 @@ config :pleroma, :config_description, [ %{ key: :ssl, type: :boolean, - description: "true to use SSL, usually implies the port 636", - suggestions: [true, false] + description: "true to use SSL, usually implies the port 636" }, %{ key: :sslopts, type: :keyword, - description: "additional SSL options", - suggestions: [] + description: "additional SSL options" }, %{ key: :tls, type: :boolean, - description: "true to start TLS, usually implies the port 389", - suggestions: [true, false] + description: "true to start TLS, usually implies the port 389" }, %{ key: :tlsopts, type: :keyword, - description: "additional TLS options", - suggestions: [] + description: "additional TLS options" }, %{ key: :base, @@ -2237,8 +2100,7 @@ config :pleroma, :config_description, [ %{ key: :active, type: :boolean, - description: "globally enable or disable digest emails", - suggestions: [true, false] + description: "globally enable or disable digest emails" }, %{ key: :schedule, @@ -2271,7 +2133,7 @@ config :pleroma, :config_description, [ children: [ %{ key: :logo, - # type: [:string, nil], + type: [:string, nil], description: "a path to a custom logo. Set it to nil to use the default Pleroma logo", suggestions: ["some/path/logo.png", nil] }, @@ -2293,37 +2155,31 @@ config :pleroma, :config_description, [ %{ key: :link_color, type: :string, - description: "", suggestions: ["#d8a070"] }, %{ key: :background_color, type: :string, - description: "", suggestions: ["#2C3645"] }, %{ key: :content_background_color, type: :string, - description: "", suggestions: ["#1B2635"] }, %{ key: :header_color, type: :string, - description: "", suggestions: ["#d8a070"] }, %{ key: :text_color, type: :string, - description: "", suggestions: ["#b9b9ba"] }, %{ key: :text_muted_color, type: :string, - description: "", suggestions: ["#b9b9ba"] } ] @@ -2346,14 +2202,12 @@ config :pleroma, :config_description, [ key: :issue_new_refresh_token, type: :boolean, description: - "Keeps old refresh token or generate new refresh token when to obtain an access token", - suggestions: [true, false] + "Keeps old refresh token or generate new refresh token when to obtain an access token" }, %{ key: :clean_expired_tokens, type: :boolean, - description: "Enable a background job to clean expired oauth tokens. Defaults to false", - suggestions: [true, false] + description: "Enable a background job to clean expired oauth tokens. Defaults to false" }, %{ key: :clean_expired_tokens_interval, @@ -2368,7 +2222,6 @@ config :pleroma, :config_description, [ group: :pleroma, key: :emoji, type: :group, - description: "", children: [ %{ key: :shortcode_globs, @@ -2415,8 +2268,7 @@ config :pleroma, :config_description, [ %{ key: :rum_enabled, type: :boolean, - description: "If RUM indexes should be used. Defaults to false", - suggestions: [true, false] + description: "If RUM indexes should be used. Defaults to false" } ] }, @@ -2475,8 +2327,7 @@ config :pleroma, :config_description, [ %{ key: :enabled, type: :boolean, - description: "Enables ssh", - suggestions: [true, false] + description: "Enables ssh" }, %{ key: :priv_dir, @@ -2512,7 +2363,6 @@ config :pleroma, :config_description, [ %{ key: :types, type: :map, - description: "", suggestions: [ %{ "application/xml" => ["xml"], @@ -2526,31 +2376,26 @@ config :pleroma, :config_description, [ %{ key: "application/xml", type: {:list, :string}, - description: "", suggestions: [["xml"]] }, %{ key: "application/xrd+xml", type: {:list, :string}, - description: "", suggestions: [["xrd+xml"]] }, %{ key: "application/jrd+json", type: {:list, :string}, - description: "", suggestions: [["jrd+json"]] }, %{ key: "application/activity+json", type: {:list, :string}, - description: "", suggestions: [["activity+json"]] }, %{ key: "application/ld+json", type: {:list, :string}, - description: "", suggestions: [["activity+json"]] } ] @@ -2578,9 +2423,7 @@ config :pleroma, :config_description, [ children: [ %{ key: :enabled, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean } ] }, @@ -2588,13 +2431,11 @@ config :pleroma, :config_description, [ group: :pleroma, key: :suggestions, type: :group, - description: "", children: [ %{ key: :enabled, type: :boolean, - description: "Enables suggestions", - suggestions: [] + description: "Enables suggestions" }, %{ key: :third_party_engine, @@ -2619,7 +2460,6 @@ config :pleroma, :config_description, [ %{ key: :web, type: :string, - description: "", suggestions: ["https://vinayaka.distsn.org"] } ] @@ -2646,7 +2486,6 @@ config :pleroma, :config_description, [ %{ key: :adapter, type: :module, - description: "", suggestions: [Pleroma.Signature] } ] @@ -2655,18 +2494,15 @@ config :pleroma, :config_description, [ group: :pleroma, key: Pleroma.Uploaders.MDII, type: :group, - description: "", children: [ %{ key: :cgi, type: :string, - description: "", suggestions: ["https://mdii.sakura.ne.jp/mdii-post.cgi"] }, %{ key: :files, type: :string, - description: "", suggestions: ["https://mdii.sakura.ne.jp"] } ] @@ -2680,19 +2516,15 @@ config :pleroma, :config_description, [ %{ key: :proxy_url, type: [:string, :atom, nil], - description: "", suggestions: ["localhost:9020", {:socks5, :localhost, 3090}, nil] }, %{ key: :send_user_agent, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean }, %{ key: :adapter, type: :keyword, - description: "", suggestions: [ [ ssl_options: [ @@ -2710,36 +2542,26 @@ config :pleroma, :config_description, [ group: :pleroma, key: :markup, type: :group, - description: "", children: [ %{ key: :allow_inline_images, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean }, %{ key: :allow_headings, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean }, %{ key: :allow_tables, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean }, %{ key: :allow_fonts, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean }, %{ key: :scrub_policy, type: {:list, :module}, - description: "", suggestions: [[Pleroma.HTML.Transform.MediaProxy, Pleroma.HTML.Scrubber.Default]] } ] @@ -2748,13 +2570,10 @@ config :pleroma, :config_description, [ group: :pleroma, key: :user, type: :group, - description: "", children: [ %{ key: :deny_follow_blocked, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean } ] }, @@ -2762,12 +2581,10 @@ config :pleroma, :config_description, [ group: :pleroma, key: :mrf_normalize_markup, type: :group, - description: "", children: [ %{ key: :scrub_policy, type: :module, - description: "", suggestions: [Pleroma.HTML.Scrubber.Default] } ] @@ -2776,12 +2593,10 @@ config :pleroma, :config_description, [ group: :pleroma, key: Pleroma.User, type: :group, - description: "", children: [ %{ key: :restricted_nicknames, type: {:list, :string}, - description: "", suggestions: [ [ ".well-known", @@ -2822,24 +2637,20 @@ config :pleroma, :config_description, [ %{ group: :cors_plug, type: :group, - description: "", children: [ %{ key: :max_age, type: :integer, - description: "", suggestions: [86_400] }, %{ key: :methods, type: {:list, :string}, - description: "", suggestions: [["POST", "PUT", "DELETE", "GET", "PATCH", "OPTIONS"]] }, %{ key: :expose, type: :string, - description: "", suggestions: [ [ "Link", @@ -2853,14 +2664,11 @@ config :pleroma, :config_description, [ }, %{ key: :credentials, - type: :boolean, - description: "", - suggestions: [true, false] + type: :boolean }, %{ key: :headers, type: {:list, :string}, - description: "", suggestions: [["Authorization", "Content-Type", "Idempotency-Key"]] } ] diff --git a/docs/api/admin_api.md b/docs/api/admin_api.md index fd608c459..9362e3d78 100644 --- a/docs/api/admin_api.md +++ b/docs/api/admin_api.md @@ -317,6 +317,7 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret ```json { + "total" : 1, "reports": [ { "account": { diff --git a/docs/config.md b/docs/config.md index 270d7fcea..3f37fa561 100644 --- a/docs/config.md +++ b/docs/config.md @@ -135,7 +135,7 @@ config :pleroma, Pleroma.Emails.Mailer, * `max_account_fields`: The maximum number of custom fields in the user profile (default: `10`) * `max_remote_account_fields`: The maximum number of custom fields in the remote user profile (default: `20`) * `account_field_name_length`: An account field name maximum length (default: `512`) -* `account_field_value_length`: An account field value maximum length (default: `512`) +* `account_field_value_length`: An account field value maximum length (default: `2048`) * `external_user_synchronization`: Enabling following/followers counters synchronization for external users. diff --git a/lib/pleroma/activity/ir/topics.ex b/lib/pleroma/activity/ir/topics.ex new file mode 100644 index 000000000..010897abc --- /dev/null +++ b/lib/pleroma/activity/ir/topics.ex @@ -0,0 +1,63 @@ +# Pleroma: A lightweight social networking server +# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/> +# SPDX-License-Identifier: AGPL-3.0-only + +defmodule Pleroma.Activity.Ir.Topics do + alias Pleroma.Object + alias Pleroma.Web.ActivityPub.Visibility + + def get_activity_topics(activity) do + activity + |> Object.normalize() + |> generate_topics(activity) + |> List.flatten() + end + + defp generate_topics(%{data: %{"type" => "Answer"}}, _) do + [] + end + + defp generate_topics(object, activity) do + ["user", "list"] ++ visibility_tags(object, activity) + end + + defp visibility_tags(object, activity) do + case Visibility.get_visibility(activity) do + "public" -> + if activity.local do + ["public", "public:local"] + else + ["public"] + end + |> item_creation_tags(object, activity) + + "direct" -> + ["direct"] + + _ -> + [] + end + end + + defp item_creation_tags(tags, %{data: %{"type" => "Create"}} = object, activity) do + tags ++ hashtags_to_topics(object) ++ attachment_topics(object, activity) + end + + defp item_creation_tags(tags, _, _) do + tags + end + + defp hashtags_to_topics(%{data: %{"tag" => tags}}) do + tags + |> Enum.filter(&is_bitstring(&1)) + |> Enum.map(fn tag -> "hashtag:" <> tag end) + end + + defp hashtags_to_topics(_), do: [] + + defp attachment_topics(%{data: %{"attachment" => []}}, _act), do: [] + + defp attachment_topics(_object, %{local: true}), do: ["public:media", "public:local:media"] + + defp attachment_topics(_object, _act), do: ["public:media"] +end diff --git a/lib/pleroma/application.ex b/lib/pleroma/application.ex index 49094704b..dabce771d 100644 --- a/lib/pleroma/application.ex +++ b/lib/pleroma/application.ex @@ -43,23 +43,9 @@ defmodule Pleroma.Application do hackney_pool_children() ++ [ Pleroma.Stats, - {Oban, Pleroma.Config.get(Oban)}, - %{ - id: :web_push_init, - start: {Task, :start_link, [&Pleroma.Web.Push.init/0]}, - restart: :temporary - }, - %{ - id: :federator_init, - start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]}, - restart: :temporary - }, - %{ - id: :internal_fetch_init, - start: {Task, :start_link, [&Pleroma.Web.ActivityPub.InternalFetchActor.init/0]}, - restart: :temporary - } + {Oban, Pleroma.Config.get(Oban)} ] ++ + task_children(@env) ++ oauth_cleanup_child(oauth_cleanup_enabled?()) ++ streamer_child(@env) ++ chat_child(@env, chat_enabled?()) ++ @@ -141,7 +127,7 @@ defmodule Pleroma.Application do defp streamer_child(:test), do: [] defp streamer_child(_) do - [Pleroma.Web.Streamer] + [Pleroma.Web.Streamer.supervisor()] end defp oauth_cleanup_child(true), @@ -163,4 +149,39 @@ defmodule Pleroma.Application do :hackney_pool.child_spec(pool, options) end end + + defp task_children(:test) do + [ + %{ + id: :web_push_init, + start: {Task, :start_link, [&Pleroma.Web.Push.init/0]}, + restart: :temporary + }, + %{ + id: :federator_init, + start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]}, + restart: :temporary + } + ] + end + + defp task_children(_) do + [ + %{ + id: :web_push_init, + start: {Task, :start_link, [&Pleroma.Web.Push.init/0]}, + restart: :temporary + }, + %{ + id: :federator_init, + start: {Task, :start_link, [&Pleroma.Web.Federator.init/0]}, + restart: :temporary + }, + %{ + id: :internal_fetch_init, + start: {Task, :start_link, [&Pleroma.Web.ActivityPub.InternalFetchActor.init/0]}, + restart: :temporary + } + ] + end end diff --git a/lib/pleroma/constants.ex b/lib/pleroma/constants.ex index ef1418543..0bf20cdd0 100644 --- a/lib/pleroma/constants.ex +++ b/lib/pleroma/constants.ex @@ -6,4 +6,16 @@ defmodule Pleroma.Constants do use Const const(as_public, do: "https://www.w3.org/ns/activitystreams#Public") + + const(object_internal_fields, + do: [ + "likes", + "like_count", + "announcements", + "announcement_count", + "emoji", + "context_id", + "deleted_activity_id" + ] + ) end diff --git a/lib/pleroma/delivery.ex b/lib/pleroma/delivery.ex new file mode 100644 index 000000000..29a1e5a77 --- /dev/null +++ b/lib/pleroma/delivery.ex @@ -0,0 +1,51 @@ +# Pleroma: A lightweight social networking server +# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/> +# SPDX-License-Identifier: AGPL-3.0-only + +defmodule Pleroma.Delivery do + use Ecto.Schema + + alias Pleroma.Delivery + alias Pleroma.FlakeId + alias Pleroma.Object + alias Pleroma.Repo + alias Pleroma.User + alias Pleroma.User + + import Ecto.Changeset + import Ecto.Query + + schema "deliveries" do + belongs_to(:user, User, type: FlakeId) + belongs_to(:object, Object) + end + + def changeset(delivery, params \\ %{}) do + delivery + |> cast(params, [:user_id, :object_id]) + |> validate_required([:user_id, :object_id]) + |> foreign_key_constraint(:object_id) + |> foreign_key_constraint(:user_id) + |> unique_constraint(:user_id, name: :deliveries_user_id_object_id_index) + end + + def create(object_id, user_id) do + %Delivery{} + |> changeset(%{user_id: user_id, object_id: object_id}) + |> Repo.insert(on_conflict: :nothing) + end + + def get(object_id, user_id) do + from(d in Delivery, where: d.user_id == ^user_id and d.object_id == ^object_id) + |> Repo.one() + end + + # A hack because user delete activities have a fake id for whatever reason + # TODO: Get rid of this + def delete_all_by_object_id("pleroma:fake_object_id"), do: {0, []} + + def delete_all_by_object_id(object_id) do + from(d in Delivery, where: d.object_id == ^object_id) + |> Repo.delete_all() + end +end diff --git a/lib/pleroma/docs/markdown.ex b/lib/pleroma/docs/markdown.ex index 8386dc2fb..68b106499 100644 --- a/lib/pleroma/docs/markdown.ex +++ b/lib/pleroma/docs/markdown.ex @@ -23,7 +23,7 @@ defmodule Pleroma.Docs.Markdown do IO.write(file, "#{group[:description]}\n") - for child <- group[:children] do + for child <- group[:children] || [] do print_child_header(file, child) print_suggestions(file, child[:suggestions]) @@ -44,6 +44,17 @@ defmodule Pleroma.Docs.Markdown do {:ok, config_path} end + defp print_child_header(file, %{key: key, type: type, description: description} = _child) do + IO.write( + file, + "- `#{inspect(key)}` (`#{inspect(type)}`): #{description} \n" + ) + end + + defp print_child_header(file, %{key: key, type: type} = _child) do + IO.write(file, "- `#{inspect(key)}` (`#{inspect(type)}`) \n") + end + defp print_suggestion(file, suggestion) when is_list(suggestion) do IO.write(file, " `#{inspect(suggestion)}`\n") end @@ -59,20 +70,19 @@ defmodule Pleroma.Docs.Markdown do defp print_suggestions(_file, nil), do: nil - defp print_suggestions(file, suggestions) do - IO.write(file, "Suggestions:\n") + defp print_suggestions(_file, ""), do: nil + defp print_suggestions(file, suggestions) do if length(suggestions) > 1 do + IO.write(file, "Suggestions:\n") + for suggestion <- suggestions do print_suggestion(file, suggestion, true) end else + IO.write(file, " Suggestion: ") + print_suggestion(file, List.first(suggestions)) end end - - defp print_child_header(file, child) do - IO.write(file, "- `#{inspect(child[:key])}` -`#{inspect(child[:type])}` \n") - IO.write(file, "#{child[:description]} \n") - end end diff --git a/lib/pleroma/flake_id.ex b/lib/pleroma/flake_id.ex index 47d61ca5f..042cf8659 100644 --- a/lib/pleroma/flake_id.ex +++ b/lib/pleroma/flake_id.ex @@ -14,7 +14,7 @@ defmodule Pleroma.FlakeId do @type t :: binary - @behaviour Ecto.Type + use Ecto.Type use GenServer require Logger alias __MODULE__ diff --git a/lib/pleroma/notification.ex b/lib/pleroma/notification.ex index b7c880c51..8012389ac 100644 --- a/lib/pleroma/notification.ex +++ b/lib/pleroma/notification.ex @@ -210,8 +210,10 @@ defmodule Pleroma.Notification do unless skip?(activity, user) do notification = %Notification{user_id: user.id, activity: activity} {:ok, notification} = Repo.insert(notification) - Streamer.stream("user", notification) - Streamer.stream("user:notification", notification) + + ["user", "user:notification"] + |> Streamer.stream(notification) + Push.send(notification) notification end diff --git a/lib/pleroma/object.ex b/lib/pleroma/object.ex index 5033798ae..3fa407931 100644 --- a/lib/pleroma/object.ex +++ b/lib/pleroma/object.ex @@ -38,6 +38,24 @@ defmodule Pleroma.Object do def get_by_id(nil), do: nil def get_by_id(id), do: Repo.get(Object, id) + def get_by_id_and_maybe_refetch(id, opts \\ []) do + %{updated_at: updated_at} = object = get_by_id(id) + + if opts[:interval] && + NaiveDateTime.diff(NaiveDateTime.utc_now(), updated_at) > opts[:interval] do + case Fetcher.refetch_object(object) do + {:ok, %Object{} = object} -> + object + + e -> + Logger.error("Couldn't refresh #{object.data["id"]}:\n#{inspect(e)}") + object + end + else + object + end + end + def get_by_ap_id(nil), do: nil def get_by_ap_id(ap_id) do diff --git a/lib/pleroma/object/fetcher.ex b/lib/pleroma/object/fetcher.ex index c1795ae0f..cea33b5af 100644 --- a/lib/pleroma/object/fetcher.ex +++ b/lib/pleroma/object/fetcher.ex @@ -6,18 +6,39 @@ defmodule Pleroma.Object.Fetcher do alias Pleroma.HTTP alias Pleroma.Object alias Pleroma.Object.Containment + alias Pleroma.Repo alias Pleroma.Signature alias Pleroma.Web.ActivityPub.InternalFetchActor alias Pleroma.Web.ActivityPub.Transmogrifier alias Pleroma.Web.OStatus require Logger + require Pleroma.Constants - defp reinject_object(data) do + defp touch_changeset(changeset) do + updated_at = + NaiveDateTime.utc_now() + |> NaiveDateTime.truncate(:second) + + Ecto.Changeset.put_change(changeset, :updated_at, updated_at) + end + + defp maybe_reinject_internal_fields(data, %{data: %{} = old_data}) do + internal_fields = Map.take(old_data, Pleroma.Constants.object_internal_fields()) + + Map.merge(data, internal_fields) + end + + defp maybe_reinject_internal_fields(data, _), do: data + + defp reinject_object(struct, data) do Logger.debug("Reinjecting object #{data["id"]}") with data <- Transmogrifier.fix_object(data), - {:ok, object} <- Object.create(data) do + data <- maybe_reinject_internal_fields(data, struct), + changeset <- Object.change(struct, %{data: data}), + changeset <- touch_changeset(changeset), + {:ok, object} <- Repo.insert_or_update(changeset) do {:ok, object} else e -> @@ -26,6 +47,17 @@ defmodule Pleroma.Object.Fetcher do end end + def refetch_object(%Object{data: %{"id" => id}} = object) do + with {:local, false} <- {:local, String.starts_with?(id, Pleroma.Web.base_url() <> "/")}, + {:ok, data} <- fetch_and_contain_remote_object_from_id(id), + {:ok, object} <- reinject_object(object, data) do + {:ok, object} + else + {:local, true} -> object + e -> {:error, e} + end + end + # TODO: # This will create a Create activity, which we need internally at the moment. def fetch_object_from_id(id, options \\ []) do @@ -57,7 +89,7 @@ defmodule Pleroma.Object.Fetcher do {:reject, nil} {:object, data, nil} -> - reinject_object(data) + reinject_object(%Object{}, data) {:normalize, object = %Object{}} -> {:ok, object} diff --git a/lib/pleroma/plugs/cache.ex b/lib/pleroma/plugs/cache.ex index a81a861d0..50b534e7b 100644 --- a/lib/pleroma/plugs/cache.ex +++ b/lib/pleroma/plugs/cache.ex @@ -20,6 +20,7 @@ defmodule Pleroma.Plugs.Cache do - `ttl`: An expiration time (time-to-live). This value should be in milliseconds or `nil` to disable expiration. Defaults to `nil`. - `query_params`: Take URL query string into account (`true`), ignore it (`false`) or limit to specific params only (list). Defaults to `true`. + - `tracking_fun`: A function that is called on successfull responses, no matter if the request is cached or not. It should accept a conn as the first argument and the value assigned to `tracking_fun_data` as the second. Additionally, you can overwrite the TTL inside a controller action by assigning `cache_ttl` to the connection struct: @@ -56,6 +57,11 @@ defmodule Pleroma.Plugs.Cache do {:ok, nil} -> cache_resp(conn, opts) + {:ok, {content_type, body, tracking_fun_data}} -> + conn = opts.tracking_fun.(conn, tracking_fun_data) + + send_cached(conn, {content_type, body}) + {:ok, record} -> send_cached(conn, record) @@ -88,9 +94,17 @@ defmodule Pleroma.Plugs.Cache do ttl = Map.get(conn.assigns, :cache_ttl, opts.ttl) key = cache_key(conn, opts) content_type = content_type(conn) - record = {content_type, body} - Cachex.put(:web_resp_cache, key, record, ttl: ttl) + conn = + unless opts[:tracking_fun] do + Cachex.put(:web_resp_cache, key, {content_type, body}, ttl: ttl) + conn + else + tracking_fun_data = Map.get(conn.assigns, :tracking_fun_data, nil) + Cachex.put(:web_resp_cache, key, {content_type, body, tracking_fun_data}, ttl: ttl) + + opts.tracking_fun.(conn, tracking_fun_data) + end put_resp_header(conn, "x-cache", "MISS from Pleroma") diff --git a/lib/pleroma/plugs/http_signature.ex b/lib/pleroma/plugs/http_signature.ex index d87fa52fa..23d22a712 100644 --- a/lib/pleroma/plugs/http_signature.ex +++ b/lib/pleroma/plugs/http_signature.ex @@ -15,7 +15,8 @@ defmodule Pleroma.Web.Plugs.HTTPSignaturePlug do end def call(conn, _opts) do - [signature | _] = get_req_header(conn, "signature") + headers = get_req_header(conn, "signature") + signature = Enum.at(headers, 0) if signature do # set (request-target) header to the appropriate value diff --git a/lib/pleroma/user.ex b/lib/pleroma/user.ex index f0306652c..fb1f24254 100644 --- a/lib/pleroma/user.ex +++ b/lib/pleroma/user.ex @@ -11,6 +11,7 @@ defmodule Pleroma.User do alias Comeonin.Pbkdf2 alias Ecto.Multi alias Pleroma.Activity + alias Pleroma.Delivery alias Pleroma.Keys alias Pleroma.Notification alias Pleroma.Object @@ -62,6 +63,7 @@ defmodule Pleroma.User do field(:last_digest_emailed_at, :naive_datetime) has_many(:notifications, Notification) has_many(:registrations, Registration) + has_many(:deliveries, Delivery) embeds_one(:info, User.Info) timestamps() @@ -148,6 +150,7 @@ defmodule Pleroma.User do Cachex.fetch!(:user_cache, key, fn _ -> {:commit, follow_state(user, target)} end) end + @spec set_follow_state_cache(String.t(), String.t(), String.t()) :: {:ok | :error, boolean()} def set_follow_state_cache(user_ap_id, target_ap_id, state) do Cachex.put( :user_cache, @@ -1640,6 +1643,18 @@ defmodule Pleroma.User do def is_internal_user?(%User{local: true, nickname: "internal." <> _}), do: true def is_internal_user?(_), do: false + # A hack because user delete activities have a fake id for whatever reason + # TODO: Get rid of this + def get_delivered_users_by_object_id("pleroma:fake_object_id"), do: [] + + def get_delivered_users_by_object_id(object_id) do + from(u in User, + inner_join: delivery in assoc(u, :deliveries), + where: delivery.object_id == ^object_id + ) + |> Repo.all() + end + def change_email(user, email) do user |> cast(%{email: email}, [:email]) diff --git a/lib/pleroma/web/activity_pub/activity_pub.ex b/lib/pleroma/web/activity_pub/activity_pub.ex index 41f6a0f1f..e1e90d667 100644 --- a/lib/pleroma/web/activity_pub/activity_pub.ex +++ b/lib/pleroma/web/activity_pub/activity_pub.ex @@ -4,6 +4,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do alias Pleroma.Activity + alias Pleroma.Activity.Ir.Topics alias Pleroma.Config alias Pleroma.Conversation alias Pleroma.Notification @@ -16,6 +17,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do alias Pleroma.User alias Pleroma.Web.ActivityPub.MRF alias Pleroma.Web.ActivityPub.Transmogrifier + alias Pleroma.Web.Streamer alias Pleroma.Web.WebFinger alias Pleroma.Workers.BackgroundWorker @@ -187,9 +189,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do participations |> Repo.preload(:user) - Enum.each(participations, fn participation -> - Pleroma.Web.Streamer.stream("participation", participation) - end) + Streamer.stream("participation", participations) end def stream_out_participations(%Object{data: %{"context" => context}}, user) do @@ -208,41 +208,15 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do def stream_out_participations(_, _), do: :noop - def stream_out(activity) do - if activity.data["type"] in ["Create", "Announce", "Delete"] do - object = Object.normalize(activity) - # Do not stream out poll replies - unless object.data["type"] == "Answer" do - Pleroma.Web.Streamer.stream("user", activity) - Pleroma.Web.Streamer.stream("list", activity) - - if get_visibility(activity) == "public" do - Pleroma.Web.Streamer.stream("public", activity) - - if activity.local do - Pleroma.Web.Streamer.stream("public:local", activity) - end - - if activity.data["type"] in ["Create"] do - object.data - |> Map.get("tag", []) - |> Enum.filter(fn tag -> is_bitstring(tag) end) - |> Enum.each(fn tag -> Pleroma.Web.Streamer.stream("hashtag:" <> tag, activity) end) - - if object.data["attachment"] != [] do - Pleroma.Web.Streamer.stream("public:media", activity) - - if activity.local do - Pleroma.Web.Streamer.stream("public:local:media", activity) - end - end - end - else - if get_visibility(activity) == "direct", - do: Pleroma.Web.Streamer.stream("direct", activity) - end - end - end + def stream_out(%Activity{data: %{"type" => data_type}} = activity) + when data_type in ["Create", "Announce", "Delete"] do + activity + |> Topics.get_activity_topics() + |> Streamer.stream(activity) + end + + def stream_out(_activity) do + :noop end def create(%{to: to, actor: actor, context: context, object: object} = params, fake \\ false) do @@ -436,6 +410,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do end end + @spec block(User.t(), User.t(), String.t() | nil, boolean) :: {:ok, Activity.t() | nil} def block(blocker, blocked, activity_id \\ nil, local \\ true) do outgoing_blocks = Config.get([:activitypub, :outgoing_blocks]) unfollow_blocked = Config.get([:activitypub, :unfollow_blocked]) @@ -464,10 +439,11 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do end end + @spec flag(map()) :: {:ok, Activity.t()} | any def flag( %{ actor: actor, - context: context, + context: _context, account: account, statuses: statuses, content: content @@ -479,14 +455,6 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do additional = params[:additional] || %{} - params = %{ - actor: actor, - context: context, - account: account, - statuses: statuses, - content: content - } - additional = if forward do Map.merge(additional, %{"to" => [], "cc" => [account.ap_id]}) diff --git a/lib/pleroma/web/activity_pub/activity_pub_controller.ex b/lib/pleroma/web/activity_pub/activity_pub_controller.ex index 705dbc1c2..01b34fb1d 100644 --- a/lib/pleroma/web/activity_pub/activity_pub_controller.ex +++ b/lib/pleroma/web/activity_pub/activity_pub_controller.ex @@ -6,6 +6,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do use Pleroma.Web, :controller alias Pleroma.Activity + alias Pleroma.Delivery alias Pleroma.Object alias Pleroma.Object.Fetcher alias Pleroma.User @@ -23,7 +24,12 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do action_fallback(:errors) - plug(Pleroma.Plugs.Cache, [query_params: false] when action in [:activity, :object]) + plug( + Pleroma.Plugs.Cache, + [query_params: false, tracking_fun: &__MODULE__.track_object_fetch/2] + when action in [:activity, :object] + ) + plug(Pleroma.Web.FederatingPlug when action in [:inbox, :relay]) plug(:set_requester_reachable when action in [:inbox]) plug(:relay_active? when action in [:relay]) @@ -54,6 +60,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do %Object{} = object <- Object.get_cached_by_ap_id(ap_id), {_, true} <- {:public?, Visibility.is_public?(object)} do conn + |> assign(:tracking_fun_data, object.id) |> set_cache_ttl_for(object) |> put_resp_content_type("application/activity+json") |> put_view(ObjectView) @@ -64,6 +71,16 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do end end + def track_object_fetch(conn, nil), do: conn + + def track_object_fetch(conn, object_id) do + with %{assigns: %{user: %User{id: user_id}}} <- conn do + Delivery.create(object_id, user_id) + end + + conn + end + def object_likes(conn, %{"uuid" => uuid, "page" => page}) do with ap_id <- o_status_url(conn, :object, uuid), %Object{} = object <- Object.get_cached_by_ap_id(ap_id), @@ -99,6 +116,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do %Activity{} = activity <- Activity.normalize(ap_id), {_, true} <- {:public?, Visibility.is_public?(activity)} do conn + |> maybe_set_tracking_data(activity) |> set_cache_ttl_for(activity) |> put_resp_content_type("application/activity+json") |> put_view(ObjectView) @@ -109,6 +127,13 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do end end + defp maybe_set_tracking_data(conn, %Activity{data: %{"type" => "Create"}} = activity) do + object_id = Object.normalize(activity).id + assign(conn, :tracking_fun_data, object_id) + end + + defp maybe_set_tracking_data(conn, _activity), do: conn + defp set_cache_ttl_for(conn, %Activity{object: object}) do set_cache_ttl_for(conn, object) end diff --git a/lib/pleroma/web/activity_pub/publisher.ex b/lib/pleroma/web/activity_pub/publisher.ex index a6322e25a..114251b24 100644 --- a/lib/pleroma/web/activity_pub/publisher.ex +++ b/lib/pleroma/web/activity_pub/publisher.ex @@ -5,8 +5,10 @@ defmodule Pleroma.Web.ActivityPub.Publisher do alias Pleroma.Activity alias Pleroma.Config + alias Pleroma.Delivery alias Pleroma.HTTP alias Pleroma.Instances + alias Pleroma.Object alias Pleroma.User alias Pleroma.Web.ActivityPub.Relay alias Pleroma.Web.ActivityPub.Transmogrifier @@ -116,7 +118,18 @@ defmodule Pleroma.Web.ActivityPub.Publisher do {:ok, []} end - Pleroma.Web.Salmon.remote_users(actor, activity) ++ followers + fetchers = + with %Activity{data: %{"type" => "Delete"}} <- activity, + %Object{id: object_id} <- Object.normalize(activity), + fetchers <- User.get_delivered_users_by_object_id(object_id), + _ <- Delivery.delete_all_by_object_id(object_id) do + fetchers + else + _ -> + [] + end + + Pleroma.Web.Salmon.remote_users(actor, activity) ++ followers ++ fetchers end defp get_cc_ap_ids(ap_id, recipients) do diff --git a/lib/pleroma/web/activity_pub/transmogrifier.ex b/lib/pleroma/web/activity_pub/transmogrifier.ex index acb3087d0..9d2ddc1cd 100644 --- a/lib/pleroma/web/activity_pub/transmogrifier.ex +++ b/lib/pleroma/web/activity_pub/transmogrifier.ex @@ -979,15 +979,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do defp strip_internal_fields(object) do object - |> Map.drop([ - "likes", - "like_count", - "announcements", - "announcement_count", - "emoji", - "context_id", - "deleted_activity_id" - ]) + |> Map.drop(Pleroma.Constants.object_internal_fields()) end defp strip_internal_tags(%{"tag" => tags} = object) do @@ -1050,7 +1042,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do with %User{local: false} = user <- User.get_cached_by_ap_id(ap_id), {:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id), already_ap <- User.ap_enabled?(user), - {:ok, user} <- user |> User.upgrade_changeset(data) |> User.update_and_set_cache() do + {:ok, user} <- user |> User.upgrade_changeset(data, true) |> User.update_and_set_cache() do unless already_ap do TransmogrifierWorker.enqueue("user_upgrade", %{"user_id" => user.id}) end diff --git a/lib/pleroma/web/activity_pub/utils.ex b/lib/pleroma/web/activity_pub/utils.ex index 258e56066..30628a793 100644 --- a/lib/pleroma/web/activity_pub/utils.ex +++ b/lib/pleroma/web/activity_pub/utils.ex @@ -33,50 +33,40 @@ defmodule Pleroma.Web.ActivityPub.Utils do Map.put(params, "actor", get_ap_id(params["actor"])) end - def determine_explicit_mentions(%{"tag" => tag} = _object) when is_list(tag) do - tag - |> Enum.filter(fn x -> is_map(x) end) - |> Enum.filter(fn x -> x["type"] == "Mention" end) - |> Enum.map(fn x -> x["href"] end) + @spec determine_explicit_mentions(map()) :: map() + def determine_explicit_mentions(%{"tag" => tag} = _) when is_list(tag) do + Enum.flat_map(tag, fn + %{"type" => "Mention", "href" => href} -> [href] + _ -> [] + end) end def determine_explicit_mentions(%{"tag" => tag} = object) when is_map(tag) do - Map.put(object, "tag", [tag]) + object + |> Map.put("tag", [tag]) |> determine_explicit_mentions() end def determine_explicit_mentions(_), do: [] + @spec recipient_in_collection(any(), any()) :: boolean() defp recipient_in_collection(ap_id, coll) when is_binary(coll), do: ap_id == coll defp recipient_in_collection(ap_id, coll) when is_list(coll), do: ap_id in coll defp recipient_in_collection(_, _), do: false + @spec recipient_in_message(User.t(), User.t(), map()) :: boolean() def recipient_in_message(%User{ap_id: ap_id} = recipient, %User{} = actor, params) do - cond do - recipient_in_collection(ap_id, params["to"]) -> - true - - recipient_in_collection(ap_id, params["cc"]) -> - true - - recipient_in_collection(ap_id, params["bto"]) -> - true - - recipient_in_collection(ap_id, params["bcc"]) -> - true + addresses = [params["to"], params["cc"], params["bto"], params["bcc"]] + cond do + Enum.any?(addresses, &recipient_in_collection(ap_id, &1)) -> true # if the message is unaddressed at all, then assume it is directly addressed # to the recipient - !params["to"] && !params["cc"] && !params["bto"] && !params["bcc"] -> - true - + Enum.all?(addresses, &is_nil(&1)) -> true # if the message is sent from somebody the user is following, then assume it # is addressed to the recipient - User.following?(recipient, actor) -> - true - - true -> - false + User.following?(recipient, actor) -> true + true -> false end end @@ -179,54 +169,59 @@ defmodule Pleroma.Web.ActivityPub.Utils do Adds an id and a published data if they aren't there, also adds it to an included object """ - def lazy_put_activity_defaults(map, fake? \\ false) do - map = - if not fake? do - %{data: %{"id" => context}, id: context_id} = create_context(map["context"]) - - map - |> Map.put_new_lazy("id", &generate_activity_id/0) - |> Map.put_new_lazy("published", &make_date/0) - |> Map.put_new("context", context) - |> Map.put_new("context_id", context_id) - else - map - |> Map.put_new("id", "pleroma:fakeid") - |> Map.put_new_lazy("published", &make_date/0) - |> Map.put_new("context", "pleroma:fakecontext") - |> Map.put_new("context_id", -1) - end + @spec lazy_put_activity_defaults(map(), boolean) :: map() + def lazy_put_activity_defaults(map, fake? \\ false) - if is_map(map["object"]) do - object = lazy_put_object_defaults(map["object"], map, fake?) - %{map | "object" => object} - else - map - end + def lazy_put_activity_defaults(map, true) do + map + |> Map.put_new("id", "pleroma:fakeid") + |> Map.put_new_lazy("published", &make_date/0) + |> Map.put_new("context", "pleroma:fakecontext") + |> Map.put_new("context_id", -1) + |> lazy_put_object_defaults(true) end - @doc """ - Adds an id and published date if they aren't there. - """ - def lazy_put_object_defaults(map, activity \\ %{}, fake?) + def lazy_put_activity_defaults(map, _fake?) do + %{data: %{"id" => context}, id: context_id} = create_context(map["context"]) - def lazy_put_object_defaults(map, activity, true = _fake?) do map + |> Map.put_new_lazy("id", &generate_activity_id/0) |> Map.put_new_lazy("published", &make_date/0) - |> Map.put_new("id", "pleroma:fake_object_id") - |> Map.put_new("context", activity["context"]) - |> Map.put_new("fake", true) - |> Map.put_new("context_id", activity["context_id"]) + |> Map.put_new("context", context) + |> Map.put_new("context_id", context_id) + |> lazy_put_object_defaults(false) end - def lazy_put_object_defaults(map, activity, _fake?) do - map - |> Map.put_new_lazy("id", &generate_object_id/0) - |> Map.put_new_lazy("published", &make_date/0) - |> Map.put_new("context", activity["context"]) - |> Map.put_new("context_id", activity["context_id"]) + # Adds an id and published date if they aren't there. + # + @spec lazy_put_object_defaults(map(), boolean()) :: map() + defp lazy_put_object_defaults(%{"object" => map} = activity, true) + when is_map(map) do + object = + map + |> Map.put_new("id", "pleroma:fake_object_id") + |> Map.put_new_lazy("published", &make_date/0) + |> Map.put_new("context", activity["context"]) + |> Map.put_new("context_id", activity["context_id"]) + |> Map.put_new("fake", true) + + %{activity | "object" => object} + end + + defp lazy_put_object_defaults(%{"object" => map} = activity, _) + when is_map(map) do + object = + map + |> Map.put_new_lazy("id", &generate_object_id/0) + |> Map.put_new_lazy("published", &make_date/0) + |> Map.put_new("context", activity["context"]) + |> Map.put_new("context_id", activity["context_id"]) + + %{activity | "object" => object} end + defp lazy_put_object_defaults(activity, _), do: activity + @doc """ Inserts a full object if it is contained in an activity. """ @@ -345,24 +340,24 @@ defmodule Pleroma.Web.ActivityPub.Utils do @doc """ Updates a follow activity's state (for locked accounts). """ + @spec update_follow_state_for_all(Activity.t(), String.t()) :: {:ok, Activity} | {:error, any()} def update_follow_state_for_all( %Activity{data: %{"actor" => actor, "object" => object}} = activity, state ) do - try do - Ecto.Adapters.SQL.query!( - Repo, - "UPDATE activities SET data = jsonb_set(data, '{state}', $1) WHERE data->>'type' = 'Follow' AND data->>'actor' = $2 AND data->>'object' = $3 AND data->>'state' = 'pending'", - [state, actor, object] - ) + "Follow" + |> Activity.Queries.by_type() + |> Activity.Queries.by_actor(actor) + |> Activity.Queries.by_object_id(object) + |> where(fragment("data->>'state' = 'pending'")) + |> update(set: [data: fragment("jsonb_set(data, '{state}', ?)", ^state)]) + |> Repo.update_all([]) - User.set_follow_state_cache(actor, object, state) - activity = Activity.get_by_id(activity.id) - {:ok, activity} - rescue - e -> - {:error, e} - end + User.set_follow_state_cache(actor, object, state) + + activity = Activity.get_by_id(activity.id) + + {:ok, activity} end def update_follow_state( @@ -413,6 +408,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do @doc """ Retruns an existing announce activity if the notice has already been announced """ + @spec get_existing_announce(String.t(), map()) :: Activity.t() | nil def get_existing_announce(actor, %{data: %{"id" => ap_id}}) do "Announce" |> Activity.Queries.by_type() @@ -495,33 +491,35 @@ defmodule Pleroma.Web.ActivityPub.Utils do |> maybe_put("id", activity_id) end + @spec add_announce_to_object(Activity.t(), Object.t()) :: + {:ok, Object.t()} | {:error, Ecto.Changeset.t()} def add_announce_to_object( - %Activity{ - data: %{"actor" => actor, "cc" => [Pleroma.Constants.as_public()]} - }, + %Activity{data: %{"actor" => actor, "cc" => [Pleroma.Constants.as_public()]}}, object ) do - announcements = - if is_list(object.data["announcements"]) do - Enum.uniq([actor | object.data["announcements"]]) - else - [actor] - end + announcements = take_announcements(object) - update_element_in_object("announcement", announcements, object) + with announcements <- Enum.uniq([actor | announcements]) do + update_element_in_object("announcement", announcements, object) + end end def add_announce_to_object(_, object), do: {:ok, object} + @spec remove_announce_from_object(Activity.t(), Object.t()) :: + {:ok, Object.t()} | {:error, Ecto.Changeset.t()} def remove_announce_from_object(%Activity{data: %{"actor" => actor}}, object) do - announcements = - if is_list(object.data["announcements"]), do: object.data["announcements"], else: [] - - with announcements <- announcements |> List.delete(actor) do + with announcements <- List.delete(take_announcements(object), actor) do update_element_in_object("announcement", announcements, object) end end + defp take_announcements(%{data: %{"announcements" => announcements}} = _) + when is_list(announcements), + do: announcements + + defp take_announcements(_), do: [] + #### Unfollow-related helpers def make_unfollow_data(follower, followed, follow_activity, activity_id) do @@ -535,6 +533,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do end #### Block-related helpers + @spec fetch_latest_block(User.t(), User.t()) :: Activity.t() | nil def fetch_latest_block(%User{ap_id: blocker_id}, %User{ap_id: blocked_id}) do "Block" |> Activity.Queries.by_type() @@ -583,28 +582,32 @@ defmodule Pleroma.Web.ActivityPub.Utils do end #### Flag-related helpers - - def make_flag_data(params, additional) do - status_ap_ids = - Enum.map(params.statuses || [], fn - %Activity{} = act -> act.data["id"] - act when is_map(act) -> act["id"] - act when is_binary(act) -> act - end) - - object = [params.account.ap_id] ++ status_ap_ids - + @spec make_flag_data(map(), map()) :: map() + def make_flag_data(%{actor: actor, context: context, content: content} = params, additional) do %{ "type" => "Flag", - "actor" => params.actor.ap_id, - "content" => params.content, - "object" => object, - "context" => params.context, + "actor" => actor.ap_id, + "content" => content, + "object" => build_flag_object(params), + "context" => context, "state" => "open" } |> Map.merge(additional) end + def make_flag_data(_, _), do: %{} + + defp build_flag_object(%{account: account, statuses: statuses} = _) do + [account.ap_id] ++ + Enum.map(statuses || [], fn + %Activity{} = act -> act.data["id"] + act when is_map(act) -> act["id"] + act when is_binary(act) -> act + end) + end + + defp build_flag_object(_), do: [] + @doc """ Fetches the OrderedCollection/OrderedCollectionPage from `from`, limiting the amount of pages fetched after the first one to `pages_left` pages. diff --git a/lib/pleroma/web/admin_api/admin_api_controller.ex b/lib/pleroma/web/admin_api/admin_api_controller.ex index 544b9d7d8..2a1cc59e5 100644 --- a/lib/pleroma/web/admin_api/admin_api_controller.ex +++ b/lib/pleroma/web/admin_api/admin_api_controller.ex @@ -442,11 +442,9 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do params |> Map.put("type", "Flag") |> Map.put("skip_preload", true) + |> Map.put("total", true) - reports = - [] - |> ActivityPub.fetch_activities(params) - |> Enum.reverse() + reports = ActivityPub.fetch_activities([], params) conn |> put_view(ReportView) diff --git a/lib/pleroma/web/admin_api/views/report_view.ex b/lib/pleroma/web/admin_api/views/report_view.ex index a25f3f1fe..51b95ad5e 100644 --- a/lib/pleroma/web/admin_api/views/report_view.ex +++ b/lib/pleroma/web/admin_api/views/report_view.ex @@ -12,7 +12,9 @@ defmodule Pleroma.Web.AdminAPI.ReportView do def render("index.json", %{reports: reports}) do %{ - reports: render_many(reports, __MODULE__, "show.json", as: :report) + reports: + render_many(reports[:items], __MODULE__, "show.json", as: :report) |> Enum.reverse(), + total: reports[:total] } end diff --git a/lib/pleroma/web/mastodon_api/controllers/mastodon_api_controller.ex b/lib/pleroma/web/mastodon_api/controllers/mastodon_api_controller.ex index 060137b80..970cfd8db 100644 --- a/lib/pleroma/web/mastodon_api/controllers/mastodon_api_controller.ex +++ b/lib/pleroma/web/mastodon_api/controllers/mastodon_api_controller.ex @@ -485,7 +485,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do end def get_poll(%{assigns: %{user: user}} = conn, %{"id" => id}) do - with %Object{} = object <- Object.get_by_id(id), + with %Object{} = object <- Object.get_by_id_and_maybe_refetch(id, interval: 60), %Activity{} = activity <- Activity.get_create_by_object_ap_id(object.data["id"]), true <- Visibility.visible_for_user?(activity, user) do conn diff --git a/lib/pleroma/web/mastodon_api/websocket_handler.ex b/lib/pleroma/web/mastodon_api/websocket_handler.ex index dbd3542ea..3c26eb406 100644 --- a/lib/pleroma/web/mastodon_api/websocket_handler.ex +++ b/lib/pleroma/web/mastodon_api/websocket_handler.ex @@ -8,6 +8,7 @@ defmodule Pleroma.Web.MastodonAPI.WebsocketHandler do alias Pleroma.Repo alias Pleroma.User alias Pleroma.Web.OAuth.Token + alias Pleroma.Web.Streamer @behaviour :cowboy_websocket @@ -24,7 +25,7 @@ defmodule Pleroma.Web.MastodonAPI.WebsocketHandler do ] @anonymous_streams ["public", "public:local", "hashtag"] - # Handled by periodic keepalive in Pleroma.Web.Streamer. + # Handled by periodic keepalive in Pleroma.Web.Streamer.Ping. @timeout :infinity def init(%{qs: qs} = req, state) do @@ -65,7 +66,7 @@ defmodule Pleroma.Web.MastodonAPI.WebsocketHandler do }, topic #{state.topic}" ) - Pleroma.Web.Streamer.add_socket(state.topic, streamer_socket(state)) + Streamer.add_socket(state.topic, streamer_socket(state)) {:ok, state} end @@ -80,7 +81,7 @@ defmodule Pleroma.Web.MastodonAPI.WebsocketHandler do }, topic #{state.topic || "?"}: #{inspect(reason)}" ) - Pleroma.Web.Streamer.remove_socket(state.topic, streamer_socket(state)) + Streamer.remove_socket(state.topic, streamer_socket(state)) :ok end diff --git a/lib/pleroma/web/rich_media/parser.ex b/lib/pleroma/web/rich_media/parser.ex index f5f9e358c..c06b0a0f2 100644 --- a/lib/pleroma/web/rich_media/parser.ex +++ b/lib/pleroma/web/rich_media/parser.ex @@ -81,6 +81,7 @@ defmodule Pleroma.Web.RichMedia.Parser do {:ok, %Tesla.Env{body: html}} = Pleroma.HTTP.get(url, [], adapter: @hackney_options) html + |> parse_html |> maybe_parse() |> Map.put(:url, url) |> clean_parsed_data() @@ -91,6 +92,8 @@ defmodule Pleroma.Web.RichMedia.Parser do end end + defp parse_html(html), do: Floki.parse(html) + defp maybe_parse(html) do Enum.reduce_while(parsers(), %{}, fn parser, acc -> case parser.parse(html, acc) do @@ -100,7 +103,8 @@ defmodule Pleroma.Web.RichMedia.Parser do end) end - defp check_parsed_data(%{title: title} = data) when is_binary(title) and byte_size(title) > 0 do + defp check_parsed_data(%{title: title} = data) + when is_binary(title) and byte_size(title) > 0 do {:ok, data} end diff --git a/lib/pleroma/web/router.ex b/lib/pleroma/web/router.ex index b0464037e..401133bf3 100644 --- a/lib/pleroma/web/router.ex +++ b/lib/pleroma/web/router.ex @@ -135,6 +135,7 @@ defmodule Pleroma.Web.Router do pipeline :http_signature do plug(Pleroma.Web.Plugs.HTTPSignaturePlug) + plug(Pleroma.Web.Plugs.MappedSignatureToIdentityPlug) end scope "/api/pleroma", Pleroma.Web.TwitterAPI do @@ -514,6 +515,7 @@ defmodule Pleroma.Web.Router do scope "/", Pleroma.Web do pipe_through(:ostatus) + pipe_through(:http_signature) get("/objects/:uuid", OStatus.OStatusController, :object) get("/activities/:uuid", OStatus.OStatusController, :activity) diff --git a/lib/pleroma/web/streamer.ex b/lib/pleroma/web/streamer.ex deleted file mode 100644 index 587c43f40..000000000 --- a/lib/pleroma/web/streamer.ex +++ /dev/null @@ -1,318 +0,0 @@ -# Pleroma: A lightweight social networking server -# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/> -# SPDX-License-Identifier: AGPL-3.0-only - -defmodule Pleroma.Web.Streamer do - use GenServer - require Logger - alias Pleroma.Activity - alias Pleroma.Config - alias Pleroma.Conversation.Participation - alias Pleroma.Notification - alias Pleroma.Object - alias Pleroma.User - alias Pleroma.Web.ActivityPub.ActivityPub - alias Pleroma.Web.ActivityPub.Visibility - alias Pleroma.Web.CommonAPI - alias Pleroma.Web.MastodonAPI.NotificationView - - @keepalive_interval :timer.seconds(30) - - def start_link(_) do - GenServer.start_link(__MODULE__, %{}, name: __MODULE__) - end - - def add_socket(topic, socket) do - GenServer.cast(__MODULE__, %{action: :add, socket: socket, topic: topic}) - end - - def remove_socket(topic, socket) do - GenServer.cast(__MODULE__, %{action: :remove, socket: socket, topic: topic}) - end - - def stream(topic, item) do - GenServer.cast(__MODULE__, %{action: :stream, topic: topic, item: item}) - end - - def init(args) do - Process.send_after(self(), %{action: :ping}, @keepalive_interval) - - {:ok, args} - end - - def handle_info(%{action: :ping}, topics) do - topics - |> Map.values() - |> List.flatten() - |> Enum.each(fn socket -> - Logger.debug("Sending keepalive ping") - send(socket.transport_pid, {:text, ""}) - end) - - Process.send_after(self(), %{action: :ping}, @keepalive_interval) - - {:noreply, topics} - end - - def handle_cast(%{action: :stream, topic: "direct", item: item}, topics) do - recipient_topics = - User.get_recipients_from_activity(item) - |> Enum.map(fn %{id: id} -> "direct:#{id}" end) - - Enum.each(recipient_topics || [], fn user_topic -> - Logger.debug("Trying to push direct message to #{user_topic}\n\n") - push_to_socket(topics, user_topic, item) - end) - - {:noreply, topics} - end - - def handle_cast(%{action: :stream, topic: "participation", item: participation}, topics) do - user_topic = "direct:#{participation.user_id}" - Logger.debug("Trying to push a conversation participation to #{user_topic}\n\n") - - push_to_socket(topics, user_topic, participation) - - {:noreply, topics} - end - - def handle_cast(%{action: :stream, topic: "list", item: item}, topics) do - # filter the recipient list if the activity is not public, see #270. - recipient_lists = - case Visibility.is_public?(item) do - true -> - Pleroma.List.get_lists_from_activity(item) - - _ -> - Pleroma.List.get_lists_from_activity(item) - |> Enum.filter(fn list -> - owner = User.get_cached_by_id(list.user_id) - - Visibility.visible_for_user?(item, owner) - end) - end - - recipient_topics = - recipient_lists - |> Enum.map(fn %{id: id} -> "list:#{id}" end) - - Enum.each(recipient_topics || [], fn list_topic -> - Logger.debug("Trying to push message to #{list_topic}\n\n") - push_to_socket(topics, list_topic, item) - end) - - {:noreply, topics} - end - - def handle_cast( - %{action: :stream, topic: topic, item: %Notification{} = item}, - topics - ) - when topic in ["user", "user:notification"] do - topics - |> Map.get("#{topic}:#{item.user_id}", []) - |> Enum.each(fn socket -> - with %User{} = user <- User.get_cached_by_ap_id(socket.assigns[:user].ap_id), - true <- should_send?(user, item) do - send( - socket.transport_pid, - {:text, represent_notification(socket.assigns[:user], item)} - ) - end - end) - - {:noreply, topics} - end - - def handle_cast(%{action: :stream, topic: "user", item: item}, topics) do - Logger.debug("Trying to push to users") - - recipient_topics = - User.get_recipients_from_activity(item) - |> Enum.map(fn %{id: id} -> "user:#{id}" end) - - Enum.each(recipient_topics, fn topic -> - push_to_socket(topics, topic, item) - end) - - {:noreply, topics} - end - - def handle_cast(%{action: :stream, topic: topic, item: item}, topics) do - Logger.debug("Trying to push to #{topic}") - Logger.debug("Pushing item to #{topic}") - push_to_socket(topics, topic, item) - {:noreply, topics} - end - - def handle_cast(%{action: :add, topic: topic, socket: socket}, sockets) do - topic = internal_topic(topic, socket) - sockets_for_topic = sockets[topic] || [] - sockets_for_topic = Enum.uniq([socket | sockets_for_topic]) - sockets = Map.put(sockets, topic, sockets_for_topic) - Logger.debug("Got new conn for #{topic}") - {:noreply, sockets} - end - - def handle_cast(%{action: :remove, topic: topic, socket: socket}, sockets) do - topic = internal_topic(topic, socket) - sockets_for_topic = sockets[topic] || [] - sockets_for_topic = List.delete(sockets_for_topic, socket) - sockets = Map.put(sockets, topic, sockets_for_topic) - Logger.debug("Removed conn for #{topic}") - {:noreply, sockets} - end - - def handle_cast(m, state) do - Logger.info("Unknown: #{inspect(m)}, #{inspect(state)}") - {:noreply, state} - end - - defp represent_update(%Activity{} = activity, %User{} = user) do - %{ - event: "update", - payload: - Pleroma.Web.MastodonAPI.StatusView.render( - "status.json", - activity: activity, - for: user - ) - |> Jason.encode!() - } - |> Jason.encode!() - end - - defp represent_update(%Activity{} = activity) do - %{ - event: "update", - payload: - Pleroma.Web.MastodonAPI.StatusView.render( - "status.json", - activity: activity - ) - |> Jason.encode!() - } - |> Jason.encode!() - end - - def represent_conversation(%Participation{} = participation) do - %{ - event: "conversation", - payload: - Pleroma.Web.MastodonAPI.ConversationView.render("participation.json", %{ - participation: participation, - for: participation.user - }) - |> Jason.encode!() - } - |> Jason.encode!() - end - - @spec represent_notification(User.t(), Notification.t()) :: binary() - defp represent_notification(%User{} = user, %Notification{} = notify) do - %{ - event: "notification", - payload: - NotificationView.render( - "show.json", - %{notification: notify, for: user} - ) - |> Jason.encode!() - } - |> Jason.encode!() - end - - defp should_send?(%User{} = user, %Activity{} = item) do - blocks = user.info.blocks || [] - mutes = user.info.mutes || [] - reblog_mutes = user.info.muted_reblogs || [] - domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(user.info.domain_blocks) - - with parent when not is_nil(parent) <- Object.normalize(item), - true <- Enum.all?([blocks, mutes, reblog_mutes], &(item.actor not in &1)), - true <- Enum.all?([blocks, mutes], &(parent.data["actor"] not in &1)), - %{host: item_host} <- URI.parse(item.actor), - %{host: parent_host} <- URI.parse(parent.data["actor"]), - false <- Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, item_host), - false <- Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, parent_host), - true <- thread_containment(item, user), - false <- CommonAPI.thread_muted?(user, item) do - true - else - _ -> false - end - end - - defp should_send?(%User{} = user, %Notification{activity: activity}) do - should_send?(user, activity) - end - - def push_to_socket(topics, topic, %Activity{data: %{"type" => "Announce"}} = item) do - Enum.each(topics[topic] || [], fn socket -> - # Get the current user so we have up-to-date blocks etc. - if socket.assigns[:user] do - user = User.get_cached_by_ap_id(socket.assigns[:user].ap_id) - - if should_send?(user, item) do - send(socket.transport_pid, {:text, represent_update(item, user)}) - end - else - send(socket.transport_pid, {:text, represent_update(item)}) - end - end) - end - - def push_to_socket(topics, topic, %Participation{} = participation) do - Enum.each(topics[topic] || [], fn socket -> - send(socket.transport_pid, {:text, represent_conversation(participation)}) - end) - end - - def push_to_socket(topics, topic, %Activity{ - data: %{"type" => "Delete", "deleted_activity_id" => deleted_activity_id} - }) do - Enum.each(topics[topic] || [], fn socket -> - send( - socket.transport_pid, - {:text, %{event: "delete", payload: to_string(deleted_activity_id)} |> Jason.encode!()} - ) - end) - end - - def push_to_socket(_topics, _topic, %Activity{data: %{"type" => "Delete"}}), do: :noop - - def push_to_socket(topics, topic, item) do - Enum.each(topics[topic] || [], fn socket -> - # Get the current user so we have up-to-date blocks etc. - if socket.assigns[:user] do - user = User.get_cached_by_ap_id(socket.assigns[:user].ap_id) - blocks = user.info.blocks || [] - mutes = user.info.mutes || [] - - with true <- Enum.all?([blocks, mutes], &(item.actor not in &1)), - true <- thread_containment(item, user) do - send(socket.transport_pid, {:text, represent_update(item, user)}) - end - else - send(socket.transport_pid, {:text, represent_update(item)}) - end - end) - end - - defp internal_topic(topic, socket) when topic in ~w[user user:notification direct] do - "#{topic}:#{socket.assigns[:user].id}" - end - - defp internal_topic(topic, _), do: topic - - @spec thread_containment(Activity.t(), User.t()) :: boolean() - defp thread_containment(_activity, %User{info: %{skip_thread_containment: true}}), do: true - - defp thread_containment(activity, user) do - if Config.get([:instance, :skip_thread_containment]) do - true - else - ActivityPub.contain_activity(activity, user) - end - end -end diff --git a/lib/pleroma/web/streamer/ping.ex b/lib/pleroma/web/streamer/ping.ex new file mode 100644 index 000000000..f77cbb95c --- /dev/null +++ b/lib/pleroma/web/streamer/ping.ex @@ -0,0 +1,33 @@ +defmodule Pleroma.Web.Streamer.Ping do + use GenServer + require Logger + + alias Pleroma.Web.Streamer.State + alias Pleroma.Web.Streamer.StreamerSocket + + @keepalive_interval :timer.seconds(30) + + def start_link(opts) do + ping_interval = Keyword.get(opts, :ping_interval, @keepalive_interval) + GenServer.start_link(__MODULE__, %{ping_interval: ping_interval}, name: __MODULE__) + end + + def init(%{ping_interval: ping_interval} = args) do + Process.send_after(self(), :ping, ping_interval) + {:ok, args} + end + + def handle_info(:ping, %{ping_interval: ping_interval} = state) do + State.get_sockets() + |> Map.values() + |> List.flatten() + |> Enum.each(fn %StreamerSocket{transport_pid: transport_pid} -> + Logger.debug("Sending keepalive ping") + send(transport_pid, {:text, ""}) + end) + + Process.send_after(self(), :ping, ping_interval) + + {:noreply, state} + end +end diff --git a/lib/pleroma/web/streamer/state.ex b/lib/pleroma/web/streamer/state.ex new file mode 100644 index 000000000..c48752d95 --- /dev/null +++ b/lib/pleroma/web/streamer/state.ex @@ -0,0 +1,78 @@ +defmodule Pleroma.Web.Streamer.State do + use GenServer + require Logger + + alias Pleroma.Web.Streamer.StreamerSocket + + @env Mix.env() + + def start_link(_) do + GenServer.start_link(__MODULE__, %{sockets: %{}}, name: __MODULE__) + end + + def add_socket(topic, socket) do + GenServer.call(__MODULE__, {:add, topic, socket}) + end + + def remove_socket(topic, socket) do + do_remove_socket(@env, topic, socket) + end + + def get_sockets do + %{sockets: stream_sockets} = GenServer.call(__MODULE__, :get_state) + stream_sockets + end + + def init(init_arg) do + {:ok, init_arg} + end + + def handle_call(:get_state, _from, state) do + {:reply, state, state} + end + + def handle_call({:add, topic, socket}, _from, %{sockets: sockets} = state) do + internal_topic = internal_topic(topic, socket) + stream_socket = StreamerSocket.from_socket(socket) + + sockets_for_topic = + sockets + |> Map.get(internal_topic, []) + |> List.insert_at(0, stream_socket) + |> Enum.uniq() + + state = put_in(state, [:sockets, internal_topic], sockets_for_topic) + Logger.debug("Got new conn for #{topic}") + {:reply, state, state} + end + + def handle_call({:remove, topic, socket}, _from, %{sockets: sockets} = state) do + internal_topic = internal_topic(topic, socket) + stream_socket = StreamerSocket.from_socket(socket) + + sockets_for_topic = + sockets + |> Map.get(internal_topic, []) + |> List.delete(stream_socket) + + state = Kernel.put_in(state, [:sockets, internal_topic], sockets_for_topic) + {:reply, state, state} + end + + defp do_remove_socket(:test, _, _) do + :ok + end + + defp do_remove_socket(_env, topic, socket) do + GenServer.call(__MODULE__, {:remove, topic, socket}) + end + + defp internal_topic(topic, socket) + when topic in ~w[user user:notification direct] do + "#{topic}:#{socket.assigns[:user].id}" + end + + defp internal_topic(topic, _) do + topic + end +end diff --git a/lib/pleroma/web/streamer/streamer.ex b/lib/pleroma/web/streamer/streamer.ex new file mode 100644 index 000000000..8cf719277 --- /dev/null +++ b/lib/pleroma/web/streamer/streamer.ex @@ -0,0 +1,55 @@ +# Pleroma: A lightweight social networking server +# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/> +# SPDX-License-Identifier: AGPL-3.0-only + +defmodule Pleroma.Web.Streamer do + alias Pleroma.Web.Streamer.State + alias Pleroma.Web.Streamer.Worker + + @timeout 60_000 + @mix_env Mix.env() + + def add_socket(topic, socket) do + State.add_socket(topic, socket) + end + + def remove_socket(topic, socket) do + State.remove_socket(topic, socket) + end + + def get_sockets do + State.get_sockets() + end + + def stream(topics, items) do + if should_send?() do + Task.async(fn -> + :poolboy.transaction( + :streamer_worker, + &Worker.stream(&1, topics, items), + @timeout + ) + end) + end + end + + def supervisor, do: Pleroma.Web.Streamer.Supervisor + + defp should_send? do + handle_should_send(@mix_env) + end + + defp handle_should_send(:test) do + case Process.whereis(:streamer_worker) do + nil -> + false + + pid -> + Process.alive?(pid) + end + end + + defp handle_should_send(_) do + true + end +end diff --git a/lib/pleroma/web/streamer/streamer_socket.ex b/lib/pleroma/web/streamer/streamer_socket.ex new file mode 100644 index 000000000..f006c0306 --- /dev/null +++ b/lib/pleroma/web/streamer/streamer_socket.ex @@ -0,0 +1,31 @@ +defmodule Pleroma.Web.Streamer.StreamerSocket do + defstruct transport_pid: nil, user: nil + + alias Pleroma.User + alias Pleroma.Web.Streamer.StreamerSocket + + def from_socket(%{ + transport_pid: transport_pid, + assigns: %{user: nil} + }) do + %StreamerSocket{ + transport_pid: transport_pid + } + end + + def from_socket(%{ + transport_pid: transport_pid, + assigns: %{user: %User{} = user} + }) do + %StreamerSocket{ + transport_pid: transport_pid, + user: user + } + end + + def from_socket(%{transport_pid: transport_pid}) do + %StreamerSocket{ + transport_pid: transport_pid + } + end +end diff --git a/lib/pleroma/web/streamer/supervisor.ex b/lib/pleroma/web/streamer/supervisor.ex new file mode 100644 index 000000000..6afe19323 --- /dev/null +++ b/lib/pleroma/web/streamer/supervisor.ex @@ -0,0 +1,33 @@ +defmodule Pleroma.Web.Streamer.Supervisor do + use Supervisor + + def start_link(opts) do + Supervisor.start_link(__MODULE__, opts, name: __MODULE__) + end + + def init(args) do + children = [ + {Pleroma.Web.Streamer.State, args}, + {Pleroma.Web.Streamer.Ping, args}, + :poolboy.child_spec(:streamer_worker, poolboy_config()) + ] + + opts = [strategy: :one_for_one, name: Pleroma.Web.Streamer.Supervisor] + Supervisor.init(children, opts) + end + + defp poolboy_config do + opts = + Pleroma.Config.get(:streamer, + workers: 3, + overflow_workers: 2 + ) + + [ + {:name, {:local, :streamer_worker}}, + {:worker_module, Pleroma.Web.Streamer.Worker}, + {:size, opts[:workers]}, + {:max_overflow, opts[:overflow_workers]} + ] + end +end diff --git a/lib/pleroma/web/streamer/worker.ex b/lib/pleroma/web/streamer/worker.ex new file mode 100644 index 000000000..5804508eb --- /dev/null +++ b/lib/pleroma/web/streamer/worker.ex @@ -0,0 +1,220 @@ +defmodule Pleroma.Web.Streamer.Worker do + use GenServer + + require Logger + + alias Pleroma.Activity + alias Pleroma.Config + alias Pleroma.Conversation.Participation + alias Pleroma.Notification + alias Pleroma.Object + alias Pleroma.User + alias Pleroma.Web.ActivityPub.ActivityPub + alias Pleroma.Web.ActivityPub.Visibility + alias Pleroma.Web.CommonAPI + alias Pleroma.Web.Streamer.State + alias Pleroma.Web.Streamer.StreamerSocket + alias Pleroma.Web.StreamerView + + def start_link(_) do + GenServer.start_link(__MODULE__, %{}, []) + end + + def init(init_arg) do + {:ok, init_arg} + end + + def stream(pid, topics, items) do + GenServer.call(pid, {:stream, topics, items}) + end + + def handle_call({:stream, topics, item}, _from, state) when is_list(topics) do + Enum.each(topics, fn t -> + do_stream(%{topic: t, item: item}) + end) + + {:reply, state, state} + end + + def handle_call({:stream, topic, items}, _from, state) when is_list(items) do + Enum.each(items, fn i -> + do_stream(%{topic: topic, item: i}) + end) + + {:reply, state, state} + end + + def handle_call({:stream, topic, item}, _from, state) do + do_stream(%{topic: topic, item: item}) + + {:reply, state, state} + end + + defp do_stream(%{topic: "direct", item: item}) do + recipient_topics = + User.get_recipients_from_activity(item) + |> Enum.map(fn %{id: id} -> "direct:#{id}" end) + + Enum.each(recipient_topics, fn user_topic -> + Logger.debug("Trying to push direct message to #{user_topic}\n\n") + push_to_socket(State.get_sockets(), user_topic, item) + end) + end + + defp do_stream(%{topic: "participation", item: participation}) do + user_topic = "direct:#{participation.user_id}" + Logger.debug("Trying to push a conversation participation to #{user_topic}\n\n") + + push_to_socket(State.get_sockets(), user_topic, participation) + end + + defp do_stream(%{topic: "list", item: item}) do + # filter the recipient list if the activity is not public, see #270. + recipient_lists = + case Visibility.is_public?(item) do + true -> + Pleroma.List.get_lists_from_activity(item) + + _ -> + Pleroma.List.get_lists_from_activity(item) + |> Enum.filter(fn list -> + owner = User.get_cached_by_id(list.user_id) + + Visibility.visible_for_user?(item, owner) + end) + end + + recipient_topics = + recipient_lists + |> Enum.map(fn %{id: id} -> "list:#{id}" end) + + Enum.each(recipient_topics, fn list_topic -> + Logger.debug("Trying to push message to #{list_topic}\n\n") + push_to_socket(State.get_sockets(), list_topic, item) + end) + end + + defp do_stream(%{topic: topic, item: %Notification{} = item}) + when topic in ["user", "user:notification"] do + State.get_sockets() + |> Map.get("#{topic}:#{item.user_id}", []) + |> Enum.each(fn %StreamerSocket{transport_pid: transport_pid, user: socket_user} -> + with %User{} = user <- User.get_cached_by_ap_id(socket_user.ap_id), + true <- should_send?(user, item) do + send(transport_pid, {:text, StreamerView.render("notification.json", socket_user, item)}) + end + end) + end + + defp do_stream(%{topic: "user", item: item}) do + Logger.debug("Trying to push to users") + + recipient_topics = + User.get_recipients_from_activity(item) + |> Enum.map(fn %{id: id} -> "user:#{id}" end) + + Enum.each(recipient_topics, fn topic -> + push_to_socket(State.get_sockets(), topic, item) + end) + end + + defp do_stream(%{topic: topic, item: item}) do + Logger.debug("Trying to push to #{topic}") + Logger.debug("Pushing item to #{topic}") + push_to_socket(State.get_sockets(), topic, item) + end + + defp should_send?(%User{} = user, %Activity{} = item) do + blocks = user.info.blocks || [] + mutes = user.info.mutes || [] + reblog_mutes = user.info.muted_reblogs || [] + domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(user.info.domain_blocks) + + with parent when not is_nil(parent) <- Object.normalize(item), + true <- Enum.all?([blocks, mutes, reblog_mutes], &(item.actor not in &1)), + true <- Enum.all?([blocks, mutes], &(parent.data["actor"] not in &1)), + %{host: item_host} <- URI.parse(item.actor), + %{host: parent_host} <- URI.parse(parent.data["actor"]), + false <- Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, item_host), + false <- Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, parent_host), + true <- thread_containment(item, user), + false <- CommonAPI.thread_muted?(user, item) do + true + else + _ -> false + end + end + + defp should_send?(%User{} = user, %Notification{activity: activity}) do + should_send?(user, activity) + end + + def push_to_socket(topics, topic, %Activity{data: %{"type" => "Announce"}} = item) do + Enum.each(topics[topic] || [], fn %StreamerSocket{ + transport_pid: transport_pid, + user: socket_user + } -> + # Get the current user so we have up-to-date blocks etc. + if socket_user do + user = User.get_cached_by_ap_id(socket_user.ap_id) + + if should_send?(user, item) do + send(transport_pid, {:text, StreamerView.render("update.json", item, user)}) + end + else + send(transport_pid, {:text, StreamerView.render("update.json", item)}) + end + end) + end + + def push_to_socket(topics, topic, %Participation{} = participation) do + Enum.each(topics[topic] || [], fn %StreamerSocket{transport_pid: transport_pid} -> + send(transport_pid, {:text, StreamerView.render("conversation.json", participation)}) + end) + end + + def push_to_socket(topics, topic, %Activity{ + data: %{"type" => "Delete", "deleted_activity_id" => deleted_activity_id} + }) do + Enum.each(topics[topic] || [], fn %StreamerSocket{transport_pid: transport_pid} -> + send( + transport_pid, + {:text, %{event: "delete", payload: to_string(deleted_activity_id)} |> Jason.encode!()} + ) + end) + end + + def push_to_socket(_topics, _topic, %Activity{data: %{"type" => "Delete"}}), do: :noop + + def push_to_socket(topics, topic, item) do + Enum.each(topics[topic] || [], fn %StreamerSocket{ + transport_pid: transport_pid, + user: socket_user + } -> + # Get the current user so we have up-to-date blocks etc. + if socket_user do + user = User.get_cached_by_ap_id(socket_user.ap_id) + blocks = user.info.blocks || [] + mutes = user.info.mutes || [] + + with true <- Enum.all?([blocks, mutes], &(item.actor not in &1)), + true <- thread_containment(item, user) do + send(transport_pid, {:text, StreamerView.render("update.json", item, user)}) + end + else + send(transport_pid, {:text, StreamerView.render("update.json", item)}) + end + end) + end + + @spec thread_containment(Activity.t(), User.t()) :: boolean() + defp thread_containment(_activity, %User{info: %{skip_thread_containment: true}}), do: true + + defp thread_containment(activity, user) do + if Config.get([:instance, :skip_thread_containment]) do + true + else + ActivityPub.contain_activity(activity, user) + end + end +end diff --git a/lib/pleroma/web/views/streamer_view.ex b/lib/pleroma/web/views/streamer_view.ex new file mode 100644 index 000000000..b13030fa0 --- /dev/null +++ b/lib/pleroma/web/views/streamer_view.ex @@ -0,0 +1,66 @@ +# Pleroma: A lightweight social networking server +# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/> +# SPDX-License-Identifier: AGPL-3.0-only + +defmodule Pleroma.Web.StreamerView do + use Pleroma.Web, :view + + alias Pleroma.Activity + alias Pleroma.Conversation.Participation + alias Pleroma.Notification + alias Pleroma.User + alias Pleroma.Web.MastodonAPI.NotificationView + + def render("update.json", %Activity{} = activity, %User{} = user) do + %{ + event: "update", + payload: + Pleroma.Web.MastodonAPI.StatusView.render( + "status.json", + activity: activity, + for: user + ) + |> Jason.encode!() + } + |> Jason.encode!() + end + + def render("notification.json", %User{} = user, %Notification{} = notify) do + %{ + event: "notification", + payload: + NotificationView.render( + "show.json", + %{notification: notify, for: user} + ) + |> Jason.encode!() + } + |> Jason.encode!() + end + + def render("update.json", %Activity{} = activity) do + %{ + event: "update", + payload: + Pleroma.Web.MastodonAPI.StatusView.render( + "status.json", + activity: activity + ) + |> Jason.encode!() + } + |> Jason.encode!() + end + + def render("conversation.json", %Participation{} = participation) do + %{ + event: "conversation", + payload: + Pleroma.Web.MastodonAPI.ConversationView.render("participation.json", %{ + participation: participation, + for: participation.user + }) + |> Jason.encode!() + } + |> Jason.encode!() + end +end diff --git a/lib/pleroma/workers/web_pusher_worker.ex b/lib/pleroma/workers/web_pusher_worker.ex index bea2baffb..61b451e3e 100644 --- a/lib/pleroma/workers/web_pusher_worker.ex +++ b/lib/pleroma/workers/web_pusher_worker.ex @@ -10,7 +10,11 @@ defmodule Pleroma.Workers.WebPusherWorker do @impl Oban.Worker def perform(%{"op" => "web_push", "notification_id" => notification_id}, _job) do - notification = Repo.get(Notification, notification_id) + notification = + Notification + |> Repo.get(notification_id) + |> Repo.preload([:activity]) + Pleroma.Web.Push.Impl.perform(notification) end end @@ -5,7 +5,7 @@ defmodule Pleroma.Mixfile do [ app: :pleroma, version: version("1.0.0"), - elixir: "~> 1.7", + elixir: "~> 1.8", elixirc_paths: elixirc_paths(Mix.env()), compilers: [:phoenix, :gettext] ++ Mix.compilers(), elixirc_options: [warnings_as_errors: true], @@ -99,9 +99,9 @@ defmodule Pleroma.Mixfile do {:plug_cowboy, "~> 2.0"}, {:phoenix_pubsub, "~> 1.1"}, {:phoenix_ecto, "~> 4.0"}, - {:ecto_sql, "~> 3.1"}, + {:ecto_sql, "~> 3.2"}, {:postgrex, ">= 0.13.5"}, - {:oban, "~> 0.7"}, + {:oban, "~> 0.8.1"}, {:quantum, "~> 2.3"}, {:gettext, "~> 0.15"}, {:comeonin, "~> 4.1.1"}, @@ -113,7 +113,7 @@ defmodule Pleroma.Mixfile do {:calendar, "~> 0.17.4"}, {:cachex, "~> 3.0.2"}, {:poison, "~> 3.0", override: true}, - {:tesla, "~> 1.2"}, + {:tesla, "~> 1.3", override: true}, {:jason, "~> 1.0"}, {:mogrify, "~> 0.6.1"}, {:ex_aws, "~> 2.1"}, @@ -133,7 +133,7 @@ defmodule Pleroma.Mixfile do {:phoenix_swoosh, "~> 0.2"}, {:gen_smtp, "~> 0.13"}, {:websocket_client, git: "https://github.com/jeremyong/websocket_client.git", only: :test}, - {:floki, "~> 0.20.0"}, + {:floki, "~> 0.23.0"}, {:ex_syslogger, github: "slashmili/ex_syslogger", tag: "1.4.0"}, {:timex, "~> 3.5"}, {:ueberauth, "~> 0.4"}, @@ -144,6 +144,7 @@ defmodule Pleroma.Mixfile do git: "https://git.pleroma.social/pleroma/http_signatures.git", ref: "293d77bb6f4a67ac8bde1428735c3b42f22cbb30"}, {:telemetry, "~> 0.3"}, + {:poolboy, "~> 1.5"}, {:prometheus_ex, "~> 3.0"}, {:prometheus_plugs, "~> 1.1"}, {:prometheus_phoenix, "~> 1.3"}, @@ -173,7 +174,8 @@ defmodule Pleroma.Mixfile do "ecto.rollback": ["pleroma.ecto.rollback"], "ecto.setup": ["ecto.create", "ecto.migrate", "run priv/repo/seeds.exs"], "ecto.reset": ["ecto.drop", "ecto.setup"], - test: ["ecto.create --quiet", "ecto.migrate", "test"] + test: ["ecto.create --quiet", "ecto.migrate", "test"], + docs: ["pleroma.docs", "docs"] ] end @@ -21,8 +21,8 @@ "decimal": {:hex, :decimal, "1.8.0", "ca462e0d885f09a1c5a342dbd7c1dcf27ea63548c65a65e67334f4b61803822e", [:mix], [], "hexpm"}, "deep_merge": {:hex, :deep_merge, "1.0.0", "b4aa1a0d1acac393bdf38b2291af38cb1d4a52806cf7a4906f718e1feb5ee961", [:mix], [], "hexpm"}, "earmark": {:hex, :earmark, "1.3.6", "ce1d0675e10a5bb46b007549362bd3f5f08908843957687d8484fe7f37466b19", [:mix], [], "hexpm"}, - "ecto": {:hex, :ecto, "3.1.4", "69d852da7a9f04ede725855a35ede48d158ca11a404fe94f8b2fb3b2162cd3c9", [:mix], [{:decimal, "~> 1.6", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm"}, - "ecto_sql": {:hex, :ecto_sql, "3.1.3", "2c536139190492d9de33c5fefac7323c5eaaa82e1b9bf93482a14649042f7cd9", [:mix], [{:db_connection, "~> 2.0", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.1.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:mariaex, "~> 0.9.1", [hex: :mariaex, repo: "hexpm", optional: true]}, {:myxql, "~> 0.2.0", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.14.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm"}, + "ecto": {:hex, :ecto, "3.2.0", "940e2598813f205223d60c78d66e514afe1db5167ed8075510a59e496619cfb5", [:mix], [{:decimal, "~> 1.6", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm"}, + "ecto_sql": {:hex, :ecto_sql, "3.2.0", "751cea597e8deb616084894dd75cbabfdbe7255ff01e8c058ca13f0353a3921b", [:mix], [{:db_connection, "~> 2.1", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.2.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:myxql, "~> 0.2.0", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.15.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm"}, "esshd": {:hex, :esshd, "0.1.0", "6f93a2062adb43637edad0ea7357db2702a4b80dd9683482fe00f5134e97f4c1", [:mix], [], "hexpm"}, "eternal": {:hex, :eternal, "1.2.0", "e2a6b6ce3b8c248f7dc31451aefca57e3bdf0e48d73ae5043229380a67614c41", [:mix], [], "hexpm"}, "ex2ms": {:hex, :ex2ms, "1.5.0", "19e27f9212be9a96093fed8cdfbef0a2b56c21237196d26760f11dfcfae58e97", [:mix], [], "hexpm"}, @@ -34,7 +34,7 @@ "ex_rated": {:hex, :ex_rated, "1.3.3", "30ecbdabe91f7eaa9d37fa4e81c85ba420f371babeb9d1910adbcd79ec798d27", [:mix], [{:ex2ms, "~> 1.5", [hex: :ex2ms, repo: "hexpm", optional: false]}], "hexpm"}, "ex_syslogger": {:git, "https://github.com/slashmili/ex_syslogger.git", "f3963399047af17e038897c69e20d552e6899e1d", [tag: "1.4.0"]}, "excoveralls": {:hex, :excoveralls, "0.11.1", "dd677fbdd49114fdbdbf445540ec735808250d56b011077798316505064edb2c", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"}, - "floki": {:hex, :floki, "0.20.4", "be42ac911fece24b4c72f3b5846774b6e61b83fe685c2fc9d62093277fb3bc86", [:mix], [{:html_entities, "~> 0.4.0", [hex: :html_entities, repo: "hexpm", optional: false]}, {:mochiweb, "~> 2.15", [hex: :mochiweb, repo: "hexpm", optional: false]}], "hexpm"}, + "floki": {:hex, :floki, "0.23.0", "956ab6dba828c96e732454809fb0bd8d43ce0979b75f34de6322e73d4c917829", [:mix], [{:html_entities, "~> 0.4.0", [hex: :html_entities, repo: "hexpm", optional: false]}], "hexpm"}, "gen_smtp": {:hex, :gen_smtp, "0.14.0", "39846a03522456077c6429b4badfd1d55e5e7d0fdfb65e935b7c5e38549d9202", [:rebar3], [], "hexpm"}, "gen_stage": {:hex, :gen_stage, "0.14.2", "6a2a578a510c5bfca8a45e6b27552f613b41cf584b58210f017088d3d17d0b14", [:mix], [], "hexpm"}, "gen_state_machine": {:hex, :gen_state_machine, "2.0.5", "9ac15ec6e66acac994cc442dcc2c6f9796cf380ec4b08267223014be1c728a95", [:mix], [], "hexpm"}, @@ -60,7 +60,7 @@ "mogrify": {:hex, :mogrify, "0.6.1", "de1b527514f2d95a7bbe9642eb556061afb337e220cf97adbf3a4e6438ed70af", [:mix], [], "hexpm"}, "mox": {:hex, :mox, "0.5.1", "f86bb36026aac1e6f924a4b6d024b05e9adbed5c63e8daa069bd66fb3292165b", [:mix], [], "hexpm"}, "nimble_parsec": {:hex, :nimble_parsec, "0.5.1", "c90796ecee0289dbb5ad16d3ad06f957b0cd1199769641c961cfe0b97db190e0", [:mix], [], "hexpm"}, - "oban": {:hex, :oban, "0.7.1", "171bdd1b69c1a4a839f8c768f5e962fc22d1de1513d459fb6b8e0cbd34817a9a", [:mix], [{:ecto_sql, "~> 3.1", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: false]}, {:postgrex, "~> 0.14", [hex: :postgrex, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm"}, + "oban": {:hex, :oban, "0.8.1", "4bbf62eb1829f856d69aeb5069ac7036afe07db8221a17de2a9169cc7a58a318", [:mix], [{:ecto_sql, "~> 3.1", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: false]}, {:postgrex, "~> 0.14", [hex: :postgrex, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm"}, "parse_trans": {:hex, :parse_trans, "3.3.0", "09765507a3c7590a784615cfd421d101aec25098d50b89d7aa1d66646bc571c1", [:rebar3], [], "hexpm"}, "pbkdf2_elixir": {:hex, :pbkdf2_elixir, "0.12.3", "6706a148809a29c306062862c803406e88f048277f6e85b68faf73291e820b84", [:mix], [], "hexpm"}, "phoenix": {:hex, :phoenix, "1.4.9", "746d098e10741c334d88143d3c94cab1756435f94387a63441792e66ec0ee974", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:phoenix_pubsub, "~> 1.1", [hex: :phoenix_pubsub, repo: "hexpm", optional: false]}, {:plug, "~> 1.8.1 or ~> 1.9", [hex: :plug, repo: "hexpm", optional: false]}, {:plug_cowboy, "~> 1.0 or ~> 2.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm"}, @@ -73,7 +73,8 @@ "plug_crypto": {:hex, :plug_crypto, "1.0.0", "18e49317d3fa343f24620ed22795ec29d4a5e602d52d1513ccea0b07d8ea7d4d", [:mix], [], "hexpm"}, "plug_static_index_html": {:hex, :plug_static_index_html, "1.0.0", "840123d4d3975585133485ea86af73cb2600afd7f2a976f9f5fd8b3808e636a0", [:mix], [{:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"}, "poison": {:hex, :poison, "3.1.0", "d9eb636610e096f86f25d9a46f35a9facac35609a7591b3be3326e99a0484665", [:mix], [], "hexpm"}, - "postgrex": {:hex, :postgrex, "0.14.3", "5754dee2fdf6e9e508cbf49ab138df964278700b764177e8f3871e658b345a1e", [:mix], [{:connection, "~> 1.0", [hex: :connection, repo: "hexpm", optional: false]}, {:db_connection, "~> 2.0", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, "~> 1.5", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm"}, + "poolboy": {:hex, :poolboy, "1.5.2", "392b007a1693a64540cead79830443abf5762f5d30cf50bc95cb2c1aaafa006b", [:rebar3], [], "hexpm"}, + "postgrex": {:hex, :postgrex, "0.15.1", "23ce3417de70f4c0e9e7419ad85bdabcc6860a6925fe2c6f3b1b5b1e8e47bf2f", [:mix], [{:connection, "~> 1.0", [hex: :connection, repo: "hexpm", optional: false]}, {:db_connection, "~> 2.1", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, "~> 1.5", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm"}, "prometheus": {:hex, :prometheus, "4.4.1", "1e96073b3ed7788053768fea779cbc896ddc3bdd9ba60687f2ad50b252ac87d6", [:mix, :rebar3], [], "hexpm"}, "prometheus_ecto": {:hex, :prometheus_ecto, "1.4.1", "6c768ea9654de871e5b32fab2eac348467b3021604ebebbcbd8bcbe806a65ed5", [:mix], [{:ecto, "~> 2.0 or ~> 3.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:prometheus_ex, "~> 1.1 or ~> 2.0 or ~> 3.0", [hex: :prometheus_ex, repo: "hexpm", optional: false]}], "hexpm"}, "prometheus_ex": {:hex, :prometheus_ex, "3.0.5", "fa58cfd983487fc5ead331e9a3e0aa622c67232b3ec71710ced122c4c453a02f", [:mix], [{:prometheus, "~> 4.0", [hex: :prometheus, repo: "hexpm", optional: false]}], "hexpm"}, @@ -89,7 +90,7 @@ "swoosh": {:hex, :swoosh, "0.23.2", "7dda95ff0bf54a2298328d6899c74dae1223777b43563ccebebb4b5d2b61df38", [:mix], [{:cowboy, "~> 1.0.1 or ~> 1.1 or ~> 2.4", [hex: :cowboy, repo: "hexpm", optional: true]}, {:gen_smtp, "~> 0.13", [hex: :gen_smtp, repo: "hexpm", optional: true]}, {:hackney, "~> 1.9", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mail, "~> 0.2", [hex: :mail, repo: "hexpm", optional: true]}, {:mime, "~> 1.1", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_cowboy, ">= 1.0.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}], "hexpm"}, "syslog": {:git, "https://github.com/Vagabond/erlang-syslog.git", "4a6c6f2c996483e86c1320e9553f91d337bcb6aa", [tag: "1.0.5"]}, "telemetry": {:hex, :telemetry, "0.4.0", "8339bee3fa8b91cb84d14c2935f8ecf399ccd87301ad6da6b71c09553834b2ab", [:rebar3], [], "hexpm"}, - "tesla": {:hex, :tesla, "1.2.1", "864783cc27f71dd8c8969163704752476cec0f3a51eb3b06393b3971dc9733ff", [:mix], [{:exjsx, ">= 3.0.0", [hex: :exjsx, repo: "hexpm", optional: true]}, {:fuse, "~> 2.4", [hex: :fuse, repo: "hexpm", optional: true]}, {:hackney, "~> 1.6", [hex: :hackney, repo: "hexpm", optional: true]}, {:ibrowse, "~> 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: true]}, {:jason, ">= 1.0.0", [hex: :jason, repo: "hexpm", optional: true]}, {:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}, {:poison, ">= 1.0.0", [hex: :poison, repo: "hexpm", optional: true]}], "hexpm"}, + "tesla": {:hex, :tesla, "1.3.0", "f35d72f029e608f9cdc6f6d6fcc7c66cf6d6512a70cfef9206b21b8bd0203a30", [:mix], [{:castore, "~> 0.1", [hex: :castore, repo: "hexpm", optional: true]}, {:exjsx, ">= 3.0.0", [hex: :exjsx, repo: "hexpm", optional: true]}, {:fuse, "~> 2.4", [hex: :fuse, repo: "hexpm", optional: true]}, {:gun, "~> 1.3", [hex: :gun, repo: "hexpm", optional: true]}, {:hackney, "~> 1.6", [hex: :hackney, repo: "hexpm", optional: true]}, {:ibrowse, "~> 4.4.0", [hex: :ibrowse, repo: "hexpm", optional: true]}, {:jason, ">= 1.0.0", [hex: :jason, repo: "hexpm", optional: true]}, {:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 0.4", [hex: :mint, repo: "hexpm", optional: true]}, {:poison, ">= 1.0.0", [hex: :poison, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.3", [hex: :telemetry, repo: "hexpm", optional: true]}], "hexpm"}, "timex": {:hex, :timex, "3.6.1", "efdf56d0e67a6b956cc57774353b0329c8ab7726766a11547e529357ffdc1d56", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.10", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 0.1.8 or ~> 0.5 or ~> 1.0.0", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"}, "trailing_format_plug": {:hex, :trailing_format_plug, "0.0.7", "64b877f912cf7273bed03379936df39894149e35137ac9509117e59866e10e45", [:mix], [{:plug, "> 0.12.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"}, "tzdata": {:hex, :tzdata, "0.5.21", "8cbf3607fcce69636c672d5be2bbb08687fe26639a62bdcc283d267277db7cf0", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"}, diff --git a/priv/repo/migrations/20190912065617_create_deliveries.exs b/priv/repo/migrations/20190912065617_create_deliveries.exs new file mode 100644 index 000000000..79071a799 --- /dev/null +++ b/priv/repo/migrations/20190912065617_create_deliveries.exs @@ -0,0 +1,12 @@ +defmodule Pleroma.Repo.Migrations.CreateDeliveries do + use Ecto.Migration + + def change do + create_if_not_exists table(:deliveries) do + add(:object_id, references(:objects, type: :id), null: false) + add(:user_id, references(:users, type: :uuid, on_delete: :delete_all), null: false) + end + create_if_not_exists index(:deliveries, :object_id, name: :deliveries_object_id) + create_if_not_exists(unique_index(:deliveries, [:user_id, :object_id])) + end +end diff --git a/priv/repo/migrations/20190917100019_update_oban.exs b/priv/repo/migrations/20190917100019_update_oban.exs new file mode 100644 index 000000000..157dc54f9 --- /dev/null +++ b/priv/repo/migrations/20190917100019_update_oban.exs @@ -0,0 +1,11 @@ +defmodule Pleroma.Repo.Migrations.UpdateOban do + use Ecto.Migration + + def up do + Oban.Migrations.up(version: 4) + end + + def down do + Oban.Migrations.down(version: 2) + end +end diff --git a/test/activity/ir/topics_test.exs b/test/activity/ir/topics_test.exs new file mode 100644 index 000000000..e75f83586 --- /dev/null +++ b/test/activity/ir/topics_test.exs @@ -0,0 +1,141 @@ +defmodule Pleroma.Activity.Ir.TopicsTest do + use Pleroma.DataCase + + alias Pleroma.Activity + alias Pleroma.Activity.Ir.Topics + alias Pleroma.Object + + require Pleroma.Constants + + describe "poll answer" do + test "produce no topics" do + activity = %Activity{object: %Object{data: %{"type" => "Answer"}}} + + assert [] == Topics.get_activity_topics(activity) + end + end + + describe "non poll answer" do + test "always add user and list topics" do + activity = %Activity{object: %Object{data: %{"type" => "FooBar"}}} + topics = Topics.get_activity_topics(activity) + + assert Enum.member?(topics, "user") + assert Enum.member?(topics, "list") + end + end + + describe "public visibility" do + setup do + activity = %Activity{ + object: %Object{data: %{"type" => "Note"}}, + data: %{"to" => [Pleroma.Constants.as_public()]} + } + + {:ok, activity: activity} + end + + test "produces public topic", %{activity: activity} do + topics = Topics.get_activity_topics(activity) + + assert Enum.member?(topics, "public") + end + + test "local action produces public:local topic", %{activity: activity} do + activity = %{activity | local: true} + topics = Topics.get_activity_topics(activity) + + assert Enum.member?(topics, "public:local") + end + + test "non-local action does not produce public:local topic", %{activity: activity} do + activity = %{activity | local: false} + topics = Topics.get_activity_topics(activity) + + refute Enum.member?(topics, "public:local") + end + end + + describe "public visibility create events" do + setup do + activity = %Activity{ + object: %Object{data: %{"type" => "Create", "attachment" => []}}, + data: %{"to" => [Pleroma.Constants.as_public()]} + } + + {:ok, activity: activity} + end + + test "with no attachments doesn't produce public:media topics", %{activity: activity} do + topics = Topics.get_activity_topics(activity) + + refute Enum.member?(topics, "public:media") + refute Enum.member?(topics, "public:local:media") + end + + test "converts tags to hash tags", %{activity: %{object: %{data: data} = object} = activity} do + tagged_data = Map.put(data, "tag", ["foo", "bar"]) + activity = %{activity | object: %{object | data: tagged_data}} + + topics = Topics.get_activity_topics(activity) + + assert Enum.member?(topics, "hashtag:foo") + assert Enum.member?(topics, "hashtag:bar") + end + + test "only converts strinngs to hash tags", %{ + activity: %{object: %{data: data} = object} = activity + } do + tagged_data = Map.put(data, "tag", [2]) + activity = %{activity | object: %{object | data: tagged_data}} + + topics = Topics.get_activity_topics(activity) + + refute Enum.member?(topics, "hashtag:2") + end + end + + describe "public visibility create events with attachments" do + setup do + activity = %Activity{ + object: %Object{data: %{"type" => "Create", "attachment" => ["foo"]}}, + data: %{"to" => [Pleroma.Constants.as_public()]} + } + + {:ok, activity: activity} + end + + test "produce public:media topics", %{activity: activity} do + topics = Topics.get_activity_topics(activity) + + assert Enum.member?(topics, "public:media") + end + + test "local produces public:local:media topics", %{activity: activity} do + topics = Topics.get_activity_topics(activity) + + assert Enum.member?(topics, "public:local:media") + end + + test "non-local doesn't produce public:local:media topics", %{activity: activity} do + activity = %{activity | local: false} + + topics = Topics.get_activity_topics(activity) + + refute Enum.member?(topics, "public:local:media") + end + end + + describe "non-public visibility" do + test "produces direct topic" do + activity = %Activity{object: %Object{data: %{"type" => "Note"}}, data: %{"to" => []}} + topics = Topics.get_activity_topics(activity) + + assert Enum.member?(topics, "direct") + refute Enum.member?(topics, "public") + refute Enum.member?(topics, "public:local") + refute Enum.member?(topics, "public:media") + refute Enum.member?(topics, "public:local:media") + end + end +end diff --git a/test/fixtures/tesla_mock/poll_modified.json b/test/fixtures/tesla_mock/poll_modified.json new file mode 100644 index 000000000..1d026b592 --- /dev/null +++ b/test/fixtures/tesla_mock/poll_modified.json @@ -0,0 +1 @@ +{"@context":["https://www.w3.org/ns/activitystreams","https://patch.cx/schemas/litepub-0.1.jsonld",{"@language":"und"}],"actor":"https://patch.cx/users/rin","attachment":[],"attributedTo":"https://patch.cx/users/rin","cc":["https://patch.cx/users/rin/followers"],"closed":"2019-09-19T00:32:36.785333","content":"can you vote on this poll?","context":"https://patch.cx/contexts/626ecafd-3377-46c4-b908-3721a4d4373c","conversation":"https://patch.cx/contexts/626ecafd-3377-46c4-b908-3721a4d4373c","id":"https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d","oneOf":[{"name":"yes","replies":{"totalItems":8,"type":"Collection"},"type":"Note"},{"name":"no","replies":{"totalItems":3,"type":"Collection"},"type":"Note"}],"published":"2019-09-18T14:32:36.802152Z","sensitive":false,"summary":"","tag":[],"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Question"}
\ No newline at end of file diff --git a/test/fixtures/tesla_mock/poll_original.json b/test/fixtures/tesla_mock/poll_original.json new file mode 100644 index 000000000..267876b3c --- /dev/null +++ b/test/fixtures/tesla_mock/poll_original.json @@ -0,0 +1 @@ +{"@context":["https://www.w3.org/ns/activitystreams","https://patch.cx/schemas/litepub-0.1.jsonld",{"@language":"und"}],"actor":"https://patch.cx/users/rin","attachment":[],"attributedTo":"https://patch.cx/users/rin","cc":["https://patch.cx/users/rin/followers"],"closed":"2019-09-19T00:32:36.785333","content":"can you vote on this poll?","context":"https://patch.cx/contexts/626ecafd-3377-46c4-b908-3721a4d4373c","conversation":"https://patch.cx/contexts/626ecafd-3377-46c4-b908-3721a4d4373c","id":"https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d","oneOf":[{"name":"yes","replies":{"totalItems":4,"type":"Collection"},"type":"Note"},{"name":"no","replies":{"totalItems":0,"type":"Collection"},"type":"Note"}],"published":"2019-09-18T14:32:36.802152Z","sensitive":false,"summary":"","tag":[],"to":["https://www.w3.org/ns/activitystreams#Public"],"type":"Question"}
\ No newline at end of file diff --git a/test/fixtures/tesla_mock/rin.json b/test/fixtures/tesla_mock/rin.json new file mode 100644 index 000000000..2cf623764 --- /dev/null +++ b/test/fixtures/tesla_mock/rin.json @@ -0,0 +1 @@ +{"@context":["https://www.w3.org/ns/activitystreams","https://patch.cx/schemas/litepub-0.1.jsonld",{"@language":"und"}],"attachment":[],"endpoints":{"oauthAuthorizationEndpoint":"https://patch.cx/oauth/authorize","oauthRegistrationEndpoint":"https://patch.cx/api/v1/apps","oauthTokenEndpoint":"https://patch.cx/oauth/token","sharedInbox":"https://patch.cx/inbox"},"followers":"https://patch.cx/users/rin/followers","following":"https://patch.cx/users/rin/following","icon":{"type":"Image","url":"https://patch.cx/media/4e914f5b84e4a259a3f6c2d2edc9ab642f2ab05f3e3d9c52c81fc2d984b3d51e.jpg"},"id":"https://patch.cx/users/rin","image":{"type":"Image","url":"https://patch.cx/media/f739efddefeee49c6e67e947c4811fdc911785c16ae43da4c3684051fbf8da6a.jpg?name=f739efddefeee49c6e67e947c4811fdc911785c16ae43da4c3684051fbf8da6a.jpg"},"inbox":"https://patch.cx/users/rin/inbox","manuallyApprovesFollowers":false,"name":"rinpatch","outbox":"https://patch.cx/users/rin/outbox","preferredUsername":"rin","publicKey":{"id":"https://patch.cx/users/rin#main-key","owner":"https://patch.cx/users/rin","publicKeyPem":"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5DLtwGXNZElJyxFGfcVc\nXANhaMadj/iYYQwZjOJTV9QsbtiNBeIK54PJrYuU0/0YIdrvS1iqheX5IwXRhcwa\nhm3ZyLz7XeN9st7FBni4BmZMBtMpxAuYuu5p/jbWy13qAiYOhPreCx0wrWgm/lBD\n9mkgaxIxPooBE0S4ZWEJIDIV1Vft3AWcRUyWW1vIBK0uZzs6GYshbQZB952S0yo4\nFzI1hABGHncH8UvuFauh4EZ8tY7/X5I0pGRnDOcRN1dAht5w5yTA+6r5kebiFQjP\nIzN/eCO/a9Flrj9YGW7HDNtjSOH0A31PLRGlJtJO3yK57dnf5ppyCZGfL4emShQo\ncQIDAQAB\n-----END PUBLIC KEY-----\n\n"},"summary":"your friendly neighborhood pleroma developer<br>I like cute things and distributed systems, and really hate delete and redrafts","tag":[],"type":"Person","url":"https://patch.cx/users/rin"}
\ No newline at end of file diff --git a/test/integration/mastodon_websocket_test.exs b/test/integration/mastodon_websocket_test.exs index 63bf73412..d02a3cc4d 100644 --- a/test/integration/mastodon_websocket_test.exs +++ b/test/integration/mastodon_websocket_test.exs @@ -11,7 +11,6 @@ defmodule Pleroma.Integration.MastodonWebsocketTest do alias Pleroma.Integration.WebsocketClient alias Pleroma.Web.CommonAPI alias Pleroma.Web.OAuth - alias Pleroma.Web.Streamer @path Pleroma.Web.Endpoint.url() |> URI.parse() @@ -19,14 +18,9 @@ defmodule Pleroma.Integration.MastodonWebsocketTest do |> Map.put(:path, "/api/v1/streaming") |> URI.to_string() - setup do - GenServer.start(Streamer, %{}, name: Streamer) - - on_exit(fn -> - if pid = Process.whereis(Streamer) do - Process.exit(pid, :kill) - end - end) + setup_all do + start_supervised(Pleroma.Web.Streamer.supervisor()) + :ok end def start_socket(qs \\ nil, headers \\ []) do @@ -43,6 +37,7 @@ defmodule Pleroma.Integration.MastodonWebsocketTest do capture_log(fn -> assert {:error, {400, _}} = start_socket() assert {:error, {404, _}} = start_socket("?stream=ncjdk") + Process.sleep(30) end) end @@ -50,6 +45,7 @@ defmodule Pleroma.Integration.MastodonWebsocketTest do capture_log(fn -> assert {:error, {403, _}} = start_socket("?stream=user&access_token=aaaaaaaaaaaa") assert {:error, {403, _}} = start_socket("?stream=user") + Process.sleep(30) end) end @@ -108,6 +104,7 @@ defmodule Pleroma.Integration.MastodonWebsocketTest do assert capture_log(fn -> assert {:error, {403, "Forbidden"}} = start_socket("?stream=user") + Process.sleep(30) end) =~ ":badarg" end @@ -116,6 +113,7 @@ defmodule Pleroma.Integration.MastodonWebsocketTest do assert capture_log(fn -> assert {:error, {403, "Forbidden"}} = start_socket("?stream=user:notification") + Process.sleep(30) end) =~ ":badarg" end @@ -125,6 +123,8 @@ defmodule Pleroma.Integration.MastodonWebsocketTest do assert capture_log(fn -> assert {:error, {403, "Forbidden"}} = start_socket("?stream=user", [{"Sec-WebSocket-Protocol", "I am a friend"}]) + + Process.sleep(30) end) =~ ":badarg" end end diff --git a/test/notification_test.exs b/test/notification_test.exs index 3be9db09b..3d2f9a8fc 100644 --- a/test/notification_test.exs +++ b/test/notification_test.exs @@ -69,16 +69,7 @@ defmodule Pleroma.NotificationTest do end describe "create_notification" do - setup do - GenServer.start(Streamer, %{}, name: Streamer) - - on_exit(fn -> - if pid = Process.whereis(Streamer) do - Process.exit(pid, :kill) - end - end) - end - + @tag needs_streamer: true test "it creates a notification for user and send to the 'user' and the 'user:notification' stream" do user = insert(:user) task = Task.async(fn -> assert_receive {:text, _}, 4_000 end) diff --git a/test/object_test.exs b/test/object_test.exs index ba96aeea4..3d64fdb49 100644 --- a/test/object_test.exs +++ b/test/object_test.exs @@ -4,10 +4,13 @@ defmodule Pleroma.ObjectTest do use Pleroma.DataCase + import ExUnit.CaptureLog import Pleroma.Factory import Tesla.Mock + alias Pleroma.Activity alias Pleroma.Object alias Pleroma.Repo + alias Pleroma.Web.CommonAPI setup do mock(fn env -> apply(HttpRequestMock, :request, [env]) end) @@ -89,4 +92,110 @@ defmodule Pleroma.ObjectTest do ) end end + + describe "get_by_id_and_maybe_refetch" do + setup do + mock(fn + %{method: :get, url: "https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d"} -> + %Tesla.Env{status: 200, body: File.read!("test/fixtures/tesla_mock/poll_original.json")} + + env -> + apply(HttpRequestMock, :request, [env]) + end) + + mock_modified = fn resp -> + mock(fn + %{method: :get, url: "https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d"} -> + resp + + env -> + apply(HttpRequestMock, :request, [env]) + end) + end + + on_exit(fn -> mock(fn env -> apply(HttpRequestMock, :request, [env]) end) end) + + [mock_modified: mock_modified] + end + + test "refetches if the time since the last refetch is greater than the interval", %{ + mock_modified: mock_modified + } do + %Object{} = + object = Object.normalize("https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d") + + assert Enum.at(object.data["oneOf"], 0)["replies"]["totalItems"] == 4 + assert Enum.at(object.data["oneOf"], 1)["replies"]["totalItems"] == 0 + + mock_modified.(%Tesla.Env{ + status: 200, + body: File.read!("test/fixtures/tesla_mock/poll_modified.json") + }) + + updated_object = Object.get_by_id_and_maybe_refetch(object.id, interval: -1) + assert Enum.at(updated_object.data["oneOf"], 0)["replies"]["totalItems"] == 8 + assert Enum.at(updated_object.data["oneOf"], 1)["replies"]["totalItems"] == 3 + end + + test "returns the old object if refetch fails", %{mock_modified: mock_modified} do + %Object{} = + object = Object.normalize("https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d") + + assert Enum.at(object.data["oneOf"], 0)["replies"]["totalItems"] == 4 + assert Enum.at(object.data["oneOf"], 1)["replies"]["totalItems"] == 0 + + assert capture_log(fn -> + mock_modified.(%Tesla.Env{status: 404, body: ""}) + + updated_object = Object.get_by_id_and_maybe_refetch(object.id, interval: -1) + assert Enum.at(updated_object.data["oneOf"], 0)["replies"]["totalItems"] == 4 + assert Enum.at(updated_object.data["oneOf"], 1)["replies"]["totalItems"] == 0 + end) =~ + "[error] Couldn't refresh https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d" + end + + test "does not refetch if the time since the last refetch is greater than the interval", %{ + mock_modified: mock_modified + } do + %Object{} = + object = Object.normalize("https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d") + + assert Enum.at(object.data["oneOf"], 0)["replies"]["totalItems"] == 4 + assert Enum.at(object.data["oneOf"], 1)["replies"]["totalItems"] == 0 + + mock_modified.(%Tesla.Env{ + status: 200, + body: File.read!("test/fixtures/tesla_mock/poll_modified.json") + }) + + updated_object = Object.get_by_id_and_maybe_refetch(object.id, interval: 100) + assert Enum.at(updated_object.data["oneOf"], 0)["replies"]["totalItems"] == 4 + assert Enum.at(updated_object.data["oneOf"], 1)["replies"]["totalItems"] == 0 + end + + test "preserves internal fields on refetch", %{mock_modified: mock_modified} do + %Object{} = + object = Object.normalize("https://patch.cx/objects/9a172665-2bc5-452d-8428-2361d4c33b1d") + + assert Enum.at(object.data["oneOf"], 0)["replies"]["totalItems"] == 4 + assert Enum.at(object.data["oneOf"], 1)["replies"]["totalItems"] == 0 + + user = insert(:user) + activity = Activity.get_create_by_object_ap_id(object.data["id"]) + {:ok, _activity, object} = CommonAPI.favorite(activity.id, user) + + assert object.data["like_count"] == 1 + + mock_modified.(%Tesla.Env{ + status: 200, + body: File.read!("test/fixtures/tesla_mock/poll_modified.json") + }) + + updated_object = Object.get_by_id_and_maybe_refetch(object.id, interval: -1) + assert Enum.at(updated_object.data["oneOf"], 0)["replies"]["totalItems"] == 8 + assert Enum.at(updated_object.data["oneOf"], 1)["replies"]["totalItems"] == 3 + + assert updated_object.data["like_count"] == 1 + end + end end diff --git a/test/support/conn_case.ex b/test/support/conn_case.ex index ec5892ff5..b39c70677 100644 --- a/test/support/conn_case.ex +++ b/test/support/conn_case.ex @@ -40,6 +40,10 @@ defmodule Pleroma.Web.ConnCase do Ecto.Adapters.SQL.Sandbox.mode(Pleroma.Repo, {:shared, self()}) end + if tags[:needs_streamer] do + start_supervised(Pleroma.Web.Streamer.supervisor()) + end + {:ok, conn: Phoenix.ConnTest.build_conn()} end end diff --git a/test/support/data_case.ex b/test/support/data_case.ex index f3d98e7e3..17fa15214 100644 --- a/test/support/data_case.ex +++ b/test/support/data_case.ex @@ -39,6 +39,10 @@ defmodule Pleroma.DataCase do Ecto.Adapters.SQL.Sandbox.mode(Pleroma.Repo, {:shared, self()}) end + if tags[:needs_streamer] do + start_supervised(Pleroma.Web.Streamer.supervisor()) + end + :ok end diff --git a/test/support/http_request_mock.ex b/test/support/http_request_mock.ex index 231e7c498..833162a61 100644 --- a/test/support/http_request_mock.ex +++ b/test/support/http_request_mock.ex @@ -1004,6 +1004,10 @@ defmodule HttpRequestMock do {:ok, %Tesla.Env{status: 200, body: File.read!("test/fixtures/tesla_mock/sjw.json")}} end + def get("https://patch.cx/users/rin", _, _, _) do + {:ok, %Tesla.Env{status: 200, body: File.read!("test/fixtures/tesla_mock/rin.json")}} + end + def get(url, query, body, headers) do {:error, "Mock response not implemented for GET #{inspect(url)}, #{query}, #{inspect(body)}, #{ diff --git a/test/test_helper.exs b/test/test_helper.exs index a927b2c3d..6a389365f 100644 --- a/test/test_helper.exs +++ b/test/test_helper.exs @@ -7,3 +7,8 @@ ExUnit.start(exclude: os_exclude) Ecto.Adapters.SQL.Sandbox.mode(Pleroma.Repo, :manual) Mox.defmock(Pleroma.ReverseProxy.ClientMock, for: Pleroma.ReverseProxy.Client) {:ok, _} = Application.ensure_all_started(:ex_machina) + +ExUnit.after_suite(fn _results -> + uploads = Pleroma.Config.get([Pleroma.Uploaders.Local, :uploads], "test/uploads") + File.rm_rf!(uploads) +end) diff --git a/test/web/activity_pub/activity_pub_controller_test.exs b/test/web/activity_pub/activity_pub_controller_test.exs index 9b78fb72d..f83b14452 100644 --- a/test/web/activity_pub/activity_pub_controller_test.exs +++ b/test/web/activity_pub/activity_pub_controller_test.exs @@ -8,6 +8,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubControllerTest do import Pleroma.Factory alias Pleroma.Activity + alias Pleroma.Delivery alias Pleroma.Instances alias Pleroma.Object alias Pleroma.Tests.ObanHelpers @@ -893,4 +894,86 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubControllerTest do assert result["totalItems"] == 15 end end + + describe "delivery tracking" do + test "it tracks a signed object fetch", %{conn: conn} do + user = insert(:user, local: false) + activity = insert(:note_activity) + object = Object.normalize(activity) + + object_path = String.trim_leading(object.data["id"], Pleroma.Web.Endpoint.url()) + + conn + |> put_req_header("accept", "application/activity+json") + |> assign(:user, user) + |> get(object_path) + |> json_response(200) + + assert Delivery.get(object.id, user.id) + end + + test "it tracks a signed activity fetch", %{conn: conn} do + user = insert(:user, local: false) + activity = insert(:note_activity) + object = Object.normalize(activity) + + activity_path = String.trim_leading(activity.data["id"], Pleroma.Web.Endpoint.url()) + + conn + |> put_req_header("accept", "application/activity+json") + |> assign(:user, user) + |> get(activity_path) + |> json_response(200) + + assert Delivery.get(object.id, user.id) + end + + test "it tracks a signed object fetch when the json is cached", %{conn: conn} do + user = insert(:user, local: false) + other_user = insert(:user, local: false) + activity = insert(:note_activity) + object = Object.normalize(activity) + + object_path = String.trim_leading(object.data["id"], Pleroma.Web.Endpoint.url()) + + conn + |> put_req_header("accept", "application/activity+json") + |> assign(:user, user) + |> get(object_path) + |> json_response(200) + + build_conn() + |> put_req_header("accept", "application/activity+json") + |> assign(:user, other_user) + |> get(object_path) + |> json_response(200) + + assert Delivery.get(object.id, user.id) + assert Delivery.get(object.id, other_user.id) + end + + test "it tracks a signed activity fetch when the json is cached", %{conn: conn} do + user = insert(:user, local: false) + other_user = insert(:user, local: false) + activity = insert(:note_activity) + object = Object.normalize(activity) + + activity_path = String.trim_leading(activity.data["id"], Pleroma.Web.Endpoint.url()) + + conn + |> put_req_header("accept", "application/activity+json") + |> assign(:user, user) + |> get(activity_path) + |> json_response(200) + + build_conn() + |> put_req_header("accept", "application/activity+json") + |> assign(:user, other_user) + |> get(activity_path) + |> json_response(200) + + assert Delivery.get(object.id, user.id) + assert Delivery.get(object.id, other_user.id) + end + end end diff --git a/test/web/activity_pub/activity_pub_test.exs b/test/web/activity_pub/activity_pub_test.exs index d0118fefa..4100108a5 100644 --- a/test/web/activity_pub/activity_pub_test.exs +++ b/test/web/activity_pub/activity_pub_test.exs @@ -38,9 +38,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubTest do stream: fn _, _ -> nil end do ActivityPub.stream_out_participations(conversation.participations) - Enum.each(participations, fn participation -> - assert called(Pleroma.Web.Streamer.stream("participation", participation)) - end) + assert called(Pleroma.Web.Streamer.stream("participation", participations)) end end end diff --git a/test/web/activity_pub/publisher_test.exs b/test/web/activity_pub/publisher_test.exs index c7d0dc3a5..df03b4008 100644 --- a/test/web/activity_pub/publisher_test.exs +++ b/test/web/activity_pub/publisher_test.exs @@ -3,7 +3,7 @@ # SPDX-License-Identifier: AGPL-3.0-only defmodule Pleroma.Web.ActivityPub.PublisherTest do - use Pleroma.DataCase + use Pleroma.Web.ConnCase import ExUnit.CaptureLog import Pleroma.Factory @@ -12,7 +12,9 @@ defmodule Pleroma.Web.ActivityPub.PublisherTest do alias Pleroma.Activity alias Pleroma.Instances + alias Pleroma.Object alias Pleroma.Web.ActivityPub.Publisher + alias Pleroma.Web.CommonAPI @as_public "https://www.w3.org/ns/activitystreams#Public" @@ -268,5 +270,69 @@ defmodule Pleroma.Web.ActivityPub.PublisherTest do }) ) end + + test_with_mock "publishes a delete activity to peers who signed fetch requests to the create acitvity/object.", + Pleroma.Web.Federator.Publisher, + [:passthrough], + [] do + fetcher = + insert(:user, + local: false, + info: %{ + ap_enabled: true, + source_data: %{"inbox" => "https://domain.com/users/nick1/inbox"} + } + ) + + another_fetcher = + insert(:user, + local: false, + info: %{ + ap_enabled: true, + source_data: %{"inbox" => "https://domain2.com/users/nick1/inbox"} + } + ) + + actor = insert(:user) + + note_activity = insert(:note_activity, user: actor) + object = Object.normalize(note_activity) + + activity_path = String.trim_leading(note_activity.data["id"], Pleroma.Web.Endpoint.url()) + object_path = String.trim_leading(object.data["id"], Pleroma.Web.Endpoint.url()) + + build_conn() + |> put_req_header("accept", "application/activity+json") + |> assign(:user, fetcher) + |> get(object_path) + |> json_response(200) + + build_conn() + |> put_req_header("accept", "application/activity+json") + |> assign(:user, another_fetcher) + |> get(activity_path) + |> json_response(200) + + {:ok, delete} = CommonAPI.delete(note_activity.id, actor) + + res = Publisher.publish(actor, delete) + assert res == :ok + + assert called( + Pleroma.Web.Federator.Publisher.enqueue_one(Publisher, %{ + inbox: "https://domain.com/users/nick1/inbox", + actor_id: actor.id, + id: delete.data["id"] + }) + ) + + assert called( + Pleroma.Web.Federator.Publisher.enqueue_one(Publisher, %{ + inbox: "https://domain2.com/users/nick1/inbox", + actor_id: actor.id, + id: delete.data["id"] + }) + ) + end end end diff --git a/test/web/activity_pub/utils_test.exs b/test/web/activity_pub/utils_test.exs index eb429b2c4..b1c1d6f71 100644 --- a/test/web/activity_pub/utils_test.exs +++ b/test/web/activity_pub/utils_test.exs @@ -87,6 +87,18 @@ defmodule Pleroma.Web.ActivityPub.UtilsTest do assert Utils.determine_explicit_mentions(object) == [] end + + test "works with an object has tags as map" do + object = %{ + "tag" => %{ + "type" => "Mention", + "href" => "https://example.com/~alyssa", + "name" => "Alyssa P. Hacker" + } + } + + assert Utils.determine_explicit_mentions(object) == ["https://example.com/~alyssa"] + end end describe "make_unlike_data/3" do @@ -300,8 +312,8 @@ defmodule Pleroma.Web.ActivityPub.UtilsTest do {:ok, follow_activity_two} = Utils.update_follow_state_for_all(follow_activity_two, "accept") - assert Repo.get(Activity, follow_activity.id).data["state"] == "accept" - assert Repo.get(Activity, follow_activity_two.id).data["state"] == "accept" + assert refresh_record(follow_activity).data["state"] == "accept" + assert refresh_record(follow_activity_two).data["state"] == "accept" end end @@ -323,8 +335,8 @@ defmodule Pleroma.Web.ActivityPub.UtilsTest do {:ok, follow_activity_two} = Utils.update_follow_state(follow_activity_two, "reject") - assert Repo.get(Activity, follow_activity.id).data["state"] == "pending" - assert Repo.get(Activity, follow_activity_two.id).data["state"] == "reject" + assert refresh_record(follow_activity).data["state"] == "pending" + assert refresh_record(follow_activity_two).data["state"] == "reject" end end @@ -401,4 +413,216 @@ defmodule Pleroma.Web.ActivityPub.UtilsTest do assert ^like_activity = Utils.get_existing_like(user.ap_id, object) end end + + describe "get_get_existing_announce/2" do + test "returns nil if announce not found" do + actor = insert(:user) + refute Utils.get_existing_announce(actor.ap_id, %{data: %{"id" => "test"}}) + end + + test "fetches existing announce" do + note_activity = insert(:note_activity) + assert object = Object.normalize(note_activity) + actor = insert(:user) + + {:ok, announce, _object} = ActivityPub.announce(actor, object) + assert Utils.get_existing_announce(actor.ap_id, object) == announce + end + end + + describe "fetch_latest_block/2" do + test "fetches last block activities" do + user1 = insert(:user) + user2 = insert(:user) + + assert {:ok, %Activity{} = _} = ActivityPub.block(user1, user2) + assert {:ok, %Activity{} = _} = ActivityPub.block(user1, user2) + assert {:ok, %Activity{} = activity} = ActivityPub.block(user1, user2) + + assert Utils.fetch_latest_block(user1, user2) == activity + end + end + + describe "recipient_in_message/3" do + test "returns true when recipient in `to`" do + recipient = insert(:user) + actor = insert(:user) + assert Utils.recipient_in_message(recipient, actor, %{"to" => recipient.ap_id}) + + assert Utils.recipient_in_message( + recipient, + actor, + %{"to" => [recipient.ap_id], "cc" => ""} + ) + end + + test "returns true when recipient in `cc`" do + recipient = insert(:user) + actor = insert(:user) + assert Utils.recipient_in_message(recipient, actor, %{"cc" => recipient.ap_id}) + + assert Utils.recipient_in_message( + recipient, + actor, + %{"cc" => [recipient.ap_id], "to" => ""} + ) + end + + test "returns true when recipient in `bto`" do + recipient = insert(:user) + actor = insert(:user) + assert Utils.recipient_in_message(recipient, actor, %{"bto" => recipient.ap_id}) + + assert Utils.recipient_in_message( + recipient, + actor, + %{"bcc" => "", "bto" => [recipient.ap_id]} + ) + end + + test "returns true when recipient in `bcc`" do + recipient = insert(:user) + actor = insert(:user) + assert Utils.recipient_in_message(recipient, actor, %{"bcc" => recipient.ap_id}) + + assert Utils.recipient_in_message( + recipient, + actor, + %{"bto" => "", "bcc" => [recipient.ap_id]} + ) + end + + test "returns true when message without addresses fields" do + recipient = insert(:user) + actor = insert(:user) + assert Utils.recipient_in_message(recipient, actor, %{"bccc" => recipient.ap_id}) + + assert Utils.recipient_in_message( + recipient, + actor, + %{"btod" => "", "bccc" => [recipient.ap_id]} + ) + end + + test "returns false" do + recipient = insert(:user) + actor = insert(:user) + refute Utils.recipient_in_message(recipient, actor, %{"to" => "ap_id"}) + end + end + + describe "lazy_put_activity_defaults/2" do + test "returns map with id and published data" do + note_activity = insert(:note_activity) + object = Object.normalize(note_activity) + res = Utils.lazy_put_activity_defaults(%{"context" => object.data["id"]}) + assert res["context"] == object.data["id"] + assert res["context_id"] == object.id + assert res["id"] + assert res["published"] + end + + test "returns map with fake id and published data" do + assert %{ + "context" => "pleroma:fakecontext", + "context_id" => -1, + "id" => "pleroma:fakeid", + "published" => _ + } = Utils.lazy_put_activity_defaults(%{}, true) + end + + test "returns activity data with object" do + note_activity = insert(:note_activity) + object = Object.normalize(note_activity) + + res = + Utils.lazy_put_activity_defaults(%{ + "context" => object.data["id"], + "object" => %{} + }) + + assert res["context"] == object.data["id"] + assert res["context_id"] == object.id + assert res["id"] + assert res["published"] + assert res["object"]["id"] + assert res["object"]["published"] + assert res["object"]["context"] == object.data["id"] + assert res["object"]["context_id"] == object.id + end + end + + describe "make_flag_data" do + test "returns empty map when params is invalid" do + assert Utils.make_flag_data(%{}, %{}) == %{} + end + + test "returns map with Flag object" do + reporter = insert(:user) + target_account = insert(:user) + {:ok, activity} = CommonAPI.post(target_account, %{"status" => "foobar"}) + context = Utils.generate_context_id() + content = "foobar" + + target_ap_id = target_account.ap_id + activity_ap_id = activity.data["id"] + + res = + Utils.make_flag_data( + %{ + actor: reporter, + context: context, + account: target_account, + statuses: [%{"id" => activity.data["id"]}], + content: content + }, + %{} + ) + + assert %{ + "type" => "Flag", + "content" => ^content, + "context" => ^context, + "object" => [^target_ap_id, ^activity_ap_id], + "state" => "open" + } = res + end + end + + describe "add_announce_to_object/2" do + test "adds actor to announcement" do + user = insert(:user) + object = insert(:note) + + activity = + insert(:note_activity, + data: %{ + "actor" => user.ap_id, + "cc" => [Pleroma.Constants.as_public()] + } + ) + + assert {:ok, updated_object} = Utils.add_announce_to_object(activity, object) + assert updated_object.data["announcements"] == [user.ap_id] + assert updated_object.data["announcement_count"] == 1 + end + end + + describe "remove_announce_from_object/2" do + test "removes actor from announcements" do + user = insert(:user) + user2 = insert(:user) + + object = + insert(:note, + data: %{"announcements" => [user.ap_id, user2.ap_id], "announcement_count" => 2} + ) + + activity = insert(:note_activity, data: %{"actor" => user.ap_id}) + + assert {:ok, updated_object} = Utils.remove_announce_from_object(activity, object) + assert updated_object.data["announcements"] == [user2.ap_id] + assert updated_object.data["announcement_count"] == 1 + end + end end diff --git a/test/web/admin_api/admin_api_controller_test.exs b/test/web/admin_api/admin_api_controller_test.exs index 516de5d0c..c497ea098 100644 --- a/test/web/admin_api/admin_api_controller_test.exs +++ b/test/web/admin_api/admin_api_controller_test.exs @@ -1309,6 +1309,7 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIControllerTest do |> json_response(:ok) assert Enum.empty?(response["reports"]) + assert response["total"] == 0 end test "returns reports", %{conn: conn} do @@ -1331,6 +1332,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIControllerTest do assert length(response["reports"]) == 1 assert report["id"] == report_id + + assert response["total"] == 1 end test "returns reports with specified state", %{conn: conn} do @@ -1364,6 +1367,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIControllerTest do assert length(response["reports"]) == 1 assert open_report["id"] == first_report_id + assert response["total"] == 1 + response = conn |> get("/api/pleroma/admin/reports", %{ @@ -1376,6 +1381,8 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIControllerTest do assert length(response["reports"]) == 1 assert closed_report["id"] == second_report_id + assert response["total"] == 1 + response = conn |> get("/api/pleroma/admin/reports", %{ @@ -1384,6 +1391,7 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIControllerTest do |> json_response(:ok) assert Enum.empty?(response["reports"]) + assert response["total"] == 0 end test "returns 403 when requested by a non-admin" do diff --git a/test/web/mastodon_api/mastodon_api_controller_test.exs b/test/web/mastodon_api/mastodon_api_controller_test.exs index 9c5322ccb..fb04748bb 100644 --- a/test/web/mastodon_api/mastodon_api_controller_test.exs +++ b/test/web/mastodon_api/mastodon_api_controller_test.exs @@ -752,7 +752,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIControllerTest do query_string = "ids[]=#{id1}&ids[]=#{id2}" conn = get(conn, "/api/v1/statuses/?#{query_string}") - assert [%{"id" => ^id1}, %{"id" => ^id2}] = json_response(conn, :ok) + assert [%{"id" => ^id1}, %{"id" => ^id2}] = Enum.sort_by(json_response(conn, :ok), & &1["id"]) end describe "deleting a status" do diff --git a/test/web/streamer/ping_test.exs b/test/web/streamer/ping_test.exs new file mode 100644 index 000000000..3d52c00e4 --- /dev/null +++ b/test/web/streamer/ping_test.exs @@ -0,0 +1,36 @@ +# Pleroma: A lightweight social networking server +# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/> +# SPDX-License-Identifier: AGPL-3.0-only + +defmodule Pleroma.Web.PingTest do + use Pleroma.DataCase + + import Pleroma.Factory + alias Pleroma.Web.Streamer + + setup do + start_supervised({Streamer.supervisor(), [ping_interval: 30]}) + + :ok + end + + describe "sockets" do + setup do + user = insert(:user) + {:ok, %{user: user}} + end + + test "it sends pings", %{user: user} do + task = + Task.async(fn -> + assert_receive {:text, received_event}, 40 + assert_receive {:text, received_event}, 40 + assert_receive {:text, received_event}, 40 + end) + + Streamer.add_socket("public", %{transport_pid: task.pid, assigns: %{user: user}}) + + Task.await(task) + end + end +end diff --git a/test/web/streamer/state_test.exs b/test/web/streamer/state_test.exs new file mode 100644 index 000000000..d1aeac541 --- /dev/null +++ b/test/web/streamer/state_test.exs @@ -0,0 +1,54 @@ +# Pleroma: A lightweight social networking server +# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/> +# SPDX-License-Identifier: AGPL-3.0-only + +defmodule Pleroma.Web.StateTest do + use Pleroma.DataCase + + import Pleroma.Factory + alias Pleroma.Web.Streamer + alias Pleroma.Web.Streamer.StreamerSocket + + @moduletag needs_streamer: true + + describe "sockets" do + setup do + user = insert(:user) + user2 = insert(:user) + {:ok, %{user: user, user2: user2}} + end + + test "it can add a socket", %{user: user} do + Streamer.add_socket("public", %{transport_pid: 1, assigns: %{user: user}}) + + assert(%{"public" => [%StreamerSocket{transport_pid: 1}]} = Streamer.get_sockets()) + end + + test "it can add multiple sockets per user", %{user: user} do + Streamer.add_socket("public", %{transport_pid: 1, assigns: %{user: user}}) + Streamer.add_socket("public", %{transport_pid: 2, assigns: %{user: user}}) + + assert( + %{ + "public" => [ + %StreamerSocket{transport_pid: 2}, + %StreamerSocket{transport_pid: 1} + ] + } = Streamer.get_sockets() + ) + end + + test "it will not add a duplicate socket", %{user: user} do + Streamer.add_socket("activity", %{transport_pid: 1, assigns: %{user: user}}) + Streamer.add_socket("activity", %{transport_pid: 1, assigns: %{user: user}}) + + assert( + %{ + "activity" => [ + %StreamerSocket{transport_pid: 1} + ] + } = Streamer.get_sockets() + ) + end + end +end diff --git a/test/web/streamer_test.exs b/test/web/streamer/streamer_test.exs index 96fa7645f..88847e20f 100644 --- a/test/web/streamer_test.exs +++ b/test/web/streamer/streamer_test.exs @@ -5,24 +5,20 @@ defmodule Pleroma.Web.StreamerTest do use Pleroma.DataCase + import Pleroma.Factory + alias Pleroma.List alias Pleroma.User alias Pleroma.Web.CommonAPI alias Pleroma.Web.Streamer - import Pleroma.Factory + alias Pleroma.Web.Streamer.StreamerSocket + alias Pleroma.Web.Streamer.Worker + @moduletag needs_streamer: true clear_config_all([:instance, :skip_thread_containment]) describe "user streams" do setup do - GenServer.start(Streamer, %{}, name: Streamer) - - on_exit(fn -> - if pid = Process.whereis(Streamer) do - Process.exit(pid, :kill) - end - end) - user = insert(:user) notify = insert(:notification, user: user, activity: build(:note_activity)) {:ok, %{user: user, notify: notify}} @@ -125,11 +121,9 @@ defmodule Pleroma.Web.StreamerTest do assert_receive {:text, _}, 4_000 end) - fake_socket = %{ + fake_socket = %StreamerSocket{ transport_pid: task.pid, - assigns: %{ - user: user - } + user: user } {:ok, activity} = CommonAPI.post(other_user, %{"status" => "Test"}) @@ -138,7 +132,7 @@ defmodule Pleroma.Web.StreamerTest do "public" => [fake_socket] } - Streamer.push_to_socket(topics, "public", activity) + Worker.push_to_socket(topics, "public", activity) Task.await(task) @@ -155,11 +149,9 @@ defmodule Pleroma.Web.StreamerTest do assert received_event == expected_event end) - fake_socket = %{ + fake_socket = %StreamerSocket{ transport_pid: task.pid, - assigns: %{ - user: user - } + user: user } {:ok, activity} = CommonAPI.delete(activity.id, other_user) @@ -168,7 +160,7 @@ defmodule Pleroma.Web.StreamerTest do "public" => [fake_socket] } - Streamer.push_to_socket(topics, "public", activity) + Worker.push_to_socket(topics, "public", activity) Task.await(task) end @@ -189,9 +181,9 @@ defmodule Pleroma.Web.StreamerTest do ) task = Task.async(fn -> refute_receive {:text, _}, 1_000 end) - fake_socket = %{transport_pid: task.pid, assigns: %{user: user}} + fake_socket = %StreamerSocket{transport_pid: task.pid, user: user} topics = %{"public" => [fake_socket]} - Streamer.push_to_socket(topics, "public", activity) + Worker.push_to_socket(topics, "public", activity) Task.await(task) end @@ -211,9 +203,9 @@ defmodule Pleroma.Web.StreamerTest do ) task = Task.async(fn -> assert_receive {:text, _}, 1_000 end) - fake_socket = %{transport_pid: task.pid, assigns: %{user: user}} + fake_socket = %StreamerSocket{transport_pid: task.pid, user: user} topics = %{"public" => [fake_socket]} - Streamer.push_to_socket(topics, "public", activity) + Worker.push_to_socket(topics, "public", activity) Task.await(task) end @@ -233,9 +225,9 @@ defmodule Pleroma.Web.StreamerTest do ) task = Task.async(fn -> assert_receive {:text, _}, 1_000 end) - fake_socket = %{transport_pid: task.pid, assigns: %{user: user}} + fake_socket = %StreamerSocket{transport_pid: task.pid, user: user} topics = %{"public" => [fake_socket]} - Streamer.push_to_socket(topics, "public", activity) + Worker.push_to_socket(topics, "public", activity) Task.await(task) end @@ -251,11 +243,9 @@ defmodule Pleroma.Web.StreamerTest do refute_receive {:text, _}, 1_000 end) - fake_socket = %{ + fake_socket = %StreamerSocket{ transport_pid: task.pid, - assigns: %{ - user: user - } + user: user } {:ok, activity} = CommonAPI.post(blocked_user, %{"status" => "Test"}) @@ -264,7 +254,7 @@ defmodule Pleroma.Web.StreamerTest do "public" => [fake_socket] } - Streamer.push_to_socket(topics, "public", activity) + Worker.push_to_socket(topics, "public", activity) Task.await(task) end @@ -284,11 +274,9 @@ defmodule Pleroma.Web.StreamerTest do refute_receive {:text, _}, 1_000 end) - fake_socket = %{ + fake_socket = %StreamerSocket{ transport_pid: task.pid, - assigns: %{ - user: user_a - } + user: user_a } {:ok, activity} = @@ -301,7 +289,7 @@ defmodule Pleroma.Web.StreamerTest do "list:#{list.id}" => [fake_socket] } - Streamer.handle_cast(%{action: :stream, topic: "list", item: activity}, topics) + Worker.handle_call({:stream, "list", activity}, self(), topics) Task.await(task) end @@ -318,11 +306,9 @@ defmodule Pleroma.Web.StreamerTest do refute_receive {:text, _}, 1_000 end) - fake_socket = %{ + fake_socket = %StreamerSocket{ transport_pid: task.pid, - assigns: %{ - user: user_a - } + user: user_a } {:ok, activity} = @@ -335,12 +321,12 @@ defmodule Pleroma.Web.StreamerTest do "list:#{list.id}" => [fake_socket] } - Streamer.handle_cast(%{action: :stream, topic: "list", item: activity}, topics) + Worker.handle_call({:stream, "list", activity}, self(), topics) Task.await(task) end - test "it send wanted private posts to list" do + test "it sends wanted private posts to list" do user_a = insert(:user) user_b = insert(:user) @@ -354,11 +340,9 @@ defmodule Pleroma.Web.StreamerTest do assert_receive {:text, _}, 1_000 end) - fake_socket = %{ + fake_socket = %StreamerSocket{ transport_pid: task.pid, - assigns: %{ - user: user_a - } + user: user_a } {:ok, activity} = @@ -367,11 +351,12 @@ defmodule Pleroma.Web.StreamerTest do "visibility" => "private" }) - topics = %{ - "list:#{list.id}" => [fake_socket] - } + Streamer.add_socket( + "list:#{list.id}", + fake_socket + ) - Streamer.handle_cast(%{action: :stream, topic: "list", item: activity}, topics) + Worker.handle_call({:stream, "list", activity}, self(), %{}) Task.await(task) end @@ -387,11 +372,9 @@ defmodule Pleroma.Web.StreamerTest do refute_receive {:text, _}, 1_000 end) - fake_socket = %{ + fake_socket = %StreamerSocket{ transport_pid: task.pid, - assigns: %{ - user: user1 - } + user: user1 } {:ok, create_activity} = CommonAPI.post(user3, %{"status" => "I'm kawen"}) @@ -401,7 +384,7 @@ defmodule Pleroma.Web.StreamerTest do "public" => [fake_socket] } - Streamer.push_to_socket(topics, "public", announce_activity) + Worker.push_to_socket(topics, "public", announce_activity) Task.await(task) end @@ -417,6 +400,8 @@ defmodule Pleroma.Web.StreamerTest do task = Task.async(fn -> refute_receive {:text, _}, 4_000 end) + Process.sleep(4000) + Streamer.add_socket( "user", %{transport_pid: task.pid, assigns: %{user: user2}} @@ -428,14 +413,6 @@ defmodule Pleroma.Web.StreamerTest do describe "direct streams" do setup do - GenServer.start(Streamer, %{}, name: Streamer) - - on_exit(fn -> - if pid = Process.whereis(Streamer) do - Process.exit(pid, :kill) - end - end) - :ok end @@ -480,6 +457,8 @@ defmodule Pleroma.Web.StreamerTest do refute_receive {:text, _}, 4_000 end) + Process.sleep(1000) + Streamer.add_socket( "direct", %{transport_pid: task.pid, assigns: %{user: user}} @@ -521,6 +500,8 @@ defmodule Pleroma.Web.StreamerTest do assert last_status["id"] == to_string(create_activity.id) end) + Process.sleep(1000) + Streamer.add_socket( "direct", %{transport_pid: task.pid, assigns: %{user: user}} |