diff options
| -rw-r--r-- | CHANGELOG.md | 1 | ||||
| -rw-r--r-- | Dockerfile | 2 | ||||
| -rw-r--r-- | docs/API/differences_in_mastoapi_responses.md | 20 | ||||
| -rw-r--r-- | docs/configuration/optimizing_beam.md | 66 | ||||
| -rw-r--r-- | lib/pleroma/notification.ex | 12 | ||||
| -rw-r--r-- | lib/pleroma/web/api_spec/operations/notification_operation.ex | 3 | ||||
| -rw-r--r-- | lib/pleroma/web/mastodon_api/views/notification_view.ex | 11 | ||||
| -rw-r--r-- | priv/repo/migrations/20200831152600_add_pleroma_report_to_enum_for_notifications.exs | 48 | ||||
| -rw-r--r-- | priv/repo/migrations/20201113060459_remove_purge_expired_activity_worker_from_oban_config.exs | 19 | ||||
| -rw-r--r-- | test/pleroma/notification_test.exs | 13 | ||||
| -rw-r--r-- | test/pleroma/web/mastodon_api/controllers/notification_controller_test.exs | 28 | ||||
| -rw-r--r-- | test/pleroma/web/mastodon_api/views/notification_view_test.exs | 22 |
12 files changed, 241 insertions, 4 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index ab73de0ea..63a8fa942 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - Mix task for sending confirmation emails to all unconfirmed users (`mix pleroma.email send_confirmation_mails`) - Mix task option for force-unfollowing relays - Media preview proxy (requires `ffmpeg` and `ImageMagick` to be installed and media proxy to be enabled; see `:media_preview_proxy` config for more details). +- Reports now generate notifications for admins and mods. - Pleroma API: Importing the mutes users from CSV files. - Experimental websocket-based federation between Pleroma instances. - Support pagination of blocks and mutes diff --git a/Dockerfile b/Dockerfile index c210cf79c..4e7c01c5d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -4,7 +4,7 @@ COPY . . ENV MIX_ENV=prod -RUN apk add git gcc g++ musl-dev make cmake &&\ +RUN apk add git gcc g++ musl-dev make cmake file-dev &&\ echo "import Mix.Config" > config/prod.secret.exs &&\ mix local.hex --force &&\ mix local.rebar --force &&\ diff --git a/docs/API/differences_in_mastoapi_responses.md b/docs/API/differences_in_mastoapi_responses.md index 3075b6b86..ba48a2ca1 100644 --- a/docs/API/differences_in_mastoapi_responses.md +++ b/docs/API/differences_in_mastoapi_responses.md @@ -129,12 +129,30 @@ The `type` value is `pleroma:emoji_reaction`. Has these fields: - `account`: The account of the user who reacted - `status`: The status that was reacted on +### ChatMention Notification (not default) + +This notification has to be requested explicitly. + +The `type` value is `pleroma:chat_mention` + +- `account`: The account who sent the message +- `chat_message`: The chat message + +### Report Notification (not default) + +This notification has to be requested explicitly. + +The `type` value is `pleroma:report` + +- `account`: The account who reported +- `report`: The report + ## GET `/api/v1/notifications` Accepts additional parameters: - `exclude_visibilities`: will exclude the notifications for activities with the given visibilities. The parameter accepts an array of visibility types (`public`, `unlisted`, `private`, `direct`). Usage example: `GET /api/v1/notifications?exclude_visibilities[]=direct&exclude_visibilities[]=private`. -- `include_types`: will include the notifications for activities with the given types. The parameter accepts an array of types (`mention`, `follow`, `reblog`, `favourite`, `move`, `pleroma:emoji_reaction`). Usage example: `GET /api/v1/notifications?include_types[]=mention&include_types[]=reblog`. +- `include_types`: will include the notifications for activities with the given types. The parameter accepts an array of types (`mention`, `follow`, `reblog`, `favourite`, `move`, `pleroma:emoji_reaction`, `pleroma:chat_mention`, `pleroma:report`). Usage example: `GET /api/v1/notifications?include_types[]=mention&include_types[]=reblog`. ## DELETE `/api/v1/notifications/destroy_multiple` diff --git a/docs/configuration/optimizing_beam.md b/docs/configuration/optimizing_beam.md new file mode 100644 index 000000000..e336bd36c --- /dev/null +++ b/docs/configuration/optimizing_beam.md @@ -0,0 +1,66 @@ +# Optimizing the BEAM + +Pleroma is built upon the Erlang/OTP VM known as BEAM. The BEAM VM is highly optimized for latency, but this has drawbacks in environments without dedicated hardware. One of the tricks used by the BEAM VM is [busy waiting](https://en.wikipedia.org/wiki/Busy_waiting). This allows the application to pretend to be busy working so the OS kernel does not pause the application process and switch to another process waiting for the CPU to execute its workload. It does this by spinning for a period of time which inflates the apparent CPU usage of the application so it is immediately ready to execute another task. This can be observed with utilities like **top(1)** which will show consistently high CPU usage for the process. Switching between procesess is a rather expensive operation and also clears CPU caches further affecting latency and performance. The goal of busy waiting is to avoid this penalty. + +This strategy is very successful in making a performant and responsive application, but is not desirable on Virtual Machines or hardware with few CPU cores. Pleroma instances are often deployed on the same server as the required PostgreSQL database which can lead to situations where the Pleroma application is holding the CPU in a busy-wait loop and as a result the database cannot process requests in a timely manner. The fewer CPUs available, the more this problem is exacerbated. The latency is further amplified by the OS being installed on a Virtual Machine as the Hypervisor uses CPU time-slicing to pause the entire OS and switch between other tasks. + +More adventurous admins can be creative with CPU affinity (e.g., *taskset* for Linux and *cpuset* on FreeBSD) to pin processes to specific CPUs and eliminate much of this contention. The most important advice is to run as few processes as possible on your server to achieve the best performance. Even idle background processes can occasionally create [software interrupts](https://en.wikipedia.org/wiki/Interrupt) and take attention away from the executing process creating latency spikes and invalidation of the CPU caches as they must be cleared when switching between processes for security. + +Please only change these settings if you are experiencing issues or really know what you are doing. In general, there's no need to change these settings. + +## VPS Provider Recommendations + +### Good + +* Hetzner Cloud + +### Bad + +* AWS (known to use burst scheduling) + + +## Example configurations + +Tuning the BEAM requires you provide a config file normally called [vm.args](http://erlang.org/doc/man/erl.html#emulator-flags). If you are using systemd to manage the service you can modify the unit file as such: + +`ExecStart=/usr/bin/elixir --erl '-args_file /opt/pleroma/config/vm.args' -S /usr/bin/mix phx.server` + +Check your OS documentation to adopt a similar strategy on other platforms. + +### Virtual Machine and/or few CPU cores + +Disable the busy-waiting. This should generally only be done if you're on a platform that does burst scheduling, like AWS. + +**vm.args:** + +``` ++sbwt none ++sbwtdcpu none ++sbwtdio none +``` + +### Dedicated Hardware + +Enable more busy waiting, increase the internal maximum limit of BEAM processes and ports. You can use this if you run on dedicated hardware, but it is not necessary. + +**vm.args:** + +``` ++P 16777216 ++Q 16777216 ++K true ++A 128 ++sbt db ++sbwt very_long ++swt very_low ++sub true ++Mulmbcs 32767 ++Mumbcgs 1 ++Musmbcs 2047 +``` + +## Additional Reading + +* [WhatsApp: Scaling to Millions of Simultaneous Connections](https://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf) +* [Preemptive Scheduling and Spinlocks](https://www.uio.no/studier/emner/matnat/ifi/nedlagte-emner/INF3150/h03/annet/slides/preemptive.pdf) +* [The Curious Case of BEAM CPU Usage](https://stressgrid.com/blog/beam_cpu_usage/) diff --git a/lib/pleroma/notification.ex b/lib/pleroma/notification.ex index 8868a910e..dd7a1c824 100644 --- a/lib/pleroma/notification.ex +++ b/lib/pleroma/notification.ex @@ -70,6 +70,7 @@ defmodule Pleroma.Notification do move pleroma:chat_mention pleroma:emoji_reaction + pleroma:report reblog } @@ -367,7 +368,7 @@ defmodule Pleroma.Notification do end def create_notifications(%Activity{data: %{"type" => type}} = activity, options) - when type in ["Follow", "Like", "Announce", "Move", "EmojiReact"] do + when type in ["Follow", "Like", "Announce", "Move", "EmojiReact", "Flag"] do do_create_notifications(activity, options) end @@ -410,6 +411,9 @@ defmodule Pleroma.Notification do "EmojiReact" -> "pleroma:emoji_reaction" + "Flag" -> + "pleroma:report" + # Compatibility with old reactions "EmojiReaction" -> "pleroma:emoji_reaction" @@ -467,7 +471,7 @@ defmodule Pleroma.Notification do def get_notified_from_activity(activity, local_only \\ true) def get_notified_from_activity(%Activity{data: %{"type" => type}} = activity, local_only) - when type in ["Create", "Like", "Announce", "Follow", "Move", "EmojiReact"] do + when type in ["Create", "Like", "Announce", "Follow", "Move", "EmojiReact", "Flag"] do potential_receiver_ap_ids = get_potential_receiver_ap_ids(activity) potential_receivers = @@ -503,6 +507,10 @@ defmodule Pleroma.Notification do [object_id] end + def get_potential_receiver_ap_ids(%{data: %{"type" => "Flag"}}) do + User.all_superusers() |> Enum.map(fn user -> user.ap_id end) + end + def get_potential_receiver_ap_ids(activity) do [] |> Utils.maybe_notify_to_recipients(activity) diff --git a/lib/pleroma/web/api_spec/operations/notification_operation.ex b/lib/pleroma/web/api_spec/operations/notification_operation.ex index f09be64cb..264a530d2 100644 --- a/lib/pleroma/web/api_spec/operations/notification_operation.ex +++ b/lib/pleroma/web/api_spec/operations/notification_operation.ex @@ -193,6 +193,7 @@ defmodule Pleroma.Web.ApiSpec.NotificationOperation do "mention", "pleroma:emoji_reaction", "pleroma:chat_mention", + "pleroma:report", "move", "follow_request" ], @@ -206,6 +207,8 @@ defmodule Pleroma.Web.ApiSpec.NotificationOperation do - `poll` - A poll you have voted in or created has ended - `move` - Someone moved their account - `pleroma:emoji_reaction` - Someone reacted with emoji to your status + - `pleroma:chat_mention` - Someone mentioned you in a chat message + - `pleroma:report` - Someone was reported """ } end diff --git a/lib/pleroma/web/mastodon_api/views/notification_view.ex b/lib/pleroma/web/mastodon_api/views/notification_view.ex index c97e6d32f..5b06a6b51 100644 --- a/lib/pleroma/web/mastodon_api/views/notification_view.ex +++ b/lib/pleroma/web/mastodon_api/views/notification_view.ex @@ -11,6 +11,8 @@ defmodule Pleroma.Web.MastodonAPI.NotificationView do alias Pleroma.Object alias Pleroma.User alias Pleroma.UserRelationship + alias Pleroma.Web.AdminAPI.Report + alias Pleroma.Web.AdminAPI.ReportView alias Pleroma.Web.CommonAPI alias Pleroma.Web.MastodonAPI.AccountView alias Pleroma.Web.MastodonAPI.NotificationView @@ -118,11 +120,20 @@ defmodule Pleroma.Web.MastodonAPI.NotificationView do "pleroma:chat_mention" -> put_chat_message(response, activity, reading_user, status_render_opts) + "pleroma:report" -> + put_report(response, activity) + type when type in ["follow", "follow_request"] -> response end end + defp put_report(response, activity) do + report_render = ReportView.render("show.json", Report.extract_report_info(activity)) + + Map.put(response, :report, report_render) + end + defp put_emoji(response, activity) do Map.put(response, :emoji, activity.data["content"]) end diff --git a/priv/repo/migrations/20200831152600_add_pleroma_report_to_enum_for_notifications.exs b/priv/repo/migrations/20200831152600_add_pleroma_report_to_enum_for_notifications.exs new file mode 100644 index 000000000..01fb90459 --- /dev/null +++ b/priv/repo/migrations/20200831152600_add_pleroma_report_to_enum_for_notifications.exs @@ -0,0 +1,48 @@ +defmodule Pleroma.Repo.Migrations.AddPleromaReportTypeToEnumForNotifications do + use Ecto.Migration + + @disable_ddl_transaction true + + def up do + """ + alter type notification_type add value 'pleroma:report' + """ + |> execute() + end + + def down do + alter table(:notifications) do + modify(:type, :string) + end + + """ + delete from notifications where type = 'pleroma:report' + """ + |> execute() + + """ + drop type if exists notification_type + """ + |> execute() + + """ + create type notification_type as enum ( + 'follow', + 'follow_request', + 'mention', + 'move', + 'pleroma:emoji_reaction', + 'pleroma:chat_mention', + 'reblog', + 'favourite' + ) + """ + |> execute() + + """ + alter table notifications + alter column type type notification_type using (type::notification_type) + """ + |> execute() + end +end diff --git a/priv/repo/migrations/20201113060459_remove_purge_expired_activity_worker_from_oban_config.exs b/priv/repo/migrations/20201113060459_remove_purge_expired_activity_worker_from_oban_config.exs new file mode 100644 index 000000000..fe31f4442 --- /dev/null +++ b/priv/repo/migrations/20201113060459_remove_purge_expired_activity_worker_from_oban_config.exs @@ -0,0 +1,19 @@ +defmodule Pleroma.Repo.Migrations.RemovePurgeExpiredActivityWorkerFromObanConfig do + use Ecto.Migration + + def change do + with %Pleroma.ConfigDB{} = config <- + Pleroma.ConfigDB.get_by_params(%{group: :pleroma, key: Oban}), + crontab when is_list(crontab) <- config.value[:crontab], + index when is_integer(index) <- + Enum.find_index(crontab, fn {_, worker} -> + worker == Pleroma.Workers.Cron.PurgeExpiredActivitiesWorker + end) do + updated_value = Keyword.put(config.value, :crontab, List.delete_at(crontab, index)) + + config + |> Ecto.Changeset.change(value: updated_value) + |> Pleroma.Repo.update() + end + end +end diff --git a/test/pleroma/notification_test.exs b/test/pleroma/notification_test.exs index 92c0bc8b6..ed2cd219d 100644 --- a/test/pleroma/notification_test.exs +++ b/test/pleroma/notification_test.exs @@ -32,6 +32,19 @@ defmodule Pleroma.NotificationTest do refute {:ok, [nil]} == Notification.create_notifications(activity) end + test "creates a notification for a report" do + reporting_user = insert(:user) + reported_user = insert(:user) + {:ok, moderator_user} = insert(:user) |> User.admin_api_update(%{is_moderator: true}) + + {:ok, activity} = CommonAPI.report(reporting_user, %{account_id: reported_user.id}) + + {:ok, [notification]} = Notification.create_notifications(activity) + + assert notification.user_id == moderator_user.id + assert notification.type == "pleroma:report" + end + test "creates a notification for an emoji reaction" do user = insert(:user) other_user = insert(:user) diff --git a/test/pleroma/web/mastodon_api/controllers/notification_controller_test.exs b/test/pleroma/web/mastodon_api/controllers/notification_controller_test.exs index 5fd518c60..9ac8488f6 100644 --- a/test/pleroma/web/mastodon_api/controllers/notification_controller_test.exs +++ b/test/pleroma/web/mastodon_api/controllers/notification_controller_test.exs @@ -75,6 +75,34 @@ defmodule Pleroma.Web.MastodonAPI.NotificationControllerTest do assert [_] = result end + test "by default, does not contain pleroma:report" do + %{user: user, conn: conn} = oauth_access(["read:notifications"]) + other_user = insert(:user) + third_user = insert(:user) + + user + |> User.admin_api_update(%{is_moderator: true}) + + {:ok, activity} = CommonAPI.post(other_user, %{status: "hey"}) + + {:ok, _report} = + CommonAPI.report(third_user, %{account_id: other_user.id, status_ids: [activity.id]}) + + result = + conn + |> get("/api/v1/notifications") + |> json_response_and_validate_schema(200) + + assert [] == result + + result = + conn + |> get("/api/v1/notifications?include_types[]=pleroma:report") + |> json_response_and_validate_schema(200) + + assert [_] = result + end + test "getting a single notification" do %{user: user, conn: conn} = oauth_access(["read:notifications"]) other_user = insert(:user) diff --git a/test/pleroma/web/mastodon_api/views/notification_view_test.exs b/test/pleroma/web/mastodon_api/views/notification_view_test.exs index 2f6a808f1..9de11a87e 100644 --- a/test/pleroma/web/mastodon_api/views/notification_view_test.exs +++ b/test/pleroma/web/mastodon_api/views/notification_view_test.exs @@ -12,6 +12,8 @@ defmodule Pleroma.Web.MastodonAPI.NotificationViewTest do alias Pleroma.Object alias Pleroma.Repo alias Pleroma.User + alias Pleroma.Web.AdminAPI.Report + alias Pleroma.Web.AdminAPI.ReportView alias Pleroma.Web.CommonAPI alias Pleroma.Web.CommonAPI.Utils alias Pleroma.Web.MastodonAPI.AccountView @@ -207,6 +209,26 @@ defmodule Pleroma.Web.MastodonAPI.NotificationViewTest do test_notifications_rendering([notification], user, [expected]) end + test "Report notification" do + reporting_user = insert(:user) + reported_user = insert(:user) + {:ok, moderator_user} = insert(:user) |> User.admin_api_update(%{is_moderator: true}) + + {:ok, activity} = CommonAPI.report(reporting_user, %{account_id: reported_user.id}) + {:ok, [notification]} = Notification.create_notifications(activity) + + expected = %{ + id: to_string(notification.id), + pleroma: %{is_seen: false, is_muted: false}, + type: "pleroma:report", + account: AccountView.render("show.json", %{user: reporting_user, for: moderator_user}), + created_at: Utils.to_masto_date(notification.inserted_at), + report: ReportView.render("show.json", Report.extract_report_info(activity)) + } + + test_notifications_rendering([notification], moderator_user, [expected]) + end + test "muted notification" do user = insert(:user) another_user = insert(:user) |