Merge remote-tracking branch 'remotes/origin/develop' into 2168-media-preview-proxy

1 files changed, 21 insertions, 0 deletions

diff --git a/docs/administration/CLI_tasks/robots_txt.md b/docs/administration/CLI_tasks/robots_txt.md
new file mode 100644
index 000000000..7eeedf571
--- /dev/null
+++ b/docs/administration/CLI_tasks/robots_txt.md
@@ -0,0 +1,21 @@
+# Managing robots.txt
+
+{! backend/administration/CLI_tasks/general_cli_task_info.include !}
+
+## Generate a new robots.txt file and add it to the static directory
+
+The `robots.txt` that ships by default is permissive. It allows well-behaved search engines to index all of your instance's URIs.
+
+If you want to generate a restrictive `robots.txt`, you can run the following mix task. The generated `robots.txt` will be written in your instance [static directory](../../../configuration/static_dir/).
+
+=== "OTP"
+
+    ```sh
+    ./bin/pleroma_ctl robots_txt disallow_all
+    ```
+
+=== "From Source"
+
+    ```sh
+    mix pleroma.robots_txt disallow_all
+    ```