Commit Graph

1310 Commits

Author SHA1 Message Date
Claire
02a87431cf
Fix error when suspending user with an already-existing canonical email block (#17036)
* Fix error when suspending user with an already-existing canonical email block

Fixes #17033

While attempting to create a `CanonicalEmailBlock` with an existing hash would
raise an `ActiveRecord::RecordNotUnique` error, this being done within a
transaction would cancel the whole transaction. For this reason, checking for
uniqueness in Rails would query the database within the transaction and avoid
invalidating the whole transaction for this reason.

A race condition is still possible, where multiple accounts sharing a canonical
email would be blocked in concurrent transactions, in which only one would
succeed, but that is way less likely to happen that the current issue, and can
always be retried after the first failure, unlike the current situation.

* Add tests
2021-11-24 17:41:03 +01:00
Takeshi Umeda
3419d3ec84
Bump chewy from 5.2.0 to 7.2.3 (supports Elasticsearch 7.x) (#16915)
* Bump chewy from 5.2.0 to 7.2.2

* fix style (codeclimate)

* fix style

* fix style

* Bump chewy from 7.2.2 to 7.2.3
2021-11-18 22:02:08 +01:00
Claire
6da135a493
Fix reviving revoked sessions and invalidating login (#16943)
Up until now, we have used Devise's Rememberable mechanism to re-log users
after the end of their browser sessions. This mechanism relies on a signed
cookie containing a token. That token was stored on the user's record,
meaning it was shared across all logged in browsers, meaning truly revoking
a browser's ability to auto-log-in involves revoking the token itself, and
revoking access from *all* logged-in browsers.

We had a session mechanism that dynamically checks whether a user's session
has been disabled, and would log out the user if so. However, this would only
clear a session being actively used, and a new one could be respawned with
the `remember_user_token` cookie.

In practice, this caused two issues:
- sessions could be revived after being closed from /auth/edit (security issue)
- auto-log-in would be disabled for *all* browsers after logging out from one
  of them

This PR removes the `remember_token` mechanism and treats the `_session_id`
cookie/token as a browser-specific `remember_token`, fixing both issues.
2021-11-06 00:13:58 +01:00
Claire
87085a5152
Fix AccountNote not having a maximum length (#16942) 2021-11-06 00:12:25 +01:00
Claire
989c67d29d
Fix handling announcements with links (#16941)
Broken since #15827
2021-11-05 21:14:35 +01:00
Takeshi Umeda
17f4e457b3
Add remove from followers api (#16864)
* Add followed_by? to account_interactions

* Add RemoveFromFollowersService

* Fix AccountBatch to use RemoveFromFollowersService

* Add remove from followers API
2021-10-18 12:02:35 +02:00
Jeong Arm
f4081d1564
Allow keeping only 1 boosts/favs on auto deleting posts (#16653)
* Allow keeping 1 boosts/favs on auto deleting posts

* Fix tests
2021-10-14 21:11:14 +02:00
Claire
9b978872cc
Fix 404 error when filtering admin action logs by non-existent target account (#16643)
Currently, there is no way for an admin to delete an account record. However,
should that happen in the future, or should an admin mistype an account id,
`/admin/action_logs?target_account_id=<non-existent-account-id>` currently
returns a 404 instead of an empty list.

With this commit, the aforementioned page returns a correct list instead
(potentially non-empty if the account has been manually deleted)
2021-10-14 20:53:11 +02:00
Claire
216570ad98
Fix scheduled statuses decreasing statuses counts (#16791)
* Add tests

* Fix scheduled statuses decreasing statuses counts

Fixes #16774
2021-10-14 19:59:21 +02:00
Claire
5159ba26e4
Fix error when rendering public pages with media attachments (#16763)
* Add tests

* Fix error when rendering public pages with media attachments

* Add tests

* Fix tests

* Please CodeClimate
2021-10-13 15:27:19 +02:00
Claire
84ceebe1c4
Fix media attachment size validation not correctly accounting for file type (#16819)
* Fix media attachment size validation not correctly accounting for file type

Fixes a regression introduced in #16724 caused by the fact that kt-paperclip
now correctly runs validations before processing, meaning that file size
verification could not rely on our before_post_processing hook.

Moved the `before_post_processing` hooks to `before_validate` to make sure
the media attachment type is set correctly before the file gets validated.

* Add tests
2021-10-06 14:49:32 +01:00
Claire
fc3ae1343d
Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Claire
db57bff11d
Stop setting a shortcode to newly-created media attachments (#16730)
* Stop setting a shortcode to newly-created media attachments

The WebUI has stopped using the “short media URL” in ages. This isn't used
anywhere except for mail notifications.

Deprecating it would allow us to eventually get rid of at least a database
column and corruption-prone index, as well as a controller.

* Fix tests
2021-09-13 18:59:37 +02:00
Claire
2b18f7a943
Fix processing mentions to domains with non-ascii TLDs (#16689)
Fixes #16602
2021-09-01 22:06:40 +02:00
Claire
5efb1ff337
Fix followers synchronization mechanism not working when URI has empty path (#16510)
* Fix followers synchronization mechanism not working when URI has empty path

To my knowledge, there is no current implementation on the fediverse
that can use bare domains (e.g., actor is at https://example.org instead of
something like https://example.org/actor) that also plans to support the
followers synchronization mechanism. However, Mastodon's current implementation
would exclude such accounts from followers list.

Also adds tests and rename them to reflect the proper method names.

* Move url prefix regexp to its own constant
2021-08-11 17:48:42 +02:00
Claire
13b08610a0
Fix crash when encountering invalid account fields (#16598)
* Add test

* Fix crash when encountering invalid account fields
2021-08-11 16:40:55 +02:00
Claire
4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Claire
763ab0c7eb
Fix owned account notes not being deleted when an account is deleted (#16579)
* Add account_notes relationship

* Add tests

* Fix owned account notes not being deleted when an account is deleted

* Add post-migration to clean up orphaned account notes
2021-08-08 15:29:57 +02:00
Claire
30ce6e395c
Fix user email address being banned on self-deletion (#16503)
* Add tests

* Fix user email address being banned on self-deletion

Fixes #16498
2021-07-14 05:35:49 +02:00
Eugen Rochko
7095c80373
Add ability to filter reports by origin of target account (#16487) 2021-07-11 11:01:38 +02:00
Eugen Rochko
771c9d4ba8
Add ability to skip sign-in token authentication for specific users (#16427)
Remove "active within last two weeks" exception for sign in token requirement

Change admin reset password to lock access until the password is reset
2021-07-08 05:31:28 +02:00
Claire
8569126c2e
Fix upgrade path from 3.4.0 (#16465)
3.4.1 dropped account_stats.lock_version, but in a way breaking the usual
upgrade path by requiring services to be reloaded after the post-migrations.

Indeed, `self.locking_column = nil` was not enough for Rails to ignore the
`lock_version` column when preparing statements on application load, resulting
in some ActiveRecord queries (typically those involving
`includes(:account_stat)`) erroring out with:

  ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR:  column account_stats.lock_version does not exist
2021-07-07 21:13:30 +02:00
Eugen Rochko
38b3419950
Fix not being able to suspend accounts that already have a canonical e-mail block (#16455) 2021-07-02 12:03:15 +02:00
Eugen Rochko
d174d12c83
Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Claire
526332c545
Fix account deletion sometimes failing because of optimistic locks (#16317)
* Fix account deletion sometimes failing because of optimistic locks

In some rare occasions[1], deleting accounts would fail with a
`StaleObjectError` exception.

Indeed, account deletion manually sets the `AccountStat` values without
handling cases where the optimistic locking on `AccountStat` would fail.

To my knowledge, with the rewrite of account counters in #15913, the
`DeleteAccountService` is now the only place that changes the counters in
a way that is not atomic.

Since in this specific case, we do not care about the previous values of the
account counters, it appears we don't need locking at all for this table
anymore.

[1]: https://discourse.joinmastodon.org/t/account-cant-be-deleted/3602

* Bump MAX_SUPPORTED_VERSION in maintenance script
2021-06-02 17:41:25 +02:00
Claire
76064e6608
Update fix-duplicates maintenance script to support latest migrations (#16231)
* Update maintenance script to support latest database migrations

* Update Account#merge_with!
2021-05-12 23:19:44 +02:00
Takeshi Umeda
c403c3695b
Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Claire
f9c2d1b9a1
Fix FollowRecommendationsScheduler failing because of unpopulated views (#16189)
Since #16173, `account_summaries` and `follow_recommendations` are not
populated at creation time, which causes concurrent refresh to fail.

As we currently only use those materialized views right after explicitly
refreshing them, this commit changes refreshes to not be performed
concurrently. This will fix the issue and ensure the refresh completes
faster while using less resources.
2021-05-09 10:39:29 +02:00
Eugen Rochko
1294f9ee4f
Remove PubSubHubbub-related columns from accounts table (#16170) 2021-05-07 19:32:58 +02:00
Eugen Rochko
74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Claire
566fc90913
Add Ruby 3.0 support (#16046)
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0

Also improve the Terrapin monkey-patch for the stderr/stdout issue.

* Fix keyword argument handling throughout the codebase

* Monkey-patch Paperclip to fix keyword arguments handling in validators

* Change validation_extensions to please CodeClimate

* Bump microformats from 4.2.1 to 4.3.1

* Allow Ruby 3.0

* Add Ruby 3.0 test target to CircleCI

* Add test for admin dashboard warnings

* Fix admin dashboard warnings on Ruby 3.0
2021-05-06 14:22:54 +02:00
Takeshi Umeda
7cb34b32f8
Add management of delivery availability in Federation settings (#15771)
* Add management of delivery availavility in Federation settings

* fix translate

* Remove useless object creation

* Fix DeepSource issue

* Add shortcut for all

* Fix DeepSource(skipcq)

* Change 'remove' to 'clear'

* Fix style

* Change class method name (exhausted_deliveries_key_by)
2021-05-05 23:39:02 +02:00
Claire
d9ae3db8d5
Improve performance of follow recommendation scheduler (#16159)
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.

This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
2021-05-05 22:04:52 +02:00
Eugen Rochko
036556d350
Fix media processing getting stuck on too much stdin/stderr (#16136)
* Fix media processing getting stuck on too much stdin/stderr

See thoughtbot/terrapin#5

* Remove dependency on paperclip-av-transcoder gem

* Remove dependency on streamio-ffmpeg gem

* Disable stdin on ffmpeg process
2021-05-05 19:44:01 +02:00
Eugen Rochko
fab65848d2
Fix empty home feed before first follow has finished processing (#16152)
Change queue of merge worker from pull to default
2021-05-04 04:45:08 +02:00
abcang
7f0c49c58a
Improve tag search query (#16104) 2021-04-25 06:33:28 +02:00
Eugen Rochko
daccc07dc1
Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Eugen Rochko
f4b7c6b619
Fix nil error when removing status caused by race condition (#16099) 2021-04-24 13:35:39 +02:00
Claire
a6564d56d6
Fix edge case where accepted follow cannot be processed because of follow limit (#16098) 2021-04-23 22:51:21 +02:00
Eugen Rochko
b3ceb3dcc4
Add canonical e-mail blocks for suspended accounts (#16049)
Prevent new accounts from being created using the same underlying
e-mail as a suspended account using extensions and period
permutations. Stores e-mails as a SHA256 hash
2021-04-17 03:14:25 +02:00
Eugen Rochko
ce2148c571
Add policy param to POST /api/v1/push/subscriptions (#16040)
With possible values `all`, `followed`, `follower`, and `none`,
control from whom notifications will generate a Web Push alert
2021-04-15 05:00:25 +02:00
Eugen Rochko
120965eb0b
Change Web Push API deliveries to use request pooling (#16014) 2021-04-12 14:25:34 +02:00
Eugen Rochko
f7117646af
Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko
619fad6cf8
Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Claire
cbd0ee1d07
Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Eugen Rochko
af8fe6e1e9
WIP (#15222) 2021-03-19 17:15:36 +01:00
Claire
741d0952b1
Improve account counters handling (#15913)
* Improve account counters handling

* Use ActiveRecord::Base::sanitize_sql to pass values instead of interpolating them

Keep using string interpolation for `key` as it is safe and using
“ActiveRecord::Base::sanitize_sql_hash_for_assignment” would require stitching
bits of SQL in a way that is not more easily checked for safety.

* Add migration hook to catch PostgreSQL versions earlier than 9.5
2021-03-19 13:14:57 +01:00
Claire
43eff898a0
Prepare Mastodon for Rails 6 (#15911)
* Fix misuse of foreign_type

* Fix use of removed "add_template_helper"

* Use response.media_type instead of response.content_type in tests

* Fix CSV export controller test on Rails 6

Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.

This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.

* Fix emoji formatting with Rails 6

* Make emoji output more idiomatic and robust

* Switch from redis-rails gem to built-in Rails redis cache storage
2021-03-17 10:09:55 +01:00
Claire
1b02d29be5
Fix not being able to change world filter expiration back to “Never” (#15858)
Fixes #15849
2021-03-12 05:25:50 +01:00
Claire
5cc45d22d3
Remove subscription_expires_at leftover from OStatus (#15857) 2021-03-12 05:25:24 +01:00
Eugen Rochko
8331fdf7e0
Add server rules (#15769) 2021-02-21 19:50:12 +01:00
Claire
acdeb162b8
Create instance actor if it hasn't been properly seeded (#15693)
An uncommon but somewhat difficult to digagnose issue is dealing with
improperly-seeded databases. In such cases, instance-signed fetches will
fail with a ActiveRecord::RecordNotFound error, usually caught and handled
as generic 404, leading people to think the remote resource itself has not
been found, while it's the local instance actor that does not exist.

This commit changes the code so that failure to find the instance actor
automatically creates a new one, so that improperly-seeded databases do
not cause any issue.
2021-02-09 18:12:54 +01:00
Eugen Rochko
acf1842896
Change max. image dimensions to 1920x1080px (1080p) (#15690)
* Change max. image size to 1920x1080px

* Change it in web UI too
2021-02-09 01:21:06 +01:00
ThibG
a044ddac5b
Fix race conditions on account migration creation (#15597)
* Atomically check for processing lock in Move handler

* Prevent race condition when creating account migrations

Fixes #15595

* Add tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-02-02 14:49:57 +01:00
abcang
7ab53f221a
Improved performance of notification preloading (#15640)
* Improved performance of notification preloading

* Remove Cacheable from Notification

* Fix test
2021-01-31 21:24:57 +01:00
abcang
c8c764dd8b
Fix N+1 query when rendering with StatusSerializer (#15641) 2021-01-31 21:24:17 +01:00
luigi
eb51e43fb4
Optimize some regex matching (#15528)
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
2021-01-22 10:09:08 +01:00
ThibG
e955ca5463
Fix sign-up restrictions based on IP addresses not being enforced (#15607)
Fixes #15606

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-21 06:18:30 +01:00
ThibG
54d4e5252b
Use Rails' index_by where it makes sense (#15542)
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:27:38 +01:00
luigi
087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
abcang
efffdd3778
Fix rubocop config and warnings (#15503)
* disable NewCops

* update TargetRubyVersion

* Fix Lint/MissingSuper for ActiveModelSerializers::Model

* Fix Lint/MissingSuper for feed

* Fix Lint/FloatComparison

* Do not use instance variables
2021-01-07 09:40:55 +01:00
ThibG
f1f96ebf02
Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-26 23:52:46 +01:00
ThibG
3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
ThibG
1cf2c3a810
Fix external user creation failing when invite request text is required (#15405)
* Fix external user creation failing when invite request text is required

Also fixes tootctl-based user creation.

* Add test about invites when invite request text is otherwise required

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 17:14:32 +01:00
Eugen Rochko
9915d11c0d
Fix unnecessary queries when batch-removing statuses, 100x faster (#15387) 2020-12-22 17:13:55 +01:00
ThibG
43961035a9
Fix some notifications not being deleted on poll/status deletion (#15402)
* Fix deleting polls not deleting notifications

* Fix fav notification deletion when deleting a toot

* Refactor DeleteAccountService spec

* Add DeleteAccountService tests for other associations and notifications

* Add favourite handling spec in status removal

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 18:22:17 +01:00
ThibG
6f51fd7435
Fix invitation links not working when invite request text is required (#15385)
Fixes #15383

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 00:47:02 +01:00
ThibG
7bf3c6e57b
Fix AccountDeletionWorker crashing and clogging sidekiq queues (#15380)
* Fix account deletion workers being queued multiple times for a single account

* Fix poll votes being unnecessarily instantiated on poll deletion

* Fix favourites being unnecessarily instantiated on status deletion

* Remove inaccurate comments

* Delete polls instead of destroying them

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-20 18:25:00 +01:00
Eugen Rochko
eb35be0431
Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
Eugen Rochko
8a95867693
Add option to obfuscate domain name in public list of domain blocks (#15355)
- Replace the middle of the domain with * characters (except for periods)
- Add SHA-256 digest of the domain name in tooltip
2020-12-18 08:30:41 +01:00
ThibG
8357969559
Fix admins being able to suspend their instance actor (#14567)
* Fix admin being able to suspend their own instance account

* Add text about the instance's own actor in admin view

* Change instance actor notice from flash message to template

* Do not list local instance actor in account moderation list
2020-12-15 17:23:58 +01:00
ThibG
1390cc194b
Add indication to admin UI of whether a report has been forwarded (#13237)
* Add indication to admin UI of whether a report has been forwarded

* Rework how forwarded status is displayed

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-15 04:30:15 +01:00
ThibG
47e507fa61
Add ability to require invite request text (#15326)
Fixes #15273

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-14 10:03:09 +01:00
Eugen Rochko
216b85b053
Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
ThibG
49eb4d4ddf
Add honeypot fields and minimum fill-out time for sign-up form (#15276)
* Add honeypot fields to limit non-specialized spam

Add two honeypot fields: a fake website input and a fake password confirmation
one. The label/placeholder/aria-label tells not to fill them, and they are
hidden in CSS, so legitimate users should not fall into these.

This should cut down on some non-Mastodon-specific spambots.

* Require a 3 seconds delay before submitting the registration form

* Fix tests

* Move registration form time check to model validation

* Give people a chance to clear the honeypot fields

* Refactor honeypot translation strings

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-10 06:27:26 +01:00
trwnh
127c543a6e
rename replies_policy enumerables (#15304) 2020-12-09 04:34:17 +01:00
Eugen Rochko
48bef17cc9
Fix slow distinct queries where grouped queries are faster (#15287)
About 2x speed-up on inboxes query
2020-12-07 12:08:30 +01:00
ThibG
fbe7519243
Fix account merging in maintenance script (#15264)
Also include AccountNote and AccountDeletionRequest to the list of classes
needing to be reassigned the merged account.
2020-12-04 02:23:51 +01:00
ThibG
9136be480f
Fix followers hash cache not being invalidated on account merge (#15256)
Also clear relationships cache.
2020-12-02 21:20:00 +01:00
Eugen Rochko
f844386809
Add tootctl accounts merge (#15201)
* Add `tootctl accounts merge`

* Update lib/mastodon/accounts_cli.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-11-23 17:50:16 +01:00
ThibG
96c1e71329
Add import/export feature for bookmarks (#14956)
* Add ability to export bookmarks

* Add support for importing bookmarks

* Add bookmark import tests

* Add bookmarks export test
2020-11-19 17:48:13 +01:00
Eugen Rochko
aa10200e58
Fix streaming API allowing connections to persist after access token invalidation (#15111)
Fix #14816
2020-11-12 23:05:24 +01:00
Eugen Rochko
8532429af7
Fix 2FA/sign-in token sessions being valid after password change (#14802)
If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token
2020-11-12 23:05:01 +01:00
ThibG
9870b175b4
Fix possible inconsistencies in tag search (#14906)
Do not downcase the queried tag before passing it to postgres when searching:
- tags are not downcased on creation
- `arel_table[:name].lower.matches(pattern)` generates an ILIKE anyway
- if Postgres and Rails happen to use different case-folding rules,
  downcasing before query but not before insertion may mean that some
  tags with some casings are not searchable
2020-11-12 18:35:23 +01:00
Takeshi Umeda
2b1a6e734f
Add follow selected followers button (#15148)
* Add follow selected followers button

* Fix unused variable

* Fix i18n normalize
2020-11-12 16:58:00 +01:00
Eugen Rochko
337dc6e0ad
Fix updating account counters when account_stat is not yet created (#15108) 2020-11-09 16:00:23 +01:00
Eugen Rochko
3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
ThibG
b29defb851
Change order of announcements in admin page to sort them newest-first (#15091)
* Change order of announcements in admin page to sort show newly-created first

Fixes #15090

* Use reverse-chronological rather than creation date only
2020-11-04 21:15:22 +01:00
Takeshi Umeda
d6fe0c94ca
Add account sensitized (#14361)
* Add account sensitized

* Fix i18n normalize

* Fix description and spec

* Fix spec

* Fix wording
2020-11-04 20:45:01 +01:00
ThibG
ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
OSAMU SATO
96761752ec
Add duration parameter to muting. (#13831)
* Adding duration to muting.

* Remove useless checks
2020-10-13 01:01:14 +02:00
Eugen Rochko
5e1364c448
Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Eugen Rochko
7d985f2aac
Remove dependency on goldfinger gem (#14919)
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.

Fix #14091
2020-10-08 00:34:57 +02:00
ThibG
78e45a5285
Add option to disable swiping motions across the WebUI (#13885)
Fixes #13882
2020-09-30 19:31:03 +02:00
Takeshi Umeda
b655a7f88f
Fix a slow query for TagFeed (#14861)
* Fix a slow query for TagFeed

* rename tags to tag_ids
2020-09-23 16:01:54 +02:00
Eugen Rochko
974b1b79ce
Add option to be notified when a followed user posts (#13546)
* Add bell button

Fix #4890

* Remove duplicate type from post-deployment migration

* Fix legacy class type mappings

* Improve query performance with better index

* Fix validation

* Remove redundant index from notifications
2020-09-18 17:26:45 +02:00
kawaguchi
5d3c8baa9a
Fix validates :sign_count of WebauthnCredential (#14806) 2020-09-16 20:16:46 +02:00
Eugen Rochko
ed099d8bdc
Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Eugen Rochko
42c4322ce7
Fix reported statuses not being included in warning e-mail (#14778) 2020-09-13 12:52:17 +02:00
tateisu
fa0c71f0d9
allow pagination by min_id and max_id (#14776)
* allow pagination by min_id and max_id

* also AccountConversation allows min_id,max_id pair

* also home,list TL allows min_id,max_id pair
2020-09-12 17:09:49 +02:00
Eugen Rochko
e6b272e5c9
Change REST API to return empty data for suspended accounts (#14765) 2020-09-11 15:16:29 +02:00
abcang
e79d719e92
Changed tag most_used to recently_used (#14760) 2020-09-07 17:47:41 +02:00
Eugen Rochko
e8bc187845
Refactor how public and tag timelines are queried (#14728) 2020-09-07 11:02:04 +02:00
ThibG
79305428a7
Add configuration option to filter replies in lists (#9205)
* Add database support for list show-reply preferences

* Add backend support to read and update list-specific show_replies settings

* Add basic UI to set list replies setting

* Add specs for list replies policy

* Switch "cycling" reply policy link to a set of radio inputs

* Capitalize replies_policy strings

* Change radio button design to be consistent with that of the directory explorer
2020-09-01 13:31:28 +02:00
dependabot[bot]
8972e5f7f6
Bump rubocop from 0.86.0 to 0.88.0 (#14412)
* Bump rubocop from 0.86.0 to 0.88.0

Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0.
- [Release notes](https://github.com/rubocop-hq/rubocop/releases)
- [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Fix for latest RuboCop

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-09-01 03:04:00 +02:00
Akihiko Odaki
ae871c4d46
Make Array-creation behavior of Paginable more predictable (#14687)
* Make Array-creation behavior of Paginable more predictable

Paginable.paginate_by_id usually returns ActiveRecord::Relation, but it
returns an Array if min_id option is present. The behavior caused problems
fixed with the following commits:
- 552e886b64
- b63ede5005
- 64ef37b89d

To prevent from recurring similar problems, this commit introduces two
changes:
- The scope now always returns an Array whether min_id option is present
  or not.
- The scope is renamed to to_a_paginated_by_id to clarify it returns an
  Array.

* Transform Paginable.to_a_paginated_by_id from a scope to a class method

https://api.rubyonrails.org/classes/ActiveRecord/Scoping/Named/ClassMethods.html#method-i-scope
> The method is intended to return an ActiveRecord::Relation object, which
> is composable with other scopes.

Paginable.to_a_paginated_by_id returns an Array and is not appropriate
as a scope.
2020-08-31 12:47:09 +02:00
Eugen Rochko
bf477cee9f
Fix videos with near-60 fps being rejected (#14684)
Fix #14668
2020-08-30 01:54:30 +02:00
santiagorodriguez96
e8d41bc2fe
Add WebAuthn as an alternative 2FA method (#14466)
* feat: add possibility of adding WebAuthn security keys to use as 2FA

This adds a basic UI for enabling WebAuthn 2FA. We did a little refactor
to the Settings page for editing the 2FA methods – now it will list the
methods that are available to the user (TOTP and WebAuthn) and from
there they'll be able to add or remove any of them.
Also, it's worth mentioning that for enabling WebAuthn it's required to
have TOTP enabled, so the first time that you go to the 2FA Settings
page, you'll be asked to set it up.
This work was inspired by the one donde by Github in their platform, and
despite it could be approached in different ways, we decided to go with
this one given that we feel that this gives a great UX.

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: add request for WebAuthn as second factor at login if enabled

This commits adds the feature for using WebAuthn as a second factor for
login when enabled.
If users have WebAuthn enabled, now a page requesting for the use of a
WebAuthn credential for log in will appear, although a link redirecting
to the old page for logging in using a two-factor code will also be
present.

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: add possibility of deleting WebAuthn Credentials

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: disable WebAuthn when an Admin disables 2FA for a user

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* feat: remove ability to disable TOTP leaving only WebAuthn as 2FA

Following examples form other platforms like Github, we decided to make
Webauthn 2FA secondary to 2FA with TOTP, so that we removed the
possibility of removing TOTP authentication only, leaving users with
just WEbAuthn as 2FA. Instead, users will have to click on 'Disable 2FA'
in order to remove second factor auth.
The reason for WebAuthn being secondary to TOPT is that in that way,
users will still be able to log in using their code from their phone's
application if they don't have their security keys with them – or maybe
even lost them.

* We had to change a little the flow for setting up TOTP, given that now
  it's possible to setting up again if you already had TOTP, in order to
  let users modify their authenticator app – given that now it's not
  possible for them to disable TOTP and set it up again with another
  authenticator app.
  So, basically, now instead of storing the new `otp_secret` in the
  user, we store it in the session until the process of set up is
  finished.
  This was because, as it was before, when users clicked on 'Edit' in
  the new two-factor methods lists page, but then went back without
  finishing the flow, their `otp_secret` had been changed therefore
  invalidating their previous authenticator app, making them unable to
  log in again using TOTP.

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>

* refactor: fix eslint errors

The PR build was failing given that linting returning some errors.
This commit attempts to fix them.

* refactor: normalize i18n translations

The build was failing given that i18n translations files were not
normalized.
This commits fixes that.

* refactor: avoid having the webauthn gem locked to a specific version

* refactor: use symbols for routes without '/'

* refactor: avoid sending webauthn disabled email when 2FA is disabled

When an admins disable 2FA for users, we were sending two mails
to them, one notifying that 2FA was disabled and the other to notify
that WebAuthn was disabled.
As the second one is redundant since the first email includes it, we can
remove it and send just one email to users.

* refactor: avoid creating new env variable for webauthn_origin config

* refactor: improve flash error messages for webauthn pages

Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
2020-08-24 16:46:27 +02:00
niwatori24
8b61bb1c3e
Cache result of SQL (#14534) 2020-08-09 15:23:42 +02:00
Takeshi Umeda
7540e235a2
Fix movie width and frame_rate returning nil (#14357)
* Fix movie width and frame_rate returning nil

* Add StreamValidationError and raise

* Fix code style
2020-07-19 22:28:27 +02:00
abcang
6a96af4d20
Fix rubocop warning (#14288)
* Fix rubocop warning

* use limit variable

* use ContextCreatingMethods option
2020-07-14 19:05:07 +02:00
ThibG
a8e84a18f1
Fix larger video files not being transcoded (#14306)
Since #14145, the `set_type_and_extension` has been moved from
`before_post_process` to `before_file_post_process`, but while the former
runs before all validations performed by Paperclip, the latter is dependent
on the order validations and hooks are defined.

In our case, this meant video files could be checked against the generic 10MB
limit, causing validation failures, which, internally, make Paperclip skip
post-processing, and thus, transcoding of the video file.

The actual validation would then happen after the type is correctly set, so
the large file would pass validation, but without being transcoded first.

This commit moves the hook definition so that it is run before checking for
the file size.
2020-07-14 18:50:19 +02:00
Eugen Rochko
99f3a55540
Add color extraction for audio thumbnails (#14209) 2020-07-05 18:28:25 +02:00
Eugen Rochko
6d23d40420
Change Redis#exists calls to Redis#exists? to avoid deprecation warning (#14191) 2020-07-01 19:05:21 +02:00
Eugen Rochko
7aaf2b44ec
Fix remote files not using Content-Type header, streaming (#14184) 2020-06-30 23:58:02 +02:00
ThibG
65506bac3f
Add user notes on accounts (#14148)
* Add UserNote model

* Add UI for user notes

* Put comment in relationships entity

* Add API to create user notes

* Copy user notes to new account when receiving a Move activity

* Address some of the review remarks

* Replace modal by inline edition

* Please CodeClimate

* Button design changes

* Change design again

* Cancel note edition when pressing Escape

* Fixes

* Tweak design again

* Move “Add note” item, and allow users to add notes to themselves

* Rename UserNote into AccountNote, rename “comment” Relationship attribute to “note”
2020-06-30 19:19:50 +02:00
Eugen Rochko
1b198d6489
Fix trying to write non-existent image remote URL attribute on preview cards (#14181)
Regression from #14145
2020-06-29 17:59:04 +02:00
Eugen Rochko
64aac30733
Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Eugen Rochko
662a49dc3f
Fix various issues around OpenGraph representation of media (#14133)
- Fix audio attachments not being represented in OpenGraph tags
- Fix audio being represented as "1 image" in OpenGraph descriptions
- Fix video metadata being overwritten by paperclip-av-transcoder
- Fix embedded player not using Mastodon's UI
- Fix audio/video progress bars not moving smoothly
- Fix audio/video buffered bars not displaying correctly
2020-06-25 01:33:01 +02:00
Eugen Rochko
72a7cfaa39
Add e-mail-based sign in challenge for users with disabled 2FA (#14013) 2020-06-09 10:23:06 +02:00
ThibG
346d9b3d2e
Add blurhash to preview cards (#13984)
Fixes #13001
2020-06-05 23:10:41 +02:00
Eugen Rochko
5d8398c8b8
Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
ThibG
51ff679b9d
Fix admin page crashing when trying to block an invalid domain name (#13884)
* Fix admin page crashing when trying to block an invalid domain name

Fixes #13880

* Fix trailing and leading spaces not being properly stripped for domain blocks
2020-06-01 03:47:20 +02:00
Takeshi Umeda
8e056bd82e
Fix csv upload (#13835) 2020-05-24 09:15:23 +02:00
Takeshi Umeda
e6706d171a
Fix searchable status without explicitly mentioning it (#13829) 2020-05-23 05:47:25 +02:00
dependabot-preview[bot]
78202e9138
Bump doorkeeper from 5.3.3 to 5.4.0 (#13733)
* Bump doorkeeper from 5.3.3 to 5.4.0

Bumps [doorkeeper](https://github.com/doorkeeper-gem/doorkeeper) from 5.3.3 to 5.4.0.
- [Release notes](https://github.com/doorkeeper-gem/doorkeeper/releases)
- [Changelog](https://github.com/doorkeeper-gem/doorkeeper/blob/master/CHANGELOG.md)
- [Commits](https://github.com/doorkeeper-gem/doorkeeper/compare/v5.3.3...v5.4.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Fix tests

* Fix use of Doorkeeper::AccessToken.find_or_create_for

* Fix tests?

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Thibaut Girka <thib@sitedethib.com>
2020-05-12 15:25:33 +02:00
Eugen Rochko
4b766f9846
Refactor monkey-patching of Goldfinger (#12561) 2020-05-10 11:41:43 +02:00
Takeshi Umeda
26b08a3c54
Add remote only to public timeline (#13504)
* Add remote only to public timeline

* Fix code style
2020-05-10 10:36:18 +02:00
ThibG
a4240fd027
Improve RSS entries for statuses (#13592)
* Improve RSS entries for statuses

- Render polls in both accounts and tags serializers
- Refactor RSS serializers
- Change title preview to include ellipsis when truncated
- Change title preview to show CW instead of toot text
- Add tests

* Remove title from OEmbed serialization

Twitter doesn't serialize title either, and tihs allows us to move the
title formatting code to the RSS serializers.
2020-05-10 09:50:54 +02:00
Taras Gogol
6748a5acb1
Fix followings list order | Issue #13538 (#13676) 2020-05-08 20:17:16 +02:00
Yamagishi Kazutoshi
e223fd8c61
Revert "improve status title (#8596)" (#13591)
This reverts commit 05756c9a14.
2020-05-03 18:48:13 +02:00
Eugen Rochko
988b0493fe
Add more tests for ActivityPub controllers (#13585) 2020-05-03 16:30:36 +02:00
kaiyou
3511528e50
Only check locally when deduplicating usernames (#13581)
When deduplicating account usernames for OAuthable users, the routine did check if any account was known with that username, including remote accounts. This caused some unnecessary deduplication, and usernames ending with unexpected trailing _1.
This fixes #13580
2020-04-30 14:39:05 +02:00
Eugen Rochko
c3ca3801f2
Add separate cache directory for non-local uploads (#12821) 2020-04-26 23:29:08 +02:00
Takeshi Umeda
46b2cc184f
Fix enable/disable relay failures (#13535) 2020-04-23 22:04:18 +02:00
ThibG
3825e1943f
Fix confusing error when failing to add an alias to an unknown account (#13480)
Follow-up to #13452, fixing broken `uri.nil?` test.

Also remove the separate check for `uri` presence, as that would result
in a “Please review 2 errors below” while only one would be listed.
2020-04-15 20:33:53 +02:00
Eugen Rochko
5edff32733
Change delivery failure tracking to work with hostnames instead of URLs (#13437) 2020-04-15 20:33:24 +02:00
ThibG
5524258da9
Fix “Email changed” notification sometimes having wrong e-mail (#13475)
* Fix “Email changed” notification sometimes having wrong e-mail

Fixes #6778

The root of the issue is that `send_devise_notification` was called before
the changes were properly commited to the database, causing the mailer to
pick previous values if running too early.

Devise's documentation provides guidance on how to handle that[1][2], however,
I have found it to not be working, as the following happens, in that order:
- `send_devise_notification` is called for the `email_changed` notification.
  In that case, `changed?` is false and `saved_changes?` is true, so
  if we use the former, we have the same issue.
- the `after_commit` hook is called
- `send_devise_notification` is called for the `confirmation_instructions`
  notification.
  In that case, `changed?` is still false, and `saved_changes?` still true,
  so if we use the latter, that second notification email is simply not
  going to be sent (as we would be queuing the notification *after*
  executing the after_commit hook).

This is because it may be called from either an `after_update` or
`after_commit` hook, the difference not being a call to `save` but the
transaction actually being committed to the database. This may arguably
be a bug in Devise, or Devise's notification.

The proposed workaround is inspired by Devise's documentation but checks
whether a transaction is open to make the call whether to immediately
send the notification or defer it to the `after_commit` hook.

[1]: https://www.rubydoc.info/github/plataformatec/devise/Devise%2FModels%2FAuthenticatable:send_devise_notification
[2]: 406915cb78/lib/devise/models/authenticatable.rb (L133-L194)

* Fix cases when sending notifications without changing the model

* Defer sending if and only if in transaction including current record
2020-04-15 16:13:44 +02:00
ThibG
f7e011919e
Fix account aliases page (#13452)
* Fix error not being displayed when adding an account alias, add error for self-references

Co-Authored-By: Mélanie Chauvel (ariasuni) <perso@hack-libre.org>

* Add “You have no aliases.” note in confusing empty aliases table

Co-Authored-By: Mélanie Chauvel (ariasuni) <perso@hack-libre.org>

Co-authored-by: Mélanie Chauvel (ariasuni) <perso@hack-libre.org>
2020-04-13 06:41:43 +02:00
Eugen Rochko
c9efb400b4
Add rate limit for reporting (#13390) 2020-04-05 14:40:08 +02:00
ThibG
89e28c7674
Fix PostgreSQL load when linking in announcements (#13250)
* Fix PostgreSQL load when linking in announcements

Fixes #13245 by caching status lookups

Since statuses are supposed to be known already and we only
need their URLs and a few other things, caching them should
be fine.

Since it's only used by announcements so far, there won't
be much statuses to cache.

* Perform status lookup when saving announcements, not when rendering them

* Change EntityCache#status to fetch URLs instead of looking into the database

* Move announcement link lookup to publishing worker

* Address issues pointed out during review
2020-04-05 12:51:22 +02:00
Eugen Rochko
f65568f1d4
Add ability to filter audit log in admin UI (#13381) 2020-04-03 13:06:34 +02:00
Eugen Rochko
0c8945e5ff
Change tootctl media remove-orphans to work for all classes (#13316)
Change `tootctl media lookup` to not use an interactive prompt
2020-03-26 01:56:41 +01:00
Jeong Arm
8c42e0b53c
Make bookmarks also searchable (#13271) 2020-03-21 03:14:10 +01:00
ThibG
61f35c1a8a
Fix reported accounts not being whitelisted when resolving a spamcheck report (#13289) 2020-03-21 03:08:09 +01:00
Eugen Rochko
bea0bb39d6
Add option to include resolved DNS records when blacklisting e-mail domains in admin UI (#13254)
* Add shortcuts to blacklist a user's e-mail domain in admin UI

* Add option to blacklist resolved MX and IP records for e-mail domains
2020-03-12 22:35:20 +01:00
Eugen Rochko
f556f79b77
Add titles to warning presets in admin UI (#13252) 2020-03-12 17:57:59 +01:00
ThibG
2c6099125d
Fix videos with unsupported colorspace not being transcoded (#13242) 2020-03-10 11:58:40 +01:00
ThibG
abd8394880
Fix MP4 (H264 + AAC) video files being needlessly re-encoded (#13239) 2020-03-09 23:15:59 +01:00
Eugen Rochko
8a2b8e0321
Change video uploads to always be converted to H264/MP4 (#13220)
Even if the container format is the same (.mp4), the codec could
be different and not playable in web browsers
2020-03-09 02:20:18 +01:00
Eugen Rochko
dc15c81e67
Change video uploads to enforce certain limits (#13218)
- Dimensions at most 1920x1200
- Frame rate at most 60
2020-03-09 02:19:07 +01:00
ThibG
b154428e14
Add federation support for the "hide network" preference (#11673)
* Change ActivityPub follower/following collections to not link first page

* Add support for hiding followers and following of remote users

* Switch to using a single `hide_collections` column

* Address code style remarks
2020-03-09 00:10:29 +01:00
Eugen Rochko
9660aa4543
Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
2020-03-08 23:56:18 +01:00
ThibG
aa67036b41
Add support for links to statuses in announcements to be opened in web UI (#13212)
* Add support for links to public statuses in announcements to be opened in WebUI

* Please CodeClimate
2020-03-08 16:10:48 +01:00
ThibG
4a4cd686c1
Add sorting by username, creation and last activity in moderation view (#13076)
* Add ability to order accounts in moderation view

* Display last status date in “Most recent activity” for remote users
2020-03-08 15:39:13 +01:00
Eugen Rochko
339ce1c4e9
Add specific rate limits for posting and following (#13172) 2020-03-08 15:17:39 +01:00
Eugen Rochko
d081a80cff
Fix "tootctl media remove-orphans" crashing when encountering invalid media (#13170)
Fixes #13168
2020-03-05 15:56:01 +01:00
ThibG
7b63c5469e
Fix announcements with fully-qualified mention to local user crashing WebUI (#13164) 2020-02-28 13:49:45 +01:00
Eugen Rochko
ff07e80533
Fix remote convertible media attachments not being saved (#13032)
Because the file name was set after loading the file, Paperclip
was flagging the differing file extension as content type spoofing

Fix #12938
2020-02-03 18:45:56 +01:00
abcang
61a7390b66
Search account domain in lowercase (#13016)
* Search account domain in lowercase

* fix rubocop error

* fix spec/models/account_spec.rb
2020-02-01 15:42:24 +01:00
Eugen Rochko
663ea84b08
Add publish/unpublish controls to announcements in admin UI (#12967) 2020-01-27 11:05:33 +01:00
Eugen Rochko
401f32f9ee
Fix expired announcements being re-published (#12964) 2020-01-26 22:43:18 +01:00
Eugen Rochko
b9d74d4076
Add streaming API updates for announcements being modified or deleted (#12963)
Change `all_day` to be a visual client-side cue only

Publish immediately if `scheduled_at` is in the past

Add `published_at` and `updated_at` to announcements JSON
2020-01-26 20:07:26 +01:00
Eugen Rochko
71921f6bc3
Fix user disabling changing activity timestamps, fix nil error (#12943) 2020-01-25 05:22:35 +01:00
Eugen Rochko
daf71573d0
Fix password change/reset not immediately invalidating other sessions (#12928)
While making browser requests in the other sessions after a password
change or reset does not allow you to be logged in and correctly
invalidates the session making the request, sessions have API tokens
associated with them, which can still be used until that session
is invalidated.

This is a security issue for accounts that were already compromised
some other way because it makes it harder to throw out the hijacker.
2020-01-24 00:20:38 +01:00
Eugen Rochko
ce1dee85b5
Fix relationships page not showing results in admin UI (#12934)
Follow-up to #12927
2020-01-24 00:20:23 +01:00
Eugen Rochko
f52c988e12
Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
Eugen Rochko
81cc86bb1f
Fix media attachments without file being uploadable (#12562)
Fix #12554
2020-01-23 21:40:03 +01:00
ThibG
43daeccccb Add “account timeline” filter category (#12918)
* Add “account timeline” filter category

Previously, no filter category applied to account timelines.

* Rename “Account timelines” into “Profiles”
2020-01-23 21:32:00 +01:00
Eugen Rochko
c0006a004d
Change followers page to relationships page in admin UI (#12927)
Allow browsing and filtering all relationships instead of just
followers, unify the codebase with the user-facing relationship
manager, add ability to see who the user invited
2020-01-23 20:33:20 +01:00
Eugen Rochko
02063c245c
Fix not all of account's active IPs showing up in admin UI (#12909) 2020-01-20 18:00:54 +01:00
Eugen Rochko
ba16d4b413
Fix search by IP not using alternative browser sessions in admin UI (#12904) 2020-01-20 17:00:41 +01:00
Eugen Rochko
6feafb8802
Various fixes and improvements (#12878)
* Fix unused role routes being generated

* Remove unused JavaScript code

* Refactor filters code to be DRYer

* Fix `.count == 0` comparisons to `.empty?` in views

* Fix filters in views
2020-01-20 15:55:03 +01:00
Eugen Rochko
3287ec8ca3
Fix file names being obfuscated on update when file didn't change (#12857)
Fix #12849
2020-01-13 12:54:33 +01:00
ThibG
57e2833f6a Remove dependency on OStatus2 gem (#12822) 2020-01-11 21:36:53 +01:00
ysksn
6f8f018e3e Refactor StatusThreadingConcern (#9626)
* Remove #filter_from_context?

* Create scope Status.with_accounts

Retrieving AR objects should be
their model's scope
2020-01-11 19:55:33 +09:00
Alexander
05756c9a14 improve status title (#8596)
* improve shown status title, useful for atom/rss

* use single quotes to satisfy codeclimate

* fix tests, make message more pretty

* fix tests

* fix codestyle

* fix codestyle

* remove atom_serializer_spec

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-01-11 06:58:16 +09:00
ThibG
51eb111503 Allow blocking TLDs, and fix TLD blocks not being editable (#12805)
Fixes #12795

It was already possible to create domain blocks for TLDs, but those
weren't enforced, nor editable. This commit changes it so that they
are enforced and editable.
2020-01-08 22:42:05 +01:00
Eugen Rochko
49b2f7c0a2
Fix base64-encoded file uploads not being possible (#12748)
Fix #3804, Fix #5776
2020-01-04 01:54:07 +01:00
Eugen Rochko
f86ee4b59f
Fix IDN mentions not being processed, IDN domains not being rendered (#12715)
This changes the REST API to return unicode domains in the `acct`
attribute instead of punycode, and to render unicode instead of
punycode on public HTML pages as well.

Fix #7812, fix #12246
2019-12-30 19:20:43 +01:00
ThibG
c0ed53b810 Fix custom emoji category creation silently erroring out on duplicate category (#12647)
Instead, just re-use the existing category if any.

Fixes #12608
2019-12-18 16:56:59 +01:00
ThibG
da2143b308 Fixes featured hashtag setting page erroring out instead of rejecting invalid tags (#12436)
* Revert "Fix ignoring whole status because of one invalid hashtag (#11621)"

This reverts commit dff46b260b.

* Fix statuses being rejected because of invalid hashtag names

* Add spec for invalid hashtag names in statuses

* Add test for featured tags controller
2019-12-17 13:31:56 +01:00
scd31
3830c0b741 Increase max backup size (#12602)
* Increased max backup size

* partially reverted schema.rb
2019-12-17 13:30:58 +01:00
Alice Gaudon
668f698077 Admin setting to disable default follows (#12566) 2019-12-16 23:55:50 +01:00
Eugen Rochko
f3d232381d
Add tootctl media remove-orphans (#12568) 2019-12-08 15:37:12 +01:00
Takeshi Umeda
f43f1e0184 Add basic support for group actors (#12071)
* Show badge on group actor in WebUI

* Do not notify in case of  by following group actor

* If you mention group actor, also mention group actor followers

* Relax characters that can be used in username (same as Application)

* Revert "Relax characters that can be used in username (same as Application)"

This reverts commit 7e10a137b8.

* Delete display_name method
2019-12-04 20:36:33 +01:00
Yamagishi Kazutoshi
6be16d02cb Update ESLint and RuboCop in Code Climate (#12534) 2019-12-02 18:25:43 +01:00
Mathieu Brunot
bd8dc9bd0c Add an LDAP Mail attribute config (#12053)
Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>
2019-12-01 18:52:21 +01:00
ThibG
911cc14481 Add follow_request notification type (#12198)
* Add follow_request notification type

The notification type already existed in the backend but was never pushed
to the front-end. This also means translation strings were also available
for the backend, from the notification mailer.

Unlike other notification types, these are off by default, to match what
I remember of Gargron's view on the topic: that follow requests should not
clutter notifications and should instead be reviewed at the user's own
leisure in the dedicated column.

Since follow requests have their own column, I've deemed it unnecessary to
add a specific tab for them in the notification quick filter.

* Show follow request link in single-column if there are pending requests, even if account isn't locked

* Push follow requests from notifications to the follow_requests list

* Offer to accept or reject follow request from the notification

* Redesign follow request notification
2019-12-01 17:25:29 +01:00
ThibG
f60cd97638 Only normalize local polls (#12515)
Before this patch, if remote poll options have leading or trailing spaces,
the information stored locally won't match them, causing federated voting to
fail.
2019-12-01 17:24:33 +01:00
Mathieu Brunot
d70268f099 Convert LDAP username (#12461)
*  Convert LDAP username #12021

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 🐛 Fix conversion var use

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 🐛 Fix LDAP uid conversion test

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 👌 Remove comments with ref to PR

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 👌 Remove unnecessary paranthesis

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>

* 🔧 Move space in conversion string

Signed-off-by: mathieu.brunot <mathieu.brunot@monogramm.io>
2019-12-01 07:21:28 +01:00
ThibG
d8f96028c5 Add ability to filter reports by target account domain (#12154)
* Add ability to filter reports by target account domain

* Reword by_target_domain label
2019-11-30 19:53:58 +01:00
ThibG
2edf6d81cb Simplify SQL query when performing account search amongst followings (#12302) 2019-11-21 17:35:04 +01:00
ThibG
dfea7368c9 Add bookmarks (#7107)
* Add backend support for bookmarks

Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.

* Add spec for bookmark endpoints

* Add front-end support for bookmarks

* Introduce OAuth scopes for bookmarks

* Add bookmarks to archive takeout

* Fix migration

* Coding style fixes

* Fix rebase issue

* Update bookmarked_statuses to latest UI changes

* Update bookmark actions to properly reflect status changes in state

* Add bookmarks item to single-column layout

* Make active bookmarks red
2019-11-13 23:02:10 +01:00
ThibG
65e13cfacf Add abilityto add oneself to lists (#12271)
* Add ability to add oneself to lists

* Change search results to include oneself when searching through followers

* Mark follow relation as optional in ListAccount
2019-11-04 13:02:01 +01:00
ThibG
650820d62d Fix remote media descriptions being cut off at 420 chars (#12262)
* Fix remote media descriptions being cut off at 420 chars

Fixes #12258

* Fix tests
2019-11-04 13:00:16 +01:00
Faye Duxovni
48f75b86ae Add setting for whether to crop images in unexpanded toots (#12126) 2019-10-24 22:51:41 +02:00
Hugo Gameiro
488dd0ff7a remove audio metadata (#12171) 2019-10-24 22:47:58 +02:00
Eugen Rochko
915f3712ae
Fix admin setting to auto-approve hashtags not affecting query (#12130)
Follow-up to #12122
2019-10-10 02:22:04 +02:00
Eugen Rochko
b5f7e12817
Remove auto-silence behaviour from spam check (#12117)
Fix #12113
2019-10-09 07:11:23 +02:00
Eugen Rochko
354fdd317e
Fix attachment not being re-downloaded even if file is not stored (#12125)
Change the behaviour of remotable concern. Previously, it would skip
downloading an attachment if the stored remote URL is identical to
the new one. Now it would not be skipped if the attachment is not
actually currently stored by Paperclip.
2019-10-09 07:10:46 +02:00
Eugen Rochko
c8bcf5cbfd
Add admin setting to auto-approve hashtags (#12122)
Change inaccurate labels on other admin settings
2019-10-09 00:30:15 +02:00
Eugen Rochko
f665901e3c
Fix performance of home feed regeneration (#12084)
Fetching statuses from all followed accounts at once takes too long
within Postgres. Fetching them one by one and merging in Ruby
could be a lot less resource-intensive

Because the query for dynamically fetching the home timeline is so
heavy, we can no longer offer it when the home timeline is missing
2019-10-06 22:11:17 +02:00
Hugo Gameiro
a7917269d3 add loglevel fatal to video and audio styles (#12088) 2019-10-06 19:48:26 +02:00
Eugen Rochko
ca22a22d7f
Fix performance of GIF re-encoding (#12057)
* Change animated GIF detection to not shell out to ImageMagick

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Change video encoding parameters to limit to 10800 video frames

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Limit GIF image size further

Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>

* Always strip metadata from video files

* Fix code style issues
2019-10-03 01:09:12 +02:00
Eugen Rochko
575dc11cb2
Fix needlessly indexing unsearchable statuses into ElasticSearch (#12041) 2019-10-02 20:04:46 +02:00
Eugen Rochko
62f60e86c2
Fix account counters being overwritten by parallel writes (#12045) 2019-10-02 04:59:37 +02:00
Eugen Rochko
5c42f47617
Fix records not being indexed sometimes (#12024)
It's possible that after commit callbacks were not firing when
exceptions occurred in the process. Also, the default Sidekiq
strategy does not push indexing jobs immediately, which is not
necessary and could be part of the issue too.
2019-10-01 01:19:11 +02:00
ThibG
3babf8464b Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko
163ed91af3
Add (back) option to set redirect notice on account without moving followers (#11994)
Fix #11913
2019-09-29 05:03:19 +02:00
Eugen Rochko
50af41a00d
Fix unscoped being used in the wrong place (#11987)
Regression from #11982
2019-09-28 05:23:32 +02:00
Eugen Rochko
3ec80c7aec
Fix preview card image not being re-fetched even if link is re-posted (#11981)
Fix #11956
2019-09-28 01:33:16 +02:00
Eugen Rochko
32ff78f749
Fix index not being used in Status.reblogs_map (#11982)
Regression from #11623
2019-09-28 01:33:02 +02:00
Eugen Rochko
ab33c4df94
Add exclude_unreviewed param to GET /api/v2/search REST API (#11977)
Make it so normal search returns even unreviewed matches, but
autosuggestions do not.

Fix #11960
2019-09-28 01:02:21 +02:00
abcang
07b057eabb Validate Web::PushSubscription (#11971) 2019-09-27 15:24:13 +02:00
Eugen Rochko
add4d4118c
Fix relays UI being available in whitelist/secure mode (#11963)
Fix relays UI referencing relay that is not functional
2019-09-27 02:13:34 +02:00
Eugen Rochko
a1f04c1e34
Fix authentication before 2FA challenge (#11943)
Regression from #11831
2019-09-24 04:35:36 +02:00
ThibG
b359974d9b Show user what options they have voted (#11195)
* Add own_votes field to poll results in REST API

Fixes #10679

* Display user votes in WebUI

* Update styling

* Add vote checkmark to public pages
2019-09-22 14:15:18 +02:00
Takeshi Umeda
b18aea91e3 Accept acct starting with @ in account migration (#11907) 2019-09-21 09:11:38 +02:00
Takeshi Umeda
f497d14b19 Addition of update activity distribution by alias, minor correction (#11905)
* Addition of update activity distribution by alias, minor correction

* Distribute Update activity after adding alias
* Add uniqueness verification to alias uri
* accept acct starting with @

* fix double-quoted to single-quoted
2019-09-21 09:11:21 +02:00
Eugen Rochko
3ed94dcc1a
Add account migration UI (#11846)
Fix #10736

- Change data export to be available for non-functional accounts
- Change non-functional accounts to include redirecting accounts
2019-09-19 20:58:19 +02:00
Eugen Rochko
d930eb88b6
Add table of contents to about page (#11885)
Move public domain blocks information to about page
2019-09-19 11:09:05 +02:00
Eugen Rochko
e1066cd431
Add password challenge to 2FA settings, e-mail notifications (#11878)
Fix #3961
2019-09-18 16:37:27 +02:00
Eugen Rochko
a4b60e9ba4
Fix TOTP codes not being filtered from logs during enabling/disabling (#11877)
Not a serious issue because they are meaningless past single use
2019-09-18 02:48:40 +02:00
han@highemelry
3919571c39 Fix audit log error when custom emoji is copied from remote server (#11876) 2019-09-17 23:20:48 +02:00
mayaeh
0e5b9e3ba0 Fix the sample account icon of Profile directory to local only (#11872) 2019-09-17 10:06:43 +02:00
Eugen Rochko
3d14b8f28f
Fix SSO login not using existing account when e-mail is verified (#11862)
Fix #11472
2019-09-16 20:42:29 +02:00
mayaeh
ef0d22f232 Add search and sort functions to hashtag admin UI (#11829)
* Add search and sort functions to hashtag admin UI

* Move scope processing from tags_controller to tag_filter

* Fix based on method naming conventions

* Fixed not to get 500 errors for invalid requests
2019-09-16 14:27:29 +02:00
Eugen Rochko
c707ef49d9
Fix 2FA challenge and password challenge for non-database users (#11831)
* Fix 2FA challenge not appearing for non-database users

Fix #11685

* Fix account deletion not working when using external login

Fix #11691
2019-09-15 21:08:39 +02:00
Eugen Rochko
0762258aec
Fix hashtags being split by ZWNJ character (#11821)
Fix #11761
2019-09-13 16:01:26 +02:00
Eugen Rochko
59da5ccb8e
Change unlisted custom emoji to not appear in autosuggestions (#11818)
Fix #11669
2019-09-13 16:01:09 +02:00
Eugen Rochko
b6381bdc7d
Change max length of media descriptions from 420 to 1500 characters (#11819)
Fix #11658
2019-09-13 16:00:34 +02:00
Eugen Rochko
c5d37f18cb
Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Tao Bror Bojlén
4fe127664b add admin setting for default search engine indexing (fix #11750) (#11804) 2019-09-11 08:44:58 +02:00
Eugen Rochko
031ca25014
Add retry for failed media downloads and tootctl media refresh (#11775) 2019-09-10 15:29:12 +02:00
Eugen Rochko
8674814825
Change tootctl to use inline parallelization instead of Sidekiq (#11776)
- Remove --background option
- Add --concurrency(=5) option
- Add progress bars
2019-09-10 13:48:48 +02:00
Eugen Rochko
1110ea1a91
Add batch actions and categories to admin UI for custom emojis (#11793) 2019-09-09 22:44:17 +02:00
Eugen Rochko
261e52268c
Add batch approve/reject for pending hashtags in admin UI (#11791) 2019-09-09 12:50:09 +02:00
Eugen Rochko
a75009a65e
Change half-life of trend decay (#11774) 2019-09-08 19:17:57 +02:00
Eugen Rochko
e445a8af64
Add timeline read markers API (#11762)
Fix #4093
2019-09-06 13:55:51 +02:00
Eugen Rochko
70ddef2654
Change trending hashtags to not disappear instantly after midnight (#11712) 2019-09-02 18:11:13 +02:00
umonaca
7235f538c6 Add more media formats (#11687)
* Add more media formats

* Removed amr file type
2019-08-30 04:30:29 +02:00
Eugen Rochko
5f591d979c
Fix non-numbers being passed to Redis and causing an error (#11697) 2019-08-30 02:49:54 +02:00
Eugen Rochko
b54b725d6b
Fix uncaught domain normalization error in remote follow (#11703) 2019-08-30 02:19:17 +02:00
Eugen Rochko
cb447b28c4
Add profile directory to web UI (#11688)
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Eugen Rochko
73ca0bb925
Add option to include reported statuses in warning e-mail (#11639) 2019-08-23 22:37:23 +02:00
Eugen Rochko
282ea17078
Add soft delete for statuses for instant deletes through API (#11623)
* Add soft delete for statuses to allow them to appear instant

* Allow reporting soft-deleted statuses and show them in the admin UI

* Change index for getting an account's statuses
2019-08-22 21:55:56 +02:00
Eugen Rochko
97192d9a77
Fix remote and staff-removed statuses leaving media behind for a day (#11638)
The reason for unattaching media instead of removing it is to support
delete & redraft functionality, but remote or staff-removed statuses
will never be redrafted, so the media should be deleted immediately
2019-08-22 04:17:12 +02:00
Eugen Rochko
dff46b260b
Fix ignoring whole status because of one invalid hashtag (#11621)
Fix #11618
2019-08-19 20:36:44 +02:00
ThibG
cb62a83a71 Add invite comments (#10465) 2019-08-19 11:40:42 +02:00