Commit Graph

733 Commits

Author SHA1 Message Date
ThibG
c656cc2191 Fix FetchLinkCardServices crashing on a tags without a target (#12159)
* Add test for links without targets

* Fix FetchLinkCardServices crashing on a tags without a target
2019-11-21 16:04:52 +01:00
ThibG
c4d45765e3 Change domain block behavior to update user records before deleting data (#12247)
When suspending or silencing a domain, one probably wants the silence or
suspend parts to take effect as soon as possible. Deleting media files can
take a while (possibly days), so perform silencing as soon as possible, and
clean up media afterwards.
2019-11-21 11:36:15 +01:00
Eugen Rochko
d14e74eff5
Add cache for OEmbed endpoints to avoid extra HTTP requests (#12403)
* add youtube oembed endpoint

* add check for oembed endpoint

* change unless for a more readable if

* clear blank lines

* endpoint via https

* Fix string literal in condition

* use cache for endpoints

* use cache for endpoints

* clean up and adding check

* clean up and remove redundant return

* add html check

* add false to return

* use double quotes

* use double quotes

* Clean up
2019-11-17 18:40:33 +01:00
ThibG
dfea7368c9 Add bookmarks (#7107)
* Add backend support for bookmarks

Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.

* Add spec for bookmark endpoints

* Add front-end support for bookmarks

* Introduce OAuth scopes for bookmarks

* Add bookmarks to archive takeout

* Fix migration

* Coding style fixes

* Fix rebase issue

* Update bookmarked_statuses to latest UI changes

* Update bookmark actions to properly reflect status changes in state

* Add bookmarks item to single-column layout

* Make active bookmarks red
2019-11-13 23:02:10 +01:00
ThibG
0d5f3ae331 Fix already-known remote private toots not being searchable by URL (#12336)
* Fix already-known remote private toots not being searchable by URL

Whenever an URL search fails, and an account is provided, try getting a
private status matching that URL from database.

* Fix searching by `id` field instead of `uri`
2019-11-11 00:04:48 +02:00
ThibG
66c1fe0495 Fix various issues with account migration (#12301)
* Fix being able to follow oneself by moving to an account that was following the old one

* Add specs

* Add spec to catch MoveWorker issue with local followers following both accounts

* Fix move worker breaking when a local account follows both source and target accounts

* Fix migration from remote to local account not sending Undo Follow

* Fix show_reblogs not being preserved for moved account's followers
2019-11-07 09:05:07 +02:00
ThibG
65e13cfacf Add abilityto add oneself to lists (#12271)
* Add ability to add oneself to lists

* Change search results to include oneself when searching through followers

* Mark follow relation as optional in ListAccount
2019-11-04 13:02:01 +01:00
nightpool
9762fe382c microformat mentions can have an implicit property (#12189)
See the first example here: http://microformats.org/wiki/microformats2#hyperlinked_person
2019-10-24 22:46:15 +02:00
Eugen Rochko
8386d9ec10
Fix hashtag timeline REST API accepting too many hashtags (#12091) 2019-10-07 03:56:21 +02:00
Eugen Rochko
eb83d6256e
Add reason param to POST /api/v1/accounts REST API (#12064)
For approval-required registrations mode
2019-10-03 17:50:59 +02:00
ThibG
3a4d994c40 Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid (#12037)
* Add test to handle suspended and missing users in BootstrapTimelineService

* Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid
2019-10-01 15:10:00 +02:00
Eugen Rochko
9ba40a6bfd
Remove HEAD request from fetching link previews (#12028)
It is not really necessary and we need to reduce requests
2019-10-01 04:54:10 +02:00
abcang
9ef5532a7e Fix follow requests N+1 (#12004) 2019-09-30 17:02:03 +02:00
ThibG
3babf8464b Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko
368a87755b
Fix account migration not affecting followers on origin server (#11980) 2019-09-29 16:23:13 +02:00
Eugen Rochko
3ec80c7aec
Fix preview card image not being re-fetched even if link is re-posted (#11981)
Fix #11956
2019-09-28 01:33:16 +02:00
Eugen Rochko
ab33c4df94
Add exclude_unreviewed param to GET /api/v2/search REST API (#11977)
Make it so normal search returns even unreviewed matches, but
autosuggestions do not.

Fix #11960
2019-09-28 01:02:21 +02:00
ThibG
18b451c0e6 Change silences to always require approval on follow (#11975)
* Change silenced accounts to require approval on follow

* Also require approval for follows by people explicitly muted by target accounts

* Do not auto-accept silenced or muted accounts when switching from locked to unlocked

* Add `follow_requests_count` to verify_credentials

* Show “Follow requests” menu item if needed even if account is locked

* Add tests

* Correctly reflect that follow requests weren't auto-accepted when local account is silenced

* Accept follow requests from user-muted accounts to avoid leaking mutes
2019-09-27 21:13:51 +02:00
Eugen Rochko
2f90a38f44
Fix unreviewed hashtag not being found by exact case-insensitive match (#11976) 2019-09-27 19:40:26 +02:00
Jeong Arm
00d7bdcc2a Fix search error when ElasticSearch is enabled but not available (#11954)
* Fallback to Database search when ES not available

* Prevent double work if ES gives 0 result

* Apply suggestion from code review
2019-09-26 18:06:08 +02:00
Eugen Rochko
4f6af87906
Change spam check to apply to local accounts and add a threshold (#11806)
Instead of detecting spam on first duplicate message, add a
threshold of 5 such messages to reduce false positives
2019-09-18 12:53:13 +02:00
Eugen Rochko
38dc51b2d6
Fix Move handler queuing jobs that will fail if account is suspended (#11864)
Don't put Move handler on cooldown if it didn't run. Skip unmerging
from timelines to save unnecessary work.
2019-09-17 08:44:45 +02:00
Eugen Rochko
50edf84898
Fix search API not resolving URL when type is given (#11822)
Fix #11771
2019-09-13 16:01:39 +02:00
Eugen Rochko
c5d37f18cb
Change deletes to preserve soft-deleted statuses in unresolved reports (#11805)
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Eugen Rochko
43f56f1291
Change account deletion page to have better explanations (#11753)
Fix deletion of unconfirmed account not freeing up the username

Add prefill of logged-in user's email in the reconfirmation form
2019-09-04 04:13:54 +02:00
Eugen Rochko
cb447b28c4
Add profile directory to web UI (#11688)
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Eugen Rochko
97192d9a77
Fix remote and staff-removed statuses leaving media behind for a day (#11638)
The reason for unattaching media instead of removing it is to support
delete & redraft functionality, but remote or staff-removed statuses
will never be redrafted, so the media should be deleted immediately
2019-08-22 04:17:12 +02:00
Eugen Rochko
cc0a55cf9a
Add more accurate hashtag search (#11579)
* Add more accurate hashtag search

Using ElasticSearch to index hashtags with edge n-grams and score
them by usage within the last 7 days since last activity. Only
hashtags that have been reviewed and are listable can appear in
searches, unless they match the query exactly

* Fix search analyzer dropping non-ascii characters
2019-08-18 03:45:51 +02:00
Eugen Rochko
70da6d6630
Fix accounts search by full/partial display name and others (#11580)
- Restrict followers counts to local users to minimize local advantage
- Fix emoji shortcodes causing error in search
- Fix search syntax parse errors not being caught
2019-08-16 13:00:30 +02:00
Eugen Rochko
8fdff2748f
Add more accurate account search (#11537)
* Add more accurate account search

When ElasticSearch is available, a more accurate search is implemented:

- Using edge n-gram index for acct and display name
- Using asciifolding and cjk width normalization on display names
- Using Gaussian decay on account activity for additional scoring (recency)
- Using followers/friends ratio for additional scoring (spamminess)
- Using followers number for additional scoring (size)

The exact match precedence only takes effect when the input conforms
to the username format and the username part of it is complete, i.e.
when the user started typing the domain part.

* Support single-letter usernames

* Fix tests

* Fix not picking up account updates

* Add weights and normalization for scores, skip zero terms queries

* Use local counts for accounts index, adjust search parameters

* Fix mistakes

* Using updated_at of accounts is inadequate for remote accounts
2019-08-16 01:24:03 +02:00
Eugen Rochko
c5661b0f92
Fix account search always returning exact match on paginated results (#11525)
Fix #11365
2019-08-11 13:14:51 +02:00
Eugen Rochko
f51c7c105f
Fix acct URIs with IDN domains not being resolved (#11520)
Fix #11494
2019-08-07 21:14:08 +02:00
ThibG
bced70469a Add domain block notes (#11515)
* Add database columns for adding notes to domain blocks/restrctions

* Add admin UI to set private and public comments when blocking a domain

* Add text for private and public comments on domain blocks

* Show domain block comments in admin UI

* Add comments to the domain block undo page

* Make UnblockDomainService more robust regarding upgraded domain blocks

* Allow editing domain blocks

* Rename button from “undo domain block” to “view domain block” in account admin UI

* Change test to unsilence silenced users from upgraded blocks
2019-08-07 20:20:23 +02:00
Eugen Rochko
5e35aa8280
Fix non-lowercase hashtags not being picked up by the streaming API (#11508)
Regression from f371b32

Fix hashtag links always being lowercase
2019-08-07 10:01:19 +02:00
Eugen Rochko
24552b5160
Add whitelist mode (#11291) 2019-07-30 11:10:46 +02:00
Eugen Rochko
f371b32137
Change hashtags to preserve first-used casing (#11416) 2019-07-28 05:59:51 +02:00
Eugen Rochko
b9fbcbfe4e
Add search syntax for operators and phrases (#11411) 2019-07-27 04:42:08 +02:00
ThibG
92569ffde8 Fix invites not being disabled upon account suspension (#11412)
* Disable invite links from disabled/suspended users

* Add has_many invites relationship to users

* Destroy unused invites when suspending an account
2019-07-26 18:55:33 +02:00
Eugen Rochko
4eeff26533
Change account domain block to clear out notifications and follows (#11393) 2019-07-25 04:17:35 +02:00
Eugen Rochko
bd87e66679
Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
ThibG
2f813b7ea1 Disable LDSigning when AUTHORIZED_FETCH is set to true (#11295) 2019-07-15 02:29:39 +02:00
ThibG
3595ce6325 Fix leaking private statuses the admin account follows (#11300)
Now that the request is signed, it can return private toots. Do not leak them.
2019-07-15 02:29:04 +02:00
Eugen Rochko
6ff67be0f6
Add a spam check (#11217)
* Add a spam check

* Use Nilsimsa to generate locality-sensitive hashes and compare using Levenshtein distance

* Add more tests

* Add exemption when the message is a reply to something that mentions the sender

* Use Nilsimsa Compare Value instead of Levenshtein distance

* Use MD5 for messages shorter than 10 characters

* Add message to automated report, do not add non-public statuses to
automated report, add trust level to accounts and make unsilencing
raise the trust level to prevent repeated spam checks on that account

* Expire spam check data after 3 months

* Add support for local statuses, reduce expiration to 1 week, always create a report

* Add content warnings to the spam check and exempt empty statuses

* Change Nilsimsa threshold to 95 and make sure removed statuses are removed from the spam check

* Add all matched statuses into automatic report
2019-07-13 16:45:50 +02:00
ThibG
4e1260feaa Fix BlockService trying to reject incorrect follow request (#11288)
Fixes #11148
2019-07-11 14:50:27 +02:00
Eugen Rochko
4e8dcc5dbb
Add HTTP signatures to all outgoing ActivityPub GET requests (#11284) 2019-07-11 14:49:55 +02:00
Eugen Rochko
5d3feed191
Refactor fetching of remote resources (#11251) 2019-07-10 18:59:28 +02:00
ThibG
85eb418e1f Fix handling of webfinger redirects in ResolveAccountService (#11279) 2019-07-10 17:10:12 +02:00
Eugen Rochko
4e92183227
Refactor domain block checks (#11268) 2019-07-09 03:27:35 +02:00
Eugen Rochko
63c7fe8e48
Refactor controllers for statuses, accounts, and more (#11249) 2019-07-08 12:03:45 +02:00
ThibG
87c1ad4ea3 Fix BackupService crashing when an attachment is missing (#11241)
* Fix BackupService crashing when an attachment is missing

For various reasons such as admin error or out-of-sync media and
database backups, it might be possible for local attachments to be lost.

This commit allows the BackupService to continue its work even if some media
file is missing.

* Change error message
2019-07-08 02:24:25 +02:00