cfd507822f
This commit fixes the follow quality issue with `PostSearchData#raw_data`: 1. URLs are being tokenized and links with similar href and characters are being duplicated in the raw data. `Post#cooked`: ``` <p><a href=\"https://meta.discourse.org/some.png\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://meta.discourse.org/some.png</a></p> ``` `PostSearchData#raw_data` Before: ``` This is a test topic 0 Uncategorized https://meta.discourse.org/some.png discourse org/some png https://meta.discourse.org/some.png discourse org/some png ``` `PostSearchData#raw_data` After: ``` This is a test topic 0 Uncategorized https://meta.discourse.org/some.png meta discourse org ``` 2. Ligthbox being included in search pollutes the `PostSearchData#raw_data` unncessarily. From 28 March 2018 to 28 March 2019, searches for the term `image` on `meta.discourse.org` had a click through rate of 2.1%. Non-lightboxed images are not included in indexing for search yet we were indexing content within a lightbox. Also, search for terms like `image` was affected we were using `Pasted image` as the filename for uploads that were pasted. `Post#cooked` ``` <p>Let me see how I can fix this image<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://meta.discourse.org/some.png\" title=\"some.png\" rel=\"nofollow noopener\"><img src=\"https://meta.discourse.org/some.png\" width=\"275\" height=\"299\"><div class=\"meta\">\n<svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#far-image\"></use></svg><span class=\"filename\">some.png</span><span class=\"informations\">1750×2000</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#discourse-expand\"></use></svg>\n</div></a></div></p> ``` `PostSearchData#raw_data` Before: ``` This is a test topic 0 Uncategorized Let me see how I can fix this image some.png png https://meta.discourse.org/some.png discourse org/some png some.png png 1750×2000 ``` `PostSearchData#raw_data` After: ``` This is a test topic 0 Uncategorized Let me see how I can fix this image ``` In terms of indexing performance, we now have to parse the given HTML through nokogiri twice. However performance is not a huge worry here since a string length of 194170 takes only 30ms to scrub plus the indexing takes place in a background job. |
||
|---|---|---|
| .. | ||
| spam_rule | ||
| anonymous_shadow_creator.rb | ||
| badge_granter.rb | ||
| color_scheme_revisor.rb | ||
| destroy_task.rb | ||
| group_action_logger.rb | ||
| group_mentions_updater.rb | ||
| group_message.rb | ||
| handle_chunk_upload.rb | ||
| notification_emailer.rb | ||
| post_action_notifier.rb | ||
| post_alerter.rb | ||
| post_owner_changer.rb | ||
| push_notification_pusher.rb | ||
| random_topic_selector.rb | ||
| search_indexer.rb | ||
| site_settings_task.rb | ||
| staff_action_logger.rb | ||
| topic_status_updater.rb | ||
| topic_timestamp_changer.rb | ||
| tracked_topics_updater.rb | ||
| trust_level_granter.rb | ||
| user_action_manager.rb | ||
| user_activator.rb | ||
| user_anonymizer.rb | ||
| user_authenticator.rb | ||
| user_destroyer.rb | ||
| user_merger.rb | ||
| user_silencer.rb | ||
| user_updater.rb | ||
| username_changer.rb | ||
| username_checker_service.rb | ||
| wildcard_domain_checker.rb | ||
| wildcard_url_checker.rb | ||
| word_watcher.rb | ||
