This repository has been archived on 2023-03-18. You can view files and clone it, but cannot push or open issues or pull requests.
osr-discourse-src/spec/requests
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
..
admin FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571) 2021-07-15 05:53:26 +03:00
api FEATURE: Per-category default slow mode duration for topics. 2021-07-21 12:32:07 +08:00
about_controller_spec.rb FIX: Correct user profile URLs in /about crawler view 2020-07-14 16:09:27 +01:00
application_controller_spec.rb PERF: Eager load Theme associations in Stylesheet Manager. 2021-06-21 11:06:58 +08:00
associate_accounts_controller_spec.rb FIX: misspelling in associate_accounts_controller_spec.rb 2021-06-04 13:55:35 +08:00
badges_controller_spec.rb FEATURE: add noindex header to badges, groups, and /my pages (#9736) 2020-05-11 15:05:42 +10:00
bookmarks_controller_spec.rb FIX: When user has already hit bookmark limit, do not error for clear_reminder! or other updates (#12658) 2021-04-09 13:06:35 +10:00
bootstrap_controller_spec.rb Code review comments. 2021-06-21 11:06:58 +08:00
categories_controller_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
clicks_controller_spec.rb DEV: Fix failling test. 2019-05-07 11:19:13 +03:00
composer_messages_controller_spec.rb DEV: Use response.parsed_body in specs (#9615) 2020-05-07 17:04:12 +02:00
csp_reports_controller_spec.rb DEV: Only include "report-sample" CSP directive when reporting is enabled (#9337) 2020-04-02 11:16:38 -04:00
directory_columns_controller_spec.rb DEV: Plugin API to add directory columns (#13440) 2021-06-22 13:00:04 -05:00
directory_items_controller_spec.rb FIX: Include user_field_ids in pagination URL for directory items (#13569) 2021-06-29 14:43:38 -05:00
do_not_disturb_controller_spec.rb DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
draft_controller_spec.rb FIX: saving drafts unconditionally increases sequence 2020-05-12 16:55:42 +10:00
drafts_controller_spec.rb DEV: Use response.parsed_body in specs (#9615) 2020-05-07 17:04:12 +02:00
email_controller_spec.rb FIX: Show Uncategorized when unsubscribing (#13832) 2021-07-26 12:19:30 +10:00
embed_controller_spec.rb FIX: Better and more secure validation of periods for TopicQuery 2021-07-23 14:24:44 -04:00
exceptions_controller_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
export_csv_controller_spec.rb DEV: Switch to new ExportUserArchive job 2020-08-28 11:46:53 -07:00
extra_locales_controller_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
finish_installation_controller_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
forums_controller_spec.rb FEATURE: Allow a cluster_name to be configured and used for /srv/status (#12365) 2021-03-15 15:41:59 +11:00
groups_controller_spec.rb DEV: extract leave_group method from the group#remove_member method (#13823) 2021-07-22 20:14:18 +04:00
hashtags_controller_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
inline_onebox_controller_spec.rb DEV: Use response.parsed_body in specs (#9615) 2020-05-07 17:04:12 +02:00
invites_controller_spec.rb FIX: Let staged users choose their username (#13678) 2021-07-12 07:57:38 +10:00
list_controller_spec.rb DEV: Return 400 instead of 500 for invalid top period (#13828) 2021-07-23 14:58:10 -06:00
metadata_controller_spec.rb PERF: cache all metadata for 60 seconds 2020-07-01 12:58:02 +10:00
notifications_controller_spec.rb FIX: Typo in NotificationsController#index not caught by tests. 2020-07-22 09:22:26 +08:00
offline_controller_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
omniauth_callbacks_controller_spec.rb FIX: Suggest current username for staged users (#13706) 2021-07-13 09:15:06 +10:00
onebox_controller_spec.rb FEATURE: Onebox local categories (#11311) 2020-11-25 10:53:05 +11:00
permalinks_controller_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
post_action_users_controller_spec.rb DEV: Cleanup ignored user logic (#11107) 2020-11-03 12:38:54 +00:00
post_actions_controller_spec.rb DEV: Use response.parsed_body in specs (#9615) 2020-05-07 17:04:12 +02:00
post_readers_controller_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
posts_controller_spec.rb FIX: Update post's raw from server response (#13438) 2021-06-18 16:26:57 +03:00
published_pages_controller_spec.rb FIX: Do not enable published page if secure media enabled (#11131) 2020-11-06 10:33:19 +10:00
push_notification_controller_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
qunit_controller_spec.rb FIX: Theme tests should work in production (#13333) 2021-06-08 22:03:59 +03:00
reviewable_claimed_topics_controller_spec.rb FIX: Make reviewable claiming work with deleted topics (#9040) 2020-02-25 15:49:23 +02:00
reviewables_controller_spec.rb FEATURE: Blocking is optional when deleting a user from the review queue. (#13375) 2021-06-15 12:35:45 -03:00
robots_txt_controller_spec.rb FEATURE: explicitly ban outlier traffic sources in robots.txt (#11553) 2020-12-23 08:51:14 +11:00
safe_mode_controller_spec.rb Code review comments. 2021-06-21 11:06:58 +08:00
search_controller_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
session_controller_spec.rb UX: suspend forever time period messages (#13776) 2021-07-20 14:42:08 +04:00
similar_topics_controller_spec.rb FIX: reindex_search job should work on model with no search data (#11819) 2021-01-25 11:23:36 +01:00
site_controller_spec.rb DEV: Make site setting type uploaded_image_list use upload IDs (#10401) 2020-10-13 16:17:06 +03:00
static_controller_spec.rb DEV: apply allow origin response header for CDN requests. (#11893) 2021-01-29 07:44:49 +05:30
steps_controller_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
stylesheets_controller_spec.rb PERF: Eager load Theme associations in Stylesheet Manager. 2021-06-21 11:06:58 +08:00
svg_sprite_controller_spec.rb FIX: Use absolute URL when redirecting SVG sprite path. 2021-06-30 11:25:05 +08:00
tag_groups_controller_spec.rb DEV: Improve tag groups test (#12848) 2021-04-27 14:05:45 +03:00
tags_controller_spec.rb FIX: Better and more secure validation of periods for TopicQuery 2021-07-23 14:24:44 -04:00
theme_javascripts_controller_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
topics_controller_spec.rb DEV: Fix flaky topics invite spec due to id collision. 2021-07-27 13:59:07 +08:00
uploads_controller_multisite_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
uploads_controller_spec.rb FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787) 2021-07-28 08:42:25 +10:00
user_actions_controller_spec.rb FEATURE - SiteSetting to disable user option to hide their profiles and presences (#10885) 2020-10-09 17:18:44 -04:00
user_api_keys_controller_spec.rb DEV: Move UserApiKey scopes to dedicated table (#10704) 2020-09-29 10:57:48 +01:00
user_avatars_controller_spec.rb DEV: Remove the remaining Travis code (#13255) 2021-06-02 20:29:47 +02:00
user_badges_controller_spec.rb FIX: simplify and improve choosing favorite badges (#13743) 2021-07-16 11:13:00 +08:00
users_controller_spec.rb FIX: Show user filter hints when typing @ in search (#13799) 2021-07-21 09:14:53 -04:00
users_email_controller_spec.rb No need to disable rate limiter after running tests (#13093) 2021-05-19 16:04:35 +04:00
webhooks_controller_spec.rb Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse"" 2020-05-23 00:56:13 -04:00
wizard_controller_spec.rb DEV: Use response.parsed_body in specs (#9615) 2020-05-07 17:04:12 +02:00