This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads` settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.
A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used.
### Starting a direct S3 upload
When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.
Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.
### Completing a direct S3 upload
Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.
1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.
2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.
We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.
3. Finally we return the serialized upload record back to the client
There are several errors that could happen that are handled by `UploadsController` as well.
Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
Multiselect data can be saved but when all are removed then data are not cleared
Ajax function is removing an empty array from request data. In that case, we should change `[]` to `null`.
We need that empty values to properly empty data.
The error was:
```
↪ Unit | Model | topic::recover [✔]
↪ Unit | Utility | emoji::emojiUnescape [✔]
↪ Unit | Utility | pretty-text::quoting a quote [✔]
↪ Unit | Utility | click-track::routes to internal urlsUnhandled request in test environment: /forum/t/1234/recover (PUT)
Error: Unhandled request in test environment: /forum/t/1234/recover (PUT)
at Pretender.server.unhandledRequest (discourse/tests/setup-tests:173:15)
at Pretender.handleRequest (pretender:400:14)
at FakeRequest.send (pretender:169:21)
at Object.send (jquery:10100:10)
at Function.ajax (jquery:9683:15)
at performAjax (discourse/app/lib/ajax:174:19)
at eval (discourse/app/lib/ajax:183:11)
at invokeCallback (ember:63104:17)
at publish (ember:63087:9)
at eval (ember:57463:16)
[✘]
```
* DEV: Don't duplicate a function
* pretender wasn't catching the request because it ran after this test finished
* restore wasn't needed, we do `sinon.restore()` after each test
The error was:
```
↪ Unit | Model | user::resolvedTimezone [✔]
↪ Unit | Utility | url::routeTo with prefixUnhandled request in test environment: /forum/u/chuck.json (PUT)
Error: Unhandled request in test environment: /forum/u/chuck.json (PUT)
at Pretender.server.unhandledRequest (discourse/tests/setup-tests:173:15)
at Pretender.handleRequest (pretender:400:14)
at FakeRequest.send (pretender:169:21)
at Object.send (jquery:10100:10)
at Function.ajax (jquery:9683:15)
at performAjax (discourse/app/lib/ajax:174:19)
at eval (discourse/app/lib/ajax:183:11)
at invokeCallback (ember:63104:17)
at publish (ember:63087:9)
at eval (ember:57463:16)
[✘]
```
A minimal reproduction:
`http://localhost:3001/qunit?seed=3&testId=da76996b&testId=e52a53e7`
The generated regular expressions did not contain \b which matched
every text that contained the word, even if it was only a substring of
a word.
For example, if "art" was a watched word a post containing word
"artist" matched.
Next Week should mean next Monday, Next Month - the first day of the next month, and so on.
Also, we'll be using the name "Next Monday" instead of "Next Week" because it's easier to understand. No one can get confused by next Monday.
The problem was happening in component integration tests on the rendering stage, sometimes the rendering would never finish.
Using time moments in the future when faking time solves the problem. Unfortunately, I don't know why exactly it helps. It was just a lucky guess after some hours I spent trying to figure out what's going on. But I've done a lot of testings, so looks like it really works. I'll be monitoring builds for some time after merging this anyway.
Unit tests seem to work alright with moments in the past. And we don't fake time in acceptance tests at the moment but I guess they would very likely be flaky with time moments from the past since they also do rendering.
I'm actually thinking of moving all fake time moments to the future (including moments in unit tests) to decrease the chances of flakiness. But I don't want to do everything in one PR, because I can accidentally introduce new flakiness.
A pretty easy way of picking time moments in the future for tests is to use the 2100 year. It has the same calendar as 2021. If a day is Monday in 2021 it's Monday in 2100 too.
Before this fix if your forum was set up with a subfolder and you
clicked on a link to a different subfolder it would not work. For
example:
subfolder: /cool
link is: /about-us
Previously it would try to resolve /about-us as /cool/about-us. With
this fix it redirects to /about-us correctly.
* UX: Improvements to reorder categories UX
Before, moving a category from, for example, position 25 to position 0 would result in switching the positions of the two categories at those positions.
Category A at position 0 would move to position 25, and Category B at position 25 would move to position 0.
Instead of switching positions, the reorder categories function should retain the order of categories except for the one being moved.
So, Category B at position 25 would still move to position 0, but Category A is merely bumped down to position 1.
This improves the UX because if a user *really* wants to switch the two categories, it results in one extra step. However in the other (what I think is normal) case, it saves the 24 other switches the user has to make to get Category A back to position 1 (you can imagine the user having to click the up arrow button repeatedly to return Category A to the top of the page). Now, imagine trying to do this with a site with 100s of categories. Yikes!
The UX improvement described above is what this commit accomplishes by redesigning the `move()` method of the reorder-categories controller. It adds some overhead to adjust the positions of all categories in between the origin and target positions, but in testing this is not noticible to the user. It's better for the computer to do extra work than the user.
* UX: Allow decimal input in reorder-categories for more precise positioning.
A common UX pattern when reordering a list of items is to allow a user to specify a target position as a decimal between two valid integer positions. The user is indicating they want the target list item to move in between the list items at the positions on either side of the target position.
For example, say there are three categories Category A at position 0, Category B at position 1, and Category C at position 3.
To move Category C in between Categories A and B, a user can now simply update Category C's position to 0.5.
It was not clear that replace watched words can be used to replace text
with URLs. This introduces a new watched word type that makes it easier
to understand.
I merged this PR in yesterday, finally thinking this was done https://github.com/discourse/discourse/pull/12958 but then a wild performance regression occurred. These are the problem methods:
1aa20bd681/app/serializers/topic_tracking_state_serializer.rb (L13-L21)
Turns out date comparison is super expensive on the backend _as well as_ the frontend.
The fix was to just move the `treat_as_new_topic_start_date` into the SQL query rather than using the slower `UserOption#treat_as_new_topic_start_date` method in ruby. After this change, 1% of the total time is spent with the `created_in_new_period` comparison instead of ~20%.
----
History:
Original PR which had to be reverted **https://github.com/discourse/discourse/pull/12555**. See the description there for what this PR is achieving, plus below.
The issue with the original PR is addressed in 92ef54f402
If you went to the `x unread` link for a tag Chrome would freeze up and possibly crash, or eventually unfreeze after nearly 10 mins. Other routes for unread/new were similarly slow. From profiling the issue was the `sync` function of `topic-tracking-state.js`, which calls down to `isNew` which in turn calls `moment`, a change I had made in the PR above. The time it takes locally with ~1400 topics in the tracking state is 2.3 seconds.
To solve this issue, I have moved these calculations for "created in new period" and "unread not too old" into the tracking state serializer.
When I was looking at the profiler I also noticed this issue which was just compounding the problem. Every time we modify topic tracking state we recalculate the sidebar tracking/everything/tag counts. However this calls `forEachTracked` and `countTags` which can be quite expensive as they go through the whole tracking state (and were also calling the removed moment functions).
I added some logs and this was being called 30 times when navigating to a new /unread route because `sync` is being called from `build-topic-route` (one for each topic loaded due to pagination). So I just added a debounce here and it makes things even faster.
Finally, I changed topic tracking state to use a Map so our counts of the state keys is faster (Maps have .size whereas objects you have to do Object.keys(obj) which is O(n).)
<!-- NOTE: All pull requests should have tests (rspec in Ruby, qunit in JavaScript). If your code does not include test coverage, please include an explanation of why it was omitted. -->
Original PR which had to be reverted **https://github.com/discourse/discourse/pull/12555**. See the description there for what this PR is achieving, plus below.
The issue with the original PR is addressed in 92ef54f402
If you went to the `x unread` link for a tag Chrome would freeze up and possibly crash, or eventually unfreeze after nearly 10 mins. Other routes for unread/new were similarly slow. From profiling the issue was the `sync` function of `topic-tracking-state.js`, which calls down to `isNew` which in turn calls `moment`, a change I had made in the PR above. The time it takes locally with ~1400 topics in the tracking state is 2.3 seconds.
To solve this issue, I have moved these calculations for "created in new period" and "unread not too old" into the tracking state serializer.
When I was looking at the profiler I also noticed this issue which was just compounding the problem. Every time we modify topic tracking state we recalculate the sidebar tracking/everything/tag counts. However this calls `forEachTracked` and `countTags` which can be quite expensive as they go through the whole tracking state (and were also calling the removed moment functions).
I added some logs and this was being called 30 times when navigating to a new /unread route because `sync` is being called from `build-topic-route` (one for each topic loaded due to pagination). So I just added a debounce here and it makes things even faster.
Finally, I changed topic tracking state to use a Map so our counts of the state keys is faster (Maps have .size whereas objects you have to do Object.keys(obj) which is O(n).)
Over the years we accrued many spelling mistakes in the code base.
This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change
- comments
- test descriptions
- other low risk areas
The user may have changed their category or tag tracking settings since a topic was tracked/watched based on those settings in the past. In that case we need to alter the reason message we show them otherwise it is very confusing for the end user to be told they are tracking a topic because of a category, when they are no longer tracking that category.
For example: "You will see a count of new replies because you are tracking this category." becomes: "You will see a count of new replies because you were tracking this category in the past."
To do this, it was necessary to add tag and category tracking info to current user serializer. I improved the serializer code so it only does 3 SQL queries instead of 9 to get the tracking information for tags and categories for the current user.
The aim of this PR is to improve the topic tracking state JavaScript code and test coverage so further modifications can be made in plugins and in core. This is focused on making topic tracking state changes easier to respond to with callbacks, and changing it so all state modifications go through a single method instead of modifying `this.state` all over the place. I have also tried to improve documentation, make the code clearer and easier to follow, and make it clear what are public and private methods.
The changes I have made here should not break backwards compatibility, though there is no way to tell for sure if other plugin/theme authors are using tracking state methods that are essentially private methods. Any name changes made in the tracking-state.js code have been reflected in core.
----
We now have a `_trackedTopicLimit` in the tracking state. Previously, if a topic was neither new nor unread it was removed from the tracking state; now it is only removed if we are tracking more than `_trackedTopicLimit` topics (which is set to 4000). This is so plugins/themes adding topics with `TopicTrackingState.register_refine_method` can add topics to track that aren't necessarily new or unread, e.g. for totals counts.
Anywhere where we were doing `tracker.states["t" + data.topic_id] = newObject` has now been changed to flow through central `modifyState` and `modifyStateProp` methods. This is so state objects are not modified until they need to be (e.g. sometimes properties are set based on certain conditions) and also so we can run callback functions when the state is modified.
I added `onStateChange` and `onMessageIncrement` methods to register callbacks that are called when the state is changed and when the message count is incremented, respectively. This was done so we no longer need to do things like `@observes("trackingState.states")` in other Ember classes.
I split up giant functions like `sync` and `establishChannels` into smaller functions for readability and testability, and renamed many small functions to _functionName to designate them as private functions which not be called by consumers of `topicTrackingState`. Public functions are now all documented (well...at least ones that are not immediately obvious).
----
On the backend side, I have changed the MessageBus publish events for TopicTrackingState to send back tags and tag IDs for more channels, and done some extra code cleanup and refactoring. Plugins may override `TopicTrackingState.report` so I have made its footprint as small as possible and externalised the main parts of it into other methods.
Headings with the exact same name generated exactly the same heading
names, which was invalid. This replaces the old code for generating
names for non-English headings which were using URI encode and resulted
in unreadable headings.
* FIX: Use theme color for anchor icon
* FIX: Do not count anchor links
* FIX: Do not count hashtags links either
* DEV: Add tests for link_count
* FIX: Disable anchors in quotes and preview
* FIX: Try building some anchor slugs for unicode
* DEV: Fix tests
The implemented helpers, are helper which might be in Ember core in the future:
- and
- or
- not
- eq
- not-eq
- lt
- lte
- gt
- gte
They follow the implementation of ember-truth-helpers: https://github.com/jmurphyau/ember-truth-helpers
Note 1: Ember rfcs are still debating going with {{not-eq}} or {{neq}}, should be easy to support in the future whatever is finally chosen.
Note 2: this commit also moves it to its own addon, and removes the {{not}} test, to simplify further updates.
There are a lot of little fixes to tests here, but the biggest issue was
too much recursion because we kept replacing the helpers over and over
again. I assume Chrome has tail recursion or something to speed this up
but Firefox hated it.
Otherwise, we can't rely on the order of attributes in rendered HTML so
I simplified most of those tests to just look for key strings in the
HTML that are rendered.
We override the default replacements rule to no longer replace "(c)", "(p)", and "(p)". Additionally, we merged the custom arrows rule into the replacement function.
This is not a security issue because regular users are not allowed to insert FA icons anywhere in the app. Admins can insert icons via custom badges, but they do have the ability to create themes with JS.
- removes the option from site settings
- deletes the site setting on existing sites that have it
- marks posts using emojis as requiring a rebake
Note that the actual image files are not removed here, the plan is to
remove them in a few weeks/months (when presumably the rebaking of old
posts has been completed).
* FIX: Subfolder replace should only affect URL prefix
Issue was reported in https://meta.discourse.org/t/-/179504
* DEV: Test subfolder handling in get-url when called twice on the same path
Original PR was reverted because of broken migration https://github.com/discourse/discourse/pull/12058
I fixed it by adding this line
```
AND topics.id IN(SELECT id FROM topics ORDER BY created_at DESC LIMIT :max_new_topics)
```
This time it is left joining a limited amount of topics. I tested it on few databases and it worked quite smooth
Follow up https://github.com/discourse/discourse/pull/11968
Dismiss all new topics using the same DismissTopicService. In addition, MessageBus receives exact topic ids which should be marked as `seen`.
* FEATURE: Ability to dismiss new topics in a specific tag
Follow up of https://github.com/discourse/discourse/pull/11927
Using the same mechanism to disable new topics in a tag.
* FIX: respect when category and tag is selected
This encompasses a lot of work done over the last year, much of which
has already been merged into master. This is the final set of changes
required to get Ember CLI running locally for development.
From here on it will be bug fixes / enhancements.
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: romanrizzi <rizziromanalejandro@gmail.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: romanrizzi <rizziromanalejandro@gmail.com>
This PR moves all of the time picking functionality from the bookmark modal and controller into a reusable time-shortcut-picker component, which will be used for the topic timer UI revamp. All of the utility JS for getting dates like tomorrow/next week/next month etc. have also been moved into a separate utility lib.
The time-shortcut-picker has a couple of options that can be passed in:
* prefilledDatetime - The date and time to parse and prefill into the custom date and time section, useful for editing interfaces.
* onTimeSelected (callback) - Called when one of the time shortcuts is clicked, and passes the type of the shortcut (e.g. tomorrow) and the datetime selected.
* additionalOptionsToShow - An array of option ids to show (by default `later_today` and `later_this_week` are hidden)
* hiddenOptions - An array of option ids to hide
* customOptions - An array of custom options to display (e.g. the option to select a post date for the bookmarks modal). The options should have the below properties:
* id
* icon
* label (I18n key)
* time (moment datetime object)
* timeFormatted
* hidden
The other major work in this PR is moving all of the bookmark functionality out of the bookmark modal controller and into its own component, where it makes more sense to be able to access elements on the page via `document`. Tests have been added to accompany this move, and existing acceptance tests for bookmark are all passing.
* Quite a few Ember-CLI / Upgrade related changes
They should all be backwards compatible. This is all to help merge our
branches.
* REFACTOR: DRY up username validation
Also avoids overwriting computed properties for compatibility with newer
Ember releases.
The bug was mentioned on [meta](https://meta.discourse.org/t/two-bugs-with-usernames-starting-with-subfolder-name/169505)
When discourse is installed on `/subfolder` and username is containing subfolder name like for example `subfolderadmin` - user URLs were incorrect.
Instead of having `/subfolder/u/subfolderadmin/summary/` we were leading to `/subfolder/uadmin/summary`.
The reason for that was incorrect check in `getUrl` helper:
```javascript
const found = url.indexOf(baseUri);
if (found >= 0 && found < 3) {
return url;
}
return baseUri + url;
```
baseUri is `/subfolder`, url is `/u/subfolderadmin` and indexOf returned position which in the end returned incorrect URL.
I think that we should check if the URL starts with baseUri and not if contains baseUri.
Paths prefixed with /tag/ are exclusively for when the tag name is the
next string in the path. Therefore, when a category is being used as
context, the path should start with /tags/ instead.
Using arrow functions changes `this` context, which is undesired in tests, e.g. it makes it impossible to setup things like pretender (`this.server`) in `beforeEach` hooks.
Ember guides always use classic functions in examples (e.g. https://guides.emberjs.com/release/testing/test-types/), and that's what it uses in its own test suite, as do various addons and ember apps.
It was also already used in Discourse where `this` was required. Moving forward, it will be needed in more places as we migrate toward ember-cli.
(I might later add a custom rule to eslint-discourse-ember to enforce this)
Before deleting a topic that has a high number of views (default of 5000), the user will be prompted with a confirmation popup. This works for all delete buttons on the topic located in: topic-timeline, topic-admin-menu, topic-footer-buttons, and post-menu if the post's ID is 1.
The delete button will be disabled while deletion is in progress, to prevent any unwanted behavior.
A site setting is also available to change the minimum amount of views required to display the confirmation popup.
All kudos are going to @RickyC0626. I only rebased with master and added few qunit tests to ensure that this feature works as expected.
Original PR: #10459
Adding a video in composer and then continuing to type into it will make the
video element flicker and restart playback on every keystroke, as the preview
is rendered. In certain configurations, this can lead to some performance
problems too.
Onebox already does the same for external videos.
* FEATURE: when we fail to ship topic timings attempt to retry
This change amends it so
1. Topic timings are treated as background requests and subject to more
aggressive rate limits.
2. If we notice an error when we ship timings we back off exponentially
The commit allows 405, 429, 500, 501, 502, 503 and 504 errors to be retried.
500+ errors usually happen when self hosters are rebuilding or some other
weird condition.
405 happens when site is in readonly.
429 happens when user is rate limited.
The retry cadence is hardcoded in AJAX_FAILURE_DELAYS, longest delay is
40 seconds, we may consider enlarging it.
After the last delay passes we give up and do not write timings to the
server.
* Address feedback
- Omit promise, no need to use promises in sendNextConsolidatedTiming
- Correct issue where >= -1 was used and > -1 was intended
- Use objects for consolidated timings instead of Array
- Stop using shift/unshift and instead use push / pop which are faster
* Move consolidated timing initialization to constructor
* Remove TODO and just console.warn if we have an issue
When the server gets overloaded and lots of requests start queuing server
will attempt to shed load by returning 429 errors on background requests.
The client can flag a request as background by setting the header:
`Discourse-Background` to `true`
Out-of-the-box we shed load when the queue time goes above 0.5 seconds.
The only request we shed at the moment is the request to load up a new post
when someone posts to a topic.
We can extend this as we go with a more general pattern on the client.
Previous to this change, rate limiting would "break" the post stream which
would make suggested topics vanish and users would have to scroll the page
to see more posts in the topic.
Server needs this protection for cases where tons of clients are navigated
to a topic and a new post is made. This can lead to a self inflicted denial
of service if enough clients are viewing the topic.
Due to the internal security design of Discourse it is hard for a large
number of clients to share a channel where we would pass the full post body
via the message bus.
It also renames (and deprecates) triggerNewPostInStream to triggerNewPostsInStream
This allows us to load a batch of new posts cleanly, so the controller can
keep track of a backlog
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
I also took the opportunity with this commit to move some test specific
stuff out of `discourse-loader` which is loaded on the front end of the
application. The test module building now happens in the `test_helper`
bundle.
This is long overdue. We had a lot of (not linted) code to initialize
our test suite as part of the Ruby `test_helper.js` bundle.
This refactor moves that out to a `setup-tests` module, which imports
all the modules properly, rather than using `require`.
It also removes the global `server` variable which some tests were using
for pretender. Those tests are fixed, and in the case of widget tests,
support for a `pretend()` was added, which mimics our acceptance tests.
One problematic test was removed, which overwrites `/posts` - this could
break tons of other tests depending on order.
The prefixing logic is moved into a `prefixProtocol` function in lib:url.
This commit also renames an incorrectly named test and uses https as default instead of http, in 2020 it's reasonable to think we most likely want https and not http. User can still specify http if required.
These are tricky because `module.exports` is used by nodejs files as a
global, which is OK. But we don't want to allow `module` in JS tests
for qunit without importing it first.
We used many global functions to handle tests when they should be
imported like other libraries in our application. This also gets us
closer to the way Ember CLI prefers our tests to be laid out.
This is where they should be as far as ember is concerned. Note this is
a huge commit and we should be really careful everything continues to
work properly.
