Merged
Conversation
gmuraru
force-pushed
the
gm-improve-tanh
branch
2 times, most recently
from
77b9dd5 to
1db29b5
Compare
gmuraru
force-pushed
the
gm-improve-tanh
branch
from
1db29b5 to
7d1e688
Compare
gmuraru
requested review from
LaRiffle,
andrelmfarias,
iamtrask and
robert-wagner
and removed request for
andrelmfarias
iamtrask
requested changes
Feb 4, 2020
Member
iamtrask
left a comment
iamtrask
left a comment
There was a problem hiding this comment.
Looking good - needs a few tests 👍
LaRiffle
reviewed
Feb 4, 2020
| # truncate outside [-maxval, maxval] | ||
| out = torch.where(tensor > maxval, 1.0, out) | ||
| out = torch.where(tensor < -maxval, -1.0, out) | ||
| return out |
Contributor
There was a problem hiding this comment.
I'm curious, 2 * sigmoid(2 * tensor) - 1 wasn't precise enough or too slow?
Member
Author
There was a problem hiding this comment.
In the PR, they say this:
Implemented improved tanh approximation that's ~43.5% more accurate (measured by total relative error) and ~33% faster (see n196073).
Before this, they also had the implementation using the sigmoid
Member
Author
gmuraru
force-pushed
the
gm-improve-tanh
branch
from
3deaeea to
b5d90c2
Compare
LaRiffle
suggested changes
Feb 6, 2020
Contributor
LaRiffle
left a comment
LaRiffle
left a comment
There was a problem hiding this comment.
Would you mind also add a test for tanh in the same way I did for exp/log etc (ie so that people can have an idea of the error rate just by looking at the test)
iamtrask
approved these changes
Feb 26, 2020
Member
iamtrask
left a comment
iamtrask
left a comment
There was a problem hiding this comment.
+1 to Theo's request for readable test but I won't block for it. :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implementation is taken from the FacebookResearch Crypten project[1].
The PR that adds this functionality in Crypten is here[2].
[1] https://github.com/facebookresearch/CrypTen
[2] facebookresearch/CrypTen#45