I believed it will be enjoyable to cross-reference the ICLR 2017 (a preferred Deep Studying convention) decisions (which fall into 4 classes: oral, poster, workshop, reject) with the variety of occasions every paper was added to somebody’s library on arxiv-sanity. ICLR 2017 determination making includes various space chairs and reviewers that determine the destiny of every paper over a interval of few months, whereas arxiv-sanity includes one individual working 2 hours as soon as a month (me), and various individuals who use it to tame the flood of papers on the market. It’s a battle between high down and backside up. Lets see what occurs.
Here are the choices for ICLR 2017. A complete of 491 papers have been submitted, of which 15 (3%)might be an oral, 183 (37.3%) a poster, 48 (9.8%)have been prompt for workshop and 245 (49.9%) have been rejected. The accepted papers might be offered at ICLR on April 24–27 in Toulon, which I’m actually wanting ahead to. Look how superb it seems to be:
However I digress.
Then again we have now arxiv-sanity, which has a library characteristic. In brief, any registered person can add a paper to their library, and arxiv-sanity will prepare a personalised SVM on bigram tfidf options of the complete textual content of all papers to make content-based suggestions to the person. For instance, I’ve various RL/generative fashions/CV papers in my library and every time there’s a new paper on these subjects it is going to come up on high in my “beneficial” tab. The overview pool of arxiv-sanity is as of now a complete of 3195 customers — that is the variety of folks with an account which have at the least one paper within the library. Collectively, these customers have to this point included 55,671 papers into their libraries, i.e. a mean of 17.4 papers.
An necessary characteristic of arxiv-sanity is that customers don’t simply upvote papers with no repercussions. Including a paper to your library has some weight, as a result of that paper will affect your suggestions. You have got an incentive to solely embody issues that actually matter to you in there. It’s intelligent proper? No? Okay wonderful.
Lengthy story brief, I loop over all papers in ICLR and attempt to discover them on arxiv utilizing an actual match on the title. Some ICLR papers should not on arxiv, and a few gained’t get matched as a result of the authors renamed them, or they comprise bizarre characters, and so forth.
For instance, lets take a look at the papers that received an oral at ICLR 2017. We get:
for oral, discovered 10/15 papers on arxiv with library counts:
64 Reinforcement Studying with Unsupervised Auxiliary Duties
44 Neural Structure Search with Reinforcement Studying
38 Understanding deep studying requires rethinking generalizatio...
28 In the direction of Principled Strategies for Coaching Generative Adversaria...
22 Studying Finish-to-Finish Objective-Oriented Dialog
19 Q-Prop: Pattern-Environment friendly Coverage Gradient with An Off-Coverage C...
13 Studying to Act by Predicting the Future
12 Amortised MAP Inference for Picture Tremendous-resolution
8 Multi-Agent Cooperation and the Emergence of (Pure) Langua...
8 Finish-to-end Optimized Picture Compression
Right here we see that we matched 10 out of 15 oral papers on arxiv, and the quantity subsequent to every one is the quantity of people that have added that paper to their library. E.g. “Reinforcement Studying with Unsupervised Auxiliary Duties” was in a library of 64 arxiv-sanity customers. I additionally needed to truncate some paper names as a result of medium.com is wrongly conceived and doesn’t allow you to change the font dimension.
Now lets take a look at the posters:
for poster, discovered 113/183 papers on arxiv with library counts:
149 Adversarial Function Studying
147 Hierarchical Multiscale Recurrent Neural Networks
140 Recurrent Batch Normalization
80 HyperNetworks
79 FractalNet: Extremely-Deep Neural Networks with out Residuals
73 Zoneout: Regularizing RNNs by Randomly Preserving Hidden Acti...
62 Unrolled Generative Adversarial Networks
52 Adversarially Discovered Inference
49 Quasi-Recurrent Neural Networks
48 Do Deep Convolutional Nets Actually Have to be Deep and Convolu...
46 Neural Photograph Modifying with Introspective Adversarial Networks
43 An Actor-Critic Algorithm for Sequence Prediction
41 A Discovered Illustration For Creative Fashion
37 Structured Consideration Networks
33 Mollifying Networks
30 DeepCoder: Studying to Write Packages
28 SGDR: Stochastic Gradient Descent with Heat Restarts
27 Studying to Navigate in Advanced Environments
27 Generative Multi-Adversarial Networks
26 Delicate Weight-Sharing for Neural Community Compression
25 Pruning Filters for Environment friendly ConvNets
24 Why Deep Neural Networks for Perform Approximation?
24 Mode Regularized Generative Adversarial Networks
24 Dialogue Studying With Human-in-the-Loop
24 Designing Neural Community Architectures utilizing Reinforcement Le...
23 PGQ: Combining coverage gradient and Q-learning
22 Frustratingly Quick Consideration Spans in Neural Language Modeli...
21 Monitoring the World State with Recurrent Entity Networks
21 Deep Probabilistic Programming
20 Density estimation utilizing Actual NVP
20 Adversarial Coaching Strategies for Semi-Supervised Textual content Classif...
19 Semi-Supervised Classification with Graph Convolutional Netwo...
19 PixelVAE: A Latent Variable Mannequin for Pure Photos
19 Studying to Optimize
19 Studying a Pure Language Interface with Neural Programmer
19 Entropy-SGD: Biasing Gradient Descent Into Large Valleys
19 Dynamic Coattention Networks For Query Answering
18 PixelCNN++: Enhancing the PixelCNN with Discretized Logistic ...
18 Generalizing Abilities with Semi-Supervised Reinforcement Learni...
18 Deep Studying with Dynamic Computation Graphs
18 Computerized Rule Extraction from Lengthy Quick Time period Reminiscence Community...
18 Adversarial Machine Studying at Scale
17 Studying by Dialogue Interactions by Asking Questions
16 Studying to Carry out Physics Experiments by way of Deep Reinforcemen...
16 Categorical Reparameterization with Gumbel-Softmax
15 Pattern Environment friendly Actor-Critic with Expertise Replay
14 Variational Lossy Autoencoder
14 Id Issues in Deep Studying
14 Bidirectional Consideration Move for Machine Comprehension
13 In the direction of a Neural Statistician
13 Recurrent Combination Density Community for Spatiotemporal Visible A...
13 On Detecting Adversarial Perturbations
12 Educated Ternary Quantization
12 Enhancing Coverage Gradient by Exploring Below-appreciated Rewa...
12 Capability and Trainability in Recurrent Neural Networks
11 SampleRNN: An Unconditional Finish-to-Finish Neural Audio Generatio...
11 Machine Comprehension Utilizing Match-LSTM and Reply Pointer
11 Latent Sequence Decompositions
11 Calibrating Power-based Generative Adversarial Networks
10 Unsupervised Cross-Area Picture Era
10 Studying to Keep in mind Uncommon Occasions
10 Freeway and Residual Networks be taught Unrolled Iterative Estima...
9 TopicRNN: A Recurrent Neural Community with Lengthy-Vary Semantic...
9 Steerable CNNs
9 Question-Discount Networks for Query Answering
9 Lossy Picture Compression with Compressive Autoencoders
9 Studying to Compose Phrases into Sentences with Reinforcement L...
8 Stick-Breaking Variational Autoencoders
8 Deep Variational Data Bottleneck
8 Batch Coverage Gradient Strategies for Enhancing Neural Conversati...
7 Discrete Variational Autoencoders
7 Knowledge Noising as Smoothing in Neural Community Language Fashions
6 Variable Computation in Recurrent Neural Networks
6 Sigma Delta Quantized Networks
6 Dropout with Expectation-linear Regularization
6 Delving into Transferable Adversarial Examples and Black-box ...
6 A Compositional Object-Based mostly Strategy to Studying Bodily Dy...
5 In the direction of the Restrict of Community Quantization
5 Tighter bounds result in improved classifiers
5 Pointer Sentinel Combination Fashions
5 On the Quantitative Evaluation of Decoder-Based mostly Generative Mode...
5 Neuro-Symbolic Program Synthesis
5 Lie-Entry Neural Turing Machines
5 Studying to superoptimize applications
5 Studying Options of Music From Scratch
5 Enhancing Neural Language Fashions with a Steady Cache
5 Deep Biaffine Consideration for Neural Dependency Parsing
4 Temporal Ensembling for Semi-Supervised Studying
4 Weight-reduction plan Networks: Skinny Parameters for Fats Genomics
4 DeepDSL: A Compilation-based Area-Particular Language for Dee...
4 DSD: Dense-Sparse-Dense Coaching for Deep Neural Networks
4 A recurrent neural community with out chaos
3 Trusting SVM for Piecewise Linear CNNs
3 The Neural Noisy Channel
3 Revisiting Classifier Two-Pattern Assessments
3 Regularizing CNNs with Regionally Constrained Decorrelations
3 Optimum Binary Autoencoding with Pairwise Correlations
3 Loss-aware Binarization of Deep Networks
3 Studying Recurrent Representations for Hierarchical Conduct ...
3 EPOpt: Studying Strong Neural Community Insurance policies Utilizing Mannequin En...
3 Deep Data Propagation
2 Phrases or Characters? Superb-grained Gating for Studying Comprehe...
2 Topology and Geometry of Half-Rectified Community Optimization
2 Most Entropy Move Networks
2 Incorporating long-range consistency in CNN-based texture gen...
2 Hadamard Product for Low-rank Bilinear Pooling
1 Multi-view Recurrent Neural Acoustic Phrase Embeddings
1 Inductive Bias of Deep Convolutional Networks by Pooling...
1 Geometry of Polysemy
1 Autoencoding Variational Inference For Matter Fashions
1 A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING
0 Deep Multi-task Illustration Studying: A Tensor Factorisati...
0 A Examine-Combination Mannequin for Matching Textual content Sequences
Some received quite a lot of love (149!), and a few little or no (0). For workshop solutions we get:
for workshop, discovered 23/48 papers on arxiv with library counts:
60 Adversarial examples within the bodily world
31 Studying in Implicit Generative Fashions
16 Shock-Based mostly Intrinsic Motivation for Deep Reinforcement Le...
14 Multiplicative LSTM for sequence modelling
13 Environment friendly Softmax Approximation for GPUs
12 RenderGAN: Producing Reasonable Labeled Knowledge
12 Generalizable Options From Unsupervised Studying
10 Programming With a Differentiable Forth Interpreter
8 Gated Multimodal Models for Data Fusion
8 Deep Studying with Units and Level Clouds
7 Unsupervised Perceptual Rewards for Imitation Studying
5 Track From PI: A Musically Believable Community for Pop Music Gen...
5 Modular Multitask Reinforcement Studying with Coverage Sketches
5 A Differentiable Physics Engine for Deep Studying in Robotics
4 Exponential Machines
4 Dataset Augmentation in Function House
3 Semi-supervised deep studying by metric embedding
2 Adaptive Function Abstraction for Translating Video to Languag...
1 Modularized Morphing of Neural Networks
1 Studying Steady Semantic Representations of Symbolic Expr...
1 Extrapolation and studying equations
0 On-line Construction Studying for Sum-Product Networks with Gauss...
0 Bit-Pragmatic Deep Neural Community Computing
and I gained’t checklist all 200-something papers that have been rejected, however lets take a look at the few that arxiv-sanity customers actually favored, however the ICLR ACs and reviewers didn’t:
for reject, discovered 58/245 papers on arxiv with library counts:
46 The Predictron: Finish-To-Finish Studying and Planning
39 RL^2: Quick Reinforcement Studying by way of Sluggish Reinforcement Lear...
35 Understanding intermediate layers utilizing linear classifier professional...
33 Hierarchical Reminiscence Networks
31 An Evaluation of Deep Neural Community Fashions for Sensible Appli...
20 Low-rank passthrough neural networks
19 Increased Order Recurrent Neural Networks
18 Including Gradient Noise Improves Studying for Very Deep Community...
16 Unsupervised Pretraining for Sequence to Sequence Studying
16 A Joint Many-Activity Mannequin: Rising a Neural Community for Multipl...
15 Adversarial examples for generative fashions
14 Gated-Consideration Readers for Textual content Comprehension
13 Extensions and Limitations of the Neural GPU
12 Warped Convolutions: Environment friendly Invariance to Spatial Transfor...
11 Neural Combinatorial Optimization with Reinforcement Studying
11 Reminiscence-augmented Consideration Modelling for Movies
10 GRAM: Graph-based Consideration Mannequin for Healthcare Representati...
9 Wav2Letter: an Finish-to-Finish ConvNet-based Speech Recognition Sy...
9 Understanding educated CNNs by indexing neuron selectivity
9 The Energy of Sparsity in Convolutional Neural Networks
9 Enhancing Stochastic Gradient Descent with Suggestions
8 In the direction of Data-In search of Brokers
8 NEWSQA: A MACHINE COMPREHENSION DATASET
8 LipNet: Finish-to-Finish Sentence-level Lipreading
7 Generative Adversarial Parallelization
7 Environment friendly Summarization with Learn-Once more and Copy Mechanism
6 Multi-task studying with deep mannequin primarily based reinforcement be taught...
6 Multi-modal Variational Encoder-Decoders
6 Finish-to-Finish Reply Chunk Extraction and Rating for Studying Co...
6 Boosting Picture Captioning with Attributes
6 Past Superb Tuning: A Modular Strategy to Studying on Small D...
5 Structured Sequence Modeling with Graph Convolutional Recurre...
5 Human notion in laptop imaginative and prescient
5 Cooperative Coaching of Descriptor and Generator Networks
Right here is the full version, which was not truncated to suit right here. There are a couple of papers on the highest of this checklist that have been presumably unfairly rejected.
Right here’s one other query — what would ICLR 2017 appear to be if it have been merely voted on by the group of arxiv-sanity customers (of the papers we will discover on arxiv)? Right here is an excerpt:
oral:
149 Adversarial Function Studying
147 Hierarchical Multiscale Recurrent Neural Networks
140 Recurrent Batch Normalization
80 HyperNetworks
79 FractalNet: Extremely-Deep Neural Networks with out Residuals
73 Zoneout: Regularizing RNNs by Randomly Preserving Hidden Acti...
64 Reinforcement Studying with Unsupervised Auxiliary Duties
62 Unrolled Generative Adversarial Networks
60 Adversarial examples within the bodily world
52 Adversarially Discovered Inference
-------------------------------------------------
poster:
49 Quasi-Recurrent Neural Networks
48 Do Deep Convolutional Nets Actually Have to be Deep and Convolu...
46 The Predictron: Finish-To-Finish Studying and Planning
46 Neural Photograph Modifying with Introspective Adversarial Networks
44 Neural Structure Search with Reinforcement Studying
43 An Actor-Critic Algorithm for Sequence Prediction
41 A Discovered Illustration For Creative Fashion
39 RL^2: Quick Reinforcement Studying by way of Sluggish Reinforcement Lear...
38 Understanding deep studying requires rethinking generalizatio...
37 Structured Consideration Networks
35 Understanding intermediate layers utilizing linear classifier professional...
33 Mollifying Networks
33 Hierarchical Reminiscence Networks
31 Studying in Implicit Generative Fashions
31 An Evaluation of Deep Neural Community Fashions for Sensible Appli...
30 DeepCoder: Studying to Write Packages
...
Once more, the complete itemizing can be found here. Notice that particularly, some ICLR2017 papers that have been rejected would have been nearly an oral primarily based on arxiv-sanity customers alone, particularly the Predictron, RL², “Understanding intermediate layers”, and “Hierarchical Reminiscence Networks”. Conversely, some accepted papers had little or no love from arxiv-sanity customers. Here’s a full confusion matrix:
And right here is the confusion matrix in text, for every cell, along with the paper titles. This doesn’t look too dangerous. The 2 teams don’t agree on the orals in any respect, agree on the posters fairly a bit, and most significantly there are only a few confusions between oral/poster and rejection. Additionally, congratulations to Max et al. for “Reinforcement Learning with Unsupervised Auxiliary Tasks”, which is the one paper that each teams agree needs to be an oral 🙂
Lastly, I learn the next Medium submit a couple of days in the past: “Ten Deserving Deep Learning Papers that were Rejected at ICLR 2017”, by Carlos E. Perez. Evidently arxiv-sanity customers agree with this submit, and all papers listed there (together with LipNet)(that we might additionally discover on arxiv) would have been accepted by arxiv-sanity customers.
An asterisk. There are a number of elements that skew these outcomes. For instance, the dimensions of arxiv-sanity person base grows over time, so these outcomes doubtless barely favor papers that have been revealed on arxiv later than earlier, as these would have come to extra person’s consideration as new papers on the positioning. Additionally, papers should not seen with equal frequencies — for example if some paper will get tweeted out by somebody widespread, extra folks will see it, and extra folks would possibly add it to their library. And at last, an excellent argument could possibly be made that on arxiv-sanity “wealthy get richer”, as a result of arxiv papers should not nameless and celebrities might get extra consideration. On this explicit case, ICLR 2017 is single-blind so this isn’t a differentiating issue.
Total, my very own conclusion from this experiment is that there’s fairly a little bit of sign right here. And we’re getting it “without spending a dime” from a backside up course of on the web, as a substitute of one thing that takes a couple of hundred folks a number of months. And as somebody who has had an excellent quantity of lengthy, painful, nerve-racking, rebuttals backwards and forwards on each submitting/reviewing sides that dragged on for a number of weeks/months, I say: Possibly we don’t want it. Or on the very least perhaps there may be quite a lot of room for enchancment.
EDIT1: somebody prompt the enjoyable concept that we add up the variety of citations of those papers in ICLR 2018 submitted/accepted papers, and see which rating “wins” on that metric. Trying ahead to that 🙂
Andrej Karpathy
