> therefore follows that a large neural network with 10 layers can do anything a human can in 0.1 seconds.
very funny ... as if ANNs are sufficiently comparable to actual neural activity. also I think it is naive to assess the "powerful"-ness of the brain to what is going on in a single neuron - it is certainly the parallel interaction which creates the human intelligence.
> And if human neurons turn out to be noisy (for example), which m...
it is pretty naive to consider noise as something of only handicapping nature - a lot of algorithms are as powerful as they are by utilizing noise and stochasticity
> What is learning? Learning is the problem of finding a setting of the neural network’s weights that achieves the best possible results on our training data.
Wrong - this is memorizing ... learning is the process leading to a low out-of-sample error.
This post is written by Ilya Sutskever, who has co-authored some of biggest breakthroughs in machine learning the last five years. Which do you think is most likely: a) That he has a naive understanding of machine learning and neuroscience, or b) that this was written informally, and without guarding against every possible way it can be misinterpreted. Please be a little charitable when interpreting other peoples writings.
He was second author on the AlexNet paper, wherein Alex Krizhevsky, Sutskever and Hinton blew everyone else out of the water on the ImageNet competition [2]. Their error rate was about 10 percentage points lower than others. Relatively speaking they had about 40% fewer errors than anyone else. This is possibly the biggest result in computer vision the last five years. So it seems a little silly to educate him on the basics of machine learning :)
well thanks for the info - but then I shift my critique to that I find it unnecessary to distort ML and biological concepts just to simplify the subject, when an accurate depiction wouldn't be much more difficult. Especially to not differentiate properly between memorization and generalization/learning is odd b/c this is one of the most prominent mistakes - it is specifically not the goal to minimize the in-sample-error! that would lead to very bad results most of the time
Actually, Ilya explains his statement regarding minimizing training errors in his comment exchange with Bengio:
"Although I didn't define it in the article, generalization (to me) means that the gap between the training and the test error is small. So for example, a very bad model that has similar training and test errors does not overfit, and hence generalizes, according to the way I use these concepts. It follows that generalization is easy to achieve whenever the capacity of the model (as measured by the number of parameters or its VC-dimension) is limited --- we merely need to use more training cases than the model has parameters / VC dimension. Thus, the difficult part is to get a low training error."
very funny ... as if ANNs are sufficiently comparable to actual neural activity. also I think it is naive to assess the "powerful"-ness of the brain to what is going on in a single neuron - it is certainly the parallel interaction which creates the human intelligence.
> And if human neurons turn out to be noisy (for example), which m...
it is pretty naive to consider noise as something of only handicapping nature - a lot of algorithms are as powerful as they are by utilizing noise and stochasticity
> What is learning? Learning is the problem of finding a setting of the neural network’s weights that achieves the best possible results on our training data.
Wrong - this is memorizing ... learning is the process leading to a low out-of-sample error.