Computer Hardware for Machine Learning
A question that comes up from time to time is:
What hardware do I need to practice machine learning?
There was a time when I was a student when I was obsessed with more speed and more cores so I could run my algorithms faster and for longer. I have changed my perspective. Big hardware still matters, but only after you have considered a bunch of other factors.
Hardware Lessons
The lesson is, if you are just starting out, you’re hardware doesn’t matter. Focus on learning with small datasets that fit in memory, such as those from the UCI Machine Learning Repository.
Learn good experimental design and make sure you ask the right questions and challenge your intuitions by testing diverse algorithms and interpreting your results through the lens of statistical hypothesis testing.
Once hardware does start to matter and you really need lots of cores and a whole lot of RAM, rent it just-in-time for your carefully designed project or experiment.
More CPU! More RAM!
I was naive when I first stated in artificial intelligence and machine learning. I would use all the data that was available and run it through my algorithms. I would re-run models with minor tweets to parameters in an effort to improve the final score. I would run my models for days or weeks on end. I was obsessed.
This mainly stemmed from the fact that competitions got be interested in pushing my machine learning skills. Obsession can be good, you can learn a lot very quickly. But when misapplied, you can waste a lot of time.
I built my own machines in those days. I would update my CPU and RAM often. It was the early 2000s, before multicore was the clear path (to me) and even before GPUs where talked about much for non-graphics use (at least in my circles). I needed bigger and faster CPUs and I needed lots and lots of RAM. I even commandeered the PCs of housemates so that I could do more runs.
A little later whilst in grad school, I had access to a small cluster in the lab and proceeded to make good use of it. But things started to change and it started to matter less how much raw compute power I had available.
Results Are Wrong
The first step in my change was the discovery of good (any) experimental design. I discovered the tools of statistical hypothesis testing which allowed me to get an idea of whether one result really was significantly different (such as better) when compared to another result.
Suddenly, the fractional improvements I thought I was achieving were nothing more than statistical blips. This was an important change. I started to spend a lot more time thinking about the experimental design.
Questions Are Wrong
I shifted my obsessions to making sure I was asking good questions.
I now spend a lot of time up front loading in as many questions and variations on the questions as I can think of for a given problem. I want to make sure that when I run long compute jobs, that the results I get really matter. That they are going to impact the problem.
You can see this when I strongly advocate spending a lot of time defining your problem.
Intuitions Are Wrong
Good hypothesis testing exposes how little you think you know. We’ll it did for me and still does. I “knew” that this configuration of that algorithm was stable, reliable and good. Results when interpreted through the lens of statistical tests quickly taught me otherwise.
This shifted my thinking to be less reliable on my old intuitions and to rebuild my institution through the lens of statistically significant results.
Now, I don’t assume I know which algorithm or even which class of algorithm will do well on a given problem. I spot check a diverse set and let the data guide me in.
I also strongly advice careful consideration of test options and use of tools like the Weka experimenter that bake in hypothesis testing when interpreting results.
Best is Not Best
For some problems, the very best results are fragile.
I used to be big into non-linear function optimization (and associated competitions) and you could expend a huge amount of compute time on exploring (in retrospect, essentially enumerating!) search spaces and come up with structures or configurations that were marginally better than easily found solutions.
The thing is, the hard to find configurations were commonly very strange or exploited bugs or quirks in the domain or simulator. These solutions were good for competitions or for experiments because the numbers were better, but not necessarily viable for use in the domain or operations.
I see the same pattern in machine learning competitions. A quick and easily found solution is lower in a given performance measure, but is robust. Often, once you pour days, weeks, and months into tuning your models, you are building a fragile model of glass that is very much overfit to the training data and/or the leaderboard. Good for learning and for doing well in competitions, not necessarily usable in operations (for example, the Netflix Prize-Winning System was not Deployed).
Machine Learning Hardware
There are big data that require big hardware. Learning about big machine learning requires big data and big hardware.
On this site, I focus on beginners starting out in machine learning, who are much better off with small data on small hardware. Once you get enough of the machine learning, you can graduate to the bigger problems.
Today, I have an iMac i7 with a bunch of cores and 8 GB of RAM. It’s a run-of-the-mill workstation and does the job. I think that your workstation or laptop is good enough to get started in machine learning.
I do need bigger hardware on occasion, such as a competition or for my own personal satisfaction. On these occasions I rent cloud infrastructure, spin up some instances and run my models, then download the CSV predictions or whatever. It’s very cheap in time and dollars.
When it comes time for you to start practicing on big hardware with big data, rent it. Invest a little bit of money in your own education, design some careful experiments and rent a cluster to execute them.
What hardware do you practice machine learning on? Leave a comment and share your experiences.
相關推薦
Computer Hardware for Machine Learning
Tweet Share Share Google Plus A question that comes up from time to time is: What hardware do I
斯坦福大學公開課機器學習:machine learning system design | data for machine learning(數據量很大時,學習算法表現比較好的原理)
ali 很多 好的 info 可能 斯坦福大學公開課 數據 div http 下圖為四種不同算法應用在不同大小數據量時的表現,可以看出,隨著數據量的增大,算法的表現趨於接近。即不管多麽糟糕的算法,數據量非常大的時候,算法表現也可以很好。 數據量很大時,學習算法表現比
Statistical Methods for Machine Learning
AS n-2 cal 元素 n) pan size AC 情況 機器學習中的統計學方法。 統計學是機器學習的一個支柱。 原始觀察僅僅是數據, 但它們不是信息或知識。數據引發問題, 例如: 什麽是最常見的或預期的觀察? 觀察的限制是什麽? 數據是什麽樣子的?
U25%(1,16) and U25%(1,168)on《C4.5:programs for machine learning》
when calculating U C
《C4.5: Programs for Machine Learning》chaper4實驗結果重現
使用自帶的vote資料集: 實驗結果如下: 剪枝前: physician fee freeze = n: | adoption of the budget resolution = y: democrat (151.0) | adoption of the budget resolution
【論文閱讀】韓鬆《Efficient Methods And Hardware For Deep Learning》節選《Learning both Weights and Connections 》
Pruning Deep Neural Networks 本節內容主要來自NIPS 2015論文《Learning both Weights and Connections for Efficient Neural Networks》。 這部分主要介紹如何剪枝網路
【博觀而約取,深研而廣求】Researcher on Stochastic Process, Variational Inference, Computer Vision and Machine Learning.
Researcher on Stochastic Process, Variational Inference, Computer Vision and Machine Learning.
the resource for machine learning
Questions and Answers What's matrix dot product in Deep Learning? Deep Neural Network with Matrices https://matrices.io/deep-neural-network-from-scrat
[Infographic] The Best Tools for Machine Learning Gengo AI
Machine learning projects can range from small datasets and standard algorithms, to much larger projects that use neural networks engines with massive data
The 50 Best Public Datasets for Machine Learning
The 50 Best Public Datasets for Machine LearningWhat are some open datasets for machine learning? After scrapping the web for hours after hours, we have cr
Facebook's PyTorch plans to light the way to speedy workflows for Machine Learning • DEVCLASS
Facebook's development department has finished a first release candidate for v1 of its PyTorch project – just in time for the first conference dedicated to
Essential libraries for Machine Learning in Python
Python is often the language of choice for developers who need to apply statistical techniques or data analysis in their work. It is also used by data scie
Computer Vision and Machine Learning
Machine learning and computer vision methods have recently received a lot of attention, in particular when it comes to data analytics. The success of deep
Top 10 Open Image Datasets for Machine Learning Research
This article would succinctly describe the best ten image datasets used for certain fundamental computer vision problems such as classification, detecti
Why Data Normalization is necessary for Machine Learning models
Why Data Normalization is necessary for Machine Learning modelsNormalization is a technique often applied as part of data preparation for machine learning.
NXP Owns the Stage for Machine Learning in Edge Devices
SAN JOSE, Calif. and BARCELONA, Spain, Oct. 16, 2018 (GLOBE NEWSWIRE) -- (ARMTECHCON and IoT World Congress Barcelona) - Mathematical advances that are dri
NXP's New Development Platform for Machine Learning in the IoT
NXP Semiconductors has launched a new machine learning toolkit. Called "eIQ", it's a software development platform that supports popular neural network fra
Free Online Course: Neural Networks for Machine Learning from Coursera Class Central
I honestly can't understand the multiple 5 star reviews presented on this site about the course. I'm giving it a 1 star which is a bit harsh I know but I'm
Marginally Interesting: Slides for Machine Learning on Streams
Tweet Yesterday I gave a talk at the Big Data Beers meetup in Berlin on
Using Amazon’s Mechanical Turk for Machine Learning Data
How to build a model from Mechanical Turk resultsAmazon Mechanical Turk will notify you when your results are ready and you will finally have a labelled da