Jekyll2022-11-18T04:15:18-08:00https://aletheap.github.io/feed.xmlLearning Out LoudOpenAI Scholar - Learning about AI and deep learningAlethea PowerShame2020-07-02T00:00:00-07:002020-07-02T00:00:00-07:00https://aletheap.github.io/posts/2020/07/shame<p>2020 has turned out to be a completely outrageous year. Not at all what I expected when I joined OpenAI’s Scholars Program. All told, it’s been extremely hard to concentrate on my work with personal medical issues, and the world falling apart. While I wish so many of the things that happened this year hadn’t, I am grateful that racial inequality is getting such high profile, and desperately needed attention. And honestly, being distracted in my program is such a tiny price for me, as a white person to pay for the good that I hope may come out of this revolutionary time.</p>
<p>With that said, my original vision for this blog (as a narrative arc where I share and de-mystify my process of learning) is no longer practical. I’ve been too distracted to keep up blogging as I’ve learned, so I’m not going to try to go back and write months worth of things all at once. Instead I’m going to jump forward to where I am now, write about what I’m learning, and try to backfill what’s needed in the future.</p>
<p>There is one thing I’ve learned over the past few months that I do think is important to highlight here, though. That’s about shame. In mid-February, I started struggling with asthma. My asthma made me unable to concentrate or work productively, which made me feel ashamed that I wasn’t doing as much or as well as I wanted to, and knew I could. I got wrapped up in that shame, and it turned into anxiety, which exacerbated the asthma, and it was a downward spiral. Through this process I did manage to solve the original environmental causes of the asthma, and I finally managed to get my anxiety under control as well. To do that, I had to embrace the parts of me that I was ashamed of. I had to accept that maybe I wasn’t going to be able to finish the program or my project at all, that maybe I was just going to fail and maybe that really was the best I could do.</p>
<p>Only when I started to look at that as a real possibility and make peace with the idea of failure, did I finally start taking the steps I needed to take to care for my health. I got a prescription for anxiety medication and I also cut myself off from the news for a while. This felt dangerous as a trans person, but it was an urgent self-care decision. When I stayed in touch with the news, my heart hurt. I don’t mean emotionally. I mean, reading the news literally caused strong physical pain in my chest. My heart started skipping beats regularly, and I stopped sleeping. For a while, my body was also losing the ability to regulate my temperature. I would be bundled up and freezing at 75 degrees but my thermometer showed I wasn’t running a fever. My allergist told me it was the first time she’d ever seen me so fragile.</p>
<p>And through all of that, I felt like it was my one shot to finally follow the career I’d been dreaming of for so long. I felt like I needed to impress my mentor and my co-workers and show them what I knew I could do. And I felt like that chance was slipping out of my hands. But the reality was so much kinder and more open-ended than that. I remember talking to Christina, one of the recruiters who runs the program, back in April and I just broke down crying. She was so kind. She told me they knew how hard this period was for all of us and that she and my mentor, Christine, were working to get us more support and an extension for our projects. I was so grateful for the compassion in my period of fear and struggle. One of the things that’s impressed me the most about OpenAI is the company’s compassion. Every person I’ve talked to there has been an incredible combination of smart and kind. On the weeks when I couldn’t get any work done at all, Christine gently supported and encouraged me. She helped me relax and set aside my belief that I had to be perfect all the time and she just told me to do what I can. That calmed me down enough that, in the end, I was able to put together a project that’s gotten a lot of praise and that I’m really quite proud of.</p>
<p>So, I’m writing this post for other people dreaming of going into AI, especially as the world teeters on the brink of precipices that might threaten you personally. If you see my work in the future and think you have to prove yourself every minute of every day to do what I’ve done, you don’t. You can be weak. You can be afraid. You can struggle. And you can still succeed. I know because I did it. You just need to be kind to yourself and you need to have people around you who are also kind to you. You deserve that kindness. If you’re part of a minority group, you may sometimes forget that you deserve kindness because you may be less accustomed to receiving it than others. But you do. And if you accept your failings, and love yourself despite them, and do your best to surround yourself with people who will treat you with kindness, then you can be successfuln and you can be happy, which is so much better than perfect.</p>Alethea Power2020 has turned out to be a completely outrageous year. Not at all what I expected when I joined OpenAI’s Scholars Program. All told, it’s been extremely hard to concentrate on my work with personal medical issues, and the world falling apart. While I wish so many of the things that happened this year hadn’t, I am grateful that racial inequality is getting such high profile, and desperately needed attention. And honestly, being distracted in my program is such a tiny price for me, as a white person to pay for the good that I hope may come out of this revolutionary time.Looking for Grammar in all the Right Places2020-07-02T00:00:00-07:002020-07-02T00:00:00-07:00https://aletheap.github.io/posts/2020/07/z-looking-for-grammar<h2 id="interpretability">Interpretability</h2>
<p>Over the course of the OpenAI Scholars Program, I became fascinated with Interpretability. Interpretability is like “mind reading” for neural networks. It’s about looking inside of networks to understand how they represent and process information. This is difficult because of how different deep learning is from traditional software engineering. In traditional software engineering, a human being writes software, and that software takes inputs and gives outputs. (If the software is a word processor, the inputs are keystrokes and clicks and the outputs are documents. If the software is a search engine, the inputs are search quesries and the outputs are links to webpages.)</p>
<p><img src="/images/looking-for-grammar/traditional_software_engineering.png" alt="Traditional Software Engineering" /></p>
<p>In deep learning, a human being creates math and feeds a bunch of training data through that math. Tthe data, the math, and the computer(s) create a piece of software, which takes inputs and outputs. The human being does not directly create the software and consequently does not plan how it will work and (most likely) does not even know how it works:</p>
<p><img src="/images/looking-for-grammar/deep_learning_software.png" alt="Deep Learning Software" /></p>
<p>It’s much harder to understand software created by math than it is to understand software created by human beings.</p>
<p>But it’s also extremely important to understand how AI systems in our lives work. These systems have a huge impact on our lives.</p>
<h2 id="gpt-2-interpretation">GPT-2 Interpretation</h2>
<p>I’m particularly fascinated by transformer-based language models, so I decided to try my hand at interpreting GPT-2, a transformer-based language model which had state of the art performance when it was released by OpenAI in early 2019. As a tractable first project, I decided to look for how GPT-2 understands English grammar. For a lay-person friendly explanation of transformers, and GPT-2 in particular, I encourage you to watch the short talk I gave about this project, here:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/J1rRYpmnUVE" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>For a deeper dive into transformers, I recommend <a href="https://jalammar.github.io/illustrated-transformer/">The Illustrated Transformer</a>.</p>
<h2 id="datasets">Datasets</h2>
<p>I decided to look at GPT-2’s representations of simple part of speech, detailed part of speech, and syntactic dependencies. To accomplish this, I began by building three large datasets of sentences, one for each of these categories. First I started with the <a href="https://nyu-mll.github.io/CoLA/">Corpus of Linguistic Acceptability</a>, which is a widely known dataset that labels sentences as grammatically and incorrect. I dropped all of the grammatically incorrect sentences and kept only the grammatically correct ones. Then I tokenized these sentences using <a href="https://spacy.io/usage/linguistic-features">spaCy</a> and the <a href="https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizer">GPT-2 BPE tokenizer</a>. I only kept sentences whose spaCy tokenization resulted in the same number of tokens as the GPT-2 tokenization. In other words, sentences that had a one-to-one correlation between words + punctuation marks and GPT-2 BPE tokens. Then, for each sentence, I labeled that sentence with the list of spaCy <code class="language-plaintext highlighter-rouge">token.pos_</code> (simple part of speech), <code class="language-plaintext highlighter-rouge">token.tag_</code>(detailed part of speech), and <code class="language-plaintext highlighter-rouge">token.dep_</code> (syntactic dependency) tags for each token in the sentence. For the final puctuation mark, I kept the punctuation intact. So here’s an example of how a sentence would get labeled for each dataset</p>
<p><strong>Example sentence: “I enjoyed this project!”</strong></p>
<table>
<tbody>
<tr>
<td><strong>Dataset</strong></td>
<td><strong>Sentence Label</strong></td>
<td><strong><code class="language-plaintext highlighter-rouge">I</code></strong></td>
<td><strong><code class="language-plaintext highlighter-rouge">enjoyed</code></strong></td>
<td><strong><code class="language-plaintext highlighter-rouge">this</code></strong></td>
<td><strong><code class="language-plaintext highlighter-rouge">project</code></strong></td>
<td><strong><code class="language-plaintext highlighter-rouge">!</code></strong></td>
</tr>
<tr>
<td><strong>simple parts of speech</strong></td>
<td><code class="language-plaintext highlighter-rouge">PRON|VERB|DET|NOUN|!</code></td>
<td>PRON (pronoun)</td>
<td>VERB (verb)</td>
<td>DET (determiner)</td>
<td>NOUN (noun)</td>
<td>!</td>
</tr>
<tr>
<td><strong>detailed parts of speech</strong></td>
<td><code class="language-plaintext highlighter-rouge">PRP|VBD|DT|NN|!</code></td>
<td>PRP (pronoun, personal)</td>
<td>VBD (verb, past tense)</td>
<td>DT (determiner)</td>
<td>NN (noun, singular or mass)</td>
<td>!</td>
</tr>
<tr>
<td><strong>syntactic dependencies</strong></td>
<td><code class="language-plaintext highlighter-rouge">nsubj|ROOT|det|dobj|!</code></td>
<td>nsubj (nominal subject)</td>
<td>ROOT (None)</td>
<td>det (determiner)</td>
<td>dobj (direct object)</td>
<td>!</td>
</tr>
</tbody>
</table>
<p>Then I used <a href="https://github.com/jcpeterson/openwebtext">openwebtext</a> to download a few months worth of webtext websites. Then I extracted the sentences from those, and added each sentence to my datasets if and only if (1) the number of spaCy tokens in the sentence matched the number of GPT-2 tokens in the sentence, and (2) the label for that sentence matched a pre-existing label in that dataset for a grammatically correct sentence from CoLA. Finally I split each of my three datasets into train, validation and test sets and ended up with:</p>
<table>
<tbody>
<tr>
<td><strong>Dataset</strong></td>
<td><strong>Labels/Grammatical Structures</strong></td>
<td><strong>Training labels with > 500 sentences each</strong></td>
<td><strong>Total Training Sentences</strong></td>
<td><strong>Total Validation Sentences</strong></td>
<td><strong>Total Test Sentences</strong></td>
</tr>
<tr>
<td><strong>simple parts of speech</strong></td>
<td>2,837</td>
<td>104</td>
<td>222,794</td>
<td>15,903</td>
<td>16,017</td>
</tr>
<tr>
<td><strong>detailed parts of speech</strong></td>
<td>3,162</td>
<td>60</td>
<td>125,756</td>
<td>8,948</td>
<td>9,226</td>
</tr>
<tr>
<td><strong>syntactic dependencies</strong></td>
<td>2,643</td>
<td>157</td>
<td>333,427</td>
<td>23,962</td>
<td>24,119</td>
</tr>
</tbody>
</table>
<h2 id="measuring-grammatical-understanding">Measuring Grammatical Understanding</h2>
<p>After building these three datasets, I replaced the <a href="https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel">language modeling linear layer</a> on top of GPT-2 with three other, independent linear layers which output probilities of grammatical tokens like this:</p>
<p><img src="/images/looking-for-grammar/grammar_classifiers.gif" alt="" /></p>
<p>I then froze GPT-2 and trained each of my grammatical classifier linear layers using cross-entropy loss. I recorded the loss after each epoch of training. In addition, I repeated this training process on top of each transfomer layer, as well as on top of the imput embedding (using no transformer layers of GPT-2). I looked at how slow/difficult it was to train these classifiers on each transformer layer and found that they trained the fastest and achieved the best loss on the middle layers of the network. Here’s what that looked like for syntactic dependency loss:</p>
<p><img src="/images/looking-for-grammar/dep_loss.png" alt="Dependency loss by layer" /></p>
<p>The horizontal axis here is epochs and the vertical axis is the number of transformer layers of GPT-2, from 0 transfomer layers (directly on top of the input embedding) to 12-transformer layers (all of gpt-2 small). The color goes from red (high loss), to green (low loss). The syntactic dependency classifier has a progressively easier and easier time classifying the incoming sentence as we increase through the first five transformer layers of GPT-2, until it got its best overall score (in the fewest epochs) at layer 5. Then at higher layers, it starts to have a harder time again.</p>
<p>This made me think the first half of the network might be focused on understanding the incoming tokens and the second half of the network might be focused on producing the outgoing tokens. Since GPT-2 is built to generate probabilities of subsequent tokens in each position, I tested my theory by shifting the grammatical labels one position to the left (so they would better match the outgoing tokens) and repeating the experiment. And here’s what I saw:</p>
<p><img src="/images/looking-for-grammar/dep_loss_shifted.png" alt="Outgoing dependency loss by layer" /></p>
<p>When the classifier was trying to produce the grammatical structure of a likely output sentence, its loss was high in the first half of the network and only got better in the second half, with the best loss score coming at layer 8. This was convincing evidence that the first half of the network was, indeed, more focused on the grammar of the incoming tokens and the second half was more focused on the grammar of the outgoing (probable) tokens. In addition, this gave a small validation that the training results for my linear layer do serve as a viable measure of informational availability at each transfomer layer.</p>
<p>It’s also interesting (and makes sense) that the outgoing classifier scored a much higher loss than the incoming classifier. Since the network does not deterministically produce output tokens, and since future input tokens are not allowed to impact past output positions (by means of attention masking in GPT-2), this means that the output is more flexible and less precisely defined than the input. So it makes sense that it would be harder to definitively say that the output token will match the grammatical structures of the input sentences. This is all a long-winded way of saying that it’s harder to describe the future than the past, because the future is not precisely known the way the past is.</p>
<p>I also ran this experiment for simple part of speech and detailed part of speech and found these results:</p>
<p><img src="/images/looking-for-grammar/pos_loss.png" alt="Simple POS loss by layer" /></p>
<p><img src="/images/looking-for-grammar/tag_loss.png" alt="Detailed POS loss by layer" /></p>
<p>We can see that it was slightly easier for the classifier to understand simple part of speech than to understand detailed part of speech. This makes sense because simple part of speech is, well, simpler. In addition, both simple and detailed part of speech have the best loss scores at layer 3, which indicates that part of speech is easier to extract from the initial embedding vectors than syntactic dependency is. This would be expected because part of speech can (often) be determined from simply knowing an individual token, but syntactic dependency requires that token, its position, and likely the other tokens in nearby positions. The embedding space could contain some part of speech information but it’s unlikely to contain much, if any, syntactic dependency information.</p>
<h2 id="hunting-wabbit-heads">Hunting Wabbit (Heads)</h2>
<p>Training the classifiers told me that incoming part of speech is iunderstood in layers 1-3 and incoming syntactic dependencies are understood in layers 1-5. So, I used truncated versions of GPT-2 (with the grammatical classifiers I trained on these trucated versions) to search for which heads played the biggest role in understanding grammar.</p>
<p>Initially, I followed the method laid out in the paper <a href="https://arxiv.org/abs/1905.10650">Are Sixteen Heads Better Than One?</a>, which was also implemented by Huggingface in their <a href="https://github.com/huggingface/transformers/blob/v3.0.1/examples/bertology/run_bertology.py">Bertology example</a>. This technique involves creating a ones tensor whose dimensions are the number of transformer layers and the number of attention heads per layer. Then I set this tensor to require gradients and multiplied the <code class="language-plaintext highlighter-rouge">[i, j]</code>th element of the tensor by the output of the <code class="language-plaintext highlighter-rouge">j</code>th attention head in layer <code class="language-plaintext highlighter-rouge">i</code>. Then I used back-propagation to calculate the jacobian of the grammatical classification loss with respect to the head coefficients. The idea was that if the derivative of the loss with respect to the coefficient of a given head is low, then this gives us some indication that that head was not extremely important for labeling the grammatical structure. And by contrast, if the derivative of the loss with respect to the coefficient of a head was high, then that head shoudlbe important for understanding the grammar of the incoming tokens.</p>
<p>Since the loss can be (and almost certainly is) a non-linear function of these coefficients, though, the derivatives with respect to the coefficients are only local, linear approximations of the importance of the heads. So, I decided to compare the results of this strategy with the impact on the loss when I pruned each individual head (which gives a definitive measurement of that head’s importance). And sadly I found that the coefficient derivative strategy did not do as good of a job of reflectiing the results as pruning the heads did.</p>
<p>So, in the end, I just looked at the impact on the loss for each grammatical structure from pruning each head. And here are the impact maps for the top 30 syntactic dependency structures in my dataset:</p>
<p><img src="/images/looking-for-grammar/head_impacts_dep.png" alt="Head Impacts" /></p>
<p>Using these maps, I slowly pruned more and more heads out of GPT-2 for each syntactic dependency structure, so as to find which combination of pruned heads produced the lowest loss. Here are the best masks for the top 30 syntactic dependency structures. Note that black means the head is pruned and white means it’s retained):</p>
<p><img src="/images/looking-for-grammar/head_impact_masks_dep.png" alt="Head Impact Masks" /></p>
<p>These masks seem extremely likely to give a map of which heads are involved in understanding each of these incoming syntactic dependency structures. We can see that similar structures require similar collections of heads, and that simpler grammatical structures require fewer heads than more complex grammatical stcutures. Finally, I looked for which heads have the most impact for structures containing each syntactic dependency label and I found that many labels seem to each be understood by a very small number heads:</p>
<p><img src="/images/looking-for-grammar/dep_head_impact_by_label.png" alt="Head impact for each dependency label" /></p>
<p>In the future, I would like to test whether the heads needed for an given sentence structure are the same as the union of the heads needed for the labels comprising that sentence structure. I would also like to open up the individual heads for each part of speech label and understand how the query, key, and value weights map clusters of tokens in the incoming from the embedding space to clusters in the much lower dimensional key/value spaces. I have a suspicion that tokens in the embedding space will cluster into parts of speech and that these cluster boundaries will be critical in the projections to lower dimension key/value spaces performed by grammatical heads.</p>Alethea PowerInterpretabilityDeep Learning Hardware2020-02-24T00:00:00-08:002020-02-24T00:00:00-08:00https://aletheap.github.io/posts/2020/02/deep-learning-hardware<p>Deep learning requires special kinds of computer hardware. This is a post about what makes that hardware so different from the traditional computer architecture, and how to get access to the right kind of hardware for deep learning. If you just want practical recommendations about how to get your hands on the right hardware, feel free to jump to <a href="#so-what-should-i-do">the end</a>. If you want to understand what makes GPUs (and TPUs) such a philosophically different approach to computing than CPUs, and why that matters for deep learning, then read on…</p>
<h2 id="history">History</h2>
<p>To really understand the difference between hardware that works well for deep learning and hardware that doesn’t, we need to cover some history. In the mid 19th century, there was a growing crisis in the mathematical world. The historical techniques that had worked well for reasoning about algebra and geometry were producing contradictions when mathematicians tried to apply them to some of the more subtle infinities and infinitesimals of calculus. So mathematicians started looking for new, more precise ways of reasoning that could resolve these contradictions.</p>
<p>This was the birth of modern formal logic. At its core, it was a way to translate math into strings of symbols, and rules for using prior strings of symbols to derive new strings of symbols that represented consistent statements about mathematics. You’re probably familiar with something very similar to this kind of deductive symbol processing from algebra:</p>
\[\begin{align}
x &= (4+5)/x\\
x &= 9 /x\\
x*x &= 9\\
x^2 &= 9\\
x &= \sqrt{9}\\
x &= 3
\end{align}\]
<p>Each step here is a string of symbols, and there are specific rules for how to convert from the string of symbols in one step to the string of symbols in the next step. For instance, to go from step 1 to step 2, we follow a rule that tells us we’re allowed to convert these five symbols: “\((4+5)\)” into this one: “\(9\)”. And similarly, to go from step 3 to step 4, we follow a rule that tells us we’re allowed to convert “\(x*x\)” into “\(x^2\)”.</p>
<p>Formal mathematical logic can be thought of as a generalization of this process, that allows one to express all of mathematics and all mathematical proofs as sequences of strings of symbols, where each string was derived from prior strings in accordance with rules of symbolic deduction. As mathematicians understood how these systems of symbolic logic worked, they started describing such systems in terms of machines that could perform the operations that would transition from one string of symbols to the next. These theoretical machines were the basis of modern computers. In fact, most computers built today still roughly follow the general architecture laid out by the mathematician, John von Neumann, in 1943.</p>
<h2 id="von-neumann-computers-and-central-processing-units-cpus">von Neumann Computers and Central Processing Units (CPUs)</h2>
<p>The <a href="https://en.wikipedia.org/wiki/Von_Neumann_architecture">von Neumann architecture</a> consists of a few basic building blocks:</p>
<ul>
<li><strong>Input</strong> - This is a way for a person to give the computer data and instructions for processing that data. On modern desktop computers, this might be a keyboard and mouse, and on servers, this might be a network connection that takes requests.</li>
<li><strong>Memory</strong> - This is where the data and instructions are stored. Today this means both RAM and Hard Drive. RAM is <em>much</em> faster than hard drives (usually thousands of times faster), but it’s not able to retain information without a constant supply of electricity. So long term storage is on hard drives, and short term storage is in RAM.</li>
<li><strong>Output</strong> - This is how the computer returns the results of the computation. On a modern desktop, this would be a screen or speakers. For a modern server this would be a reply sent through the network connection.</li>
<li><strong>Arithmetic Unit</strong> - This is what actually performs the symbolic manipulation to go from one or more strings of symbols to the next string of symbols.</li>
<li><strong>Control Unit</strong> - This keeps track of which string(s) the Arithmetic Unit needs to process for the current step.</li>
</ul>
<p><strong><em>Today, the Arithmetic Unit and Control Unit are bundled together into a “Central Processing Unit” or CPU.</em></strong></p>
<p>The von Neumann architecture, and von Neumann-style CPUs in particular, are excellent at performing the kinds of sequential symbolic processing that computers were originally designed for. This worked well for many years while the primary computer interface was a prompt where the user would input a string of symbols, which the computer would process to calculate the next string of symbols.</p>
<p><img src="/images/terminal_math.gif" alt="Computer Terminal" /></p>
<p>Eventually though, it turned out that formal logic was only a tiny fraction of the diversity of ways that human beings engage with information (who knew?!), and it wasn’t long before people wanted to use these new, fast electrical information machines for more diverse tasks. So the Graphical User Interface (GUI) was born:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/Cn4vC80Pv6Q" frameborder="10" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>With the growth of graphical interfaces, CPUs were used to compute more abstract and complex things than before, including windows and icons on higher resolution screens. This was only possible because the CPUs of the early 1980s had become fast enough to perform millions of symbolic processing steps per second.</p>
<p>For a number of years, transistor sizes shrank according to <a href="https://en.wikipedia.org/wiki/Moore%27s_law">Moore’s Law</a>, and CPU speed continued to increase accordingly. And so CPUs were able to draw more and more complex and rapidly changing user interfaces including many video games. But ultimately, this was a <em>hack</em>. The basic design of CPUs was developed by mathematicians to perform the kind of sequential symbolic operations that happen in mathematical proofs, which are a profoundly different kind of task than drawing complex and rapidly changing images on a screen. As the video game industry took off, it was clear that CPUs could not perform sequential operations fast enough, and a new type of parallel computational unit would have to be invented. And so the Graphical Processing Unit (or GPU) was born.</p>
<h2 id="graphical-processing-units-gpus">Graphical Processing Units (GPUs)</h2>
<p>GPUs were a fundamental departure from the von Neumann architecture. Where CPUs were designed to operate on strings of symbols, performing <em>one computation at a time</em>, GPUs were designed to operate <em>in parallel</em> on huge collections of numbers to calculate whole expanses of a visual field simultaneously. But GPUs were originally thought of as just a tool to draw pictures on the screen.</p>
<p>Additionally, the long history of von Neumann computers meant that the majority of programming languages were designed for such computers. Today this includes languages like C, C++, Java, Javascript, Python, Swift, Go, Ruby, Rust, etc. Since these programming languages were all designed for von Neumann computers, this means that all of the software written in these languages runs on CPUs. This includes every major operating system, every word processor, every web browser, every email client, etc. This software legacy means that any GPU attached to a modern computer will be a <em>peripheral</em> component. And today, GPUs connect to traditional CPU-based computers using a protocol known as PCI (Peripheral Component Interconnect).</p>
<p>In servers and desktops, a GPU is one or more chips attached to a physical expansion card known as a “graphics card” or a “video card”. These expansion cards get their name from their card-like shape, which plugs into a PCI slot on a computer’s motherboard:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/nyDxrTHDjXQ?start=177" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>In laptops, televisions, smart phones, and smartwatches today, there are no PCI slots and consequently no expansion cards to plug into them. People do still sometimes refer to GPUs in laptops as “graphics cards” out of habit, even though this is technically inaccurate. Laptop GPUs are soldered directly onto motherboards as shown here in red:</p>
<p><img src="http://www.laptoprepair101.com/wp-images/motherboard-NVIDIA-problem/fix-failed-laptop-nvidia-chip-01.jpg" alt="" /></p>
<p>The computing paradigm of GPUs was so radically different than traditional CPUs that GPUs were originally largely isolated from the software running on CPUs. GPUs were only exposed to such software via higher level graphics APIs designed for rendering images on screens. Eventually, though, GPU manufacturers started to understand the unique computational power of GPUs, and they created programming languages like CUDA, with low level APIs for programmers to write code that could directly execute in parallel on GPUs.</p>
<p><strong>The subsequent shift from von Neumann style symbolic processing CPUs to massively parallel calculations on GPUs was the core technological advancement that made the modern Deep Learning Era possible.</strong></p>
<p>Deep learning is inspired by the neural information processing patterns of the human brain. And human brains are fundamentally not von Neumann machines. Human beings evolved from single cell organisms and we inherited the cellular structure of most life on Earth. Evolution didn’t have high speed, high energy silicon transistors available to build our brains, and formal mathematical logic wasn’t the top priority for the early stages of our ancestors’ cognitive development. We had to develop ways to use cells to perform analog, spatial tasks like moving toward light, or recognizing and fleeing from predators. And consequently, animal and human brains are large collections of cellular neurons that process information in massively parallel ways, much more like GPUs than CPUs. This is why GPUs are so much better suited for running software simulations of neural networks than CPUs are.</p>
<p>To quote Jeremy Howard, “<a href="https://www.youtube.com/watch?v=4u8FxNEDUeg&t=44m03s">When we say Python <em>[on a CPU]</em> is too slow, we don’t mean twenty percent too slow; we mean thousands of times too slow</a>.”</p>
<h2 id="nvidia">Nvidia</h2>
<p>While many companies today make GPUs for video games, one company in particular, Nvidia, realized the revolutionary importance of GPU computing for deep learning earlier than the others, and they prioritized making high quality programming languages and software libraries for deep learning tools to build on. In the video game space, AMD is the biggest competitor to Nvidia, and their GPUs have the hardware capabilities needed for deep learning, but the AMD software stack is nowhere near as robust as Nvidia’s. So nearly all modern deep learning libraries today are built for Nvidia GPUs.</p>
<p>At this point its worth mentioning that Python is a very popular language for deep learning. But Python runs on CPUs, not GPUs. What makes Python a viable candidate for deep learning is its ability to transparently call code written in other languages, including languages for GPUs. So for instance, in <a href="https://pytorch.org/">PyTorch</a> when you see <code class="language-plaintext highlighter-rouge">model.to(torch.device("cuda"))</code>, this is where Python is loading code written in CUDA into the GPU to execute there. If you run a PyTorch deep learning program without this line, you’ll find it’s thousands of times slower than with this line. This is because the code would execute on the CPU rather than the GPU.</p>
<p>Since nearly all deep learning libraries today are (ultimately) built on CUDA, Nvidia’s GPU programming language, this means Nvidia effectively has a monopoly on GPUs suitable for running deep learning programs. Interestingly though, Nvidia does not have a monopoly on GPUs capable of running video games. Nvidia must compete on price with AMD in the desktop/laptop GPU space, but doesn’t have to compete with anyone in the deep learning server space. This has led Nvidia to differentiate their GPUs into two lines: one for personal computers and one for servers. While these two lines are not drastically different in computational capacity, the licensing for the personal computer GPUs forbids using them in datacenters. This allows Nvidia to charge much more money for datacenter GPUs than it does for comparable personal computer GPUs.</p>
<p>Given how critical GPUs are for deep learning, one of the most important advancements for deep learning today would be porting deep learning libraries like <a href="https://www.tensorflow.org/">Tensorflow</a> to run on AMD GPUs. This would break Nvidia’s monopoly on commerical deep learning hardware and drastically drive down pricing.</p>
<h2 id="so-what-should-i-do">So what should I do?</h2>
<p>If you’re new to deep learning, you’re going to need access to some kind of computer with a fast Nvidia GPU. You can either access such a computer online through a cloud provider, or you can buy or build your own. As I described above, Nvidia has a monopoly on deep learning GPUs, so cloud providers will have to pay much more for GPUs in the datacenters than you would have to pay for a comparable personal GPU for your own computer. This might lead you to assume that buying your own computer with a GPU is the way to go. That would probably be true if not for Google. Google is offering very basic, and limited access to cloud servers with GPUs for the low, low cost of… NOTHING.</p>
<h3 id="google-colab">Google Colab</h3>
<p>That’s right, Google has a cloud service called <a href="https://colab.research.google.com/notebooks/intro.ipynb">Colaboratory</a> (or “Colab”, for short), which offers free access to basic GPUs, through a <a href="https://jupyter.org/">Jupyter Notebook</a> interface for up to 12 hours at a time, for free. For someone just starting in deep learning, this is an excellent way to get easy access to a server with the right kind of hardware (and already pre-configured with the right software). The service offered through Colab is fast enough to start learning about deep learning, but there’s a good chance you’ll eventually hit the limits of what you can do with it. If those limits are primarily about speed, you can try upgrading to the $10/month Colab Pro plan which guarantees you faster GPUs and longer execution times.</p>
<h3 id="your-own-hardware">Your own hardware</h3>
<p>If you outgrow Colab, then it may be time to buy or build your own server. Rather than dig deep into the details of what GPU you should get, I’ll point you to <a href="https://timdettmers.com/2019/04/03/which-gpu-for-deep-learning/">Tim Dettmer’s excellent page</a>, which he updates with each new hardware release. As of Spring 2020, Tim recommends an RTX 2070 for most newish users (or RTX 2080 Ti or potentially RTX Titan upgraded). These recommendations will likely change when Nvidia releases new video cards in mid-2020. If you would like to buy a pre-configured desktop or laptop, I would recommend buying from:</p>
<ul>
<li><a href="https://system76.com/">System76</a> - They sell high quality Linux machines with options to add the kinds of GPUs you need for deep learning. This is a good source for a person wanting to break into the field.</li>
<li><a href="https://lambdalabs.com/">Lambda Labs</a> - They sell <em>high end</em> deep learning workstations, preconfigured with multiple GPUs and deep learning software. This is a good source for a small business trying to start a small AI team.</li>
</ul>
<p>If you want something a little cheaper and have a desktop around that you’re willing to run Linux on and install a GPU in, then you should go for <a href="https://ubuntu.com/download">Ubuntu Linux</a> which is the most popular Linux used in AI today, or <a href="https://system76.com/pop">Pop!_OS</a>, which is a compatible derivative of Ubuntu that has a lot of additional stuff built in to make it nicer to use. As for where to buy a GPU, <a href="https://www.newegg.com/">Newegg</a> amd <a href="https://www.ebay.com">Ebay</a>, are both good choices. If you’re buying multiple GPUs to put in a desktop, make sure to buy “<a href="https://www.youtube.com/watch?v=0domMRFG1Rw">blower style</a>” GPUs, which vent the heat out the back of the machine.</p>
<h3 id="cloud-services">Cloud Services</h3>
<p>If you outgrow a desktop or two, especially if maintaining the OSes on them becomes too complex, then you may want to consider renting cloud based compute from <a href="https://cloud.google.com/">Google Cloud Platform</a>, <a href="https://aws.amazon.com/">Amazon Web Services</a>, or <a href="https://azure.microsoft.com/">Microsoft Azure</a>. Note that cloud based servers are much more expensive on an ongoing basis than running your own desktop, and they also use a completely different line of GPUs.</p>
<p>If you’re going with cloud servers, It’s important to understand the idea of virtual machines (or VMs). A virtual machine is a piece of software which acts like it’s a physical computer, even though it’s just software running on another computer. Here’s a picture of Microsoft Windows running inside of Virtual Machine software, which is in turn running on macOS, which is in turn running on a physical (Mac) computer:</p>
<p><img src="/images/Virtual_Machine.png" alt="Image of Microsoft Windows running on a virtual machine, which is running on macOS, which is running on a physical compuetsr" /></p>
<p>When you rent servers from a cloud provider, you’re renting access to this kind of virtual machine, which is in turn running on the physical computers that the cloud company has installed in their datacenters. For your purposes, the machine you rent will act like a physical computer (though it may be slightly slower than a physical computer, since some of its hardware is emulated).</p>
<p>People sometimes refer to cloud virtual machines as “VMs”, “servers”, “nodes”, or “compute nodes”, and cloud providers have different “size” VMs you can rent. For instance, on the Azure Sponsorship account we get through OpenAI, Azure offers what they call “NC6_Promo” VMs, which each have 6 (virtual) CPUs and 1 GPU. The next size up that they offer is the “NC12_Promo”, which has 12 (virtual) CPUs and 2 GPUs. Then the next size up (and the maximum size we have access to) is an “NC24_Promo”, which has 12 (virtual) CPUs and 4 GPUs.</p>
<p>For reference, here are lists of popular Nvidia GPUs, in rough order of performance, as of early 2020:</p>
<p><strong>Personal Computer Nvidia GPUs (fastest to slowest):</strong></p>
<ol>
<li>RTX Titan</li>
<li>RTX 2080 Ti</li>
<li>RTX 2080 Super</li>
<li>RTX 2080</li>
<li>RTX 2070</li>
<li>RTX 2060</li>
<li>GTX 1080 Ti</li>
<li>GTX 1080</li>
<li>GTX 1070</li>
<li>GTX 1650</li>
<li>GTX 1060</li>
</ol>
<p><strong>Cloud GPUs (fastest to slowest):</strong></p>
<ol>
<li>V100</li>
<li>P100</li>
<li>P4</li>
<li>T4</li>
<li>K80 (this is really just two K40 GPUs on one graphics card)</li>
<li>K40</li>
</ol>
<h3 id="limited-benchmarks">Limited benchmarks</h3>
<p>As part of the OpenAI Scholars program, we receive some Microsoft Azure credit. I decided to use some of my credit to benchmark K40/80 GPUs against Google Colab and a GTX 1080 Ti in my home desktop: I built an image recognition network (similar to <a href="https://arxiv.org/abs/1512.03385">ResNet-34</a>) and I trained it on a few thousand images. I timed each epoch and averaged them to get these numbers. Note that this may not quite reflect the performance of the actual video cards since all of the cloud benchmarks were running inside of virtual machines and the benchmark on my desktop was not:</p>
<table>
<thead>
<tr>
<th>Server</th>
<th>GPU Kind</th>
<th># GPUs</th>
<th>speed (lower is better)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Azure NC6_Promo</td>
<td>K40</td>
<td>1</td>
<td>61.5 seconds</td>
</tr>
<tr>
<td>Google Colab</td>
<td>P100</td>
<td>1</td>
<td>37.0 seconds</td>
</tr>
<tr>
<td>Azure NC24_Promo</td>
<td>K40</td>
<td>4</td>
<td>22.4 seconds</td>
</tr>
<tr>
<td>Google Colab Pro</td>
<td>P100</td>
<td>1</td>
<td>22.4 seconds</td>
</tr>
<tr>
<td>My home desktop</td>
<td>1080 Ti</td>
<td>1</td>
<td>20.6 seconds</td>
</tr>
</tbody>
</table>
<p>Note that the Azure Servers technically have K80 graphics cards, but a K80 graphics card is just two K40 GPUs on one card, and Azure describes each virtual machine GPU as “Half of a K80”, which means in practice that they are K40s. Also note that the Google Colab Pro server had more RAM than the Google Colab free server did.</p>
<h2 id="bonus-round-tpus">Bonus Round: TPUs</h2>
<p>I mentioned above that Nvidia has a monopoly on deep learning GPUs. That’s true, but there is technically one other kind of hardware built for deep learning that you may want to know about: Google’s Tensor Processing Units (TPUs). TPUs are custom built by Google for deep learning and they’re only available to users through Google Cloud Platform and Google Collaboratory. TPUs are sufficiently restrictive that many deep learning programs built for GPUs won’t easily run on them, but if you can <a href="https://towardsdatascience.com/running-pytorch-on-tpu-a-bag-of-tricks-b6d0130bddd4">get your code to run on them</a>, they’re much faster than even the fastest Nvidia GPUs. And they’re also available for free or very cheap through Colab, so it could be worth checking out!</p>Alethea PowerDeep learning requires special kinds of computer hardware. This is a post about what makes that hardware so different from the traditional computer architecture, and how to get access to the right kind of hardware for deep learning.Starting from Scratch2020-02-14T00:00:00-08:002020-02-14T00:00:00-08:00https://aletheap.github.io/posts/2020/02/starting-from-scratch<h1 id="starting-from-scratch">Starting From Scratch</h1>
<p>I’ve been excited about AI since I was about seven years old. When I was a kid, my Dad told me stories about the sci-fi books he read, and I always thought the most interesting ones were about AI. I loved imagining that one day computers might think like people do, and that this could give us new insights into humanity. By the time I was in high school, I was dreaming of how to transfer my mind into a robot body and wondering whether I would still be conscious. I don’t know if these things will be possible in my lifetime, but they sparked my imagination, and my interest in AI has only grown over time.</p>
<p>I’ve spent many years reading news stories about AI and thinking “I wish I could do that”. But I haven’t studied machine learning and I don’t have a PhD, and honestly I’ve felt intimidated pursuing the field. I didn’t know whether someone like me could even get into this. So, after a number of years of doing other work and dreaming of AI, I finally decided I had to try (with some encouragment of my wonderful partner). :-) So, I applied for the <a href="https://openai.com/blog/openai-scholars-spring-2020/">OpenAI Scholars Program</a> and, much to my surprise, I was accepted into the 2020 cohort.</p>
<p>Now that I’m in the program, I want to share my process learning about something exciting and new to me. My hope is that seeing someone else starting from scrtach will inspire you to start (this or something else) from scratch as well. That’s why I’ve called this blog “Learning Out Loud”. I plan to learn out loud in front of you (and of course the pun is that I’m also going to be talking out loud about learning, both human and artificial). I know, my witty puns are <em>suuuuper</em> funny. ;-P Anyway, I hope you enjoy reading about my process of discovery, and I especially hope that if you’re excited and unsure whether to pursue this kind of work, then this blog can help give you the spark of courage and excitement to push you forward too. :-)</p>Alethea PowerI've been excited about AI since I was about seven years old....