Camp Pre-work (required for everyone)

For all the following, it is fine to not do an exercise or reading if you don’t expect to learn anything. The only goal is that we can build together on all the following content during the camp.

Transversal

• Read Half-assing it with everything you've got, which is a precise description of the philosophy of this camp.
• You should be familiar with most of the content in these videos: Playlist - ML4G (there are 3 hours of videos, but skip what you already know)
• and this playlist from 3Blue1Brown.

AI Safety

• Skim chapters 1 and 2 of the AI Safety Textbook. Skip what you already know. I encourage you to comment and ask questions. We will organize sessions to discuss each one of those chapters, so being familiar with the notions presented is very important.

Implementation

We’ll be using PyTorch for all the implementations.

• Learn and practice the very basics of PyTorch: complete this notebook Tuto Numpy to Pytorch - Colab (solution)
• Deep Learning with PyTorch: A 60 Minute Blitz: all 4 pages, click next at the bottom left, content on GPU at the end is not necessary and optional. Run the notebooks, experiment, modify the code to understand what it does and why it is like this. At the end you’ll have trained a classifier on CIFAR10

You’ll also need familiarity manipulating high dimensional tensors if you want to understand the details of the transformer’s implementation.

• Read about the benefits of the einops library here.
• Do the manipulating high dimensional tensors tutorial. The goal here is not to pass all the exercises, but to get familiar with the manipulation of those objects. There are a lot of exercises in this notebook: stop once you can solve them without documentation on the 1st/2nd try. Don’t spend more than 2 hours on this.
  • You may need the documentation for einops.

Tip: Remember that you can ask ChatGPT for help explaining some of the concepts explored here. You can also ask it to ask you questions to test your understanding.

Extra resources

We also provide extra resources on the topics that will be useful for the camp. They are meant to bring you up to speed on topics you might be less comfortable with. You are not expected to dive thoroughly in every link, but instead spend time wisely depending on your weaknesses.

Please select the areas that you need to review depending on your strengths.

Mathematics