Keeping up with ML

And trying to remain sane

Generated by Bing Image Creator

Keeping up with ML is hard, period. Doing that, vis-a-vis a day job is harder. Add all the beautiful things that come with adulting ( YAAAAAAYYYY I AM AN ADULT NOW…..AM I?…WAIT, WTF, ME?) makes it seem like a seemingly labyrinthine task.

So how do I do it? Well, I try, somehow, to do whatever I can to remain relevant in a field that seems to be reinventing itself every 12 months ( do I use GANs now or Diffusion models? LSTM or Transformers, which transformer? And don’t get me started on LLMs.)

Reading Papers

This one should be obvious: while I can’t do much research on my own while in a job that has nothing to do with it, I can still stand on the shoulders of giants by reading the latest papers. With the crazy amount of work published almost daily, finding relevant works is tricky, but the following sources work well for me.

• Twitter — Follow the right folks on Twitter, and your feed becomes the best source of staying updated
• PapersWithCode — Has the list of the top papers floating around in social media/GitHub.
• Youtubers — @YannicKilcher, @hu-po, @gabrielmongaras, @TheAIEpiphany- make life super-easy by covering the top papers being released in an easy-to-understand fashion.

In the beginning, it used to take me a lot of time to understand if a paper was actually good, and I spent a lot of time reading papers which weren’t that good. Now, with years of practice ( I read my first ML paper four years back ), I can mostly glaze through a paper's abstract/some figures, take a call on its quality and then decide if I want to spend time going deeper into it.

If I really like a paper, or if I want to increase my understanding of a particular field, I follow the Feynman method and try to learn it by teaching it. This led to the birth of my own YouTube channel, RR with deku, where I go deep into papers I find interesting, much inspired by @YannicKilcher.

Blogs/YT/Books

While information on the most cutting-edge research topics is mostly found in research papers, there is a lot to be said about the wonderful blogs written by experts and amateurs alike. If I have to learn about a new topic, I usually start by reading a few blogs about it on the internet since they explain the concepts in the most user-friendly manner.

While the blogs I read are usually whatever surfaces in the Google results, here are some of the blogs I follow

• Lilian Weng’s blog (https://lilianweng.github.io/) Has great overviews/survey sort of blogs on popular topics like Diffusion Models/LLMs/RL. Good to get a 100 feet view of the topics
• Eugene Yan’s blog (https://eugeneyan.com/) — Has good real-world ML/MLOps/Career/Engineering articles.
• Chip Huyen’s blog (https://huyenchip.com/blog/) — Similar to above

Aside from these, I also like to follow the engineering blogs of some of the major companies using ML to get an idea of how ML is actually used in products instead of just focusing on the algorithms; some good ones are

• Netflix’s tech blog (https://netflixtechblog.com/)
• Uber’s Engineering blog (https://www.uber.com/en-IN/blog/engineering/)
• Spotify’s blog (https://engineering.atspotify.com/, https://research.atspotify.com/blog/)
• Pinterest's blog (https://medium.com/pinterest-engineering)

Similarly, YouTube has played a big big role in my self-learning. It has videos on literally anything I want to learn more about. You can also watch a course from Stanford/MIT to gain structured knowledge about stuff OR watch the proceedings of a popular conference to get a whirlwind tour of all that’s happening in the field.

Recently, I have been picking up a lot more code walkthroughs through YT, in which some wonderful people go through the code of the models that make DL happen, and I have found that to be very helpful; some of the good ones are

• The AI Ephipany — Aleksa has a lot of very detailed walkthroughs of the GitHub repos of all the popular Diffusion Models, which I found to be very helpful.
• Machine Learning With Phil — My go-to channel for anything related to implementing Reinforcement Learning
• Umar Jamil — Has really good implementation videos for LLMs

While I don’t read a lot of technical books, I find them to be a good source of structured information. But I’ll still rate them lower than Blogs/YT. Some good technical books I read this year are

• Designing Machine Learning Systems
• Machine Learning Design Patterns
• System Design Interview — An insider’s guide

OpenSource/Side-projects

The above was mostly dealing with the theoretical end of things. But theory would only take you so far, and I am someone who likes to hack things a bit and I love to code.

This year, doing open-source contributions has emerged as a decent way for me to keep working on my machine-learning implementation skills. Open-source can be a hard nut to crack, and I have failed multiple times previously in the process. Thankfully, this year, I was able to contribute to Huggingface Diffusers (https://github.com/huggingface/diffusers). Contributing three models to their core pipelines ( Kandinksy, Consistency Models, Blip Diffusion). I also made a small fix (PR) to the OpenAI Consistency Models repo. I think the best way to get started with open-source is to get familiar with any one project, dive deep into it and then just look for something you can add to it.

Apart from open-source, I did a few side projects, like implementing a diffusion model from scratch in Pytorch, Jax. This was motivated by me wanting to learn exactly how a diffusion model works, and I thought the best way to do so would be to implement one from scratch. The same goes for learning JAX. Similarly, I wanted to play around with LLMs a bit, so I ended up creating an AMA bot for my personal website (blog)!

I think that is most of what I do to keep up with ML, along with my job. The most important thing driving me is probably curiosity and wanting to get better and better at my craft and remain relevant. I try, but it's a long journey!