Writing • Eugene Yan
eugeneyan
all machinelearning engineering datascience production learning llm career python productivity recsys leadership 🔥 🛠 life ≥20min writing deeplearning misc survey teardown omscs mechanism 🩷 eval lazada 📬 til ai agile
How to Work and Compound with AI
Context as infra, taste as config, verification for autonomy, scale via delegation, closing the loop.
03 May 2026  ·  13 min  ·  ai productivity mechanism
2025 Year in Review
An eventful year of progress in health and career, while making time for travel and reflection.
14 Dec 2025  ·  5 min  ·  life
Product Evals in Three Simple Steps
Label some data, align LLM-evaluators, and run the eval harness with each change.
23 Nov 2025  ·  9 min  ·  eval engineering production
Advice for New Principal Tech ICs (i.e., Notes to Myself)
Based on what I've learned from role models and mentors in Amazon
19 Oct 2025  ·  14 min  ·  leadership engineering career 🩷
Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs
An LLM that can converse in English & item IDs, and make recommendations w/o retrieval or tools.
14 Sep 2025  ·  23 min  ·  llm recsys learning 🛠 🩷
Evaluating Long-Context Question & Answer Systems
Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
22 Jun 2025  ·  28 min  ·  llm eval survey
Exceptional Leadership: Some Qualities, Behaviors, and Styles
What makes a good leader? What do good leaders do? And commando, soldier, and police leadership.
18 May 2025  ·  4 min  ·  leadership
Building News Agents for Daily News Recaps with MCP, Q, and tmux
Learning to automate simple agentic workflows with Amazon Q CLI, Anthropic MCP, and tmux.
04 May 2025  ·  8 min  ·  llm learning 🛠
An LLM-as-Judge Won't Save The Product—Fixing Your Process Will
Applying the scientific method, building via eval-driven development, and monitoring AI output.
20 Apr 2025  ·  5 min  ·  eval llm engineering
Frequently Asked Questions about My Writing Process
How I started, why I write, who I write for, how I write, and more.
30 Mar 2025  ·  13 min  ·  writing
Improving Recommendation Systems & Search in the Age of LLMs
Model architectures, data generation, training paradigms, and unified frameworks inspired by LLMs.
16 Mar 2025  ·  43 min  ·  recsys llm teardown 🔥
Building AI Reading Club: Features & Behind the Scenes
Exploring how an AI-powered reading experience could look like.
12 Jan 2025  ·  10 min  ·  llm learning 🛠 🩷
2024 Year in Review
A peaceful year of steady progress on my craft and health.
22 Dec 2024  ·  6 min  ·  life
Seemingly Paradoxical Rules of Writing
With regard to writing, there are many rules and also no rules at all.
01 Dec 2024  ·  2 min  ·  writing 🩷
How to Run a Weekly Paper Club (and Build a Learning Community)
Benefits of running a weekly paper club, how to start one, and how to read and facilitate papers.
24 Nov 2024  ·  6 min  ·  learning productivity mechanism
My Minimal MacBook Pro Setup Guide
Setting up my new MacBook Pro from scratch
17 Nov 2024  ·  5 min  ·  engineering misc
39 Lessons on Building ML Systems, Scaling, Execution, and More
ML systems, production & scaling, execution & collaboration, building for users, conference etiquette.
03 Nov 2024  ·  10 min  ·  machinelearning engineering production leadership
AlignEval: Building an App to Make Evals Easy, Fun, and Automated
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
27 Oct 2024  ·  14 min  ·  llm eval learning 🛠 🩷
Building the Same App Using Various Web Frameworks
FastAPI, FastHTML, Next.js, SvelteKit, and thoughts on how coding assistants influence builders' choices.
08 Sep 2024  ·  8 min  ·  learning engineering python 🛠
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
18 Aug 2024  ·  49 min  ·  llm eval production survey 🔥
How to Interview and Hire ML/AI Engineers
What to interview for, how to structure the phone screen, interview loop, and debrief, and a few tips.
07 Jul 2024  ·  21 min  ·  machinelearning career leadership 🔥
Prompting Fundamentals and How to Apply them Effectively
Structured input/output, prefilling, n-shots prompting, chain-of-thought, reducing hallucinations, etc.
26 May 2024  ·  17 min  ·  llm production 🔥
What We've Learned From A Year of Building with LLMs
From the tactical nuts & bolts to the operational day-to-day to the long-term business strategy.
12 May 2024  ·  1 min  ·  llm engineering production leadership 🔥
Building an AI Coach to Help Tame My Monkey Mind
Building an AI coach with speech-to-text, text-to-speech, an LLM, and a virtual number.
07 Apr 2024  ·  4 min  ·  llm ai life 🛠
Task-Specific LLM Evals that Do & Don't Work
Evals for classification, summarization, translation, copyright regurgitation, and toxicity.
31 Mar 2024  ·  33 min  ·  llm eval survey
Don't Mock Machine Learning Models In Unit Tests
How unit testing machine learning code differs from typical software practices
25 Feb 2024  ·  6 min  ·  machinelearning engineering python
How to Generate and Use Synthetic Data for Finetuning
Overcoming the bottleneck of human annotations in instruction-tuning, preference-tuning, and pretraining.
11 Feb 2024  ·  42 min  ·  llm survey
Language Modeling Reading List (to Start Your Paper Club)
Some fundamental papers and a one-sentence summary for each; start your own paper club!
07 Jan 2024  ·  6 min  ·  llm learning
2023 Year in Review
An expanded charter, lots of writing and speaking, and finally learning to snowboard.
31 Dec 2023  ·  8 min  ·  life
Push Notifications: What to Push, What Not to Push, and How Often
Sending helpful & engaging pushes, filtering annoying pushes, and finding the frequency sweet spot.
24 Dec 2023  ·  18 min  ·  teardown recsys machinelearning production
Out-of-Domain Finetuning to Bootstrap Hallucination Detection
How to use open-source, permissive-use data and collect less labeled samples for our tasks.
05 Nov 2023  ·  12 min  ·  llm eval machinelearning python
Reflections on AI Engineer Summit 2023
The biggest deployment challenges, backward compatibility, multi-modality, and SF work ethic.
15 Oct 2023  ·  7 min  ·  llm ai misc
Evaluation & Hallucination Detection for Abstractive Summaries
Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
03 Sep 2023  ·  23 min  ·  llm eval survey
How to Match LLM Patterns to Problems
Distinguishing problems with external vs. internal LLMs, and data vs non-data patterns
13 Aug 2023  ·  6 min  ·  llm production
Patterns for Building LLM-based Systems & Products
Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
30 Jul 2023  ·  66 min  ·  llm engineering production 🔥
Obsidian-Copilot: An Assistant for Writing & Reflecting
Writing drafts via retrieval-augmented generation. Also reflecting on the week's journal entries.
11 Jun 2023  ·  6 min  ·  llm engineering 🛠
Some Intuition on Attention and the Transformer
What's the big deal, intuition on query-key-value vectors, multiple heads, multiple layers, and more.
21 May 2023  ·  8 min  ·  deeplearning llm
Open-LLMs - A list of LLMs for Commercial Use
It started with a question that had no clear answer, and led to eight PRs from the community.
07 May 2023  ·  1 min  ·  llm misc
Interacting with LLMs with Minimal Chat
Should chat be the main UX for LLMs? I don't think so and believe we can do better.
30 Apr 2023  ·  3 min  ·  llm misc 🛠
More Design Patterns For Machine Learning Systems
9 patterns including HITL, hard mining, reframing, cascade, data flywheel, business rules layer, and more.
23 Apr 2023  ·  20 min  ·  machinelearning engineering production recsys
Raspberry-LLM - Making My Raspberry Pico a Little Smarter
Generating Dr. Seuss headlines, fake WSJ quotes, HackerNews troll comments, and more.
16 Apr 2023  ·  2 min  ·  llm 🛠
Experimenting with LLMs to Research, Reflect, and Plan
Also, shortcomings in document retrieval and how to overcome them with search & recsys techniques.
09 Apr 2023  ·  14 min  ·  llm deeplearning learning 🛠 🔥
LLM-powered Biographies
Asking LLMs to generate biographies to get a sense of how they memorize and regurgitate.
19 Mar 2023  ·  11 min  ·  llm misc
How to Write Data Labeling/Annotation Guidelines
Writing good instructions to achieve high precision and throughput.
12 Mar 2023  ·  6 min  ·  machinelearning mechanism survey
Content Moderation & Fraud Detection - Patterns in Industry
Collecting ground truth, data augmentation, cascading heuristics and models, and more.
26 Feb 2023  ·  16 min  ·  teardown machinelearning production
Mechanisms for Effective Technical Teams
End of week debrief, weekly business review, monthly learning sessions, and quarter review.
05 Feb 2023  ·  7 min  ·  mechanism leadership
Mechanisms for Effective Machine Learning Projects
Pilot & copilot, literature review, methodology review, and timeboxing.
22 Jan 2023  ·  7 min  ·  mechanism machinelearning productivity
Goodbye Roam Research, Hello Obsidian
How to migrate and sync notes & images across devices
15 Jan 2023  ·  2 min  ·  til productivity
What To Do If Dependency Teams Can’t Help
Seeking first to understand, earning trust, and preparing for away team work.
08 Jan 2023  ·  3 min  ·  leadership misc
2022 in Review & 2023 Goals
Travelled, wrote, and learned a lot, L5 -> L6, gave a keynote at RecSyS, and started a meetup.
24 Dec 2022  ·  8 min  ·  life
Autoencoders and Diffusers: A Brief Comparison
A quick overview of variational and denoising autoencoders and comparing them to diffusers.
11 Dec 2022  ·  3 min  ·  deeplearning
Text-to-Image: Diffusion, Text Conditioning, Guidance, Latent Space
The fundamentals of text-to-image generation, relevant papers, and experimenting with DDPM.
27 Nov 2022  ·  19 min  ·  deeplearning llm survey
RecSys 2022: Recap, Favorite Papers, and Lessons
My three favorite papers, 17 paper summaries, and ML and non-ML lessons.
02 Oct 2022  ·  14 min  ·  recsys engineering production
Writing Robust Tests for Data & Machine Learning Pipelines
Or why I should write fewer integration tests.
04 Sep 2022  ·  19 min  ·  engineering machinelearning production 🩷
Simplicity is An Advantage but Sadly Complexity Sells Better
Pushing back on the cult of complexity.
14 Aug 2022  ·  10 min  ·  machinelearning engineering production 🔥
Uncommon Uses of Python in Commonly Used Libraries
Some off-the-beaten uses of Python learned from reading libraries.
31 Jul 2022  ·  10 min  ·  python engineering 🔥
Why You Should Write Weekly 15-5s
15 minutes a week to document your work, increase visibility, and earn trust.
26 Jun 2022  ·  4 min  ·  mechanism productivity career
Design Patterns in Machine Learning Code and Systems
Understanding and spotting patterns to use code and components as intended.
12 Jun 2022  ·  13 min  ·  machinelearning engineering python 🔥
What I Wish I Knew About Onboarding Effectively
Mindset, 100-day plan, and balancing learning and taking action to earn trust.
22 May 2022  ·  11 min  ·  mechanism career
Bandits for Recommender Systems
Industry examples, exploration strategies, warm-starting, off-policy evaluation, and more.
08 May 2022  ·  14 min  ·  teardown recsys machinelearning
How to Measure and Mitigate Position Bias
Introducing randomness and/or learning from inherent randomness to mitigate position bias.
17 Apr 2022  ·  7 min  ·  recsys
Counterfactual Evaluation for Recommendation Systems
Thinking about recsys as interventional vs. observational, and inverse propensity scoring.
10 Apr 2022  ·  8 min  ·  recsys eval machinelearning
Traversing High-Level Intent and Low-Level Requirements
How they differ and why they work better in different situations.
20 Mar 2022  ·  7 min  ·  engineering productivity misc
Data Science Project Quick-Start
Hard-won lessons on how to start data science projects effectively.
06 Mar 2022  ·  7 min  ·  datascience engineering productivity
Mailbag: How to Define a Data Team's Vision and Roadmap
I'm heading into a team lead role and would like to define the vision and roadmap.
18 Feb 2022  ·  3 min  ·  leadership datascience 📬
Red Flags to Look Out for When Joining a Data Team
What to consider for in terms of data, roadmap, role, manager, tooling, etc.
13 Feb 2022  ·  8 min  ·  datascience machinelearning career
How to Keep Learning about Machine Learning
Beyond getting that starting role, how does one continue growing in the field?
19 Jan 2022  ·  6 min  ·  learning career machinelearning
2021 Year in Review
Met most of my goals, adopted a puppy, and built ApplyingML.com.
28 Nov 2021  ·  7 min  ·  productivity life
Informal Mentors Grew into ApplyingML.com!
More than two dozen interviews with ML Practitioners sharing their stories and advice
25 Nov 2021  ·  1 min  ·  machinelearning career 🛠
5 Lessons I Learned from Writing Online (Guest post by Susan Shu)
Susan shares 5 lessons she gained from writing online in public over the past year.
07 Nov 2021  ·  6 min  ·  writing
What I Learned from Writing Online - For Fellow Non-Writers
Write before you're ready, write for yourself, quantity over quality, and a few other lessons.
17 Oct 2021  ·  7 min  ·  writing
RecSys 2021 - Papers and Talks to Chew on
Simple baselines, ideas, tech stacks, and packages to try.
03 Oct 2021  ·  5 min  ·  recsys deeplearning production survey
The First Rule of Machine Learning: Start without Machine Learning
Why this is the first rule, some baseline heuristics, and when to move on to machine learning.
19 Sep 2021  ·  8 min  ·  machinelearning 🔥
Reinforcement Learning for Recommendations and Search
Focusing on long-term rewards, exploration, and frequently updated item.
05 Sep 2021  ·  13 min  ·  teardown recsys deeplearning
Bootstrapping Labels via ___ Supervision & Human-In-The-Loop
How to generate labels from scratch with semi, active, and weakly supervised learning.
01 Aug 2021  ·  12 min  ·  teardown machinelearning
Mailbag: How to Bootstrap Labels for Relevant Docs in Search
Building semantic search; how to calculate recall when relevant documents are unknown.
20 Jul 2021  ·  1 min  ·  machinelearning 📬
Influencing without Authority for Data Scientists
Show them the data, the Socratic method, earning trust, and more.
04 Jul 2021  ·  9 min  ·  leadership
System Design for Recommendations and Search
Breaking it into offline vs. online environments, and candidate retrieval vs. ranking steps.
27 Jun 2021  ·  13 min  ·  teardown production engineering recsys 🔥
Patterns for Personalization in Recommendations and Search
A whirlwind tour of bandits, embedding+MLP, sequences, graph, and user embeddings.
13 Jun 2021  ·  25 min  ·  teardown recsys machinelearning deeplearning
The Metagame of Applying Machine Learning
How to go from knowing machine learning to applying it at work to drive impact.
02 May 2021  ·  12 min  ·  machinelearning career 🩷
Search: Query Matching via Lexical, Graph, and Embedding Methods
An overview and comparison of the various approaches, with examples from industry search systems.
25 Apr 2021  ·  21 min  ·  teardown machinelearning production 🔥
My Impostor Syndrome Stories (Guest Post by Susan Shu)
Even high achieving individuals experience impostor syndrome; here's how Susan learned to manage it.
18 Apr 2021  ·  8 min  ·  career life learning
How to Live with Chronic Imposter Syndrome
More education, achievements, and awards don't shoo away imposter syndrome. Here's what might help.
11 Apr 2021  ·  9 min  ·  career life learning
Planning Your Career: Values and Superpowers
What do you deeply care about? What do you excel at? Build a career out of that.
04 Apr 2021  ·  11 min  ·  career life
Choosing Problems in Data Science and Machine Learning
Short vs. long-term gain, incremental vs. disruptive innovation, and resume-driven development.
21 Mar 2021  ·  12 min  ·  datascience machinelearning leadership
Seven Habits that Shaped My Last Decade
I wish I started sooner. All have improved my life and several have compounding effects.
14 Mar 2021  ·  15 min  ·  life misc
How to Write Design Docs for Machine Learning Systems
Pointers to think through your methodology and implementation, and the review process.
07 Mar 2021  ·  15 min  ·  writing machinelearning engineering
How to Write Better with The Why, What, How Framework
Three documents I write (one-pager, design doc, after-action review) and how I structure them.
28 Feb 2021  ·  10 min  ·  writing engineering productivity 🩷 🔥
Feature Stores: A Hierarchy of Needs
Access, serving, integrity, convenience, autopilot; use what you need.
21 Feb 2021  ·  19 min  ·  teardown machinelearning engineering 🔥
How to Win a Data Hackathon (Hacklytics 2021)
What the top teams did to win the 36-hour data hackathon. No, not machine learning.
14 Feb 2021  ·  6 min  ·  datascience engineering misc
Growing and Running Your Data Science Team
What I learned about hiring and training, and fostering innovation, discipline, and camaraderie.
31 Jan 2021  ·  16 min  ·  leadership datascience
You Don't Really Need Another MOOC
Stop procrastinating, go off the happy path, learn just-in-time, and get your hands dirty.
24 Jan 2021  ·  9 min  ·  learning
Mailbag: How to Get Experienced DS Resume Noticed by Recruiters?
How to increase the chances of getting called up by recruiters?
16 Jan 2021  ·  5 min  ·  career datascience 📬
Real-time Machine Learning For Recommendations
Why real-time? How have China & US companies built them? How to design & build an MVP?
10 Jan 2021  ·  21 min  ·  teardown machinelearning recsys production 🔥
2021 Roadmap: Sharing, Helping, and Living More
A public roadmap to track and share my progress; nothing mission or work-related.
03 Jan 2021  ·  4 min  ·  productivity life
2020 Retrospective: New Country, New Role, New Habit
Wrapping up 2020 with writing and site statistics, graphs, and a word cloud.
20 Dec 2020  ·  8 min  ·  productivity life
Catch the Flying Daggers
A short story on flying daggers and life's challenges.
11 Dec 2020  ·  2 min  ·  learning life
How I’m Reflecting on 2020 and Planning for 2021
Time to clear the cache, evaluate existing processes, and start new threads.
06 Dec 2020  ·  4 min  ·  productivity life
Alexey Grigorev on His Career, Data Science, and Writing
How he switched from engineering to data science, what "senior" means, and how writing helps.
29 Nov 2020  ·  23 min  ·  career writing
Mailbag: What's the Architecture for your Blog?
How did you set up your site and what's an easy way to replicate it?
24 Nov 2020  ·  1 min  ·  misc 📬
What Machine Learning Can Teach Us About Life - 7 Lessons
Data cleaning, transfer learning, overfitting, ensembling, and more.
22 Nov 2020  ·  11 min  ·  machinelearning life
How to Prevent or Deal with a Data Science Role or Title Mismatch
Interview questions you should ask and how to evolve your job scope.
15 Nov 2020  ·  8 min  ·  datascience career
Applied / Research Scientist, ML Engineer: What’s the Difference?
A personal take on their deliverables and skills, and what it means for the industry and your team.
08 Nov 2020  ·  11 min  ·  datascience machinelearning engineering career
Chip Huyen on Her Career, Writing, and Machine Learning
Setbacks she faced, overcoming them, and how writing changed her life.
01 Nov 2020  ·  11 min  ·  career machinelearning writing
Data Discovery Platforms and Their Open Source Solutions
What questions do they answer? How do they compare? What open-source solutions are available?
25 Oct 2020  ·  16 min  ·  teardown datascience engineering
Why I switched from Netlify back to GitHub Pages
DNS server snafus led to email & security issues. Also, limited free build minutes monthly.
21 Oct 2020  ·  3 min  ·  misc
Why Have a Data Science Portfolio and What It Shows
Not 'How to build a data science portfolio', but 'Whys' and 'Whats'.
18 Oct 2020  ·  15 min  ·  datascience learning career
How to Install Google Scalable Nearest Neighbors (ScaNN) on Mac
Step-by-step walkthrough on the environment, compilers, and installation for ScaNN.
14 Oct 2020  ·  3 min  ·  python machinelearning til
How Prototyping Can Help You to Get Buy-In
Building prototypes helped get buy-in when roadmaps & design docs failed.
11 Oct 2020  ·  7 min  ·  productivity
Is Writing as Important as Coding?
As careers grow, how does the balance between writing & coding change? Hear from 4 tech leaders.
04 Oct 2020  ·  13 min  ·  writing leadership
RecSys 2020: Takeaways and Notable Papers
Emphasis on bias, more sequential models & bandits, robust offline evaluation, and recsys in the wild.
27 Sep 2020  ·  16 min  ·  recsys deeplearning production survey
Appreciating the Present
What if the alternative was nothingness?
26 Sep 2020  ·  1 min  ·  life
Routines and Tools to Optimize My Day (Guest Post by Susan Shu)
For years I've refined my routines and found tools to manage my time. Here I share it with readers.
20 Sep 2020  ·  16 min  ·  productivity
How to Accomplish More with Less - Useful Tools & Routines
My tools for organization and creation, autopilot routines, and Maker's schedule
13 Sep 2020  ·  11 min  ·  productivity
Migrating Site Comments to Utterances
A step-by-step of how to migrate from json comments to Utterances.
07 Sep 2020  ·  1 min  ·  til python
How to Test Machine Learning Code and Systems
Checking for correct implementation, expected learned behaviour, and satisfactory performance.
06 Sep 2020  ·  14 min  ·  machinelearning engineering python
Mailbag: Parsing Fields from PDFs—When to Use Machine Learning?
Should I switch from a regex-based to ML-based solution on my application?
04 Sep 2020  ·  4 min  ·  machinelearning 📬
How Reading Papers Helps You Be a More Effective Data Scientist
Why read papers, what papers to read, and how to read them.
30 Aug 2020  ·  6 min  ·  learning
Mailbag: I'm Now a Senior DS—How should I Approach this?
Becoming a senior after three years and dealing with imposter syndrome.
27 Aug 2020  ·  2 min  ·  career datascience 📬
Embrace Beginner's Mind; Avoid The Wrong Way To Be An Expert
How not to become an expert beginner and to progress through beginner, intermediate, and so on.
23 Aug 2020  ·  11 min  ·  learning
NLP for Supervised Learning - A Brief Survey
Examining the broad strokes of NLP progress and comparing between models
16 Aug 2020  ·  23 min  ·  llm deeplearning survey
Unpopular Opinion: Data Scientists Should be More End-to-End
Why (and why not) be more end-to-end, how to, and Stitch Fix and Netflix's experience
09 Aug 2020  ·  17 min  ·  datascience machinelearning leadership 🔥
Adding a Checkbox & Download Button to a FastAPI-HTML app
Updating our FastAPI app to let users select options and download results.
05 Aug 2020  ·  3 min  ·  engineering python til
What I Did Not Learn About Writing In School
Surprising lessons I picked up from the best books, essays, and videos on writing non-fiction.
02 Aug 2020  ·  11 min  ·  learning writing 🩷
Georgia Tech's OMSCS FAQ (based on my experience)
Why OMSCS? How can I get accepted? How much time needed? Did it help your career? And more...
26 Jul 2020  ·  15 min  ·  learning omscs 🔥
How to Set Up a HTML App with FastAPI, Jinja, Forms & Templates
I couldn't find any guides on serving HTML with FastAPI, thus I wrote this to plug the hole on the internet.
23 Jul 2020  ·  3 min  ·  engineering python til
Why You Need to Follow Up After Your Data Science Project
Ever revisit a project & replicate the results the first time round? Me neither. Thus I adopted these habits.
19 Jul 2020  ·  12 min  ·  mechanism datascience productivity
What I Do During A Data Science Project To Deliver Success
It's not enough to have a good strategy and plan. Execution is just as important.
12 Jul 2020  ·  7 min  ·  mechanism datascience productivity
How to Update a GitHub Profile README Automatically
I wanted to add my recent writing to my GitHub Profile README but was too lazy to do manual updates.
11 Jul 2020  ·  3 min  ·  engineering python til
The 85% Rule: When Giving It Your 100% Gets You Less than 85%
I thought giving it my all led to maximum outcomes; then I learnt about the 85% rule.
09 Jul 2020  ·  2 min  ·  productivity life 🩷
My Notes From Spark+AI Summit 2020 (Application-Specific Talks)
Part II of the previous write-up, this time on applications and frameworks of Spark in production
05 Jul 2020  ·  15 min  ·  machinelearning deeplearning production survey
My Notes From Spark+AI Summit 2020 (Application-Agnostic Talks)
Sharing my notes & practical knowledge from the conference for people who don't have the time.
28 Jun 2020  ·  11 min  ·  machinelearning deeplearning survey
How to Set Up a Python Project For Automation and Collaboration
After this article, we'll have a workflow of tests and checks that run automatically with each git push.
21 Jun 2020  ·  20 min  ·  engineering production python productivity
Mailbag: Qns on the Intersection of Data Science and Business
Does DS have business requirements? When does it make sense to split DS and DE??
21 Jun 2020  ·  3 min  ·  datascience 📬
Why Are My Airflow Jobs Running “One Day Late”?
A curious discussion made me realize my expert blind spot. And no, Airflow is not late.
17 Jun 2020  ·  3 min  ·  engineering production til
What I Do Before a Data Science Project to Ensure Success
Haste makes waste. Diving into a data science problem may not be the fastest route to getting it done.
15 Jun 2020  ·  12 min  ·  mechanism datascience productivity
What I Love about Scrum for Data Science
Initially, I didn't like it. But over time, it grew on me. Here's why.
07 Jun 2020  ·  10 min  ·  mechanism agile leadership datascience
How to Apply Crocker's Law for Feedback and Growth
Crocker's Law, cognitive dissonance, and how to receive (uncomfortable) feedback better.
31 May 2020  ·  8 min  ·  learning career
A Practical Guide to Maintaining Machine Learning in Production
Can maintaining machine learning in production be easier? I go through some practical tips.
25 May 2020  ·  16 min  ·  machinelearning engineering production
6 Little-Known Challenges After Deploying Machine Learning
I thought deploying machine learning was hard. Then I had to maintain multiple systems in prod.
18 May 2020  ·  14 min  ·  machinelearning engineering production
How to Write: Advice from David Perell and Sahil Lavingia
An expansion of my Twitter thread that went viral.
09 May 2020  ·  4 min  ·  writing
A Hackathon Where the Dinkiest Idea Won. Why?
What I Learnt about evaluating ideas from first-hand participation in a hackathon.
03 May 2020  ·  7 min  ·  datascience lazada
Serendipity: Accuracy’s Unpopular Best Friend in Recommenders
What I learned about measuring diversity, novelty, surprise, and serendipity from 10+ papers.
26 Apr 2020  ·  11 min  ·  recsys survey
How to Give a Kick-Ass Data Science Talk
Why you should give a talk and some tips from five years of speaking and hosting meet-ups.
18 Apr 2020  ·  7 min  ·  datascience writing
Commando, Soldier, Police and Your Career Choices
Should I join a start-up? Which offer should I accept? A simple metaphor to guide your decisions.
12 Apr 2020  ·  6 min  ·  career
Stop Taking Regular Notes; Use a Zettelkasten Instead
Using a Zettelkasten helps you make connections between notes, improving learning and memory.
05 Apr 2020  ·  6 min  ·  writing learning productivity 🔥
Writing is Learning: How I Learned an Easier Way to Write
Writing begins before actually writing; it's a cycle of reading -> note-taking -> writing.
28 Mar 2020  ·  7 min  ·  writing learning
Simpler Experimentation with Jupyter, Papermill, and MLflow
Automate your experimentation workflow to minimize effort and iterate faster.
15 Mar 2020  ·  6 min  ·  productivity python
My Journey from Psych Grad to Leading Data Science at Lazada
How hard work, many failures, and a bit of luck got me into the field and up the ladder.
27 Feb 2020  ·  12 min  ·  career lazada 🔥
Beating the Baseline Recommender with Graph & NLP in Pytorch
Beating the baseline using Graph & NLP techniques on PyTorch, AUC improvement of ~21% (Part 2 of 2).
13 Jan 2020  ·  17 min  ·  recsys deeplearning python 🛠
Building a Strong Baseline Recommender in PyTorch, on a Laptop
Building a baseline recsys based on data scraped off Amazon. Warning - Lots of charts! (Part 1 of 2).
06 Jan 2020  ·  14 min  ·  recsys deeplearning python 🛠
OMSCS CS6200 (Introduction to OS) Review and Tips
OMSCS CS6200 (Introduction to OS) - Moving data from one process to another, multi-threaded.
15 Dec 2019  ·  7 min  ·  omscs learning engineering
OMSCS CS6750 (Human Computer Interaction) Review and Tips
OMSCS CS6750 (Human Computer Interaction) - You are not your user! Or how to build great products.
03 Sep 2019  ·  8 min  ·  omscs learning
Goodbye Wordpress, Hello Jekyll!
Moving off wordpress and hosting for free on GitHub. And gaining full customization!
25 Aug 2019  ·  1 min  ·  misc
OMSCS CS6440 (Intro to Health Informatics) Review and Tips
OMSCS CS6440 (Intro to Health Informatics) - A primer on key tech and standards in healthtech.
04 Aug 2019  ·  7 min  ·  omscs learning
OMSCS CS7646 (Machine Learning for Trading) Review and Tips
OMSCS CS7646 (Machine Learning for Trading) - Don't sell your house to trade algorithmically.
11 May 2019  ·  9 min  ·  omscs learning machinelearning python
What does a Data Scientist really do?
No, you don't need a PhD or 10+ years of experience.
30 Apr 2019  ·  8 min  ·  career datascience
Data Science and Agile (Frameworks for Effectiveness)
Taking the best from agile and modifying it to fit the data science process (Part 2 of 2).
02 Feb 2019  ·  14 min  ·  mechanism agile datascience productivity
Data Science and Agile (What Works, and What Doesn't)
A deeper look into the strengths and weaknesses of Agile in Data Science projects (Part 1 of 2).
26 Jan 2019  ·  13 min  ·  agile datascience productivity 🔥
OMSCS CS6601 (Artificial Intelligence) Review and Tips
OMSCS CS6601 (Artificial Intelligence) - First, start with the simplest solution, and then add intelligence.
20 Dec 2018  ·  8 min  ·  omscs learning machinelearning python
OMSCS CS6460 (Education Technology) Review and Tips
OMSCS CS6460 (Education Technology) - How to scale education widely through technology.
25 Aug 2018  ·  9 min  ·  omscs learning
OMSCS CS7642 (Reinforcement Learning) Review and Tips
OMSCS CS7642 (Reinforcement Learning) - Landing rockets (fun!) via deep Q-Learning (and its variants).
30 Jul 2018  ·  6 min  ·  omscs learning deeplearning python
Building a Strong Data Science Team Culture
Culture >> Hierarchy, Process, Bureaucracy.
12 May 2018  ·  5 min  ·  leadership datascience lazada
OMSCS CS7641 (Machine Learning) Review and Tips
OMSCS CS7641 (Machine Learning) - Revisiting the fundamentals and learning new techniques.
27 Dec 2017  ·  4 min  ·  omscs learning machinelearning python
My first 100 days as Data Science Lead
How being a Lead / Manager is different from being an individual contributor.
25 Sep 2017  ·  5 min  ·  leadership datascience lazada
OMSCS CS6300 (Software Development Process) Review and Tips
OMSCS CS6300 (Software Development Process) - Java and collaboratively developing an Android app.
13 Aug 2017  ·  5 min  ·  omscs learning engineering
How to get started in Data Science
Tools and skills to pick up, and how to practice them.
25 Jun 2017  ·  9 min  ·  datascience learning career
OMSCS CS6476 (Computer Vision) Review and Tips
OMSCS CS6476 Computer Vision - Performing computer vision tasks with ONLY numpy.
15 May 2017  ·  5 min  ·  omscs learning deeplearning python
One way to help a data science team innovate successfully
If things are not failing, you're not innovating enough. - Elon Musk
19 Feb 2017  ·  3 min  ·  leadership datascience lazada
Product Categorization API Part 3: Creating an API
Or how to put machine learning models into production.
13 Feb 2017  ·  8 min  ·  machinelearning production python 🛠
Image search is now live!
A web app to find similar products based on image.
14 Jan 2017  ·  4 min  ·  deeplearning python production 🛠
Product Classification API Part 2: Data Preparation
Cleaning up text and messing with ascii (urgh!)
11 Dec 2016  ·  8 min  ·  machinelearning python 🛠
Image classification API is now live!
A simple web app to classify fashion images into Amazon categories.
27 Nov 2016  ·  2 min  ·  deeplearning python production 🛠
I'm going back to school
Got accepted into Georgia Tech's Computer Science Masters!
02 Nov 2016  ·  5 min  ·  omscs learning
SortMySkills is now live!
A card sorting game to discover youl passion by identifying skills you like and dislike.
23 Oct 2016  ·  4 min  ·  misc 🛠
Product Classification API Part 1: Data Acquisition
Parsing json and formatting product titles and categories.
11 Oct 2016  ·  9 min  ·  machinelearning python 🛠
Thoughts on Functional Programming in Scala Course (Coursera)
Learning Scala from Martin Odersky, father of Scala.
31 Jul 2016  ·  4 min  ·  learning
DataKind Singapore’s Latest Project Accelerator
Guest post of how DataKind SG worked with NGOs to frame their problems and suggests solutions
17 Sep 2015  ·  8 min  ·  datascience
210 posts, 31 talks, 19 prototypes, 422,737 words, and countless hours.
Join 11,800+ readers getting updates on machine learning, RecSys, LLMs, and engineering.
Get email updates
I'm a Member of Technical Staff at Anthropic. I work to bridge the field and the frontier, and help build safe, reliable AI systems that scale. I've led ML/AI teams at Amazon, Alibaba, Lazada, and a Healthtech Series A, and write about LLMs, RecSys, and engineering at eugeneyan.com.