Gagandeep's Blog

How Cursor boosted my productivity in 2025

Sun, 18 Jan 2026 00:00:00 +0530

Photo by Aerps.com on Unsplash

Introduction

I’ve been working with Cursor for almost a year now. I’m a Senior Engineer at Branch International, where I’ve been for about 1.5 years. My job mostly involves updating old code or rewriting parts of the system, along with some projects for adding new features. Writing new code is often easier, but updating old code is harder — you have to make it better and easier to maintain without breaking anything that’s already working. Before using Cursor, I used Sublime Text for more than 10 years and always preferred fast, simple editors over full-featured IDEs like IntelliJ. I mainly write backend code in Ruby with a bit of frontend work.

2025 was significant

ChatGPT was launched in 2022, making AI accessible to the general public for the first time. Shortly thereafter, tools like GitHub Copilot emerged, leading to a rapid increase in adoption among developers. I personally utilized ChatGPT as a mock interviewer during my interview preparation in early 2024. Cursor was released in late 2024, and I began using it in December of that year. Transitioning to a new editor after more than a decade was challenging, but a few plugins allowed me to recreate my Sublime setup, including shortcuts, extensions, and themes. I have not looked back since. The year 2025 was also pivotal for Cursor as a company. A startup founded by first-time MIT entrepreneurs successfully captured over 50% of the market share, despite facing competition from Microsoft-backed GitHub Copilot and other similar tools.

On a personal note, my daughter was born in March 2025. This change initially threw off my work routine, but with some disciplined deep-work sessions and AI assistance, I managed to get back on track and even boost my productivity.

Improved Productivity with AI

My role at Branch is filled with exciting opportunities to connect through meetings—stand-ups, 1:1s, unblocking sessions, interviews, and sometimes process-related discussions. To make the most of my time and keep meetings from taking over, I follow a few strategies:

I start my day early, around 8-9 am (depending on how well the baby lets us sleep 😂). This allows me to carve out focus slots before meetings begin.
I push back or skip meetings where my presence isn’t crucial. Saving 30–60 minutes of uninterrupted time is more valuable than being passively present.
Weekly 1:1s without progress can become repetitive. Addressing this early frees up time for both parties to focus on what truly matters.
I try to schedule one interview per day. Design interviews already last more than an hour so it’s hard to take multiple interviews in a day.
I prioritize attending mandatory org-wide meetings and enjoy the flexibility of watching recordings for the rest.

After all this, the time left for engineering tasks is limited, so I need to ensure I don’t waste any of it. I was fortunate to read Deep Work by Cal Newport a few years ago, which helped me apply some ideas to optimize my working style. I also wrote a detailed post on this topic if you’re interested in reading more). Since my role involves a lot of refactoring and rewriting, AI tools help me stay productive despite all these meetings.

AI @ Branch

Before moving ahead, I’d like to explain how Branch views AI, as its usage becomes increasingly important when the organization encourages widespread adoption. In December 2024, I began using Cursor with a personal subscription. Some developers were already using GitHub Copilot or similar autocomplete tools, but none reported a significant productivity boost. Initially, I used Cursor for autocomplete, but after a few weeks, I started utilizing its agent mode, which allows interaction with AI to accomplish tasks. Having coded independently for over a decade, it took time to trust AI and grant it more control. As more developers in our organization began using Cursor, they reported increased productivity. This topic was discussed in our weekly engineering calls, leading to the creation of a working group to evaluate available tools for potential adoption by the entire team. Options like Copilot, Claude Code, and Cursor were considered, and the team began evaluating them.

The AI landscape is constantly evolving, making it challenging to conclude research. However, by mid-year, Branch had enterprise accounts for multiple AI tools, allowing developers to choose based on their preferences, IDE, and comfort level. Additionally, we hosted a hackathon where participants from outside the engineering team built projects without prior coding knowledge. This was just the beginning, and the organization is still working on optimizing workflows for all teams using the available tools.

My Cursor Usage

I started using Cursor like you all did - tab tab tab 🤣. These tools quickly grasp the context of your code. If you’re still writing all the code yourself, you’ll notice it peeking into your work and completing a line of code for you, suggesting the next line, or even the entire method. This is the most basic use of any AI-based editor.

Below, I’ll talk about my setup, some tweaks, and my usage style that boosted my productivity in 2025.

Understanding Different models - Cost vs. quality

Just like other tools, Cursor supports multiple LLM models from different vendors ranging from OpenAI’s GPT models to Anthropic’s Opus/Sonnet, Google’s Gemini and their own recently launched model Composer 1. I won’t go into comparisons of these models but you must have a basic understanding of when to use different models. When choosing between different AI models, it’s important to consider their strengths and limitations. Thinking models such as Opus and Sonnet are better suited for complex tasks due to their advanced reasoning capabilities and ability to handle nuanced queries. However, these models can be slower and may require more computational resources. On the other hand, models like Composer are designed for speed and efficiency, making them ideal for tasks that require quick responses but are not overly complex. Additionally, context windows play a crucial role in determining how much information a model can process at once. Larger context windows allow models to consider more information simultaneously, which is beneficial for understanding intricate problems. However, “larger context” also means slower processing times for a long running thread.

I’ve been using Cursor in auto mode for many months. It picks a model based on the task, but sometimes the responses were too slow, so I started trying different models. Here are a few reasons for the slowness:

When the context window was about 80-90% full, it became large enough to slow down future conversations.
At around 98-99%, many conversations just stopped, and I had to start a new one, losing all the context 😔.
Sometimes, there was random slowness, which suggested their API might have been down or slow, like any other backend service.

Thankfully, by September 2025 Cursor launched context summarisation that triggers automatically. You can also use /summarize command to trigger it on-demand.

Just like everyone else, I love coding quickly, so I tried out Cursor’s Composer model. I usually give smaller tasks with clear context to Cursor, and Composer-1 handled these simple tasks pretty well. I chatted with my colleagues about it, and they found it to be an average model for anything more complex. These days, I use a mix of Sonnet, Opus, and Composer for my projects.

Configuring how agents work

There are various ways in which you can provide a good initial context to the agent before it starts working on your task according to your prompt. As we’re using both Copilot and Cursor in our organisation, few folks from a working group have spent some time defining the configuration for both of these tools. Copilot searches for .github/copilot-instructions.md in your repository rules whereas Cursor has multiple ways of defining rules and configuring the agent (eg. using AGENTS.md). When you are defining these configurations, consider including things like:

Project’s coding standards: Provide detailed guidelines for designing and organizing components within the codebase. This includes specifying folder structures, naming conventions, and rules for using components like models, services, and controllers. It ensures consistency and adherence to best practices across the project.
Workflow Selection Criteria: Clearly define the conditions under which each workflow should be selected. This includes specifying when product, design, coding, testing, and review workflows are applicable. This ensures that the appropriate workflow is triggered based on the task requirements.
Testing and Domain Specifications: Outline the testing plan, including the types of tests and their locations. This helps in maintaining a clear understanding of the project’s structure and ensures comprehensive testing coverage.
Error Handling and Logging: Define how errors should be handled within each workflow. Include guidelines for logging errors and exceptions to ensure that issues can be tracked and resolved efficiently.
Security and Compliance: Outline any security protocols or compliance requirements that need to be adhered to during the development process. This could include data protection measures, access controls, and encryption standards.

In brief, treat the agent as an integral part of your engineering team by providing it with a comprehensive understanding of the problem you’re addressing. This approach ensures that each task the agent undertakes aligns with the expected quality standards. However, be mindful not to overload the initial context with excessive details, as this could consume a significant portion of the context window or lead to some instructions being overlooked. For example, if there are some very specific instructions, don’t include them in this configuration. You can mention those instructions or provide some files as a reference while solving that particular task.

Planning is still the most important step

A detailed task description is crucial for successful AI integration, as the quality of the output is directly related to the clarity of the input—essentially, garbage in, garbage out. By providing clear and comprehensive instructions, you enable AI to deliver more accurate and useful results.

I’ve written some internal posts at my organization on how to effectively break down and estimate tasks. Throughout my career, I have observed that many engineers face challenges with medium to large projects due to lack of proficiency in breaking them down into milestones and tasks. Projects often begin smoothly, but confusion tends to arise midway, leading to a rushed conclusion. This typically results in the submission of large pull requests with a significant blast radius.

With AI as your assistant, planning still remains one of the most important task in SDLC. Those who have reaped the most benefits from AI are typically individuals who excel at planning and breaking tasks. They understand that AI is not a magic wand that can replace the need for skilled software engineers. but a powerful tool that can enhance productivity when used correctly.

I’m not great at writing long prompts, so I’ve been handling my projects by breaking them down myself. I often have design discussions with AI when needed and then assign individual tasks to the tools to solve. This approach has worked well and quickly for me because:

Not everything is outsourced to AI so I completely understand the problem statement. At times, it’s easy to lose context of the problem at hand by completely letting AI solve it.
Smaller problems are solved faster, so I can see the results immediately. I can then proceed to testing and raising smaller PRs for the individual tasks.
The code might be generated using AI but it will be committed against my name. In the future, if something breaks then the ownership is on me, not on AI. So it’s very important to completely go through the generated code. Even if the code is completely covered by test cases, I don’t get a feeling of satisfaction unless the I completely understand the generated and it looks maintainable enough. By working with smaller diffs, it’s easier to review the generated code completely before commiting

Just like speed, I’m also a fan of smaller feedback loops so I wrote a very short blog on it years ago.

Deeper integrations

Cursor also has integration with our issue tracker (Linear) and version control system (Github). By simply tagging @cursor on a ticket, we can launch a background agent that reads the requirement, understands it, and directly raises a pull request on Github with explanation of the changes. A developer can simply check the diff, see if it meets the requirements & coding standards, and merge the PR 🚀.

Our role as engineers is evolving from primarily writing code to focusing on creating comprehensive and detailed tasks from Product Requirement Documents (PRDs) and design documents. This shift allows AI to handle these tasks efficiently and accurately.

Understanding legacy code

Most of us join teams with existing codebases, often over a decade old, which can be difficult to understand at times. While those who witnessed the evolution of the system can grasp the reasons behind changes, newcomers may struggle to comprehend the current state. Code that spans multiple files or has high cyclomatic complexity is challenging to interpret.

AI tools are quite useful in these situations. I’ve been using AI extensively to understand and rewrite legacy systems. Both Cursor and Copilot have a read-only “Ask” mode which is particularly useful for learning and exploration, unlike the default “Agent” mode that is more suited for updating the code.

It’s hard for humans to unlearn — once a certain way of thinking or solving problems is ingrained, stepping back and approaching it with a completely fresh perspective takes conscious effort but AI can provide a fresh perspective on problems and suggest solutions as soon as you start a new chat window.

While detailed prompts are recommended to execute tasks as per your expectation, it has been beneficial for me to ask AI random questions like “Why is this test case slow”. AI would then inspect the file, our test setup, and all the related files to give a fresh perspective into our setup. Without AI, you would generally treat most of the setup as source of truth. Other examples of random questions include “How can this function be optimized?” or “What are potential security vulnerabilities in this code?” These inquiries can reveal valuable insights and improvements. Additionally, AI can assist in design discussions, offering innovative ideas and solutions that might not be immediately apparent to human developers.

Running shell commands

Often, the agent might need to run a script or execute a shell command. While it might seem risky to let AI access beyond the editor, there are safe ways to do it. Currently, both Cursor and Copilot can run shell commands from the editor’s AI agent and use the output to proceed.

These tools use an allow list to make this process quicker and safer. When an agent needs to run a shell command, it offers three choices: Skip, Run, and Add to allow list. Adding a command to the allow list means that next time, it can run without your permission. This feature is very helpful because I often found Cursor running read-only commands like wc, awk, grep, and sort safely, so it made sense to whitelist these commands. Without this feature, you would have to constantly approve commands in your editor.

It’s not safe to whitelist commands like rm, but since I frequently commit my code, I’ve whitelisted them too. Because Cursor doesn’t run with sudo privileges, this setup is both safe and efficient. For the past two months, I’ve been working on projects that required rewriting the controller layer for many modules. One of my testing goals was to ensure that the JSON output from the old and new controllers matched. If there were differences, I needed to know exactly what changed. I used AI-written scripts to compare each nested key in the JSONs to avoid regression issues. Directly asking AI agents to compare the JSONs could work but might be less precise, so I had it write a script for this task.

For writing tests

Tests are essential for identifying regressions and instilling confidence in the codebase over time. They ensure that new changes do not disrupt existing functionality and help maintain the software’s integrity. Many developers now use AI to write tests, including those who previously avoided writing them. I have long advocated for writing tests, even before the widespread adoption of AI agents. There are two main approaches to writing tests: one targets achieving 100% code coverage, while the other focuses on covering meaningful scenarios. I will outline my approach to writing tests:

I’m not into Test-Driven Development (TDD), but I ensure that I write tests for most of the features I touch.
As the code is already written and I maintain a clean coding style, I can easily identify which code paths need testing.
I determine the types of tests required, which generally include unit tests (for individual classes), controller tests (to test a controller’s journey), and end-to-end tests (covering complete scenarios involving multiple controller invocations).
To avoid reviewing subpar AI-generated code, I provide an initial structure for the AI to follow, allowing it to handle the repetitive tasks.

A sample in Ruby might look like this:

RSpec.describe <file> do
  # leave setup data for AI

  describe '#method_to_test1' do
    context 'when scenario 1' do
      # leave this for AI to fill
    end

    context 'when scenario 2' do
      # leave this for AI to fill
    end
   ...
  end

  describe '#method_to_test2' do
    context 'when scenario 1' do
      # leave this for AI to fill
    end

    context 'when scenario 2' do
      # leave this for AI to fill
    end
  end
end

I’ve been encouraging my juniors to write tests this way (since before the AI era) — first list the scenarios, then dive into the details. Doing it this way ensures you think through the scenarios carefully without getting into the specifics of setup code or mocks. If you start writing tests for one scenario without defining all of them, you might have to switch your focus between thinking about cases and writing the actual test cases. Fortunately, the grunt work can now be outsourced 😎.

I’ve also tried without giving AI the structure and I didn’t get good output. But it was some time ago, maybe the models have improved now. Or a better idea would be to frame the prompt like this:

Write tests cases for @this_file. Refer @that_file for understanding how to organise the scenarios.

MCP servers

If you have not yet explored MCP servers, you’re missing out on some serious automation. Most of the popular websites have rolled out their MCP servers and people are already building agents over them 🚀. While I’ve not developed any full fledged agentic workflows yet, these are the things that I’ve used MCP for:

Debugging high impact production issue - Our major backend web application is a monolith that’s deployed twice a week. Every month, I get to handle my team’s on call for a week. In August 2025, I got an issue assigned that was causing problem in a specific module which was essential for all markets, so it had a major impact. Seeing the charts, it was clear that something started breaking after the most recent deployment. While rolling back the entire release was an option, it is not always the preferred path because a release contains work of multiple developers and unless it’s a SEV0, you would not want to revert everyone’s work. But I was not able to catch the issue by manually looking into the code as there were no recent changes around the impacted code or it’s parent classes. I had recently integrated Github MCP with Cursor, and I thought of debugging this issue with MCP integration. Because each release going to production has a PR, I gave the PR link and the exception trace to Cursor and asked it to debug the same using Github MCP tools. Within a minute, it was able to point out that a framework upgrade has also went live within the same release and some methods we used in our module could have been impacted in the newer version. It gave me a direction to look into and that was the exact issue which I fixed later and released within next one hour.
Filing >100 tickets - We use Linear as our issue tracker and I had a use case of filing many linear tickets with a given title and details. Apart from this, the tickets had to be assigned to people from multiple teams so I had to ensure that the project is linked to their team’s board. I spent half hour with Cursor and I was able to file 100+ tickets with accurate details and team information. In a pre-AI world, this would anyways be done with a script but the effort of writing that by hand got eliminated.
Frontend tasks - I recently got assigned some UI work for an urgent task. While I’ve been a full-stack engineer but my inclination has been towards backend only. I googled and found out that our design platform (Figma) also has a MCP server that can be connected to AI editors. Within minutes, I could access designs from my editor and give commands to build specific parts of the page. This entirely eliminated the mental effort in building UI structure by hand.

Avoiding AI Slop

AI slop code refers to low-quality, often buggy, insecure, or nonsensical code generated rapidly by AI tools, lacking true value or originality, and is a significant concern as it can introduce errors like missed security checks, hallucinated functions, and inefficiency, but can be avoided by experienced developers using AI as a supplement, not a replacement, and always validating the output.

It’s very common to get excited with everything happening in the AI world and put it in the driver’s seat, but that can quickly backfire. AI models work by predicting the next token based on patterns learned from large datasets. They don’t truly understand your system, its constraints, or its long-term trade-offs. Because of this, the output may look correct and even pass tests, yet still be suboptimal, brittle, or misaligned with your architecture. Your organisation has hired you, not an AI agent, so it’s your responsibility to ensure that while AI output helps you move faster, it doesn’t quietly introduce slop into the codebase.

Few months ago, when more people in our team started using AI tools, I got to see a lot of obvious comments after every few lines in their pull requests. If you’ve been a Ruby developer, you know that the style guide says this about comments:

Good code is its own best documentation. As you’re about to add a comment, ask yourself, “How can I improve the code so that this comment isn’t needed?”. Improve the code and then document it to make it even clearer.

Having coding in Ruby for more than a decade, I generally write close to zero comments in my code (feel free to call me extremist here 😂). There are very rare cases sometimes when you see a specific complexity in the codebase that would take time to refactor so in those cases I leave a useful comment. This is one example of AI slop but there can be many more. For example, if you’re designing web pages using AI and you don’t give them sufficient context of your overall application and design scheme, the AI agent would end up redefining a lot of unneeded CSS inline just to match the design.

In my knowledge, there are two easy ways to avoid such slop:

Define the Agent.md, copilot-instructions.md or similar files to setup top level behaviour for the agent.
Ensure your prompts are clear enough so that no slop is generated.

Even then if you get any such sloppy code, it’s your responsibility to clean that up manually before committing.

Is AI eliminating software engineers?

From my personal experience, and from listening to respected industry voices, I’ve concluded that AI today can help you build apps from scratch and even get some paying users. But building serious applications is still fundamentally a system design problem that needs experienced engineers. You need a simpler stack, fewer proxies, fewer network hops, and carefully optimised code to run systems efficiently at scale.

Also, in the realm of serious software development, writing code is just one part of the job. A significant amount of effort goes into prioritizing tasks, deciding on product behavior, managing releases, providing post-release support, ensuring observability, debugging through logs and metrics, and handling incidents. Automating parts of code writing, even though they still need testing, reviewing, merging, and releasing, optimizes about 15-20% of the overall process.

Still AI saves a lot of valuable time for the engineers and significantly reduces costs, which is truly impressive. So if you ask me: Do we still need developers?. I’d say absolutely. We probably need fewer of them.

Two edged sword

AI is undeniably powerful and already indispensable. The challenge is to use it as a leverage tool, not as a crutch, so that we don’t trade long-term engineering strength for short-term speed. While I regularly use AI in my work, I see a few long-term issues that are worth talking about:

Today vs 5-10 Years Ago - I started coding professionally in 2015, when StackOverflow was the OG. Until around 2022, someone who coded fast was usually someone who had coded for years and had built both speed and expertise by writing code for a long time. That relationship has changed. AI now makes everyone a designer, a poet, and a coder. The problem is not capability, it’s learning. A large part of the learning is shifting from humans to machines. For example, I’m fluent in Ruby and reasonably comfortable with Java and JavaScript. If you ask me to write C or C++, I’ll still open an editor like Vim and rely on what I learned in college 12–13 years ago. But if you ask me to write Go or Rust today, I’ll most likely describe the idea to an AI editor. I’ll ship a “hello world” in seconds and probably an MVC app in days, but I may never become truly comfortable with those languages without AI. Human brains get comfortable with syntax by writing it again and again. Outsourcing that repetition to AI speeds up delivery, but it slows down internalisation. While writing this, I also remembered my first company’s interview process, where I wrote C++ code with pen and paper 🤠. Will people ever do that again? Maybe I’m overthinking or sounding old, but the idea is simple: programming languages are slowly becoming interfaces for people who don’t deeply understand their syntax, treating engineers and non-engineers alike. The upside is that new developers will ship and debug production code much faster than we ever did. The downside is that it may take them much longer to become fluent in any one language. Whether this is good or bad — only time will tell.
Long-Term Maintainability - A big worry is “AI slop.” Code gets approved not because it’s well understood, but because it works and passes tests. Developers might accept suggestions they don’t fully understand, especially when rushed. Over time, this results in codebases that are harder to understand, harder to change, and fragile in unexpected ways.
Skill Atrophy and Debugging Depth - When AI writes most of the scaffolding and glue code, developers spend less time building a mental model of the system. This can show up during incidents. Debugging distributed systems, performance issues, or subtle data bugs still requires deep understanding, not just good prompts. If that understanding erodes, incident recovery times can increase, even if development felt faster initially.
Lock-In and Rising Costs - Currently, AI tools are relatively affordable due to growing adoption and high competition. However, as teams and individuals become increasingly dependent on these tools, pricing power will shift. When these tools become essential for development, costs are likely to rise. At that point, opting out will be mentally challenging. While the costs might still be more justifiable than hiring additional engineers, only time will reveal how cost-effective these tools will remain.

Personal targets for 2026

With 2025 bringing so many improvements in these AI models, I’m excited to see what we have for 2026 and beyond. For this year, I’m looking to try out a few things:

Agentic workflows -Automation has been around in software engineering for a long time, but creating it has become easier with Agentic workflows. I explored tools like n8n in late 2025 when my manager made a small agent to set up a daily alert on some reports. Building agentic workflows on these platforms is like drawing diagrams. It’s easier because of the visuals, and it lowers costs by cutting out

the need for software engineers. While the tool is very powerful, I didn’t have any ideas to implement, so I didn’t create any agentic workflows last year. I’m usually not great with ideas, but I’m hopeful that this year I’ll develop an agentic workflow for personal use or a production application. Let’s see if I can achieve this 😎.
Cursor commands (more) - I’ve heavily used cursor this year and often searched for old chats when I had to perform similar tasks so that I’m not required to provide the same prompt again. While this is doable and context window isn’t a problem after context summarisation feature, a better method would be to create reusable prompts and commit them to the respective repo. I’ll try to find such use cases and create their cursor commands to help both myself and the wider team.
Agent skills (more) - I got to hear about this very recently. The idea is that agent can pick a skill to perform a specific task and using a markdown file, it would know how to do that task in your preferred way. I believe this would mean that you no longer need long, carefully crafted prompts or repeated context-setting every time you need to perform these tasks since the agent already knows how to perform the task, what steps to follow, and what constraints to respect.
End to end execution with AI - For small projects, I generally start implementing my tasks without creating detailed technical tasks from the PRD. Even for technical projects, I’ve seen that my project description is detailed but the individual tasks have minimal information. Recently, some folks in my organisation are exploring AI for end to end execution. The flow looks like:
1. Create a design doc from PRD (manually or with AI’s help).
2. Use AI to create project, tasks and milestones from design doc. These tasks would include required details with code snippets, or any other thing that would help the agent understand the context deeply.
3. Run agents (sequentially or parallelly) to execute each task and monitor the generated code.
Parallel agents / subagents - Till now, my way of using AI has been to give it a task and stay glued to the editor screen because the tasks are small enough and agent respond mostly within a minute. But the real power lies in giving them different tasks simultaneously to improve efficiency.

Conclusion

As the AI landscape has evolved in 2025, it feels increasingly realistic to pursue these goals and even beyond. The steady pace of improvement in AI tools opens up new possibilities for efficiency and experimentation, but it also comes with trade-offs that require thoughtful adoption rather than blind optimism. With a focus on learning, adapting, and using these tools deliberately within real-world constraints, I see room to make meaningful progress without losing sight of engineering fundamentals.

I’d genuinely like to learn from others here. So if you’ve developed useful workflows, or discovered patterns that work well in real production environments, please share them in comments.

Redis Production Insights

Fri, 17 Oct 2025 00:00:00 +0530

Photo by Luke Chesser on Unsplash

Background

I began my first full-time role in June 2015. Since then, two technologies have consistently been part of my journey: Rails and Redis. Every company and project I have been involved with has utilized these technologies alongside others. In November 2015, Antirez (the author of Redis) wrote a blog post revealing a security vulnerability in Redis. This vulnerability allowed hackers to gain access to the machine where Redis was running through SSH if the Redis server was operating in non-protected mode (which was the default at that time) and the bind address was set to 0.0.0.0. The project I worked on was a website and host monitoring SaaS tool developed in 2010. At that time, creating a VPC was not mandatory, and all machines could run in a single flat network hierarchy called EC2-Classic (which AWS recently retired). This project also provided me the opportunity to move the entire system (from app servers to databases) inside the VPC by relaunching them gradually. The concepts I learned about VPC, networking, and security 8-9 years ago remain relevant today 😁.

Returning to the vulnerability — because we had a monitoring product that checked website uptime from different regions worldwide (similar to this), we had our website probing service running in various AWS regions and even on smaller cloud providers like Linode (Linode was a very simple and tiny cloud provider then). The application we ran was a small Ruby service with an attached Redis, deployed across multiple regions and providers. Back then, Docker and Kubernetes were not mainstream, and server setup was a mix of bash scripts and manual interventions. While our publicly exposed servers had basic SSH key authentication and services like fail2ban to handle repeat offenders, Redis was mostly running on default configuration, with the server bound to 0.0.0.0. This meant that someone could connect to Redis from outside if port 6379 (the default Redis port) was not protected through a firewall or security groups. As I mentioned, we were running this service on multiple cloud providers, which were not as feature-rich as AWS.

Many of our VMs running Redis were hacked, and this issue impacted numerous internet businesses in the following days. Fortunately, we resolved the issue by binding the machine to the localhost address (127.0.0.1), and a future upgrade of Redis enabled protected-mode by default. This was my first production encounter with a security breach that directly impacted our business because we lost access to those servers, and the feature stopped working. This taught me an important lesson - while default settings are acceptable for experimentation, they might not be suitable for running a production application.

In this blog post, I want to share my experience running Redis in production over the years. Although I haven’t managed a very complex Redis infrastructure, these experiences should still benefit those who operate databases or are generally interested in learning about them.

What is Redis?

Before using Redis, I was only familiar with SQL databases, so Redis seemed very different. I referred to it as a “data-structures db” because it implemented many commonly used data structures like lists, hashes, sets, etc. In all the codebases I’ve reviewed, the most common use case for Redis is as a key-value store (essentially string-based GET, SET). However, Redis supports many more features beyond this:

SET allows the EX parameter to set expiring keys, a very useful feature for cache invalidation.
SET also allows the NX parameter to set the key only if it doesn’t exist. SET with the XX parameter does the opposite. At the application layer, libraries return a boolean value to indicate if the key was set or not. This is very useful for implementing distributed locks. In fact, Redis is very popular for implementing distributed locking.
The List data structure in Redis allows pushing and popping from both the left and right sides, making it useful as both a stack and a queue. In fact, the most popular background job libraries for Python and Ruby use Redis lists for queuing.
Redis also supports Geospatial calculations. A use case for this could be storing map locations of multiple fuel stations in your area and then using Redis commands to determine which one is closest to your location. Or perhaps list all the fuel stations in ascending order of their distance, within 3 km. Isn’t that cool?
Redis also supports PubSub and Streaming. PubSub enables real-time, fire-and-forget message broadcasting, making it perfect for instant updates like chat messages or live stock prices where no history is needed. In contrast, Streams provide a durable, replayable event log with consumer groups, ideal for processing workflows or pipelines that need message persistence and ordered delivery.
Redis also supports approximate data structures like Bloom filters, HyperLogLogs and has support even for TimeSeries data.

While the most common usages are for String, List, and Hash data structures, Redis supports a wide range of other features out of the box. If you’re new to Redis or not familiar with these features, I recommend trying them on your local machine.

Use cases solved

At it’s core, Redis is a NoSQL database that supports lots of data structures and also solves for many other use cases like streaming, time series data sampling, geospatial computation, etc. In this section, we’ll explore more about Redis use cases for providing standard solutions to common problems:

Caching - If you’re using Redis in production, there’s a high likelihood you’re using it for caching. Developers love Redis for caching due to its in-memory nature, which helps improve service response times and reduce load on SQL databases.
Queueing - As previously explained, Redis is widely used as a queue to solve the problem of distributed background processing. You don’t need to write anything beyond business logic, as the problem is common and already solved by libraries in your preferred language using Redis as a queue.
Rate limiting - Building a Rate Limiter is a common problem, especially when exposing APIs to the public. There are multiple algorithms to implement a Rate Limiter that can be built over Redis using its supported data structures.
Distributed locking - If you’re from a SQL background, you might have used Row locks (like SELECT FOR UPDATE in Postgres) for similar purposes. However, many times you’ll have use cases to lock something not yet persisted in a SQL database. Redis locks are handy in those cases.

Enough of selling Redis, let’s discuss some learnings.

Learnings and Recommendations

Do I need the multiple Redices?

I’ve seen applications grow from a few thousand users to millions. In a simple stack, we usually start with one Redis instance. As the product grows, more developers realize use cases for Redis and continue building on top of it. However, there are instances when you should consider adding a second Redis machine to your infrastructure:

Sharing across services - As with any other database, we should avoid sharing a Redis machine across different services. A Redis instance shouldn’t be considered as a dump-all cache for multiple services. The primary reason for this is the single point of failure, and no one would want that kind of setup.
Having different use cases - In a previous organization, we had an application with four Redis instances connected to it. Each had a different use case and its own configuration. Separating databases based on use cases helps in tuning and managing them separately. Some people might not do this initially and later realize that separating them initially was a good idea.
Separating cache vs. persisted storage - This is a special case of previous point (different use cases). Sometimes it’s still ok to not separate Redices for use cases because they’re small enough but it’s never okay to combine use case of cache vs persisted storage. Most of the people put Redis in a place where flushing it doesn’t impact anything (apart from some intermittent CPU peaks in the primary database) because the data can be repopulated. But there are some use cases where you can’t afford to drop database because Redis is the primary database. One very good example for this is when you use Redis as a database for background job processing (example1, example2). In such cases, you cannot afford to lose jobs even if the Redis process crashes, making it important to run Redis in persisted mode. Depending on your Redis version and current configuration, you might need to tune it for your persistence use case, balancing durability and performance.

Based on use cases, persistence, latency, and other tuning requirements, you might want to separate Redis early on to avoid migration overhead.

Using expiring keys

I’ve seen apps where the count of keys easily grows beyond 10 million. It’s not bad to have this many keys as long as you have the memory available, but in some cases, it might not be justified given the app size and use cases. A primary reason for ever-growing keys in Redis is not setting expiry for keys. On analyzing key patterns on various Redis machines, I’ve often come across keys for which the corresponding application code was deleted long ago, but the developers didn’t clean up the respective keys. As Redis is an in-memory database, useless keys mean wasted RAM. I’m not saying all keys should be expiring, but if you’re considering running in cache mode, your application code should manage repopulating Redis as needed. Some good examples of expiring keys are:

When Redis is used for distributed locking, we can set an expiry on the lock key. This TTL is generally higher (or a few times) than the task execution time. This ensures that if the application logic fails to clean up the lock key, it gets automatically cleaned up, avoiding infinite locking.
We often use Redis for storing temporary passwords and OTPs, and in most product specifications, there is an expiry associated with this OTP. Setting TTL on such keys helps build this feature without additional application logic.

If you see your Redis memory graph in an ‘ever-increasing trend”, it might be a good time to revisit if you really need all those keys 😁.

Caching invalidation & application hooks

There are only two hard things in Computer Science: cache invalidation and naming things.

-- Phil Karlton

Although this is not directly related to Redis, I often see the problem of cache invalidation discussed among people who use Redis for caching. So I thought of sharing a few best practices. Caching is a powerful technique for improving application performance, but it comes with the challenge of cache invalidation. Properly managing cache invalidation is crucial to ensure your application serves accurate data. Here are some key points to consider when setting up cache invalidation hooks:

Understand the Data Lifecycle: Before setting up cache invalidation, it’s important to understand the lifecycle of the data being cached. Identify when data changes and how those changes should be reflected in the cache. I’ve often seen people caching the response of an entire API to save time in querying and serialization. This is good if the API is Restful, but if it queries multiple data sources to generate that response, there might be multiple hooks on which you need to reset the cache. So take care of such edge cases.
Suitable Hooks for Invalidation: We should update the cache once the data in primary data source has been updated. For a Rails application, we use after_commit hook for the respective model to update the respective cache. I asked ChatGPT for parallels in other languages/frameworks and got this for your reference 😁

Set Expiry Times: ome people use expiry times as a band-aid for cache invalidation. This could be to defer the fix for the actual bug in some cases (e.g., when you don’t know all the hooks to invalidate the cache). A similar case where we used such expiring keys in the past was to update feature rollouts for a user. To see enabled features for a user, we earlier relied on a SQL-backed system that performed checks on at least four levels before telling if the feature is enabled on the user or not. But as the system grew in complexity and we added an authorization layer, such checks became more widespread. The query to check enabled features got flagged as one of the most used queries in our APM tool, and we fixed this in two phases: In the first phase, we cached this data for 5 minutes and ensured that we don’t query from the SQL database within these 5 minutes. While this might not be acceptable in many use cases, it was okay for this use case as the latency of 5 minutes for turning a feature on/off was acceptable to product and business teams. As we got some breathing space, we figured out all the hooks in the different models to invalidate this cache. We could have completely removed the TTL, but we rather set it to a higher value (e.g., 7 days) to ensure that we don’t cache data for users who don’t come to our app often.
Use Versioning: Implementing versioning for cache keys can help manage cache invalidation. By appending a version number to cache keys, you can easily invalidate old entries by incrementing the version number when data changes. For systems where we have timestamps like updated_at, developers often use that as a versioning key because it naturally tracks an update in the system.

By carefully setting up cache invalidation hooks and strategies, you can maintain the integrity of your application’s data while leveraging the performance benefits of caching.

Tuning the configuration

This is true for all databases, not just Redis. Whenever you’re planning to run anything in production, it’s good to know about various parameters that you can tune. For ages, we’ve been operating single-node centralized databases that persist data on disk, and they generally run fine with default settings unless you hit some metric (IO/CPU/RAM) really hard. But the same is not true for distributed NoSQL databases. Based on the configuration, such databases can be tuned to compromise consistency or availability for speed, scalability, and fault tolerance.

All databases have at least one configuration file available to tune different parameters for your use case. There are sufficient comments in the file for documentation purposes (although their websites have more detailed explanations, if needed). You can access these files to tune the database only if you’re self-managing the database installation on a virtual machine like AWS EC2. If you’re using managed services like AWS RDS, AWS Elasticache, etc., then you’d be exposed to a limited set of parameters through their GUI (like Parameter Groups in AWS).

To see a sample Redis conf, you can check this link or google “redis stable conf” . I’ll talk about a few useful config parameters for Redis:

Persistence: Redis, by design, is an in-memory data store, which makes it extremely fast but raises an important question — what happens to your data when the process crashes or the machine restarts? To handle this, Redis supports multiple persistence mechanisms that balance speed, durability, and operational overhead differently. Broadly, Redis offers two major forms of persistence: RDB (snapshotting) and AOF (append-only file). The RDB persistence mechanism takes point-in-time snapshots of your dataset and writes them to disk as a .rdb file. Redis forks a background process (also called as BGSAVE or background save) to perform the dump, ensuring the main process remains responsive. It reads your save configuration from the redis.conf to decide when it should run the background save process. I’ll explain this with an example of save parameter.
```
 save 900 1
 save 300 10
 save 60 10000
```
This configuration means: save after 1 change within 15 minutes, or save after 10 changes within 5 minutes, or save after 10,000 changes within 60 seconds. These rules can be customized or even disabled with save "" for a super-fast cache-like setup. The other setup is AOF where Redis provides finer-grained durability than RDB by logging every write operation Redis performs. On restart, Redis replays the AOF log to rebuild the in-memory dataset. The two important parameters here are: appendonly (should be set to yes to use this mode) and appendfsync . The second property defines how frequently you flush to the disk (possible values: no, everysec, always). In most production environments, appendonly everysec is a safe balance — ensuring sub-second durability without blocking throughput. For extremely fast ephemeral caches, appendonly no can be used for maximum performance. If you set this property to always, Redis will enter into fully durable mode. This means — every time Redis executes a write command (e.g. SET, LPUSH, INCR), it appends that command to the Append-Only File (AOF) on disk, and then calls the fsync() system call immediately afterward — before acknowledging success to the client. This guarantees that even if the OS or machine crashes immediately after acknowledgment, the data is physically on disk and recoverable.

Note on fsync: fsync is a system call that ensures all pending writes for a given file descriptor are flushed from the kernel’s page cache to the physical storage device. By default, when you call write(), the data goes into the kernel’s buffer cache, not immediately to disk — the OS decides when to flush it. fsync() forces that flush right now and waits until the device confirms that data is persisted.
Databses - There’s a controversial feature inside Redis. Redis by default supports 16 databases (numbered 0 to 15) which we can choose using the SELECT command. I’ve not used this in many projects but in of the use-cases, this feature helped me separate data into different namespaces. I was talking to someone few months ago and they mentioned that this feature is considered as an anti-pattern. To understand a bit more, I did some research and found why some people don’t like this feature:
- Redis cluster mode doesn’t support databases because of implementation complexity. So it’s supported only if you’re running a single node Redis (with/without replicas).
- All DBs share same memory, persistence, and config. So there’s no isolation in the way data is stored in memory or on disk. This also means that the configuration we chose for the server applies to all the databases.
- There are commands like FLUSHDB to flush one database (the one you’re currently on), but there’s also a FLUSHALL command that wipes all databases, easy to cause total data loss by mistake.
- As the backup files (RDB or AOF) contains all the databases together, you cannot selectively restore one of them in event of crash or restart. You need to wait for the entire file dump to load in memory.
- Like I used it — there were slightly different use cases in my project so I thought it’s better to separate the namespace but this could easily create the noisy neighbour problem and the monitoring tools can hardly differentiate between metrics based on database number.
  
  Due to all these reasons, even the creator of Redis recommends using different Redis instances rather than using different database numbers.
Replication - Redis supports asynchronous replication with extensive tunable parameters, but achieving reliability requires balancing latency, durability, and consistency:
- Replication is asynchronous by default, meaning replicas may lag or lose writes if the primary crashes before sync.
- Use repl-diskless-sync yes for faster initial syncs and reduced I/O overhead; otherwise, RDB-based syncs (repl-diskless-sync no) can block during snapshot creation.
- Tune repl-backlog-size and repl-backlog-ttl to maintain replication continuity across brief disconnects — preventing full resyncs.
- Enable min-replicas-to-write and min-replicas-max-lag to protect against acknowledging writes when replicas are stale or disconnected.
- Always monitor replication lag via INFO replication — even small lags can break consistency assumptions in queue or locking workloads.
Memory management - By carefully managing memory settings, you can ensure that Redis operates efficiently and reliably, whether used as a cache or for persistent storage.
- When using as a cache, it is useful to configure parameters like maxmemory and maxmemory-policy to evict data based on algorithms like LRU, LFU, etc. If there are no keys that match the criteria for eviction and maximum memory is reached, Redis will report errors for write commands.
- When using for persistent storage, you should set maxmemory-policy to noeviction and leave 30-40% free memory. This buffer protects against fragmentation, background save forks, and transient spikes ensuring Redis never evicts queued jobs or stalls under memory pressure.
Tuning Data structures performance - There are certain parameters that influence how Redis internally represents its core data structures. Tuning them can significantly impact memory efficiency and CPU performance. Let’s understand these for a few data structures:
1. Hashes
  - hash-max-listpack-entries 512
  - hash-max-listpack-value 64
  Redis stores small hashes in a compact, contiguous area for better cache locality. Once a hash exceeds either threshold (more than 512 fields or a field value longer than 64 bytes), Redis automatically converts it into a standard hash table. Increase these limits for smaller hashes to save memory; decrease if you have frequent large-field updates to avoid costly re-encodings. Instagram optimised this parameter for their use case back in 2011 (read more).
2. Lists
  - list-max-listpack-size -2
  - list-compress-depth 0
  Small lists are also stored as flat contiguous memory. list-max-listpack-size controls the maximum size of each listpack node; -2 means Redis targets listpack nodes of roughly 8 KB. list-compress-depth defines how many nodes at the list’s ends remain uncompressed — 0 disables compression entirely. For read-heavy queues, keep compression low (0–1) for fast access. For archival or long lists, allow deeper compression (2–3) to save memory.
3. Sets
  - set-max-intset-entries 512
  - set-max-listpack-entries 128
  - set-max-listpack-value 64
  Redis represents small sets of integers as intsets, and small generic sets as listpacks. Once a set exceeds these thresholds (by count or member size), it upgrades to a full hash table internally. Increase thresholds for workloads with many small sets to reduce overhead. Lower thresholds if set membership changes frequently to avoid repeated re-encodings.
Note: I’ve heard of these configurations many times but never got a chance to optimise Redis for these use cases. Although these look like advanced configuration, these could be useful for you if your use-case revolve around specific data structures only and you’re running on a huge scale, it might be worth experimenting with these settings. Also, before playing around with configurations for any data structure, have a good idea about what all data structures you are using in your application. For example, Redis implements Geospatial data structures internally using sorted set (ZSET). So if you change configuration for zset-max-listpack-entries or zset-max-listpack-value it would also impact the performance and storage for Geospatial keys (incase you’re using those).

Beyond these, Redis exposes several other configuration parameters that impact

Security (eg. bind, protected-mode, requirepass, masterauth)
Define clustering (for sharding data across nodes)
Performance (timeout, tcp-keepalive, latency-monitor-threshold)
Maintainability (maxclients, loglevel, rename-command), etc.

Redis is single-threaded (wait… what?)

Redis is famously known as a single-threaded data store — a design choice that often raises eyebrows among developers used to multi-threaded applications and databases. The assumption is natural: single-threaded means slow, right? In this case, the answer is “no — not really.” When Redis is called single-threaded, it means that only the core command execution loop is one single thread. All commands are executed sequentially, one after another. Each incoming request is parsed, executed, and responded to before moving to the next one — with no context switches or locking. This design avoids the biggest enemy of high-performance systems: concurrency overhead.

Despite being single-threaded, Redis is capable of handling hundreds of thousands of operations per second on a single core. Here’s why:

Redis stores all data in RAM so there’s no disk I/O during command execution. Each operation typically completes in microseconds.
Since there’s only one command executor, there’s no need for fine-grained locking, atomic sections, or concurrency control — all commands are inherently atomic (hence safe for distributed locks).
Redis multiplexes I/O events in a single loop (similar to Node.js). This means one thread can serve thousands of connections efficiently.
Redis batches replies and writes them efficiently, minimizing expensive kernel transitions.
Operations like background save fork a child process does not impact the command execution directly (“indirect” impact is explained below).

Redis proves that single-threaded doesn’t mean slow — when data lives in memory, network latency and locking costs matter more than CPU parallelism. Its design trades concurrency for deterministic speed, making it one of the fastest databases per core ever built. But there are some cases where command execution can be impacted:

Redis supports Lua scripting but those are executed within same single-threaded event loop. So large/complicated lua scripts can slow down this thread.
Commands with high time complexity can slow down this event loop. For example: If you’re running LRANGE on a large list or SMEMBERS on a large set, it might take time to execute them and any commands that are fired afterwards. Redis official documentation mentions time complexity of all the commands.
DEL command can be slow if you’re trying to free up large memory blocks synchronously. UNLINK can be used in those situations.
There’s a very popular and notorious command in Redis called KEYS. I took a production Redis down for a few seconds during my first encounter with Redis 😛. Because this iterates the entire keyspace and production systems (generally) have millions of keys, this command takes a lot of time to execute.
Background save for large data sets (>10GB as per ChatGPT) can have some intermittent impact on the command execution thread.

Safely analyzing data with KEYS command

There are multiple ways for doing this:

First method is to avoid this as much as possible. Revisit your use case and check for alternate methods.
Redis documentation would recommend using SCAN instead of KEYS and use of the pointer (cursor) returned with each output to get next set of keys but you would need some application level logic to follow the cursor iteratively and filter keys by names incase you’re searching for some specific pattern of keys (eg. keys starting with word alert). For this specific purpose, I’ve tried a workaround:
1. Setup a replica of this Redis.
2. Disconnect the replica from MASTER (command: REPLICAOF NO ONE).
3. Ensure there are no connections to this replica now, except your current session (use CLIENT LIST to see connected clients and MONITOR to see if someone is running any other command).
4. Now run KEYS * here so that there’s no impact elsewhere.

If your purpose is doing some analytics, somewhat old data might not cause any problems.

How I saved 75% cost due to single core usage

By now, we know two things about Redis:

It’s uses a single CPU Core.
It’s an in-memory database. So for production machines, you need boxes with large amount of RAM.

In one of my prev organisations, we had four Redis machines connected to a monolith application:

One was for background queueing (full persistence needed, can’t lose data).
One was for some alert thresholds (persistence needed).
One was a pure cache use case (speed preferred)
One was for acting as a database for a high performance system (using lists/sets mostly) (persistence needed).

Out of these, three were hosted on VMs (AWS EC2) and one on managed service (AWS Elasticache). The organisation was trying to cut costs from all sides and our own production infra was a significant component of that cost. After cutting all corners, we saw another opportunity — we could merge these Redices into one big EC2 machine. For some, this might sound like a stupid decision because it would mean a single point of failure for the application (i.e. if this EC2 goes down, everything goes down). But we still did moved ahead to this setup because cost cutting were our only targets.

We took a machine with 4 cores, ran four Redis processes on different ports and exported data from existing machines, saving 75% on Redis costs. One learning that I got in this process is — you can’t create replica of a Elasticache Redis outside Elasticache ecosystem (basically EC2 cannot act as replica of Elasticache hosted Redis). So in order to copy data, we had to export it to S3 and copy that backup file on the new machine which increased our migration downtime by a few minutes.

Deployment options and Cloud providers

Redis can be deployed in several topologies, each balancing availability, consistency, and operational complexity differently. From a single instance to fully distributed clusters, the choice depends on whether your priority is simplicity, high availability, or scalability. I have mostly operated first two setups but I’ll share four of them for your reference:

Standalone: A single Redis instance handling all reads and writes. It is simple, fast, and ideal for development environments or small production caches with no fault tolerance if it goes down.
Master–Replica: A primary node manages writes while replicas asynchronously copy data for read scalability and redundancy, offering better availability but manual recovery on failure.
Sentinel: A lightweight layer that monitors master–replica setups, automatically promotes replicas during failures and updates clients, bringing high availability without clustering.
Cluster: A distributed Redis setup that shards data across multiple masters with replicas for each, enabling horizontal scalability and built-in failover for large, production-scale workloads.

Beyond this, I also wanted to touch upon an important thing that is kind of deployment - self managed vs. provider managed. For provider managed Redis services (eg. AWS ElastiCache, Azure Cache or GCP Memorystore), the provider takes care of the heavy lifting from provisioning to scaling, failover, patching, and monitoring, making them production-ready but that convenience comes with trade-offs in control and flexibility. Some reasons are:

Limited configuration control - You can only modify parameters exposed via their UI (eg. parameter groups). Many performance and persistence knobs remain locked. One reason for this is that Redis license has changed multiple times since 2021 and the newer one doesn’t let the cloud provider run the original Redis as a managed service. So all of them fork Redis 6.x (till which license was relaxed) and maintain their own implementation. Because the API layer and protocol is compatible with the original Redis, developers hardly care about this fact.
Restricted operational access - Direct shell access (redis-cli, INFO all with full privileges) is limited or read-only. You can’t inspect /var/log/redis or the underlying OS, so debugging memory fragmentation, CPU throttling, or swap usage becomes harder.
Limited control during freezes or failovers - If a node hangs (e.g., fork stall, full memory pressure), you can’t SSH or restart the process. You must wait for the cloud provider’s control plane to detect failure and trigger replacement — which can take minutes. For production use cases, this can cause serious damage by the time this gets fixed. It did much damage for one of the teams I worked with, and they started hating Redis because of this.
Version lag and feature delays - Managed offerings usually trail open-source Redis by a few releases. New features (e.g., active-active replication, ACL improvements, new data encodings) arrive late or in enterprise-only tiers.
Higher cost for flexibility - While managed Redis saves operational effort, it’s significantly more expensive per GB of memory. Running Redis on self-managed EC2 is often 50% cheaper.

Popularity and alternatives

For over a decade, Redis has been the standard choice for in-memory data store because of its speed, simplicity, and versatility. As of 2025:

Redis continues to rank among the top 10 databases on the DB-Engines index, often #1 in the key-value category.
Major companies rely on it for both caching and primary storage of ephemeral workloads.
Virtually every cloud platform offers Redis (or compatible software) as a first-class managed service.

While providers with managed Redis services have different forks of Redis due to license reasons, there have been parallel initiatives by many open source and closed source teams leading to softwares which are 100% API compatible with Redis. To name a few:

AWS Memory DB is a durable, in-memory, Redis-compatible database service that delivers ultra-fast performance with multi-AZ durability. MemoryDB stores all data in memory for sub-millisecond reads, but persists every write to a distributed transactional log (across multiple AZs in AWS). If the cluster or all nodes crash, MemoryDB can rebuild the in-memory state from that log — unlike ElastiCache, which would rely on periodic snapshots.
Garnet is a research project by Microsoft Research. It uses the RESP wire protocol, making it compatible with standard Redis clients in many languages but unlike Redis’s single-thread command execution, Garnet’s storage layer is designed to scale across threads within a node. It can work over memory, SSD, and even remote storage (e.g., Azure storage) to support datasets larger than memory.
DiceDB is an open-source, in-memory, reactive database written in Go. It is Redis protocol compatible while adding features like subscriptions / query reactivity (push updates rather than polling). It is designed to better utilize multi-core hardware rather than being stuck with a single-threaded event loop. I was following the creator of this project for a while, who had created this mostly for learning purposes and had not tested this on large number of production applications yet.
DragonflyDB is a high-performance drop-in Redis replacement written in C++. It is fully API-compatible with Redis, supports multi-threaded execution, and often 2–3× faster in mixed workloads. It uses lock-free structures and shared-nothing architecture, avoiding Redis’s single-thread bottleneck. It is known to be extremely efficient with memory and CPU — ideal for modern multi-core servers.

Debugging performance bottlenecks

To ensure optimal performance of Redis in production, it’s essential to monitor and debug performance bottlenecks effectively. Here’s a concise guide:

INFO Command: The INFO command in Redis is a versatile tool that provides a comprehensive overview of the server’s status and performance metrics. When executed, it returns a wealth of information categorized into sections such as server, clients, memory, persistence, stats, replication, CPU, and keyspace. This command is invaluable for monitoring and diagnosing the health of your Redis instance. For latency analysis, the INFO command can help identify potential issues by providing insights into metrics like the number of connected clients, memory usage, and command statistics. High memory usage or a large number of connected clients can contribute to increased latency, and the INFO command helps pinpoint these areas.
Slow logs: They are essential for identifying performance bottlenecks by tracking commands that exceed a specified execution time threshold. Configured through parameters like slowlog-log-slower-than and slowlog-max-len, slow logs capture detailed entries of long-running commands, including their execution time and timestamp. By analyzing these entries, you can pinpoint specific commands or patterns causing delays, guiding you in optimizing command performance or adjusting Redis configurations. I’ve used this a few times to figure out usage of some commands on large data structures causing slowness in command execution thread.
Latency Doctor: Utilize the LATENCY DOCTOR command to analyze and report on latency spikes, providing insights into potential causes. When you run the LATENCY DOCTOR command, Redis examines its internal latency history and generates a report. This report includes details about various latency events, such as their frequency and potential causes. The command analyzes different aspects of the server’s operation, including command execution, memory management, and I/O operations. Based on the insights provided by LATENCY DOCTOR, you can take specific actions to mitigate latency issues.
Connection Pooling: Always implement connection pooling on the client side to manage Redis connections efficiently. This reduces the overhead of establishing new connections by reusing existing ones, which most client libraries support.
Monitoring Tools: Employ monitoring tools that can instrument Redis metrics, such as Datadog, New Relic, or AWS CloudWatch. Key metrics to monitor include:
- Memory Usage: Track used and peak memory to prevent out-of-memory errors.
- Command Latency: Monitor the time taken to execute commands to identify slow operations.
- Keyspace Hits/Misses: Analyze cache efficiency by observing the ratio of hits to misses.
- Evicted Keys: Keep an eye on the number of keys evicted due to memory limits, which can indicate the need for configuration adjustments.

By using these strategies and tools, you can effectively monitor and address performance bottlenecks in your Redis environment.

Closing note

Reflecting on my journey with Redis over the years, it’s clear that this powerful tool has been more than just a component in my tech stack; it’s been a constant companion. From my early days of grappling with security vulnerabilities to tweaking the nuances of Redis configurations, each experience has enriched my understanding and appreciation of this versatile database.

As I share these insights, my hope is that they resonate with you, whether you’re just starting out or are well-versed in the world of Redis. Remember, while technical knowledge is crucial, it’s the personal experiences and challenges overcome that truly shape our expertise. Here’s to embracing the learning curve and continuing to explore the endless possibilities that such databases offer. Thank you for sparing time to read this.

If you liked this post, you might also like these:

Having worked with small teams mostly, I’ve been a proponent of keeping the tech stack very simple. I shared my thoughts around the same.
I wrote a post early this year about habits, productivity and deep work.
If you’re transitioning from monolith to microservices and are looking for a scalable way to migrate data, read this post once.

My Financial Journey

Fri, 06 Jun 2025 00:00:00 +0530

Photo by micheile henderson on Unsplash

Background

I began working as a Software Engineer in June 2015, which means I’ve been earning for nearly 10 years. But if you ask me when I seriously started investing, the answer is “about three years ago.” In this blog post, I’ll share what happened around that time and how my views on money and expenses have changed over the years. This article might draw some ideas from the book — The Psychology of Money, but I won’t be acting as another financial influencer suggesting which stocks to buy 😃.

Three mistakes

My first salary was for about 16 days of work as I joined the company on 15th of June. About Rs. 20000 was credited to my account and my aim (before reaching home) was to take out some money to give to my parents. I was not a kind of person who would reach home with some gifts for everyone (as shown in movies 🤣). The only struggle for me was to find the ATM of the same bank so that I can set the PIN of debit card and then do the first withdrawal. While my new schedule was tiring (including travel, ramping up at work, etc.), being employed and having a steady income was definitely motivating. With this steady income, I did the following things (that sound like mistakes today):

I wanted a new phone for myself but my sister also needed one. So I got one for her first (around August 2015) because the one I wanted was not yet launched. As both of us now had latest phones, I thought I should get one for my parents too. By September 2015, all four of us were having new phones. With a salary of ~ 40K/month, I got phones worth 55K within first four months to make everyone happy.
As my father thought that I should start investing, I met someone who did LIC policies. I got two policies done by them before end of financial year without doing any market research. In the next financial year, I took another one. Today, I have three running LIC policies that are mix of money back and long term policies. After that I also started putting some amount in PPF.
In 2019, I drove from Delhi to Amritsar (and back) on our Hyundai Santro. After coming back, I thought of purchasing an automatic car without any strong reasons (okay, there was one reason — a good salary increment). I purchased a brand new Honda Amaze CVT. In last 6 years, I’ve driven it < 27000 Kms.

I think many of you might be having very similar stories of such expenses, so I hope you can relate with these. Although my parents always recommended me to save money and avoid useless expenses, they never gave any serious financial advice. And I think the core problem is that the system we are raised in expects “every good and sensible thing to come from elders”. Ideally, if it’s your money, then you should take care of investing it (not your parents). I’ll share my current perspective to the above expenses:

The amount I spent on phones was < 1.5month of my salary and half of it was on Credit card (because of No Cost EMI). To many of you, this might sound okay because even I have seen people earning a similar amount today, and still getting a iPhone Pro Max from their first cheque. While today I’m calling this as a mistake, it was not a very expensive mistake. And I think it was important for me to commit these mistakes so that I can learn from them and avoid making even bigger mistakes in future.
My father’s friend who did my LIC policies is a very good family friend and also a very good gentlemen. He did not hide any details of the policies from me or my father. Being a senior citizen (~70 yrs at that time), he recommended me to invest in safer options rather than in private companies or stock market. Because I was influenced by my father and his friend, I decided to go for safer options (like FDs, PPF and LIC). For first 3-4 years these were the only instruments that I was investing in. I even had a C.A. (who was of my age) but he never recommended me any good options and was happy filing my tax at the end of financial year. Investing a good amount in LIC policies seems like a mistake because the XIRR is very low. Although these policies have terminal benefits too, but that’s insignificant unless you die young.
I got a very good increment in Feb 2019 that made me think that I should upgrade our car. As my father also likes cars, he didn’t stop me 😆. Both of us started seeing some pre-owned cars and finally landed to the top model, automatic Honda Amaze that costed 9.5L at that time. My father was retiring in the same year so I thought I’ll use it to go to office later. With COVID kicking in and my remote role (since first COVID), we’ve not driven it much. Today, I think that purchasing a new car is the fastest way to burn hard earned post-tax salary (okay - not the fastest, having an AWS account is still the fastest).

From 2019 - 2022

I met another CA through a common friend who started advising me to invest in slightly riskier options, given my young age and risk taking ability. The best thing about this person was that he didn’t act like policy agents or middlemen who recommended you specific funds or schemes to invest in. He focused on basic investing knowledge — the concept of asset classes, returns that beat inflation and possible investment options in the market (like stocks, mutual funds). These things were pretty new for me at that time. He convinced me to invest in Mutual funds but due to some issue with my Identity card (Aadhar) to mobile linking, this didn’t happen. I still got introduced to financial literacy, that I could learn more about.

That same year, my sister got engaged and married, and then COVID hit in early 2020, leading to salary cuts and a big drop in the market. Later in 2020, I got engaged too, and then married in 2021. After the wedding, my expenses went up for a few months (as you’d expect), and I started thinking seriously about investing. April 2022 was when I started my first set of SIPs.

In the past decade, financial literacy among salaried individuals has improved significantly. Back in 2015, most people I spoke with relied on traditional savings methods like FDs, RDs, PPF, and EPF. Now, even college graduates are trying their hand at trading (though that has a dark side too). A major reason for this change is mobile-first platforms like Zerodha and Groww, which have made investing more accessible and user-friendly. As a result, terms like expense ratio, asset allocation, LTCG, and STCG are no longer alien. Additionally, financial influencers, or “finfluencers,” have played a key role in educating the public, offering insights and tips on smart investing, and promoting a more informed approach to personal finance. I personally know folks who started earning in the past few years and are actively investing in stocks as well as IPOs. But again, this post is not to talk about which asset class is better so let’s talk about some fundamentals.

Understanding the basics

Managing money isn’t just about numbers, or creating plans on spreadsheets — it’s about behaviour, mindset, and self-awareness. You don’t need a finance degree to build wealth. What you really need is a good grip on a few timeless principles that most people overlook in the race for returns or status. (Few of these draw ideas from Psychology of money that I read somewhere in 2023):

Spend less than what you earn - This is one of the most basic rules of personal finance, yet also the hardest to follow: don’t let your spending grow just because your income does. Real wealth begins when you resist that urge and continue living within your means. That doesn’t mean you need to give up all pleasures or avoid nice things—but if you don’t draw a line somewhere, you’ll always feel like it’s not enough, no matter how much you earn. Lifestyle upgrades can easily turn into a trap. The more you try to impress others, the more you link your happiness to appearances—and that’s a cycle that’s hard to break. When your self-worth depends on showing success, it can start to feel like you’re constantly falling behind. True financial peace comes when you no longer feel the need to prove anything to anyone—not even yourself.
Wealth is what you don’t see - Just because someone drives a fancy car doesn’t mean they’re rich. In fact, it might mean the opposite. Wealth is invisible — it’s not the money spent, but the money sitting quietly in savings or investments.
Money is more about psychology than just numbers - The best financial plan isn’t the one with the highest returns. It’s the one you can stick to through good times and bad. Your ability to control impulses, stay patient, and avoid comparison matters more than technical knowledge.
Save beyond goals -It’s great to have a recurring deposit or SIP in place for planned goals like your child’s education, a bigger home, or a new car. But life doesn’t always follow a plan. Many important or difficult expenses come without warning and that’s why you don’t need a specific reason to save. Saving gives you options, freedom and most importantly peace of mind when life takes an unexpected turn. Take layoffs, for example — they’ve become increasingly common and can affect anyone, regardless of their role or the size of the company. It’s never easy to navigate a layoff, especially in a tough job market. But the first thing you need during such a time is a financial cushion that helps you cover your expenses for the next few months. That safety net gives you breathing room to make thoughtful decisions, rather than desperate ones. Without it, people often experience serious mental stress and end up taking whatever job comes first (even if it pays less or doesn’t align with their goals).
Don’t act like you’re in a race - Money related matters are personal. Our goals, life circumstances, and risk appetite are unique, and so are your financial decisions. For instance, I bought my first car at 26. Around the same time, a few people I know bought their first homes (something I still haven’t done). On paper, they invested in an appreciating asset, while I spent on something that loses value over time. But both choices had their own place and purpose in our lives. That’s why comparing financial decisions doesn’t really help. What matters more is being clear about your priorities and staying consistent with your plan. It’s easy to get influenced by what others are doing, but real financial progress comes when you stop reacting to others and focus on what makes sense for you.

The above is a gist of what I learned from the books and blogs, but there are a few more personal lessons I want to share about saving and managing money:

We usually learn by making mistakes. It’s great if you can learn from other people’s mistakes, but when it comes to personal finance, most of us end up learning the hard way—after making a few of our own. It’s generally less costly to make money mistakes when you’re younger and earning less, so don’t stress too much if you slip up early on.
Have a clear idea of your monthly expenses. At the beginning of the month (when your salary comes), know exactly how much you need for essentials: groceries, bills, rent, small luxuries, vacations, or any planned expenses. Add a small buffer on top, and then invest the rest. If you wait until the end of the month to invest, chances are you’ll spend more than you intended. Money sitting idle in a savings account has a tendency to be consumed. Set up automatic transfers for SIPs, RDs, or any other investments, just like your EMIs to ensure disciplined investing.
Build an emergency fund. Aim to set aside at least 4–6 months’ worth of expenses (not your salary). So if your salary is ₹2L per month but monthly expenses are ₹1L, your emergency fund should be at least ₹4–6L. Keep this in an easily accessible & safe place like a fixed deposit or a similar liquid fund, so that you can access it quickly when something unexpected happens.
Don’t ignore insurance—especially health insurance. If you’re living in a Tier-1 city and your parents are aging or already retired, a medical emergency can shake your finances. Insurance won’t prevent health issues, but it’s a solid safety net when something unexpected happens. It’s not just about your parents; your own health cover also matters. Beyond this, if you can afford - do also get a term insurance.
Plan ahead for annual expenses. Large annual expenses can feel like a sudden burden if you don’t prepare for them. For instance, if your parents’ health insurance premium is ₹48K per annum, it’s a lot easier to save ₹4K every month than to part with ₹48K in one go. This works well for school fees, festival travel, or any other similar large planned purchase.
Think & invest long-term. Once you’ve built your emergency fund, and have some financial stability, explore options for long-term investing based on your age and risk appetite. Mutual funds, for instance, aren’t great for short-term gains (except maybe debt funds), but over 10–20 years, the power of compounding really kicks in. The idea is to plant now and harvest later.
Understand different asset classes and diversify. Inflation slowly eats away at your savings, so your goal should be to beat inflation, not just save. Learn about different investment options: fixed deposits, recurring deposits, ELSS, PPF, mutual funds, stocks, real estate, gold etc. Diversification protects your portfolio during market ups and downs.
Look beyond shiny numbers, look at real returns. Few options may sound like a good investment but eat away at your returns. For example:
- Gold: We’ve been buying gold for generations, but always buying jewellery as an investment isn’t smart. Making charges (10–20% or more) are sunk costs. If you buy something for ₹1.2L (₹1L gold + ₹20K making charges), you’ll only get ₹1L or less if you sell it immediately. If you want to invest in gold, consider gold funds or gold bonds.
- Mutual Funds: In a bull run, short-term returns might tempt you to sell early. But remember, STCG (Short-Term Capital Gains Tax) at 15% and exit load (varies per fund) can reduce your actual gains.
- Real Estate: Buying and selling property within 1–2 years can be expensive. Taxes, broker fees, registration charges, and the hassle itself often outweigh any short-term profit.
Stay curious and keep learning. Personal finance isn’t a one-time thing you learn and forget. As you grow, your income, goals, and responsibilities evolve. Revisit your financial plan every 1–2 years. Read blogs, listen to podcasts, talk to financially savvy friends, or even consult a planner once in a while. The more you learn, the fewer regrets you’ll have later.
Start now, however small. Many individuals delay saving or investing, waiting for the “right time” or a higher salary. However, the truth is that the earlier you begin, the more advantageous it is for your financial future. Even if you start with small amounts, investing consistently over a long period can significantly increase your wealth. This approach allows your investments to benefit from the power of compounding, where the returns on your investments generate their own returns over time

Conclusion

Money can feel overwhelming, but it really doesn’t have to be. A few intentional habits, some patience, and regular check-ins can go a long way. I’m still learning and making some mistakes. But it’s better to learn with a plan than to keep drifting without one. If you’re reading this, chances are you’ve had your own turning points too — maybe it was the first time you ran out of money before the month ended, or when you impulsively spent your bonus on something that didn’t feel worth it later. Whatever your journey looks like, I’m sure you’ve had moments of confusion, growth, or even regret when it comes to managing money. That’s perfectly normal—we’ve all been there in some way. The important thing is that we’re trying, learning, and doing a little better each year.

I’d love to hear your story too—whether it’s a mistake that taught you something valuable, a trick that helped you save more, or a mindset shift that changed the way you think about money. Drop your thoughts in the comments below.

References

Growth Through Unlearning

Sun, 26 Jan 2025 00:00:00 +0530

Photo by Tim Mossholder on Unsplash

Notice if you’re describing a problem in terms of a solution you’ve already chosen — this can be a mental block for a lot of engineers. We start out by comparing problems to solve, but find ourselves talking in terms of technology or architecture we “should” be using to make everything better.

-Tanya Reilly, The Staff Engineer’s Path

Introduction

In 2022-2023, I spent at least two afternoons each week discussing architecture with two senior colleagues at my company. We had hired many people, expecting growth, and decided to transition from a monolith to microservices. During one discussion about a central authorization service, I kept drawing ideas from a system I had built in the monolith a few years ago. This is common among engineers: they often jump straight to solutions without fully understanding the problem, suggesting what they already know. This issue isn’t limited to engineering teams; even product and design teams can overlook problem details and plan to create solutions similar to what they’ve done before.

Many industry leaders like hiring people with similar past experience, but that shouldn’t stop you from finding better solutions. When you face a similar problem, some parts might be different, so the same solution might not work directly. If you overlook these small but important details, you might implement an unoptimized solution.

The tale of two RBAC systems

In the past, I had developed a Role-Based Access Control (RBAC) system for a customer-facing application using SQL tables. This system was designed to manage user permissions efficiently by assigning roles to users and controlling access based on these roles. A few months after the implementation, we began optimizing the SQL queries for our application to enhance performance. During this process, we discovered that the tables associated with the RBAC feature were among the top 10 most frequently accessed tables. This was because the data from these tables was loaded before almost every API call, which significantly impacted the application’s performance.

As an improvement, we decided to implement caching for this data. We set up a cache to store the this data and configured it to invalidate whenever a new role was assigned to a user. Since role assignments were not very frequent, this approach was feasible and effective. Within a month, we successfully eliminated 4-5 database queries that were previously executed @ 40 queries/sec.

A few years later, I encountered another RBAC system in a different application, and here were the differences between the two:

The system was designed for internal users (instead of customers) and there was no chance that the customer side would need this functionality due to the nature of the business.
It was built using configuration files and one SQL table, instead of multiple SQL tables that use joins to get required data.
In the newer system, changing roles required a deployment because these configuration files were part of the repository code. In my old system, we had an admin page to update the database entry (and a callback to refresh the cache as well).

When I first saw this new system, it seemed like a misnomer, but this is how it was perfect for their use case:

Since this was built for internal users, there was no need to update the data often. Plus, the number of internal users was under a thousand.
There was no need for a caching layer because the configuration files were only loaded once, during app startup.
The team wanted to track changes to the permission system, and they thought using Git was the best way to do it 🤓. Sure, a maker-checker system could have been created for logging changes if SQL tables were used, but that requires separate effort.

So which system is better? The one built on SQL tables or the one that mostly used configuration files 🤔?Take a moment to think before you continue reading.

Seasoned engineers would sum it up in two words: It depends 😆.

Both systems worked well for their specific use cases without causing any issues for the end user. If I were to design the second system after creating the first one, I might have made the same mistake of using SQL tables and joins because that would have been the “standard solution” I remembered. But now, looking at both, I see that each was built according to its requirements without over-engineering for the future. In fact, over-engineering is very common in software engineering (been there, done that).

Again quoting from the Staff Engineer’s Path.

An engineer who is not busy can be inclined to make work for themselves. When you see a vastly over-engineered solution to a straightforward problem, that’s often the work of a staff engineer who should have been assigned to a harder problem.

For this issue, building a solution based on SQL tables would be considered over-engineering because:

There was no need to update roles dynamically (without deployment).
Implementing the same on the SQL layer would call for a separate maker-checker system to monitor the changelog.
Additionally, future work might involve developing a caching layer since the data rarely changed.

Comfort resists innovation

Over the past few years, a notable portion of my work has focused on simplifying systems that have become increasingly complex due to ongoing product changes implemented by various developers. As I work on enhancing these systems, it is crucial to first gain a thorough understanding of their current behavior. This often involves interviewing individuals who have been involved with the system for a long time. These individuals tend to be very familiar and comfortable with the system as it stands, having witnessed the gradual increase in complexity and understanding the reasons behind each modification.

This sense of comfort can be problematic because it prevents them from envisioning a newer, more streamlined version of the system that would be easier for everyone to comprehend. Their familiarity with the existing complexity, combined with limited exposure to effective design principles, often leads them to believe that the current state is the optimal way to construct the system. This mindset results in a lack of motivation to pursue incremental improvements that could make the system more efficient and user-friendly. Consequently, they may not plan or implement small, reversible changes that could gradually enhance the system’s design and functionality.

In this case, unlearning is important so that you can think of the system from a fresh perspective and think of an alternate, simplified architecture. This would help you:

To recognize that some of what you “know” might not be universally correct or applicable everywhere.
To let go of fixed patterns that prevent you from adapting to new situations. In order to grow, it’s important to not rely on such patterns.
To shed the confidence in solutions or methods simply because they worked in the past. Assumptions based on past experiences can lead to flawed conclusions or solutions when circumstances change.

Some personal examples

Software Engineers need to constantly unlearn the old concepts so that they can build efficient systems. Throughout my career, I’ve encountered numerous instances that have taught me the importance of being willing to unlearn outdated methods.

These are some instances where unlearning helped me:

If you’re used to working with monolithic systems and switch to microservices (or join a team using microservices), you need to unlearn some habits and adopt new ones. In a monolithic system, you rarely deal with timeouts because all the data is usually in one database that your app is always connected to. But in a system that constantly gets data from other services, you have to consider timeouts, retries, and fallbacks (if available). You’ll also learn new concepts like circuit breakers, centralized logging, inter-service authentication, etc. If you stick to the old monolithic way of coding in a microservices setup, you’ll create inefficient solutions. (Learn more here).
The simplest and slowest way to build a module is to execute all statements sequentially. For instance, in an e-commerce system, when an order is placed, tasks like sending notifications to the user, notifying the warehouse, and processing the payment are often performed one after the other. This sequential approach can cause inefficiencies and delays. By adopting asynchronous processing frameworks, you can enhance performance by allowing each step to be handled independently and concurrently. It’s crucial to unlearn traditional methods and embrace the intricacies of asynchronous distributed systems to develop scalable and efficient solutions.
My first company operated in a “work from office” mode. During my second role, we transitioned from being “in-office” to “work from home” due to the COVID pandemic. Now, I am part of a fully remote team with members spread across different time zones. The skills required to thrive in these varied work environments differ significantly. For instance, in a remote setting, effective written communication is crucial to ensure that messages are clear and not misinterpreted by colleagues from diverse cultural backgrounds. Additionally, it’s important to recognize that team members may not always be available simultaneously. Therefore, it’s essential to plan your workday strategically. Tasks requiring input from others in different time zones should be scheduled when they are available, while you can focus on tasks that can be completed independently during other times. This approach helps maintain productivity and ensures smooth collaboration across the team.
The transition from on-premise to cloud virtual machines, and now to Kubernetes pods, has introduced a more ephemeral nature to what we consider a server. I recall a high-rate ingestion service in our old organisation that buffered data temporarily to disk when the consumer was down. This application ran on AWS EC2 machines, which are less ephemeral than Kubernetes pods, and the design worked well despite using instance stores that do not persist through EC2 stop/start cycles. In a Kubernetes environment, designing this would require a different approach because pods can be terminated for various reasons and with higher frequency. To handle this, you might need to implement persistent storage solutions like using Persisted volumes, StatefulSets or external storage systems such as Amazon EBS to ensure data durability and availability when pods are frequently restarted or rescheduled. This shift necessitates a rethinking of how data persistence and application state are managed in cloud-native architectures.

Unlearning old habits was crucial for my growth as it allowed me to adapt to new technologies and methodologies. By shedding outdated practices and embracing continuous learning, I was able to contribute to the development of more efficient and scalable systems.

Conclusion

In this brief blog post, I’ve highlighted the significance of unlearning and its vital role in fostering innovation and adaptability for professionals across various fields. By letting go of preconceived notions and outdated solutions, we can tackle problems with fresh perspectives, resulting in more efficient and context-appropriate outcomes. Embracing unlearning as a practice can significantly contribute to personal and professional growth, enabling individuals to stay agile and responsive in an ever-evolving landscape.

If you liked this post, please read these too:

This is my second blog post related to growth. The first one talks about habits, productivity and deep work
It’s important to manage your time when you’re thinking about growth. I shared my thoughts on the same an year ago.
If you’re transitioning from monolith to microservices and are looking for a scalable way to migrate data, read this post once.

Habits, Productivity & Deep work

Sun, 05 Jan 2025 00:00:00 +0530

Photo by Carl Heyerdahl on Unsplash

Background

I resigned from my first company in December 2018 when I decided to switch from a services company to an early-stage product startup. Although the previous company wasn’t treating me badly, there was a trend of people leaving after 2-4 years, which was on my mind, along with other factors. My previous project, which I worked on for more than 2.5 years, was almost wrapped up, and I was being assigned random tasks that I didn’t enjoy. Finally, I joined this product startup, LocoNav, on February 5th, 2018.

While onboarding at my current organization, Branch, took almost a week due to extensive documentation and mature processes, the situation at LocoNav was completely different at that time. Within the first four hours, I was logged into their production server using my new laptop to check a log rotation issue. This might seem unusual to those who haven’t experienced the early startup environment, but for those who have, it’s not surprising. You need to be productive from day one and ensure your efforts align with everyone else’s. With all this hustle came a lot of stress, and there were many reasons for this stress. Here are a few:

I had moved from a services company. Although it was also a startup, its processes and hierarchies were more formal. Sometimes, I would think, We should have the “xyz process” from my old company to avoid this problem.
Not just me, but most of us came from service or stable companies. So, working at this fast pace was totally new for us. Everyone found it tough to deal with the uncertainty, juggling different tasks, and the ever-changing requirements. When I joined, there were no product managers. So, we were building a product without a product manager.
I aimed for perfection in many tasks, and that was a bit of a mistake. I didn’t realize that in early-stage startups, much of what we build might be discarded, so aiming for perfection from the start isn’t necessary. Trying to balance perfection and delivery, I ended up stressing myself and sometimes overworking. (Though many APIs I wrote in 2018-19 are still running in their production 😀).

In this blog, I’m sharing what I’ve learned from working in both early-stage and mature startups. This will cover:

Finding a balance between perfection, speed, and the stress of delivering.
How much context-switching is beneficial and its impact on your productivity.
The value of deep work and how most of our jobs require it, but our environment isn’t set up for it.
How your habits and environment affect your work.

Many of these insights are common to any corporate role. However, based on my experience as a Software Engineer, most examples will pertain to software design. So, let’s start 🚀

Perfection & Speed

I was quite a perfectionist when I joined LocoNav. Out of stress, I would write emails to my founder, VP, and a few teammates, sharing my thoughts and asking for their suggestions on the problems we were facing. These weren’t product-related issues but suggestions on solving team problems (e.g., Should we hire more?) or individual problems (e.g., How to improve productivity?). While everyone appreciated my efforts to address these issues, not everyone was interested in solving them. Everyone has a unique perspective, and what seemed problematic to me might have seemed normal to someone else, possibly due to their past experience with similar situations.

The question for the need of perfection is tricky, and is very subjective to your current working environment. For example:

If you’re a billion-dollar stock exchange firm or bank, perfection in your work is a basic requirement.
If you’re a startup dealing with money, most of your work needs to be perfect. However, your idea of perfection will likely be less strict than that of a billion-dollar company because your processes are less mature.
If you’re a company manufacturing medical equipment, even one defect can be dangerous or life-threatening. So, your standards for perfection will be very high.
If you’re a company not heavily regulated due to the nature of your business, your standards for perfection might not be very high. Doing things in a better way takes time and money, and no one wants to spend that money unless necessary.

Do you really need perfection everywhere?

I think most of the companies that build something new or cutting edge do not face a lot of compliance (at least initially). For example:

Stock companies that started between 2010-2020 can show how compliance has become stricter over time. In 2010, they didn’t take all the measures they do now. Each step requires time and money, which startups often lack.
SaaS companies in Europe, or those handling European user data, had to change how they store, process, and delete user data after GDPR was introduced. Making these changes is expensive, especially for large products.
Many AI startups have been creating amazing products since ChatGPT appeared, but there isn’t a governing body to set limits on AI use to ensure it’s safe. For example, the CTO of OpenAI wasn’t sure if YouTube’s public videos were used to train Sora, their video generation model.

Startups often operate in fast-paced environments where speed and innovation are prioritized over strict processes and documentation. Without external regulatory requirements setting clear quality benchmarks, teams lean towards quick fixes and ad-hoc solutions rather than building for long-term stability and scalability. This lack of enforced structure can lead to inconsistent practices, technical debt, and overlooked edge cases in both product and operational workflows. No one wants to shutdown their company or go to jail for doing something that’s illegal (at least, most of us don’t want to!). So if you’re under a governing body, ensure perfection accordingly. Otherwise feel free to experiment and take some risks (or technical debts).

I think this gives you an idea that the level of perfection depends on kind of business you’re building.

Stress & Context switching

Handling stress in a startup requires balance and resilience. In the early days, we worked six days a week, with one day from home and five in the office. This wasn’t a set rule, but many of us followed it. I didn’t burn out quickly, but I made sure to rest when needed. Negotiate and align priorities with your teammates and manager because time is the most valuable asset for a startup. It’s crucial to have leaders who are supportive and practical. While everyone is expected to work quickly and deliver almost daily, not everyone will be a 5x or 10x developer.

I remember many days when I would come home utterly exhausted, unable to even hold a conversation with anyone. My family was concerned and thought I had made a mistake by leaving a stable job to join a startup. At that time, I also had little idea of what the future held for me. Despite this, I was driven by the motivation and support I received from my team and the leaders around me. There were two major sources of stress during this period:

In the first 3 months, I mainly worked on building the API layer for their new Android app. We were designing screens, building APIs, and developing the mobile frontend all at once. This simultaneous work caused a lot of back and forth, leading to rework. While redoing tasks is common in startups, it caused frustration for everyone. Too much parallel work can be counterproductive, showing our inexperience with working at such a fast pace.
Since everyone needed to handle multiple roles and most things were unstable, I was involved in many tasks besides just building the API layer. I had to switch between different tasks, which sometimes meant I accomplished nothing concrete on a particular day. This was very discouraging for me.

One clear lesson I learned from observing my day was that to make an impact, I needed to prioritize tasks. In my previous role, this kind of prioritization wasn’t necessary because things were stable, and the layers of management above me handled it. Gradually, I became someone who learned to say ‘No’ when something couldn’t be done in a given time frame or when I was already overloaded with tasks. In some teams, this might be seen negatively, but in our small team, where everyone was making honest efforts, people understood that if I said no, it truly meant I was busy.

Can we get a lot done with lots of context switching?

The short answer is “No”, but I have two takes here. Both of these conclusions come from my personal experience and what I learnt by reading.

If you often switch between tasks while working, you might initially feel more stressed and frustrated, as you’ll see small progress in many tasks but nothing substantial by the end of the day. Over time, you’ll learn to choose fewer tasks and complete some of them. From my personal experience, trying to handle 10 tasks a day was never effective for me. However, focusing on 3-5 tasks that needed minimal input before completion made me happier. When starting something new, I found that I needed 2-4 uninterrupted hours (without context switching) to create a solid plan for the coming days or weeks. I’ve worked closely with people who can’t switch between tasks as efficiently as I do. This behaviour is explained by Robin Sharma in the book “The 5 AM Club”, where he highlights the brain’s amazing neuroplasticity (an ability to adapt, rewire, and grow with consistent effort and habits). So, context switching works for me because my mind has adapted to this pattern over months and years of practice.
Not everyone can handle a lot of stress, so not everyone can switch tasks efficiently. It’s wrong to expect everyone to multitask well. I’ve written about efficient multitasking in a separate blog if you want more details. As a colleague, if I see someone struggling with multiple tasks, I always recommend prioritizing a few and completing those instead of making little progress on all of them.

Deep work

I’ve been working remotely since March 2020, so I stay connected with my colleagues through instant chat apps like Slack. When Facebook was launched in 2004, “online connections” were seen positively, but over time, people realized how addictive these sites can be. While you can cut down on using apps like Facebook, Instagram, Twitter, and Snapchat, you can’t stop using tools like Slack, Zoom, Meet, and Teams because you’re expected to be online during work hours and respond to calls and messages. You might be wondering - Why is this guy comparing Facebook to Slack?

The answer is simple - all these apps steal your attention and reduce your ability to perform Deep Work. Apps like Slack are designed with instant notifications and visual cues (eg. red badges and notification sound) that trigger our brain. Each notification creates a sense of urgency, pulling us away from focused tasks. The fear of missing out (FOMO) on important updates further compels us to check messages frequently. Additionally, the platform’s design encourages constant context-switching, fragmenting our attention and reducing cognitive capacity for deep, uninterrupted work. Over time, these repeated interruptions diminish our ability to focus deeply, making us more reactive and less capable of sustained, meaningful problem-solving. If you find it challenging to advance in your role, your weak attention span might be a significant factor. Referring to the career progression of software engineers, it’s relatively straightforward to advance from a Software Engineer to a Senior Software Engineer. However, progressing beyond the Senior Engineer level requires demonstrating a broader impact, which can only be achieved through focused, uninterrupted time dedicated to cognitively demanding tasks. Deep work fosters a systems-level perspective, enabling the identification of patterns, anticipation of potential challenges, and proposal of innovative improvements. It also provides the opportunity to learn and refine expertise in critical areas that contribute to organizational success. Without consistent deep work, engineers may become ensnared in reactive workflows, unable to deliver the strategic value expected at the Senior+ level.

Do our jobs encourage Deep work?

Unfortunately, many work environments, especially startups, don’t really encourage deep work. With tasks often feeling urgent, we tend to reward visibility over impact, which unintentionally promotes shallow work, like responding to emails, attending long meetings, or giving constant status updates. These activities can create an illusion of productivity because they are easy to measure and immediately visible. On the other hand, the focused, uninterrupted effort needed for solving complex problems often doesn’t show immediate results and can be overlooked in places focused on short-term goals. In my opinion, we need more mature managers who appreciate deep work for its long-term benefits.

Some personal productivity hacks

These habits have helped me to get more work done without overworking:

Whether it’s email or Slack, I’ve set up filters everywhere. If you’re in a channel that no longer interests you, or if there are emails you’re not interested in but are sent to a group, create filters for them. Check these messages occasionally instead of getting notified every time a new message arrives. Similar optimisations include muting channel notifications, leaving those channels, unsubscribing to some newsletters that you no longer need, or setting up email filters to ensure emails with certain subjects or sender IDs don’t land in your inbox (allowing you to check them later).
I typically begin my workday around 8:00 - 8:30 am. This routine has been a part of my life for more than five years now, and it has significantly contributed to my ability to focus on crucial tasks before the rest of the team starts their day (given flexible working hours and distributed teams).
Although I get a lot done in the morning, I also set up Do Not Disturb (DND) on Slack when needed. It’s better to clearly indicate that you’re busy rather than ignoring messages.
I keep my mobile internet off during work hours (except at lunch) to avoid distractions from push notifications.
I keep a simple text file open in my editor all the time. It’s got a list of stuff I need to focus on, plus all the raw notes about how things are going. Since the list covers everything from my own tasks to PR reviews, design document reviews, and other things, it’s my go-to spot for figuring out what to do next. Prioritizing is just a matter of moving a few lines around. This setup also makes it easy to pick up where I left off after vacations or long weekends without feeling overwhelmed or lost.
I push all shallow work to a specific part of my day. These are minor tasks with medium to low priority, which I can tackle after completing deep work. On another note, there are often tiny tasks that take less than two minutes. I choose to do these tasks immediately because scheduling them actually wastes more time.
I maintain a clutter-free physical workspace, ensuring that my desk and surrounding area are organized and tidy. This helps me focus better and reduces distractions. Additionally, I relocated to a quieter place a few months ago, which has significantly improved my ability to concentrate and work efficiently (more details in the next section).

Habits and environment

Your room is a reflection of your mind. The state of your room, whether it’s organized, cluttered, or decorated in a specific way, can often reveal aspects of your personality, thought process, and overall mental state. A tidy room signifies a clear mind and a messy room indicates a more chaotic inner world. I’m sharing this because I moved to a cleaner workspace in October 2024, and it has improved my focus. Having worked from home for a long time, it was important for me to set up a space away from regular household distractions, which I had become accustomed to working with.

During the time I set up my new desk, I added three books to my bookshelf: Atomic Habits, Deep Work, and Zero to One. While the third book focuses on entrepreneurship, the first two helped me adjust my habits and work patterns to be more productive.For example, I used to struggle to read regularly and would save reading for the weekends because my weekdays were busy. However, in Atomic Habits, the author shares a simple but useful tip:

It’s easy not to practice the guitar when it’s tucked away in the closet. It’s easy not to read a book when the bookshelf is in the corner of the guest room. It’s easy not to take your vitamins when they are out of sight in the pantry. When the cues that spark a habit are subtle or hidden, they are easy to ignore.

He suggests placing a book under your pillow to encourage more frequent reading, and that’s exactly what I did. In the morning, I would wake up and spend 10 minutes reading, repeating the same before bed. I thought I was “busy” during the week, but making this small change wasn’t difficult since I was using that time to scroll through social media anyway. As a result, I finished reading these three books in just 9 weeks (something that would have taken me at least 16-20 weeks if I only read on weekends). Some days, I got so interested in the topic that I spent 20-30 minutes reading instead of just 10. It was definitely better than spending those 20-30 minutes on Instagram 😉. This habit lasted for about 4 months, but now the streak is broken, so I read 3-4 nights a week instead of 6-7 (still much better than just weekend reading). One reason for this change is that I switched back to technical books, which require more focus and the right mood to read.

While it’s important to have good habits, it’s even more crucial to learn how to fit them into your schedule, so you don’t struggle with prioritizing what’s important. Reading Atomic Habits really helped me with that. Let me share two more habits I’ve developed over the years that I believe have helped me grow:

Boredom is important

Many people who struggle to grow beyond a certain point often claim they are extremely busy—whether genuinely or by filling their time with low-impact tasks that create an illusion of “busyness.” While constant activity might feel satisfying, it often leaves little room for strategic thinking or mental clarity. Surprisingly, boredom plays a vital role in growth. It’s in these quiet, unoccupied moments that your mind can wander, reflect, and make unexpected connections. Boredom is not idleness. it’s about giving your brain the space to process thoughts, spark creativity, and identify priorities. Without moments of stillness, you risk being trapped in a cycle of reactive tasks, never pausing to think deeply or plan effectively for the future. Sadly, social media apps eliminate boredom by constantly feeding quick dopamine hits through endless scrolling, notifications, and bite-sized content. This prevents our minds from wandering, reflecting, or engaging in deeper thought.

Simplicity is clarity

This habit is deeply rooted in my mindset, drawing inspiration from both religious beliefs and practical experiences. On the religious side, there’s an emphasis on leading a simple and minimal life, free from unnecessary complications. This perspective has taught me the value of clarity & focus, which extend far beyond personal life. Similarly, as an engineer, simplicity isn’t just a preference while building large systems — it’s a functional necessity. Simple systems are inherently easier to understand, debug, and improve. They lower the cognitive load on those working with them, allowing teams to collaborate effectively, onboard faster, and make informed decisions with confidence. On the flip side, I’ve also worked on (and even built 🫣) systems that were architected in a way that they were not only hard to understand but harder to extend. So seek simplicity as a principle in every aspect of your life.

Conclusion

In this article, I shared my journey transitioning from a services company to an early-stage product startup, highlighting the challenges and lessons learned along the way. We explored the delicate balance between perfection and speed, the impact of stress and context switching on productivity, and the importance of deep work for personal and professional growth. Additionally, I shared personal productivity hacks and the significance of habits and environment in enhancing work efficiency. By reflecting on these experiences, I hope to provide valuable insights for those navigating similar paths in dynamic work environments.

Before ending this blog post, I’d like to share photos of my existing setup (that was in the middle of our living room) and current setup (isolated on a separate floor where I use to work out during COVID).

Refactoring: Lessons Learnt So Far

Sun, 24 Nov 2024 00:00:00 +0530

Photo by Nina Mercado on Unsplash

Background

Refactoring is the process of restructuring code, without changing its original functionality.

Over the past three years, my role has allowed me to pick a lot of refactoring tasks, which have enhanced my skills and understanding around Refactoring. Despite its proven benefits, many developers and teams approach refactoring with hesitation, weighed down by myths that it is risky, time-consuming, or equivalent to a complete rewrite. However, refactoring is none of these. When done correctly, it can significantly reduce technical debt, improve readability, enhance test coverage, and simplify debugging.

This blog aims to demystify refactoring by exploring its importance, addressing common misconceptions, and providing practical guidelines for approaching it the right way. Whether you’re tackling a legacy codebase or refining new code, understanding the essence of refactoring can empower you to build more sustainable software. Let’s delve into why and how refactoring should be a regular part of your development process, not an intimidating task.

Why should you refactor code?

After nearly a decade of writing software, I’ve learned that refactoring is unavoidable. While it’s possible to delay it, putting it off only makes the task costlier and more complicated down the line. Ignoring refactoring may offer short-term relief, but in the long run, the cost of addressing the accumulated technical debt grows exponentially.. The below points should help you understand why you need to prioritize refactoring regularly:

To untangle Code Ownership

One common complaint among developers working in a fast-paced startup environment is that the code for different modules often becomes tightly coupled, sometimes even within a single file. This situation arises because, in the beginning, the business logic is straightforward. Each time a developer needs to make a change, they perceive it as just another simple change. Over the years, this approach results in a file that resembles a god object, which seems to know everything but has lost sight of its original purpose. Refactoring plays a crucial role in untangling this complex web of code. By doing so, it allows different teams to take ownership of specific files or modules. This can be effectively managed using tools like GitHub CodeOwners, which help assign responsibility and maintain clarity in code management. Through refactoring, the codebase becomes more organized, enabling teams to work more efficiently and reducing the risk of errors when changes are made.

To reduce cost of debugging

Recently, I was on a call with another engineer, and we were trying to find problem in the production code. A section of the application wasn’t functioning correctly, so we started by reading the code starting from the controller to trace the potential path a user might have taken. The controller file was cluttered, containing most of the business logic in one file. It was filled with numerous methods that called one another, each with lengthy lists of arguments that were difficult to guess. We spent over an hour trying to pinpoint the issue, but even after all that time, we weren’t entirely confident about the root cause. Having dealt with debugging production issues in the past where the code was better organized, I realized that these engineers were losing countless hours every time a problem like this came. If you find yourself in a similar situation where valuable productive time is being wasted on debugging due to tangled code, it’s crucial to have a conversation with your managers. Discuss the importance of planning for refactoring in the future. By doing so, you can improve the code structure, making it easier to identify and fix issues, ultimately saving time and resources in the long run.

To reduce techical debt

Technical debt is the cost of future rework that results from prioritizing speed over long-term design in software development.

Companies often ship code that needs more work before adding new features. While launching the initial product is the top priority, addressing tech debt often gets delayed. For example, when working on a new module, you might hardcode values in an existing class. As more requirements come in, adding more hardcoded values or if-else statements just creates a mess. This means you need to refactor the code before building more on it. Remember this quote: First make the change easy, then make the easy change.

To improve engineering culture

The cleaner and more well-structured your code is, the better it becomes as a reference for others. When you take the time to improve and maintain high-quality code, it not only benefits the project at hand but also sets a positive example for the rest of the team. New team members will be motivated by its clarity and organization, inspiring them to adopt similar practices in their own work. This fosters a culture of excellence, where everyone strives to write better code. On the flip side, if the codebase is cluttered and unorganised, there’s a high likelihood that others will follow that same pattern and create similarly messy code. It becomes a cycle, where poor practices multiply. As the team grows, this culture of quality spreads, impacting the overall productivity and the risk of technical debt based on the shape of initial code.

The habits you instill in your codebase today can shape the quality of work throughout the entire team tomorrow. Before we learn how to refactor, let’s clear up some myths about refactoring.

Myths around refactoring

It’s another name for “Rewriting”

Before we talk more about refactoring, it’s important to understand basic differences between Complete Rewrite and Refactoring:

Aspect	Complete Rewrite	Refactoring
Scope	Entire codebase is discarded and rebuilt.	Improves the existing code incrementally.
Risk	High risk of introducing new bugs or missing features.	Lower risk as existing functionality is preserved.
Time & Cost	Time-consuming and often more expensive.	Typically faster and more cost-effective.
Goal	Start fresh with a new design or approach.	Improve code quality, maintainability, and performance.
Disruption	Can cause significant disruption to development and team workflow.	Less disruptive, as changes are small and incremental.

I’ve seen developers (of all experience ranges) raising PRs with thousands of lines of changes across multiple files, claiming they’ve “tested everything thoroughly”. They think this one PR will solve all their problems, but that’s rarely true. In most cases, the real issues begin after merging the PR. Don’t take this negatively if you’re following the same approach currently. Even I’ve been that developer who has merged large PRs without breaking anything on production, and I’ve seen others do it too, because they took the mental stress of testing every case and fixing all review comments on the large PR (and again test everything). The only problem with this approach is that it doesn’t scale for everyone in the team. While everyone can raise large PRs,

Not everyone can test them thoroughly. And even if they do it once, they don’t do it with the same effort after fixing the review comments where they might have changed something in logic or in design.
Also, not everyone can review large PRs. As a result, many such large changes go live without a good code review. In my experience, reviewing large PRs can easily take more than an hour and not everyone is ready to put that much effort. These days I often don’t review large PRs. I spent ~ 5 minutes in understanding the structure of changed files and then leave a comment suggesting how the author can raise smaller PRs that are easy to review.

Due to how developers have been approaching “refactoring”, management and leadership have started seeing it in the same way as “rewriting.” While raising a large PR for a new feature is safer (since most of the code is new), it is considered very risky for refactoring. If your code change breaks a stable module in production, it will confirm your manager’s belief that refactoring is as risky as rewriting. So, make sure you avoid the “big bang” approach.

It’s meant for legacy applications

Refactoring is linked to legacy systems because legacy systems often accumulate large technical debt over time, but it’s just as important for new and evolving applications. Even in modern codebases, you can gather small debts like long classes/methods, duplicate code, unclear variable names, etc. Refactoring lets developers improve and optimize code continuously without a major overhaul. I often refactor code I wrote last week or last month. When a change makes a method or class messy, I find better ways to rewrite that code. It gives a different kind of satisfaction 😌.

It’s risky, hard and unscoped

The favorite thing for an engineer to do is build new features. The second favorite (for a lot of us) is doing a big-bang rewrite of an old system. But there are two issues with large rewrites:

They have more chances of breaking production because of size of change.
They have less chances of getting shortlisted by your manager, again, because of it’s size. And because such tasks do not create any immediate business value, they gets shelved.

If your manager has been a Senior IC person in the past, there might suggest you to split the project in order to get it prioritized. But if they are someone who’s not much into tech these days, and they do not suggest something like this, your refactoring project won’t get picked ever. So irrespective of how technical your manager is, it’s your role to understand the scope of the project and break it into parts that can be shipped. If you’ve never done it, please start doing it from now. Shipping new features can go in large PRs (although I don’t recommend you doing that) but refactoring projects should never go as large PRs. And because many developers have been executing them wrongly, refactoring is seen as risky, hard and unscoped.

It’s a one time event

A significant amount of technical debt shows that your team hasn’t focused on refactoring for a long time. If you compare the cost of refactoring a system today with the cost three years from now, you’ll see that doing it today is easier and cheaper (unless the product is being retired in the next three years). This is because if experienced team members leave, new ones will struggle to understand the messy code that has built up over the years. Refactoring is a skill that everyone develops over time. Once you successfully complete a small refactoring project, you gain the confidence to handle more tasks and improve the system further. As you master this skill, it’s important to share your knowledge with others, so everyone can help improve the existing codebase. Ideally, refactoring shouldn’t be something you do only once or twice a year. It should be a continuous and intentional effort to keep technical debt from becoming too much to handle.

Refactoring is “moving code across files”

“Don’t mistake motion for progress” - Alfred A. Montapert

In web applications, there is a common principle that your controllers should remain small and focused. This means they should primarily call the service layer and handle the response based on the service’s output. As a result, developers who find themselves writing extensive code within controllers might feel compelled to move that code elsewhere. While this instinct is understandable, it is crucial to approach this task with careful consideration and planning. Instead of simply cutting and pasting lengthy methods from one file to another, which only addresses the superficial goal of reducing controller size, a more thoughtful strategy is required. True refactoring involves a deeper analysis of each method’s purpose and determining the most appropriate location for it. This process requires developers to think critically about the design and structure of their classes, ensuring that each method is placed in a context where it logically belongs. By engaging in a well-planned refactoring activity, developers can achieve more than just smaller controllers.

I believe you now have a solid understanding of what refactoring is—and what it isn’t. Now, let’s dive deeper into how to approach refactoring and explore the key challenges to watch out for during the process.

Approaching Refactoring

It is done in isolation

When someone is working on a feature, they’re often motivated to fix the code around it. As a result, we get to see PRs where someone lints an entire file rather than the method they changed, or improved the logic of the methods that is getting called from their code. Their intention is to improve or optimise the code but the change could probably be taken up in a separate PR. Just like you don’t mix drinking with driving, avoid mixing feature changes with refactoring. Make sure that if you want to do the noble act of improving the existing code, raise a separate PR for it so that it can be individually tested, reviewed and released (and even reverted, incase something goes wrong). This will ensure that your feature changes do not get blocked due to review comments on this refactoring changes.

It should be guided by tests

Test cases are incredibly important when you are refactoring any part of a system. They serve as a safety net, ensuring that the changes you make do not unexpectedly break an existing functionality. For example, if you have a comprehensive set of unit tests for a large file, these tests can guide you through the process of splitting the file into smaller, more manageable pieces. With unit tests in place, you can confidently refactor, knowing that any errors introduced during the process will be quickly identified. This allows you to focus on improving the code’s structure and readability without worrying about unintended side effects.

What if I don’t have test cases for some code? - Incase you don’t have test cases, I recommend building a strong automated test suite first, or find ways to manually test the system to confirm everything works after refactoring. Please do not prioritize refactoring without a good way of testing out changes. I’ve been moving code from controllers to services, and from large services to smaller, more manageable services using this approach. I make sure my controller tests cover 100% of the controller and service code. Once that’s done, I know that any changes I make will be caught if they break any part of the API contract.

Refactoring should use established patterns

While many believe that code is poetry — and I wholeheartedly support the idea of creative problem-solving, it’s equally important to follow established patterns and best practices when refactoring code. These patterns have been developed and refined by the community over decades of experience while solving similar problems. They encapsulate lessons learned from solving common challenges, ensuring solutions are robust, efficient, and easier for others to understand. Established patterns, like design patterns (e.g., Singleton, Strategy, Factory), architectural styles (e.g., modular monoliths, microservices), solutions to common code smells (like long class, long methods, data clumps), and practices specific to a language or framework act as a shared vocabulary among developers. When you follow these patterns, you’re not just solving a problem, you’re solving it in a way that others can immediately recognize, adapt, and build upon. This reduces cognitive overhead for team members, accelerates onboarding, and improves collaboration. While building a solution in way that it feels “uniquely yours” is a tempting feeling, these (overly creative) solutions can become difficult for others to maintain, debug, or extend over time. By grounding your refactoring efforts in proven patterns, you ensure the code remains accessible and future-proof.

Avoid Premature abstraction

Abstraction is a great concept, but premature abstraction makes things complex. A common rule in software engineering suggests to avoid having an abstraction till there are at least three repetitions of the code fragment that you’re planning to abstract out. While “three” might not be an appropriate number for everyone, it should be a good starting point for you if you don’t have anything else in mind. Waiting for a few repetitions of the code ensures that you have some data around what to generalize and what not to. While duplication of code is considered bad, generalizing prematurely might lead to speculative designs that are very different from the real world needs. Write code that is clear, direct, and solves the immediate problem effectively. Refactor and abstract only when the need becomes evident—when similar patterns emerge across different parts of the system, or when a particular piece of functionality needs to be reused or extended.

Keep changes small and reversible

I think we’ve already discussed about the benefits of small PRs, so keeping the changes small is an obvious thing. Another important thing is to ensure that changes are reversible. As refactoring can touch very critical parts of your application, it is important to plan for failure. The simplest rollback plan would be to turn off the feature flag incase something goes wrong (or rollback the previous release). But that only works for deployments that didn’t modify any data. If you’ve modified some data in your database, it might be hard to rollback. As a precaution, try to follow an approach where rolling back data changes is also easy. For example, split your deployments like this:

Add a new column that has new data. Push the new code along with this but don’t flip the feature flag yet.
Back populate the data in new column and add relevant callbacks in the system to ensure new data gets copied.
When previous data is also copied, flip the feature flag and see if everything is going well. If something goes wrong, immediately turn off the feature.
After few weeks/months (based on amount of testing you need), drop the old column, delete the old code and drop the if-else code handling the feature flag.

Based on the scale of data and complexity of project, the above approach can sound normal or an overkill to you. But trust me, I’ve seen things going wrong just because developers didn’t plan well for rollbacks for data related changes.

Document changes

Documenting changes during refactoring is essential for maintaining clarity, ensuring team alignment, and supporting future development. It captures the rationale behind the changes, helping others understand why specific decisions were made, especially if they impact dependencies or architectural patterns. Documentation preserves context to avoid undoing improvements and provides a historical record for long-term projects. It also simplifies onboarding by giving new team members a clear view of the code’s evolution. Good documentation highlights what was refactored, why it was needed, and how it aligns with the project’s goals, helping avoid confusion and duplication of effort. Including notes in commit messages, pull requests, or internal wikis ensures the refactoring’s intent is communicated effectively. This practice not only fosters collaboration but also aids debugging by clarifying the purpose and impact of the changes, ensuring the refactored code remains accessible and adaptable.

Monitor performance, check for bugs

When refactoring, it’s crucial to monitor performance and check for bugs to ensure stability and identify unintended side effects. While refactoring doesn’t guarantee performance improvements or fewer bugs, it can pave the way for these by simplifying the codebase and addressing inefficiencies. A well-refactored codebase should also make future optimizations (such as adding a caching layer) easier to plan and implement. Additionally, cleaner code helps in debugging and enhances maintainability (as already explained above). Use comprehensive test suites to validate changes and check for regressions and APM tools to monitor real-world performance after deployment.

Value contributions

This message is for team leads and engineering managers: one of the most common reason why developers struggle with refactoring tasks is a lack of motivation. And often, this lack of motivation comes from how teams undervalue refactoring compared to business deliverables. As Eileen pointed out in her latest RailsConf talk, a good engineering culture starts at the top. Many of us have worked at organizations where messy code is shipped in the name of urgency, and the architecture becomes too complex to maintain over time. If we’re to take actionable insights from her experience, here’s what leadership can do to address these challenges:

Prioritize Refactoring - Treat refactoring tasks with the same importance as product development. Assign dedicated bandwidth for these efforts. Spending “few hours a day” don’t work and often lead to half-hearted results.
Align Teams - Ensure engineers, product managers, and other stakeholders understand that refactoring is just as crucial as building features. A healthy codebase supports long-term business goals.
Invest in Education - Not all team members naturally excel at refactoring. If specific skills are needed to address technical debt or architectural issues, invest time and resources to train your team.

Eileen, with over a decade of experience at companies like Basecamp, GitHub, and Shopify, emphasized these points from her journey. While I haven’t worked at large organizations, I’ve faced similar challenges in small-to-medium startups. And seeing the comments on her talk, these problems persist even in teams following microservices architecture. So we can conclude that the challenges of maintaining a clean, scalable codebase are universal.

Conclusion

While refactoring is essential for maintaining a clean and efficient codebase, it’s important to respect the existing code, even if it appears messy. This code was often written under tight deadlines to meet urgent business needs, and it has successfully supported the business for months and years. Recognizing the value of this “mess” is crucial, as it reflects quick thinking of developers who ensured the continuity and success of the business, which is crucial for any organisation.

Thank you for reading this post, have a good day 😊.

1:1 conversations with 10 Software Engineers

Tue, 15 Oct 2024 00:00:00 +0530

Photo by Dylan Ferreira on Unsplash

Background

I’m always looking for opportunities to engage with people who share similar interests and experiences. Recently, I shared a Google Form to connect with others, and I was thrilled to receive about 10 responses! Most of the engineers were based in India, with one from Switzerland. It was great to chat about their work, challenges, and career growth, and I learned a lot from these conversations.

After some of these discussions, I posted a few interesting insights from our calls. I’m now writing this (detailed) post to share some highlights and reflections, as many of the conversations included requests for advice that could benefit a wider audience.

What all we discussed?

In this section, I am sharing all the key points that we discussed during our calls. Each of these points will be explored in detail in the sections that follow. Most of the engineers had 4-7 years of experience and wanted advice on career growth along with potential future work and challenges. Here’s what we talked about:

Why am I doing this for free? Do I offer paid mentorship? - Not everyone asked this but two separate people asked each of these questions.
How to prepare for Interviews? - Half of them were either in their notice period or actively searching for new opportunities. I was able to provide detailed insights on this topic, having recently gone through a job transition myself. Additionally, I shared thoughts on various learning resources because all available content is not of good quality.
How do large teams work? - It’s fascinating to observe how massive products like YouTube, Amazon S3, or Uber are developed and maintained. These platforms are supported by extensive teams that collaborate across different regions and time zones. Understanding the dynamics of such large teams can provide valuable insights into how complex projects are managed and delivered successfully.
How do I hire people? - I’ve given interviews at about 6-8 companies before switching this time. But at LocoNav, I had interviewed > 200 people for various engineering roles including Senior Engineers, Managers, QA engineers, Devops. So I shared some insights here based on my experience.
What are cultural differences between Indian and non-Indian teams? - I discussed this with someone working outside India who also had experience working in India. Many others have shared similar insights, especially those who have dealt with “not so good” bosses.
How do you handle/convey delays across layers of stakeholders? - Handling this can be challenging, especially when it involves end customers. We’ll discuss a framework that helps keep everyone satisfied.
Are Big Tech engineers are superior to us? - Someone had this in their mind, so had to discuss around this inferiority complex. Another guy had a question - Why my friends in big tech companies get paid so much if they do same work as we do?
How to grow as an Engineer (not a Java Engineer or Ruby Engineer) - It’s important to be skilled in at least one language, but it’s even more important to stay flexible and ready to learn anything new that comes your way.
How to know about scale without working on scale? - Not everyone gets a chance to work on scale or work on certain things like Devops, but many companies seek that experience. How to counter that?
How I generally approach learning anything new? - I was asked this question in an interview also. I’ll also share what I call as continuous learning.
How often your managers sync and how does it impact your growth? - I’ve been managed by all sorts of people — from those who didn’t sync in months to the current org where managers mostly sync weekly.
How to know about team’s culture before joining? - While salary details are clear before you join, it’s tough to understand a company’s culture until you start working there. However, there’s a simple trick you can use to learn about the company before accepting an offer.
How to be good at real world HLD? - As I spoke with mid-level engineers aiming to become senior engineers, they were eager to learn about real-world system design. In many companies, you rarely get the chance to design large systems because high-level designs are costly to redo, and most people work on systems that are already designed. Despite this, it’s crucial to understand core system design concepts so that even when making small changes to an existing design, you are aware of their implications.
I’m not working on Kubernetes or Microservices, will I be easily able to switch? - While I believe that it’s not important to get your hands dirty in every single hot technology to be relevant in the current market, you must remain updated with a few things.

Let’s discuss about each of these one by one. This is going to be a long blog post because it’s summary of more than 13 hours of discussion. If you find this really long or parts of it not relevant to you, feel free to read specific parts using table of contents on the top.

Why am I doing this for free? Do I offer paid mentorship?

Many engineers, especially those at Big Tech companies, offer paid mentorships through platforms like Topmate, charging varying fees. While having a second income is fine, I’ve received help throughout my 9-year career without being charged. It doesn’t feel right to start charging for sharing the same knowledge. My religious beliefs also motivate me to offer help for free. I’ve encouraged others to help for free whenever possible. My job pays well, so I don’t want to charge for an hour of conversation. Additionally, not all calls are about mentorship; some people just want honest chats about work, experiences, productivity, team culture, and more. As of now, I don’t plan to offer paid mentorship services. Feel free to message me on LinkedIn if there’s anything I can help with.

How to prepare for Interviews?

As half of the engineers were in their notice period, this was a pretty common question — How to prepare for interviews and what all things to study. I’ve already written in detail about my interview experiences, resume improvement journey, low level design interviews and job titles. I’ll still share more based on their specific questions.

Learn about different companies and target accordingly - Talking specifically about engineering roles, there are different types of companies like services companies, consultant companies, early/mid-age startups (Seed/Series-A/Series-B), Scale-ups (Series C+, post IPO), and Big-Tech companies. To explain a bit about each of them:
- Services companies typically include (but not limited to) Infosys, TCS, Accenture, Cognizant etc.
- Consultant companies could include MBB, Thoughtworks, etc. Similar to the first one, you are working for others (not on your own products) but the role consultant-based where you completely handles client, not just code.
- Early/mid age startups could include any company that is going 0-1 (building MVP) or 1-10 (secured some funding and building for real customers). People in such companies are expected to be generalists, who can wear multiple hats and can get work done really fast.
- Scale ups could include public listed companies. I could categorize them as young, agile, fast-moving but having lot of governance and processes because of being a public entity.
- Big Tech (as you’d expect) includes FAANG, and similar sized companies like Atlassian, Uber, Stripe etc. I’d call any other company as BigTech if they have >1000 engineers. In comparison to any other kind of company, here you’d find a lot of internal tooling.
  
  There’s a trend of people moving from services companies and startups to Big Tech companies, but I’ve also seen people move from startups to other startups or from Big Tech to startups. For example, I started at a services company (not as big as the ones mentioned) and then moved to startups. So I’d recommend you to decide types of companies you want to target and prepare for them accordingly. There are many websites, videos, and interview experience blogs that explain the interview processes for each of these.
Process familiarity - All companies have their own process for selecting candidates. While common rounds like DSA, LLD, HLD, and HM exist in most companies, always get the details from your recruiter. If you’re interviewing with a large company, the recruiter will likely give you a document explaining the process with resources to help you prepare. In smaller companies, you often need to ask questions to get clarifications. I’ve met engineers who didn’t know the total number of rounds because the recruiter didn’t explain, and they didn’t ask. You must ask these questions to understand what is expected in each round so you can prepare accordingly.
Resources for preparation - A lot of my work over the past 2-3 years has focused on refactoring and migrating systems, which has improved my low-level design skills. One way to prepare is by taking on similar work in your current role to naturally enhance your design skills. However, this approach doesn’t work for DSA and HLD rounds. That’s why I view the LLD round differently from the DSA/HLD rounds. Generally, people in roles like SE, SSE, or Staff, who are individual contributors, are involved in building or maintaining systems, writing, and deleting a lot of code. Engaging in similar work will automatically improve your low-level design skills as you create and refine system designs. On the other hand, most of us don’t use the advanced data structures that are tested in interviews, making DSA a topic specifically for interview preparation. Similarly, few engineers get the opportunity to build or modify high-level designs, so they don’t improve practically in System Design. Working at an early-stage startup in my last role gave me a solid understanding of cloud and system design, which many engineers don’t get. To succeed in these rounds, I recommend spending a few weeks or months (depending on your current skills and preparation time) to master the basics of these areas:
1. Resume building - Please please please spend time in building and continuously improving your resume. Eliminate all spelling mistakes, punctuation errors, and any other beginner mistakes. For a detailed overview on how I did it, read this blog.
2. DSA - Leetcode helped me improve my DSA skills, but for my role (Senior/Staff) the questions were mostly easy-medium level. Based on your level and companies you’re targeting, you might be asked a combination of easy/medium/hard questions, so prepare accordingly.
3. LLD - I studied from multiple youtube channels and solved some problems in Ruby and Java (available here) so that I can solve them on a decent speed during the interview. I did not spend a single penny on a paid resource for LLD. If you want to know more about different kinds of LLD interviews I gave, I’ve written a detailed blog on it already. Feel free to check it out.
4. HLD - I was skeptical on my HLD skills so I took a paid course but that was badly structured. It was enough to confuse any beginner but having a good idea of those concepts already, I was able to navigate through it and complete about 70% of it before giving interviews. Later, someone told me about Educative.io and it looked much better organised. I’m not pushing you to use the same resource because of a few reasons. First of all, it’s a paid course and I do not promote anything paid on my blogs. Secondly, it’s a text based course while most of the other courses are video based. I’m fond of reading so I purchased it, you may not like it if you’re not into reading. And third, their well-known system design course (Grokking) is very long. So I’d recommend this only if you want to learn the concepts beyond interviews and the course falls within your budget. Although I don’t endorse it, I can vouch for the quality of content.
5. Behavioural / Hiring manager - Watch some free youtube videos to understand the structure of this round and what people generally ask. Beyond that, just be true to yourself and be good at communicating your thoughts. You might struggle here if you’re not an efficient communicator.
Where to apply? - There are plenty of websites could help you. Again, I didn’t spend anything for my job search but these are the websites I used - LinkedIn jobs (set relevant job alerts), Tophire (I got Branch’s offer through them), Bigshyft (I still get calls from them 😂), Instahyre (Atlassian reached out through this) and Hirist. I also registered on Indeed and Naukri but never got relevant openings through them. You can find and register on more platforms but at least register yourself on these.

I hope this helps for interview preparation. Moving on to the next topic 😎.

How do large teams work?

Let me add more context to this question — This was asked by someone who was working for a startup for about 4 years, and was wondering how things would unfold if their team scaled 2x-3x in strength. How would the projects be managed and what challenges would exist for them being an old member of the team who is transitioning more towards a managerial role. This is a common question for people working in small teams. They sometimes wonder how large teams operate to make something as large as Youtube, Google Maps, or Amazon S3.

I’ve been in a habit of reading a lot of content in the past few years from which I shared the following insights:

First of all, no large system is built in a day, or week, or even a month. Smart engineers start building things in the most stupid way, and iterate on them as they find bottlenecks in the system that prevents it from scaling further, so that they can replace those specific parts of the system. I remember my last company’s co-founder (who was also a Ruby engineer) using a hacky way for partitioning tables in Postgres 9.3. If you’ve used that version of Postgres vs Postgres 12+, you’d know that the current partitioning features didn’t even exist in that version. But at that time we didn’t have enough choices because we were pretty early stage and had limited exposure to good tech. Once it was no longer scalable for us and the cost became very high, the team handling that system redesigned the same into a separate data layer that used a suitable technologies. It’s very important to start stupid initially and satisfy the business needs so that you can sell whatever you made. For example, it would be hard to assume that YouTube used MySQL to begin with, and they made it work till a massive scale.
Having large systems doesn’t mean that you have really large teams that run them in a monolith fashion. For example, Amazon’s two pizza rule is very popular in the software industry that says — No team should be big enough that it would take more than two pizzas to feed them. Ideally, this is a team of less than 10 people: smaller teams minimize lines of communication and decrease overhead of bureaucracy and decision-making. Also, because we’re talking about large teams and about Amazon, this article is a great reference. The most popular blob storage service, Amazon S3 alone is composed of more than 300 micro-services, and you cannot assume all the teams building them to come in one giant standup thrice a week to talk about their work. It’s obvious that a lot of engineers and managers would be managing these services so that they can focus on specific areas. At the same time, it’s very easy to talk about good engineering management at that scale, than doing it efficiently. So I always appreciate how these large organisations remain agile and build massive applications for their customers.
If your company is building really large systems, they also invest in building Internal tools so that developers in different teams follow same standards and do not reinvent the wheel for solving similar problems. In my previous role, I’ve worked in the developer productivity team for more than one year. As our team was scaling in strength and developers were breaking the system into microservices, we were building internal tools for configuration management and deployment so that we could easily deploy the applications to EC2 machines, dockers or kubernetes. We were required to support multiple deployment systems because we had not transitioned fully to kubernetes and different applications (both old and new) were getting deployed to different platforms.
Not all the processes of large organisations make sense for small organisations, or for those organisations that are scaling to a larger one but I’d like to emphasize on one very important rule that impacts the success of projects — Amazon’s Single threaded leadership principle. In startups, we generally don’t hire people as new initiatives begin. As a result, same set of people are juggling (and sometimes struggling) with multiple initiatives. If there are some gaps in the quality of delivered products, it’s acceptable to the leadership because they know that people are trying their best. With teams scaling to a larger strength, it is important to revisit this culture. If it is not acceptable going forward, it’s important to improve the team’s culture in a way where people are not overwhelmed with projects and whatever they deliver is of much better quality. The Single Threaded Leadership Principle might make some sense for the scaling organisation because at a specific point, teams might be catering enterprise clients where quality is very important.

How do I hire people?

I’ve been contributing to hiring efforts from my second year as a Software Engineer, where I use to take DSA round. I’ve taken hundreds of interviews since that time, which include the same for Engineers (junior / senior / lead), Managers, QAs and Devops. I personally believe that it’s very important to understand the candidate and their aspirations, rather than just throwing a problem in front of them and talking about it for an hour. Interviews of a candidate should happen in a way where each interview helps you to paint a clearer picture of their behaviour and skills rather than every round starting with the old “please introduce yourself” question. To hire anyone in future, I’d look for these things:

Technical excellence - If it’s a hands-on technical role, it’s very important to ensure that the candidate understands code and should be fluently coding in one language of their choice. I always focus on the basics rather than evaluating someone on a fancy leetcode problem. As an example — while interviewing Devops people, my favorite interview question use to be “You’re trying to SSH to a machine but the command is unresponsive unless you cancel it. How would you debug this?”. I want to see how people respond to such basic questions —What clarifications do they take on this small question? Do they talk about network barriers like security groups or VPC route tables? Do they consider the fact that if the authentication key is wrong, then you immediately get an error rather than having a hung connection. I appreciate good debugging skills.
Ability to break a problem - You could learn about this while evaluating design rounds, or when they explain some of their projects. IMO, if someone can break problems into parts, they can work on things incrementally rather than doing big-bang releases/refactors. Also, these people estimate projects better than those who do not understand the power of small changes. This is an under-rated technical skill.
Programming language agnostic - I’ve done one wrong hiring in the past where the candidate was too much tied with Spring boot framework (not just Java). They told about the same after joining but I was not managing projects that were Java based. In the end I had to move them in some other team because the other option was to let them go. As a learning, I’d not expect a person to be polyglot already, but open enough to work in any language.
Quality of work - When evaluating their solution in any round, check if their solution meets the ask of the problem statement. How well do they handle the edge cases? Do they ask enough clarifications where required (or they just assumed those)?
Getting things done - While quality of work matters, I’ve seen people getting stuck on small things and not having enough bias for action. Having worked in startups, I think this is something I’d look for all kinds of roles. If you’re a good engineer who writes optimised code but do you not push the barriers that come your way, you might not be able to release something real that gets used by the customers.
Cultural aspects - In small organizations, while we may not conduct formal behavioral interviews, all interviewers are vigilant about observing a candidate’s behavior throughout the interview process. This careful observation is crucial to ensure that we do not end up hiring someone who might not be a good fit for the team. From my experience, I have witnessed the consequences of hiring the wrong person, ranging from fresh graduates to Heads of Engineering (HOEs). Depending on their role and influence within the team, a poor hiring decision can lead to significant negative impacts, even within a short timeframe.

To summarize:

Good basics + great debugging skills.
Ability to break problems and some past experience working on quality projects.
Bias for action, humble and eager to learn.

What are cultural differences between Indian and non-Indian teams?

I think all of you have heard of a few cases in the recent past where employees died at work or committed suicide due to work pressure. It’s very unfortunate to see the same companies having great working culture outside India and highly toxic culture in India. I wrote about blameless culture recently where I shared some examples of how good or bad culture propagates from the top to bottom, or from old employees to new employees. It all boils down to a few people on the top that we call as the leadership. These people define:

Code values and principles.
Quality of work.
Communication Style.
Response to mistakes / failures.
And most importantly - Work life balance.

For me, the 16-hour workday philosophy is nonsense. If I don’t have time to think about other things in life, I can’t grow personally or professionally. I’ve worked with US teams in my first organization, and now I work with both US and Africa teams (in addition to the India team). Fortunately, the overall culture encourages taking leave whenever needed, whether for a personal emergency, rest, or vacation. People in the US don’t work beyond a certain limit, but that doesn’t mean they haven’t created good products or scaled companies.

In a nutshell,

Do not over-work regularly. Doing the same for a few days should be fine based on the business needs and your physical/mental health.
If you or your team overworks everyday, figure out what could be wrong (some ideas here and here for engineering teams).
Do not value any work or opportunity above your health or family.
Fix the company culture if something is wrong. If the leaders do not support you, find a new job.
Take your work seriously. In the 8-9 hrs you put, give your 100%.

How do you handle/convey delays across layers of stakeholders?

Let’s consider this hierarchy: developers (including you) → project manager → business teams → customer. Anyone who has successfully launched a product knows that initial estimates are often inaccurate. The accuracy of your estimates depends on many factors, such as:

How undefined is the problem? - Do we need to check if it’s feasible? Are there dependencies on other teams?
How much experience does our team have in building something like this? - This includes the experience of senior engineers, product managers, and designers.
How well did we break down the problem while estimating? - If you break it down well, you’ll likely complete half of the software design, and your estimates will be closer to accurate.
How new is the team? - If there are many new members, there is a possibility of inaccurately estimating their speed in developing production-ready systems.

Beyond this, each layer should understand that the previous layers might exceed their estimates, so they should include their own buffers. For example, if developers estimate 6 weeks, managers should say 7-8 weeks, and the business team should assume 8-10 weeks. It’s okay to under-commit and over-deliver (but not the reverse).

If your team can follow this kind of framework, that’s excellent. It means you won’t have to struggle as much to convince customers who have been waiting for a feature. A note for managers: your team might sometimes struggle with accurate estimations. However, that doesn’t mean you should provide your own estimates to the business team. Non-engineers should avoid estimating on behalf of engineers. If you find that the team’s estimates often miss the mark, take the time to discuss and reflect with them.

Are Big Tech engineers are superior to us?

I respect great engineers and managers who have built really large systems without which the world would have been very different. At the same time, do not think that they are of a different breed. I think engineers in startups typically work in environments with fewer resources, less structure, and more uncertainty. They wear multiple hats, tackling everything from backend to frontend, cloud and even take product decisions. This exposure forces them to develop a holistic understanding of technology and the business. In contrast, engineers at Big Tech firms may specialize in a narrow domain within a larger system, where their role and responsibilities are more clearly defined. While specialization can lead to deep expertise in specific areas, it may also mean less exposure to the end-to-end development process, product strategy, or direct customer interaction.

The perception of Big Tech engineers being superior is often tied to the prestige of working at a well-known company and the assumption that these engineers have passed rigorous technical interviews. Another key difference is the sense of ownership and impact. In startups, engineers are closer to the product and customers. The features they build, the bugs they fix, and the architectural decisions they make have a direct influence on the company’s trajectory. This proximity to the business side creates a different kind of pressure but also provides unique learning opportunities that many Big Tech engineers may not experience. Ultimately, the skills and experiences gained in startups can be just as valuable as those acquired in Big Tech. Both environments have their strengths, and neither is inherently superior to the other. So the engineers in startups shouldn’t view themselves as “inferior” simply because they don’t work for a large company.

On the other hand, big tech companies manage vast products, extensive business operations, and significant overall revenue. In such environments, even small optimizations made by engineers can lead to substantial financial benefits. For instance, a minor improvement in code efficiency or system performance can save millions of dollars due to the scale at which these companies operate. Therefore, it’s important not to assume that engineers in big tech are performing the same tasks as those working on an app serving only 1,000 users. The scale, complexity, and potential impact of their work are vastly different. Engineers in big tech often deal with challenges related to scalability, security, and global user bases, which require a different set of skills and approaches compared to those needed in smaller-scale applications.

How to grow as an Engineer (not a Java Engineer or Ruby Engineer)

I’ve been fortunate to be mentored by someone who was a polyglot programmer. Although I’ve majorly written only Ruby in production throughout my career, I’m mentally aligned to think about problems without tying them with language specific constructs. Knowing multiple languages help you to apply good things of one language in another language and build something in a unique way. With this knowledge, you can also plan to build certain parts of a system in a different language that is more optimised for the use case, especially when you get a chance to split them into services.

In many of these discussions that I had with these engineers, I told them to start thinking of solving problems without talking about the language. Also, whenever given a chance, do not shy away from learning a new programming language. The maturity that comes with this mindset also helps you to write build documentations where you use more generic / industry standard terms rather than using the names of specific libraries that were used to implement the same (eg. calling part of system as background job processor rather than celery, jobrunnr or sidekiq).

How to know about scale without working on scale?

This question came from someone who was asked about database query optimisation in an interview. While they knew about indexes and a few other things that help you optimise queries, they had never done something like this in the past. I think there are two ways to learn about scale:

Build something (in a company or as a personal project) that scales to millions of users.
Or, see how others did it.

By “others,” I mean learning from the experiences and mistakes of other people. Almost all large tech companies have blogs where they publish case studies, failure stories, and success stories. In addition, there are independent bloggers who write about their experiences in building and debugging large systems. Subscribe to some of these blogs, and you’ll notice a difference in your knowledge in the next three months.

If you’re not sure where to start from, I’ll give you some ideas:

Think of the big tech companies that your admire (examples - Github, Shopify, Stripe, Instagram, Slack).
Follow some good substack newsletters. I really like Pragmatic engineer’s newsletter (although a lot of content is paid).
A lot of people reading this might already be familiar with Arpit’s youtube channel. I really admire the quality of this content.
You could consider subscribing to my blog if you find the content good enough 🫣.

How I generally approach learning anything new?

There are generally two kinds of things that anyone is learning:

If you’re learning something to quickly build a prototype where you do not care about quality, use any source to learn. It could be a 10 minute youtube video, or a brief documentation, or ChatGPT prompts, that help you write some code, or understand a few basic concepts about a new technology.
If you’re serious about learning that topic, still follow step 1 so that you build some interest and then learn from a book. I’m old an school guy here, who would prefer books over videos. I think these days, anyone can put their camera on a tripod stand to record a video after a little bit of research. Here the quality of content might be good, average, or bad. But if you’re picking a book of a good publisher, the chances of getting a good quality content are much higher. This is my own theory, feel free to disagree 😂.

I learnt docker back in 2022 using this method. Earlier I just created a Dockerfile for one microservice that I was building. It wasn’t optimised at all (I copied it from somewhere). Later I read DIAMOL to get a deeper understanding of Docker, after which I did a lot of optimisations in different Dockerfiles and wrote this detailed article to help the community.

Continuous learning - Your work might be very challenging, but it can still limit your learning to certain areas. Make sure to set aside some time regularly (weekly or monthly, if not daily) for learning. I understand that not every day is the same. Some days are more stressful, while others are more relaxed. Create a list of blogs you want to read or videos you want to watch, and learn from them whenever you have some free time.

How often your managers sync and how does it impact your growth?

In my first organization, there was no formal manager hierarchy in my team. We all reported to one of the three founders. Our project was in maintenance mode, so we didn’t have a senior resource like a Tech Lead assigned, as they were in other teams. I wouldn’t say my growth was halted because of this, but now that I have managers who sync weekly, I see the benefits of regular manager check-ins. Many people in these calls mentioned their managers don’t sync with them regularly. Here, regular doesn’t mean weekly but rather on a fixed schedule. This depends on factors like:

The company culture might not encourage managers to have regular meetings with all their direct reports.
Your manager might be on a technical role like tech lead and not a “manager-only” role where they have a tight schedule with their own deliverables.
You might be part of a fast-paced team where these meetings aren’t prioritized because delivering products is the main focus. The team may be learning a lot during this process, but that doesn’t ensure your goals align with the company’s goals.

In my previous role, I met with my manager once every 2-3 months, and I was managing a team of 7 people myself. Since I hadn’t experienced regular meetings with my managers before and wondered, “What would we even discuss every week?” I didn’t schedule regular meetings with my team. Instead, I told them to reach out to me at least once a month if they felt the need. In my current role, all managers meet with their direct reports weekly. They discuss everything from professional to personal topics to help you feel comfortable and aligned for growth. In short, managers should set up regular meetings (weekly, every two weeks, or at most monthly) to ensure that:

Their direct reports do not feel disconnected from them, especially in a remote setup. A strong connection is important to ensure that employees are motivated and feel comfortable enough to give feedback for the team’s overall improvement.
If there are performance issues or expectation gaps, they should be addressed early, rather than during the appraisal cycle. This ensures you have enough time to fix those gaps promptly.
Your career goals are aligned with the company’s goals.
These meetings can also cover topics beyond work, like how you are planning an extended weekend or how everyone is doing at home. After all, a healthy team is like a family :)

How do I know about team’s culture before joining?

A strong team culture fosters productivity by creating a supportive environment, while a toxic culture can decrease morale, leading to disengagement, lack of collaboration, and ultimately, low productivity. More than compensation, culture is crucial, yet many people overlook it before joining a company. If you’ve only experienced companies with good culture, you might not recognize bad culture till you get to see it first-hand. It’s important to know:

If the team engages in productive meetings or they just setup long meaningless meetings with a lot of folks. Do meetings have agenda and end with either action items or followup meetings?
How is failure handled? Does the team engage in blame game or fixes the root cause and learns from it?
How do people generally feel about the product they’re working on? Are they proud of it?
Do people encourage sharing feedbacks openly or does everyone think “they’re right”?

At this point, you might be thinking — how do I know about these things before joining? It’s not that hard. I’d do all this to confirm the same:

During your interviews, pay attention to how organized and punctual the interviewers are. Notice their communication style, how they evaluate your solutions, and the feedback they provide. A big red flag for me in design rounds is when people have fixed solutions in mind and force you to change your design to match theirs. Such people won’t let you innovate even after you join the team.
Check online reviews on some websites to get a sense of the company. Remember, some reviews can be biased, so this should not be the only thing you consider.
Talk to current or former employees to see how they feel about their work. Ask specific questions (like those listed above) instead of just asking, “How do you feel working at <company_name>?” Make sure you ask good questions. If they’re unhappy, they likely won’t want someone else to join and be unhappy too. To get a better idea, talk to at least two people.
In discussions with your recruiter or hiring manager, ask about work-life balance, leadership principles, and other factors that affect productivity. Compare their responses to what employees have shared about their experiences.

How to be good at real world HLD?

This response slightly overlaps with a previous section. You don’t easily get the chance to build something large or join a team scaling their systems. The best thing you can do without much investment is to learn from others’ designs and mistakes.

IIf you work for a company with large systems, talk to the people who built them or have been working on them for a long time. Read documentation to understand design decisions and how they’ve evolved. Look for ADR documents in large companies to learn the reasons behind these decisions.
If you’re on a small team with scaling products, it’s a great opportunity to learn how to optimize the system for scaling without overspending. Understand how indexes speed up queries and which types you need. Learn about caching and cache invalidation (the second hardest thing in computer science). Learn how you can write modular code to make future changes easier.
If you’re on a team with unstable products, find out what causes outages and read past outage postmortems. You’ll learn more from outage calls than from regular work. Don’t miss this learning opportunity (I’ve also shared my insights from debugging production outages here).
If none of these apply to you, learn from others through newsletters, blogs, and videos that explain these concepts.

Being good at HLD is not building scalable systems on day 1, but starting with a stupid solution and innovating on the go. And in this process, you should know (or learn) what part of systems can potentially be a bottleneck so that you plan your changes accordingly.

I’m not working on Kubernetes or Microservices, will I be easily able to switch?

No system needs micro-services architecture, an EKS cluster or Kafka to begin with. Even when they acquire real customers, they still might not need these technologies. Mature technical leaders know the challenge of introducing fancy parts in the infrastructure, and they make sure that you do not have them unless you can pay for the operational cost of running and maintaining them. It’s easy to spin a Redis cluster, an EKS cluster or even an API gateway with a few clicks on your favourite cloud provider. But it’s not easy to get away with these technologies once you’re in. For example, an EKS cluster has a starting cost of ~ $70-$80. In a similar cost, you can launch 2 small EC2 machines behind an Application loan balancer along with a small RDS. While EKS solves for your “scaling needs”, you don’t need to solve them on day 1. I remember my last company was running a single EC2 server that was also running background jobs in a 4 core, 16G machine when I joined in 2018. It worked pretty well for them at that time and we didn’t introduce an ALB till late 2018 because that has a fixed monthly cost + data transfer cost that no one wants to pay from the first day.

This question came from someone who had 5+ YOE experience but no experience working on microservices. My suggestion here is that you should read about microservices and when it’s good to have them in your system. I’ve worked on transitioning monolith to microservices after reading this book, and trust me that it’s not everyone’s cup of tea. You need a dedicated team to work on it rather than a few engineers working on it part-time. Even then, there are high chances that you end up building a more convoluted architecture (some thoughts shared here). For you as an engineer, it’s important to know all the challenges so that whenever you get a chance to do such a migration, your chances of being successful are high.

Working on distributed systems isn’t something everyone gets to do, but it’s important to understand their pros and cons. This way, future interviewers can assess your knowledge, and you won’t appear uninformed about these systems. So be it microservices, kubernetes or containers — know little bit about everything even when you don’t get to work on them in production.

Conclusion

Thank you for taking the time to read through this post. I hope the insights were worth the time you spent. Hearing firsthand about the diverse challenges, growth experiences, and problem-solving approaches was a reminder of how valuable open dialogue is for learning and professional development. Whether you’re navigating similar paths or different ones, I hope these reflections offer useful perspectives. Do share this with your friends who are looking for similar advice. Let me know your thoughts in the comments. See you in the next blog post 🚀.

Problems with generic code

Sun, 22 Sep 2024 00:00:00 +0530

Photo by Kier in Sight Archives on Unsplash

Background

I’m a Senior Engineer with nearly 9 years of experience in the software industry. In one of my previous projects, I had to migrate the background job processing engine of a Ruby-based application from delayed-job to sidekiq. For those not familiar with Ruby, think of these as two libraries that help you run async jobs in a distributed system, similar to JobRunr in Java projects or Celery in Python projects. We encountered several issues with the old library and realized that Sidekiq would be a much better option as we scaled. To summarize:

Sidekiq runs multiple threads per process to execute jobs, while Delayed Job runs one thread per process. This means Delayed Job uses more memory for executing the same set of jobs per unit time, especially if your workloads are IO-bound.
Delayed Job uses SQL table to store jobs, whereas Sidekiq uses Redis. Redis, being an in-memory database, is much faster than traditional disk-based SQL databases.
Sidekiq is a very popular Ruby library, so there are many extensions available for performing other common tasks. These include running crons and ensuring unique jobs. Additionally, Sidekiq’s author has started supporting a drop-in replacement for Redis for storing jobs.
While many open-source extensions exist for Sidekiq, companies can get more features and premium support with Sidekiq Pro.Migration phase

We added both libraries together for a few months and gradually migrated from DJ to Sidekiq. The two most common uses of background jobs were:

Calling an instance method asynchronously, like loading a model entry from the database and calling a method on it asynchronously (e.g., Model.find(some_id).process_it_async).
Calling a class method asynchronously, which might initialize more classes or call methods as needed (e.g., SomeClass.process_class_method_async(some_data).)

At that time, I was a typical startup engineer with limited knowledge of code quality, focused primarily on getting things done. Consequently, I wrote a small class that facilitated the migration of many usages to Sidekiq:

class GenericAsyncWorker
  include Sidekiq::Worker

  def perform(klass, method, args)
    if args[:id]
      klass.constantize.find(args[:id]).send(method)
    else
      klass.constantize.send(method, args)
    end
  end
end

This is not the exact code but gives an idea of what I wrote. In short:

If args had an id, I loaded the model object and called method on it.
If not, I called the class method with the given args.

For non-Ruby developers:

include Sidekiq::Worker makes this class a background job processor.
find loads a table row from the database based on ID. It’s part of Rails’ ORM.
send dynamically calls methods of a class.

There was a very kind Senior Engineer who reviewed my code later and said, “this is not good, this is too generic.” I didn’t understand what they meant at the time. My thought process was — it’s working and it helped complete the migration faster, so what’s the problem?

Before you read on, take a moment to think about whether this is good or bad.

The Problems

Even though the code above works, it breaks key principles of good, maintainable code:

According to the S in SOLID principles, a class should have one reason to change. I also apply this to methods — a method should do one thing really well. This method is trying to do two things. You might think the method is small and only doing two things, so what’s the issue? Read on to understand.
The code inside the if statement expects to load the model object and then call method on it. What if someone wants to pass arguments to that method also? Would we add another argument or if statement to handle that?
What if this class is a plain Ruby class (not a Rails model) that needs to initialize the constructor and then call the method? Another flag to handle that?
As you might know, we assign queues to process specific jobs. Queues are chosen based on the urgency of the task. If many tasks start running with this generic code, how do we determine the urgency of this job class?

By just looking at the code for a minute, I quickly spotted these problems. I laugh now, thinking I wrote this back in 2019. But that’s a good sign because if you don’t like your old code, it means your design skills are matured 😇.

The above is an example of Generic Code. Such classes evolve into long methods with a lot of arguments, creating a mess. Over time, multiple developers change the code in these classes to fit new use-cases, often breaking existing ones. There are only two ways to solve such problems:

Don’t create such classes in the first place.
If you created them somehow, gradually move to specific classes and delete the old one later.

I don’t suggest spending time untangling the mess built up over the years by different developers unless you have test cases and very strong reasons to do so. It’s better to create specific classes for each use case and migrate to them slowly.

Let’s talk about testing

Note: Many developers and teams don’t write unit tests. If you’re one of them, I strongly recommend you start. They make future maintenance easier and provide feedback on your design.

If you write tests regularly, you’ll see that this class is hard to test. The reason is simple - it has a lot of generic behavior, making it difficult to identify all the combinations you need to test. From my past experience, if a class is hard to test, it likely needs refactoring.

I still see developers writing generic code for production projects. This isn’t necessarily a problem, as they are on the same learning journey I once experienced. When I encounter this, I take the time to discuss it with them. I emphasize the importance of writing specific, maintainable code. I share my experiences and lessons learned over the years.

Extending the same thought process to “tables”

Developers love writing code that goes beyond their current needs. As we saw earlier, they often create solutions that are more complex than necessary. But this issue isn’t just limited to classes — we also tend to generalize tables and columns to handle more problems than they were meant to solve.

Let’s imagine a developer named “John” created a table called users to handle authentication for their app. At first, this table had only six columns:

                                 Table "public.users"

Column         |            Type             | Nullable  |         Default
---------------+-----------------------------+-----------+----------+-----------------------
 id            | integer                     | not null  | nextval('users_id_seq'::regclass)
 first_name    | character varying           |           |
 last_name     | character varying           |           |
 email         | character varying           | not null  |
 created_at    | timestamp without time zone | not null  |
 updated_at    | timestamp without time zone | not null  |

As time went by, they realized they needed more columns like middle_name, country_code, language, and timezone. So, they added those columns as needed (sounds good, right?). As their product expanded to multiple countries, they needed to track specific attributes for each country. They decided to add a jsonb column to the users table to handle these “country-specific” attributes because some keys were valid in one country but not in another. Adding separate columns for each attribute seemed like overkill. Thinking ahead, they named this field extra_attributes.

A month later, another developer, Lee, decided to add their data to the extra_attributes column since it was a similar use case (but not dependent on country). Within a year, this column had different values for different users, and no one knew all the use cases for this column. This is why I sometimes don’t like NoSQL schemas — they are too flexible unless you have proper safeguards. To avoid this:

John could have named this column something other than extra_attributes to indicate it was for storing country-specific attributes only (better naming).
John could have created different tables with a 1:1 relation to users for such use cases. Anyone could check the country_code column and figure out which table(s) are relevant for a specific country’s user (better design).

Whenever we introduce a bit of generic behavior in our code or database design, it is likely to be misused later. This happens because everyone thinks differently and wants to deliver quickly 🚀.

This is just the first level of misuse. Another common generic mistake is creating tables like tags or attachments. Once these tables exist, everyone adds their own use cases, making them very large. I’ve seen cases where more than 50% of data is in a generic table, just to avoid adding a column to the primary table. While querying the primary table is easy, querying this table requires a large index, which can be even bigger than the table itself. The problems with large generic table are:

You can’t archive old data even if the index and table size keep growing. So you end up paying the cost of keeping all this data in hot storage (eg. SSDs).
You can’t partition the table easily.
Queries become slower over time.
Internal tasks like Vacuuming take longer.
When you upgrade your database, these tables take more time to analyze and start serving data.

And all these problems exist because there are too many use cases being served, and you have no idea how your solution could affect the users. Just like with generic classes, the solution is to eliminate this table and gradually migrate the data to specific columns (or perhaps specific tables).

Conclusion

While generic code and database design can offer flexibility and quick solutions in the short term, they often lead to significant long-term challenges. Over-reliance on generic patterns makes testing difficult, as it’s hard to test all the supported cases. Similarly, abusing generic database structures can result in performance bottlenecks, scalability issues, and maintainability problems. The key takeaway is to strike a balance between flexibility and specificity. While some level of generalization is beneficial for code reuse and rapid development, it is essential to maintain a clear structure, define boundaries, and optimize for performance and future growth to avoid technical debt.

Thank you for reading! Have a good day.

If you liked this post, please read these too:

Multitasking is not that cool.
Are you planning to transition from monolith to microservices? Read this once.
If you’re planning data migration from one system to another, read this.

Blameless culture

Sun, 22 Sep 2024 00:00:00 +0530

Photo by krakenimages on Unsplash

Background

Mistakes are inevitable in any project, especially when it comes to complex software projects. However, the way we respond to those mistakes can make or break the motivation and productivity of our teams. This post discusses how fostering a blameless culture can enhance team productivity.

What is a Blameless Culture?

Have you ever heard of teams where individuals blame each other when a product release is delayed? Or perhaps, unfortunately, worked with such teams? It is quite disheartening to be part of a team where such inefficiencies outweigh productive work. A few reasons why this happens are:

If managers do not invest sufficient time in designing a system that promotes transparency,
Or when most of the team members encourage the blame-game, and no one steps up to highlight that this approach harms team motivation,
Or where it is normal to delay releases or shift responsibilities, rather than maintaining clarity.

The above is exactly opposite of how Blameless culture looks like. In a Blameless culture, we emphasize on learning and improving but without pointing fingers. So when a mistake happens, the focus is on analysing the root cause and fixing it. In such a culture, people are open to communicate and accept their mistakes rather than hiding them or shifting blames.

Role of processes

Let’s say you work in a startup with less than 10 people. There are high chances that you won’t have a process defined for anything. So if you are working in the leadership team and someone comes and asks for a leave, or wants to get their laptop repaired, or needs a comfortable chair — you will need to define a process for them. On the contrary, in a large organisation you will see a process and budget defined for almost everything. While processes are important for any organisation, startups can’t prioritise defining these processes, as their goal is to get the first few paying customers for their product(s). At the same time, they can’t even inherit the processes from a large company because a lot of those will not make sense for a startup.

As of now, I have over 9 years of industry experience, all in startups (including one at a very early stage). I believe that team members should be open to learning new processes to tackle new challenges. They shouldn’t have a fixed mindset that makes them think in just one way. What worked at your previous company might not work at the next one. So, having an open mindset helps you adapt to the environment and come up with new, specific, and innovative solutions.

Processes in startups evolve as the company grows. For instance, they determine which leave plan is effective and what kind of release schedule works best for their software. The evolution of these processes also addresses many cultural issues. I recall in my previous role, we began to develop a culture of shifting blame to individuals and teams when our product manager inquired “Why is the Android release delayed?”

For the frontend team, it was the QA team that approved the API too late.
For the QA team, it was the backend team whose estimates were off by days or weeks.
For the backend team, it was the product team that didn’t consider all scenarios, leading to changes in the code design after they started building the backend..

We saw this as a chance to improve our processes. It took us 3-4 retrospective meetings to identify what needed improvement. Our open mindset helped us eliminate the wrong culture and address the root cause of our problem. I also wrote a detailed blog post about it many years later.

Culture <> Accountability

Blameless culture doesn’t mean avoiding accountability; it means redefining it. Accountability is about ownership of tasks and outcomes. When an issue arises, the goal is to improve the process and systems that allowed the problem to occur, rather than penalizing the individual (or team) involved. This empowers team members to take responsibility without fear.

Another story I remember is from a recent interaction with a junior colleague in my last role. Both of us were in our notice period at that time. I met them face to face for the first time on their last working day 😅. They were moving to a large investment bank as a Software Engineer. Moving from a startup to a big company that deals with money, they were worried about what would happen if their code caused a loss of real money in production. My shortest answer to them was —

If it’s easy to cause loss of money in their systems, it’s a process problem and not an engineer’s fault. There should be sufficient processes and checks in place to make sure this doesn’t happen very easily.

Similarly, every now and then, I see a meme where an intern claims to accidentally destroy the production database. The problem is not the intern, but the policies that gave them write access to production database 😄.

Founding Team’s Impact

The founding team has a profound impact on a company’s culture and processes, Founders establish the core values that reflect their work ethic and vision, setting the tone for how employees interact, make decisions, and solve problems. Whether intentional or not, the founding team’s leadership style influences the company’s norms — whether it encourages open communication, innovation, or strict hierarchy. I recently interacted with an old friend who works as a Senior Business analyst. They recently changed their job and told that the culture is very poor. In their words:

Our managers do not treat their reportees well. If you don’t answer their Slack messages quickly, they’ll ping others in the team asking, “Where is XYZ person and why aren’t they responding?” It doesn’t matter if the person went to the washroom for five minutes. They expect immediate responses to everything.

When I asked about their skip manager’s behavior, they said, “Everyone is the same here. Skip managers shout at managers, and managers shout at us.” 😿 This shows how a bad culture spreads from top to bottom. Another example is a story shared by The Pragmatic Engineer in their blog. It’s about Stripe’s strong writing culture. Because their CEO and CTO write a lot, it encourages everyone to do the same. As a result, all engineers and managers write long internal documents. I wrote about the impact of the founding team on the rest of the team on my LinkedIn a few months ago:

If you’re an old member of a team, your work is beyond your work profile. One major role is to ensure a positive team culture. Culture propagates in two ways:

1. Vertically (CEO -> Dept leads -> managers -> seniors -> freshers). 2. Horizontally (Among peers / among old and new people).

If there is a blame-game culture in a team of 10 members, then chances are that 11th member will also start doing that, or will leave. On the positive side, if there’s a bar of “high quality” code or processes in a team, then the new member will have to match that in order to survive.

Conclusion

If you work in a team with cultural issues, collaborate with your peers and manager to improve it. If you’re in a startup or a growing team, make sure the culture develops positively. Small habits have a big impact when others follow them, whether good or bad.

Thank you for reading! Have a good day.

Unseen risks of Inheritance

Sun, 25 Aug 2024 00:00:00 +0530

Photo by FlyD on Unsplash

Background

I started learning Object-Oriented Programming (OOP) with C++ during my college days, and Inheritance is one of the key concepts you can’t miss when learning OOP. In fact, inheritance is one of the four core principles of OOP, along with Encapsulation, Abstraction, and Polymorphism. In our textbooks, the examples of inheritance were quite simple. The one I remember involved a Shape class, where classes like Circle, Square, and Rectangle inherited from Shape and override the area method. Unfortunately, real-life classes are never that simple 😄.

Problems with Inheritance

I have written a significant amount of code using the Inheritance pattern, which complicated the overall design of my system over time. A common issue was that many classes in the hierarchy contained methods that were not useful to them. These methods were inherited from somewhere in the inheritance chain. This problem often arises when you use this pattern early in the design and fail to refactor the design when things become more complicated. So basically, if you stick with the design and keep adding more behavior to the classes without rethinking the design for new requirements, you’ll end up with the same problem. As a result, your classes might have methods that don’t make much sense for them. A few other problems that come with Inheritance are:

Tight coupling - Inheriting classes from each other inevitably increases coupling. While changes in a subclass do not affect the parent class, the reverse is not true. In a lengthy inheritance chain, any modifications to the base class design can impact all subclasses.
Long chain - Inheritance is beneficial only when the chain is kept short. In a long chain of classes, it becomes challenging to trace the origin of method definitions and understand where they are overridden. Such a design complicates debugging and maintenance.
Rigid design - Because of the coupling introduced by inheritance, the design becomes less flexible and more rigid. When you make modifications to the base class, these changes can unintentionally affect all derived classes. This can lead to unexpected bugs or require changes in multiple places throughout the codebase.

Why people still use it?

We all know that low coupling, ease of debugging and flexibility are the traits of a good design. So why should we use Inheritance if we do not get all of this? The answer to this is — Inheritance is beautiful, only if you know how to do it nicely. For example, the Java’s collection framework hierarchy neatly implements inheritance:

The above design is better because:

Clean Abstraction - It uses abstract classes and interfaces. Your language might not provide the same functionality (e.g., Ruby doesn’t have interfaces), but the general idea of breaking down the functionality should help.
Refactoring - They clearly define the role of each level, and they have been heavily refactoring the hierarchy since the collections framework was released in 1998.
Design Patterns - Beyond just inheritance, the framework also uses some design patterns internally to simplify the design.
Inheritance Chain - Although there is no strict rule on how long or wide your inheritance hierarchy should be, it’s good to keep an eye on it so you don’t end up with 10-20 levels of inheritance.

Beyond these points, the classes in the Java collection framework adhere to the SOLID principles, which are crucial for maintaining a robust and flexible design.

Clean Inheritance

There’s an excellent talk by Sandi Metz from RailsConf 2014 (link) where they explain how they refactored a code with high cyclomatic complexity to a much simpler design using Inheritance. While they are on it, they also explain when inheritance is good. Quoting from the same talk:

Despite of what you have have heard, Inheritance is not evil, and I can tell you exactly when it’s safe to use it. Here’s what you want - You want a shallow, narrow hierarchy. You don’t want it to be deep, and you don’t want it to be wide, alright? If that’s the problem you have, there’s no better solution that inheritance and you’re free to use it. Inheritance is not, evil but sometimes we are.

Although I’ve provided the gist of the talk in context of inheritance, there are a lot of other good things that you can learn from the talk, so you must watch it.

Other patterns

I also wrote a bit about the use of design patterns in the Java collections framework. That’s something I’d like to explain further. As a developer, you should always look for better ways to write cleaner, simpler code. So beyond inheritance, try these things:

Small classes - The entire idea of maintainable software comes from writing small, extensible classes that can work together to complete a task. Make sure that your classes do a single task nicely. If you feel that the new requirements are leading to some bloat in the existing class, plan to refactor it rather than adding behaviour that doesn’t belong to it. I’ll touch on refactoring safely in my next blog.
Design patterns - There’s nothing new that I’m suggesting here. It’s important for all engineers to read about design patterns at least once so that you do not make the same mistakes that these patterns solve. I’ve personally found this website excellent for design patterns (and code smells) but it’s ok if you’ve read GOF or some other book. I’ve personally used Facade, Adapter, Decorator, Proxy, Strategy and a few more design patterns in my projects to simplify the design.
Composition over Inheritance (more here) - A lot of code that we write using Inheritance can be simplified using Composition pattern. By composing objects from simpler, reusable components, you create more flexible and maintainable code. Composition allows you to change or extend behaviours dynamically without altering existing code, unlike inheritance that leads to rigid hierarchies and tight coupling.

Conclusion

While inheritance has been a cornerstone of Object-Oriented Design, it’s important to recognise its limitations and explore alternatives that offer more flexibility and maintainability. As your software evolves, adopting these approaches can lead to more maintainable code.

Gagandeep's Blog

How Cursor boosted my productivity in 2025

Photo by Aerps.com on Unsplash

Introduction

2025 was significant

Improved Productivity with AI

AI @ Branch

My Cursor Usage

Understanding Different models - Cost vs. quality

Configuring how agents work

Planning is still the most important step

Deeper integrations

Understanding legacy code

Running shell commands

For writing tests

MCP servers

Avoiding AI Slop

Is AI eliminating software engineers?

Two edged sword

Personal targets for 2026

Conclusion

Redis Production Insights

Photo by Luke Chesser on Unsplash

Background

What is Redis?

Use cases solved

Learnings and Recommendations

Do I need the multiple Redices?

Using expiring keys

Caching invalidation & application hooks

Tuning the configuration

Redis is single-threaded (wait… what?)

Safely analyzing data with KEYS command

How I saved 75% cost due to single core usage

Deployment options and Cloud providers

Popularity and alternatives

Debugging performance bottlenecks

Closing note

My Financial Journey

Photo by micheile henderson on Unsplash

Background

Three mistakes

From 2019 - 2022

Understanding the basics

Conclusion

References

Growth Through Unlearning

Photo by Tim Mossholder on Unsplash

Introduction

The tale of two RBAC systems

Comfort resists innovation

Some personal examples

Conclusion

Habits, Productivity & Deep work

Photo by Carl Heyerdahl on Unsplash

Background

Perfection & Speed

Do you really need perfection everywhere?

Stress & Context switching

Can we get a lot done with lots of context switching?

Deep work

Do our jobs encourage Deep work?

Some personal productivity hacks

Habits and environment

Boredom is important

Simplicity is clarity

Conclusion

Further reading

Refactoring: Lessons Learnt So Far

Photo by Nina Mercado on Unsplash

Background

Why should you refactor code?

To untangle Code Ownership

To reduce cost of debugging

To reduce techical debt

To improve engineering culture

Myths around refactoring

It’s another name for “Rewriting”

It’s meant for legacy applications

It’s risky, hard and unscoped