Published 15th March, 2024

Developing From Scratch: 2016 vs. now

Development   Scratch   BuildingAndLearning

cover-photo

Photo by Brands&People on Unsplash

Background

One year into the industry, I wasn’t sure if I knew enough about full-stack software development. I was working on a maintenance project, where we were mostly scaling down the system, doing small bug-fixes, doing a lot of debugging and writing a little bit of new code. I visited my college in 2016 where I met some juniors and the placement head. I discussed the idea of creating an Alumni website for our college because no such connection existed between the college and it’s alumni. While I was into full-stack development but I didn’t like doing frontend a lot. So I discussed the idea with my friend Ekas (a year junior to me and just out of college) and we both started working on it. This was completely out of fun, not for profit project. This was our tech stack at that time:

  1. Backend - Ruby on Rails,

  2. Frontend - JQuery/Bootstrap,

  3. Database - MySQL (all user data), Redis (for some caching, probably redundant),

  4. Cloud provider - Linode (Reason: Non fancy platform & straight forward billing),

  5. Issue tracker - Trello,

  6. Version control - Gitlab initially (now Github, source code),

  7. Communication - Gmail/Whatsapp,

  8. Domain - Godaddy,

  9. CDN - Cloudflare (Wasn’t needed, but I needed a free SSL Cert that they provided).

Like many other side projects, the stack was pretty straight forward - We chose what the best we knew. For example, if I knew about LetsEncrypt for SSL certificate, I’d not have used Cloudflare because CDN was not required at that time.

Developing something in 2024

In past 8 years, software landscape has seen major changes. And with recent advancements in AI, we’re able to write code faster. Many low-code and no-code tools are also available if you need to build simple things (even beyond static websites). For example, I had to revamp my portfolio website from this to current two months ago. It took me just a few hours to create the current version because the skeleton and design was generated by ChatGPT. As recent as this week, Devin is released, who is world’s first AI Software Engineer (although a bold claim to make).

Given all the improvements over time, let’s discuss the choices we have if you are (or even I am) developing an MVP or side project today:

  1. Backend - Although my major backend experience has been in Rails and it will be a biased opinion if I say Rails is faster to develop than any other framework. But I’ve seen many people and organisations use Rails to quickly build MVPs. So if I was building something that really needs a complete dynamic and full-stack website of it’s own (with UI), I’d still prefer Rails. If there’s a simpler requirement where I just need to develop APIs on the backend, and frontend is a mobile platform, I might be inclined to use serverless platform to start with. Reason for choosing serverless is simple - the deployment strategies are simpler and I’d pay for the time I’m using the backend, rather than running servers full-time. Ruby on Jets is one such option if you’re a Rails developer. It’s just a wrapper over AWS Lambda, Gateway and databases. You can also use Lambda directly rather than using this fancy wrapper, and write your functions in Java, NodeJS, Python, Go and other platforms supported by Lambda. Another stack is very popular these days that uses Google Cloud services like Cloud functions as backend, Firestore as a NoSQL database. If you’re planning to use AWS, you might want to consider serverless databases like Aurora Serverless (for SQL) and Elasticache Serverless (for Redis/Memcached) or the famous DynamoDB.

  2. Frontend - In the past 4-5 years, I’ve worked quite less on the frontend side. My project’s frontend is in ReactJS and I’ve never written code in it. For some internal portals and for my personal portfolio, I still get to work on HTML, CSS (sass), Bootstrap and Jquery. If I get someone to handle the frontend part, I might want to use React/Next/Vue on the frontend side with a modern CSS framework like Tailwind. But if I’m handling the frontend myself, I’d still use the old stack. It’s not that I don’t want to learn something new, but because I’d prioritise delivery speed over anything for the MVP. This is the single most important thing to consider when building MVP. Unless there’s a significant speed boost in using a new technology, don’t run towards it while validating your idea/product. AFAIK, if I use a modern framework and write code in Typescript, I’d need a transpiler like webpack to convert it into JS code before deploying. And I’d not want to add more complexity in the beginning unless it’s worth.

  3. Deployment / Hosting - If you plan to use cloud functions / lambdas for your backend, you might not need any place to host your server. Your code could be deployed easily by the UI provided by the platform. Incase you’re still need to run your app server on a virtual machine, there are a few options like

    1. Deploying app directly on VM.

    2. Deploying app as a container on VM (behind Nginx proxy maybe)

    3. Deploying app as a container on managed platform (like AWS ECS)

    4. Deploying app as a container on orchestration platform (like AWS EKS)

    If you don’t have experience with Docker or Kubernetes, just use the first option. it’s like deploying app on the server in the same way you’d setup on your local machine (with minimal differences). If you have experience with Docker and Kubernetes, still prefer to choose a service that’s cheap to start with. For example, you don’t need a full-blown EKS cluster to start with as it would cost $70-75 per month just to have the cluster running (while you could get a t4g.xlarge [4CPU / 16GB] machine in that cost). Given my expertise with containers and ease of deployment, I’d prefer 2nd or 3rd option.

  4. Database - I see a lot of new folks preferring NoSQL databases for all kinds of use cases. All new shiny tech looks good unless you fall face operational complexities. That being said, I’d prefer SQL database if following conditions are satisfied:

    1. The requirements enforce some relation between the entities (that can be easily modelled as foreign keys).

    2. The data is not just append only (i.e. it probably needs updation).

    3. There are (probably) transactional requirements.

    Even if 2/3 are true, it makes sense to use an SQL database. If not, you can go for a NoSQL database. There are many choices in both of these, which are already discussed in the first point.

  5. Cloud Provider - If you’re having experience with any cloud provider and understand their billing well, choose them blindly (just kidding). Cloud billing is the most complex thing ever invented by these guys. I’ve seen people struggling on their personal account (with their own credit card) when they’re not able to understand what they’re getting billed for. Even if you sit with your project’s Devops team and ask them to explain the components of this month’s bill, I bet there will be parts they can’t explain completely. Discussing cloud bills is a complete topic on it’s own and the subject is so complex that there are dedicated companies that help you lower your cloud bill (eg. 1, 2). So if you have experience using AWS and you either understand their billing, or at least know how you can terminate everything and remove your credit card, then choose that provider. For your use case (i.e. an MVP), there won’t be huge difference in cloud costs. If you don’t have experience with cloud providers, do one of the following:

    1. Choose any one provider, learn how to setup budget alerts and remove your credit card if something goes out of hands.

    2. Find a dead simple one - I chose Linode because I had used it in my project and they gave only one service (plain unix VM). There was no additional service provided by them (like machine images, container hosting, security groups, load balancers) and hence billing was very simple. The server I ran back in 2016 costed me $10 per month (nothing more, nothing less).

  6. Issue tracker - Use anything that’s free. You could even use google sheets to keep things simple.

  7. Communication - Whatsapp has grown into a very nice communication tool but nothing beats a tool like Slack 😍. I just checked that they provide a free plan with limited features and 3 month message history. If that works for you, spin a new slack instance.

  8. Version control - Github is a popular choice here. I’d happily use it because they now allow private repos for free. So incase you’re building something that could grow into a paid product later, you’d want to keep your code private. I’ve used Gitlab and bitbucket also. Almost all of them support CI/CD setup incase you’re looking to have that for your project.

  9. Domain - You need a domain for your project unless you’re happy hosting as a subdomain of .herokuapp or .netlify . I have a few domains and I find Godaddy cheap for domains. You can compare providers and see whatever works for you. There’s nothing special with domain providers. There are ways to migrate from one provider to another (it takes 1-2 days) if you need to do that in future. Also, you can move DNS management (that routing thing) to some other provider if you want. So in my organisation, we have DNS on AWS Route 53 while domain management is with Godaddy. That’s doable if you need to do that in future.

  10. CDN - I think to start with, I won’t need it unless I’m creating paid videos for my DSA/System Design course 🤣 (jokes apart: because videos are static resources that could be served fast using a CDN). If we need a CDN, we can choose between popular ones like Akamai, Cloudflare or AWS Cloudfront. I don’t have any strong opinions here.

More things? Of course, we did not talk about Load balancers, API Gateways, Firewalls and many other components. In most of the cases, you won’t need them when starting up. The idea was to outline the choices available today vs those available back then. There were lesser cloud providers with fewer services, Docker was fairly new, managed docker services were limited, and Linode was dead simple (and not owned by Akamai). While choosing your stack, just make sure you’re not too much locked into one provider. For example, while Google’s stack sounds simple and cheap to begin with, you’ll be locked with it completely if you build on it. It’s not impossible to migrate but it could be hard or expensive. Another example could be of AWS RDS - If you’re using a Postgres/MySQL database hosted on RDS, that doesn’t completely lock you with AWS as Postgres could be self hosted outside AWS also. But if you use something like DynamoDB, then you can’t move outside AWS unless you find an alternative, change your code and move all your data to the new datastore. If you write your code in a modular fashion and follow good design patterns (like a facade over the classes that interact with DynamoDB), you could easily switch to new database. But if your code is coupled deeply, it could take time to refactor everything and take yourself out from AWS.

Conclusion

As software developers, we often face imposter syndrome and feel like we don’t know nothing beyond our company projects. Building something in parallel could be a way to sharpen your skills and increase confidence. If you have time, you can build something in a new tech stack. If not, feel free to choose your existing stack and build it. Sadly, I got too busy with my work that I did not spare enough time to build anything else like the Alumni website. But thankfully I started writing last year and I enjoy that (I hope you too enjoy reading the blogs 😊).