
In addition to being a Software Engineer, I'm also a [Photographer](https://vinayakakv.substack.com/). I take a lot of photos, and as a result, I ran out of free storage space on Google Photos. Rather than paying for a recurring subscription to Google for additional storage, I decided to self-host my photos. I was inspired by a [friend](https://www.linkedin.com/in/vinyasns/) who was already doing this.

## The Software

There are many self-hosted photo solutions, each with its own niche. Here are a few that I know of:

- [PhotoPrism](https://www.photoprism.app/)
- [Ente](https://ente.io/): Focused on secure storage of photos, with encryption at rest.
- [Immich](https://immich.app/): Offers a Google Photos-like UI and experience, along with an app.

I decided to go with Immich because the setup is straightforward—just pull the `docker-compose.yml`, configure the `.env` file, and run it!

## Takeout and the First Backup

Google Photos provides an option to export all your photo data, but there's a catch—the EXIF metadata for the photos is stored in a separate `.json` file. Thankfully, there are tools available to merge this information back into the EXIF metadata. I used [GooglePhotosTakeoutHelper](https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper) for this task.

Once the photos were prepared, the upload process was straightforward. Immich provides a CLI, which can also be run inside Docker, to batch upload the photos. The CLI handles deduplication automatically. Given the large volume of photos I had, the first upload took most of the day to complete.

## Daily Backup

Most of my photos are taken on my smartphone, and the Immich app can automatically recognize and back them up to the server. However, when the server and the smartphone client are not on the same network, communication becomes challenging.

To address this, the server needs either a static IP or a dynamic DNS setup to be accessible from outside the local network. However, exposing your server to the public internet comes with risks. Without proper firewall configurations, it can become a potential security vulnerability.

Fortunately, [Tailscale](https://tailscale.com/) comes to the rescue in such cases. It creates a VPN where devices can be manually added by installing a daemon. Once on the VPN, the devices can communicate as if they were on the same local network—all **without requiring a central VPN server!** Their [technical architecture](https://tailscale.com/blog/how-tailscale-works) document provides an excellent summary of how this is achieved.

With the server and smartphone on the same Tailscale network, the backup process becomes seamless. Thanks to Tailscale's efficient architecture, communication remains fast even over a mobile 5G network.

## To the Pi

Initially, I hosted the server on my laptop, which I turned on only when backups were needed. However, this quickly became inconvenient when I wanted to access photos remotely—it meant I had to keep the laptop on and awake at all times.

Soon, I decided to upgrade the setup to a [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/) with 8GB of RAM. While it doesn't match the performance of Apple Silicon for image processing, it is sufficient for my needs. I exported the PostgreSQL database from the laptop, which dumps SQL `INSERT` statements, and imported it into the Pi's database. The external hard disk storing the photos required no changes.

> The ease with which I migrated the application from one host to another highlights the beauty of abstraction, where an application, along with its dependencies and runtime, is bundled as a _container_. These containers are the foundation of modern cloud applications, enabling effortless scheduling and scaling.

## Pi Shenanigans

When I purchased the Raspberry Pi, I didn’t have a micro-HDMI to HDMI adapter, which meant I couldn’t see any error messages displayed on the screen. Initially, I attempted to boot the Pi from a USB pendrive instead of a MicroSD card due to past experiences with overheated and damaged memory cards. While I successfully burned the image onto the pendrive, the Pi refused to boot.

After deciding to abandon the pendrive idea, I opted to use a MicroSD card instead. However, I quickly discovered that I didn’t have a MicroSD adapter to connect the card to the SDXC port on my MacBook. Luckily, I found an old adapter lying around—but it was missing the lock switch. This caused the card to mount in read-only mode, leaving me worried that the card itself might be defective.

Curious, I opened the adapter and found that the connections inside were mostly 1:1, with no complex circuitry for the lock mechanism. I learned that [any firm obstruction in place of the missing lock switch could force it into the "unlocked" state](https://www.youtube.com/watch?v=oKrn-0jXZCU). With this simple fix, I successfully burned the image and finally booted up the Pi.

## Storage Architecture

While it's possible to format a plain hard disk and use it to store photos, migrating to another disk can become quite challenging, especially when dealing with a large number of small files. Achieving fault tolerance would typically require hardware-level RAID, which is not cost-effective at a smaller scale. Additionally, extending the storage by adding another disk and exposing it as a single volume to applications is often not straightforward with this approach.

[Logical Volume Manager (LVM)](https://wiki.archlinux.org/title/LVM) offers a solution to the challenges mentioned above. I learned about LVM from [a friend](https://www.linkedin.com/in/gregdaydevops/), and it immediately resonated with me. LVM allows for software-level RAID, the ability to extend volumes by adding additional disks, and even growing or shrinking partitions online (subject to the underlying file system). These features make it an incredibly flexible and powerful tool for managing storage.

For now, I use a simple setup with a single disk, a single volume, and a single BTRFS partition. BTRFS also provides storage-level data compression, which is a bonus. If needed, additional storage can be added easily, and RAID configurations can be set up for fault tolerance in the future.

## Is it Worth It?

While paying Google for storage might seem like an easy option, it comes at the cost of relinquishing ownership of your data. In some cases, Google can [lock you out of your data](https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html). Photo data is highly valuable and can [reveal a lot of personal information](https://theyseeyourphotos.com/), which could even have financial implications.

On the other hand, self-hosting has its own downsides, such as maintenance. You need to treat your system as if it were a production-critical setup, with regular database backups, data replication, and hardware checks. However, the level of maintenance is entirely up to you — you’re free to do more or less depending on your comfort and requirements. The key trade-off is that you’re the owner, and with that ownership comes full responsibility.


How I self host a Photo server with Raspberry Pi

Self-Hosting with Pi


It is the early 2010s, and you have a bunch of Java applications to deploy. There are good servers like [Tomcat](https://tomcat.apache.org/) and [Jetty](https://jetty.org/index.html). You procure a VM in the cloud, set up the OS and servers, deploy the application, and expose it to the public. All is good, until...

## There is More Traffic!!

Congrats! You have grown your customer base, and your application is experiencing traffic that cannot be handled by a single VM. Now, you decide to horizontally scale the system by adding more VMs. You do not want to keep writing the same commands to configure and deploy applications to every machine, so you decide to write a bunch of Ansible and Shell scripts to handle these tasks for you. Thus, `deploylib` is born.

Now, you don't want to be running these scripts from your machine and want to give control back to the developers. So, you also create a Jenkins VM with appropriate permissions, write a `Deploy` job with suitable parameters that `ssh`es into every machine and handles the deployment. Cool, we’ve solved the problem, right? Not quite yet...

## Rolling Deployment

During a deployment, you don’t want all machines to go down together when updating to a new version of the application. You want the process to happen in groups so that you can abort the deployment if you notice anything going wrong in the earlier groups. To achieve this, you logically define the groups, update your `deploylib` to handle it, and break the `Deploy` job into phases to ensure everything runs smoothly.

Whenever there is a new version of the application, you run the `Deploy` job, and the apps go to production _automagically_ with no downtime. A phenomenal achievement! But one day, you wake up to a midnight page that one of the production-critical applications is down...

## Hanged State Machines

The particular version of the app was running smoothly in production for 15 days—until it decided not to. You check the application logs, run `htop`, and observe that there is a memory leak. The engineering team is busy with other business priorities, and fixing the memory leak is not one of them. Memory leaks, after all, require dedicated effort to identify and fix.

A bright engineer comes up with a workaround — just restart the applications every day, but outside business hours! All applications are essentially giant state machines, and a restart is often an easy way to fix them, as they return to their initial state upon restarting. Since deployments already restart the apps, you configure Jenkins to schedule the `Deploy` job daily, using the last deployed version.

Nice! At least now you can sleep without interruptions.

## Growth Brings New Challenges

More growth is always good news since it brings in more revenue. Confident that the system you’ve built can be easily scaled to handle new traffic, you decide to add two more groups. However, as soon as the new groups are added, a database alert — its CPU utilization is way too high.

The architecture you designed is horizontally scalable, yes — but your poor PostgreSQL instance is not. To address this issue, you decide to take the following steps:

1. Vertically scale the PostgreSQL instance for now, or better yet, enable the cloud provider’s autoscaling feature.
2. Move business-critical objects to a horizontally scalable NoSQL database, such as MongoDB.
3. Break down your monolith into microservices, with each microservice owning its own database, ensuring non-critical apps don’t interfere with critical ones.

All great! You implement #1 immediately. Points #2 and #3 take some time, but they are eventually completed.

## Microservice Heaven

You split your monolith app into multiple microservices, carefully defining the responsibilities of each microservice and exposing APIs to facilitate communication between them. To handle asynchronous communication, you deploy a message broker. Each microservice now has its own database, and fortunately, one or two production-critical microservices receive the most database resources.

However, not everything is shiny in the world of microservices. To sync certain business _facts_ across them, you end up with some data duplication. You decide this is a worthwhile tradeoff for the functionality it brings—after all, every engineering decision involves tradeoffs.

To deploy these microservices, you provision a VM for each one and scale it based on its needs. Naturally, your `Deploy` job requires updates—along with managing _groups_, you now need to handle each microservice individually. Taking the time to address this, you parameterize the job to accept the microservice as an input while carefully accounting for the rolling deployment process.

At this stage, `deploylib`—your set of shell scripts for automating deployment — is growing significantly. It now includes the deployment logic, microservice configurations, and more, all centralized in one place.

## Welcome, Kubernetes!

While you have automated most of the deployment setup, it’s still not completely free from human intervention. Services occasionally fail in the middle of the night, despite running the `Deploy` job every 12 hours now. The job has grown so complex that it sometimes catastrophically fails, leaving the system in a half-new, half-old state.

You decide to migrate the applications to Kubernetes (k8s) to take advantage of its automated recovery and autoscaling functionalities. While the microservices you’ve built should theoretically be easy to containerize, several challenges arise due to the way the apps were architected:

- The application runtime doesn’t work well in containers, as it relies on direct access to hardware resources.
- The build step generates numerous artifacts, requiring careful orchestration to produce a container image.
- You need to maintain backward compatibility — apps migrated to k8s must still be able to communicate with apps outside of it, and vice versa.
- The apps have a significant startup time because they load all application-critical data from the database into an in-memory cache layer to meet SLA requirements. The database access itself is slow, as it is highly normalized and requires around 17 joins for even simple queries!

While these challenges are addressable, you still have an affection for your `deploylib` framework and decide to repurpose it for gradually deploying new k8s apps. By leveraging a combination of tools such as `envsubst`, `kubectl apply -f`, `kubectl wait`, `kubectl get`, and others, you integrate k8s deployments into `deploylib`.

Although Kubernetes offers native rolling deployments, your architecture requires a different approach. To accommodate this, you create `Service` sets exposed behind a single `Ingress` and modify your `Deploy` job to handle rolling deployments. This results in Kubernetes performing rolling deployments within your `Deploy` job's rolling deployments — a "double roll"! 🙈

## Was it Really Needed?

Yes, the circumstances of the time demanded it, and you believed a simple set of deploy scripts would make life easier. However, over time, `deploylib` evolved into a [36k+ LOC shell script monster](https://bsky.app/profile/kelseyhightower.com/post/3l7m23hnv242k), with utilities written for almost everything. Even after Kubernetes was introduced, you stuck with the framework due to business requirements for making the migration _fast_. Ironically, the work ultimately slowed down because `deploylib` became a bottleneck that tied you up!

Modern GitOps-based Kubernetes deployment tools like [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) and [FluxCD](https://fluxcd.io/) can greatly simplify the deployment process. Additionally, tools like [external-dns](https://kubernetes-sigs.github.io/external-dns/v0.14.0/) and [external-secrets](https://external-secrets.io/latest/) can make Kubernetes deployments fully self-service, eliminating the need for a dedicated ops person.

After careful technical consideration and review, you decide to adapt these modern tools and finally retire `deploylib`, hoping that this new system doesn’t turn into the next `deploylib`.

## Is Shell Really Good for Large Projects?

While shell scripts can simplify and automate a few workflows, any large collection of shell scripts becomes difficult to debug due to several factors, including:

- **Variable scoping:** A variable can exist before the script runs and might persist after the script ends. While `local` can help, there still needs to be a robust way to communicate between different scripts.
- **Many ways to do the same thing:** For example, should we use `[ ]` (a.k.a. `test`) or `[[ ]]` (a shell built-in) for a condition? To check if a variable is empty, is `-z` preferable, or should we use `$var == ""`?
- **Syntax hacks:** Constructs like `${var/find/replace}`, `${var%substr}`, `${var:-default}`, and `${var##substr}` can be confusing and make life hard for readers unfamiliar with these conventions.

In such cases, a dedicated programming language with optional static typing can be a better choice, as long as the program has a consistent CLI. After all, [`kubectl`](https://github.com/kubernetes/kubectl) is a `Go` project, not a collection of shell scripts! [Kamal](https://github.com/basecamp/kamal) also deserves an honorable mention in the deployment world as a project of this kind.

Some tools, like [zx](https://github.com/google/zx), bring shell primitives to higher-level programming languages, offering the best of both worlds—for example, you can use features like `Promise.race()` to manage processes efficiently!

> This article is inspired by [Dear Sir, You Have Built a Compiler](https://rachit.pl/post/you-have-built-a-compiler/)


A tale of how a simple deployment script turned into a monster that could rival Kubernetes.

Dear Sir, You Have Built Kubernetes

Operators are a fundamental part of a programming language. They allow expressing expressions, mathematical or otherwise, elegantly and reduce the cognitive burden on the reader. Some programming languages take the concept of operators one step further and allow them to have different meanings depending on the context in which they are used, making them even more powerful. This approach is popularly known as "**Operator Overloading**." As with many things in software engineering, any flexibility comes with its own set of trade-offs. In this article, we will explore a few cases where operator overloading can be helpful or problematic.

## The Ways of Operator Overloading

Every language has its own way of defining operator overloads. I was introduced to the concept in C++ while following the book by [Daniel Liang](https://www.amazon.in/C-Programming-Language-Y-Daniel-Liang/s?rh=n%3A14447041031%2Cp_27%3AY.+Daniel+Liang). In C++, operators are special functions whose names begin with the keyword `operator`. For example, an addition operation for complex numbers can look like this:

```cpp
// test.cpp
// compile with `clang++ -std=c++20 test.cpp`
#include <iostream>

#include <format>

using namespace std;

class Complex {
 private: float real, imaginary;

 public: Complex(float real, float imaginary) {
 this -> real = real;
 this -> imaginary = imaginary;
 }

 Complex operator + (Complex & another) {
 return Complex(this -> real + another.real, this -> imaginary + another.imaginary);
 }

 string toString() {
 return format("{} + {} i", this -> real, this -> imaginary);
 }
};

int main() {
 Complex c1(2, 3), c2(1, 0);
 Complex c3 = c1 + c2; // This works
 cout << c3.toString() << endl; // 3 + 3 i
}
```

Apart from the `operator+` function, we also see the `<<` operator used for stream output, which can also be used for bit shifts. The arrow operator `->` is used for pointer member access (i.e., `a->b` and `(*a).b` are equivalent) since `this` is a pointer. C++ allows all of these operators to be overloaded as well, but it does not allow changing their precedence.

Other languages like Python and Kotlin require the implementation of special methods as part of the class in order for them to be exposed as operators. For example, we can achieve the above result in [Kotlin](https://pl.kotl.in/XkrlBsH0F):

```kotlin
data class Complex(val real: Double, val imaginary: Double) {
 operator fun plus(other: Complex): Complex {
 return Complex(this.real + other.real, this.imaginary + other.imaginary)
 }
}

fun main() {
 val c1 = Complex(2.0,3.0)
 val c2 = Complex(1.0,0.0)
 val c3 = c1 + c2
 println(c3)
}
```

As we can see from the above examples, operator overloading makes the interface (API, if you will) of the class more _natural_, and the expressions more readable, making them feel closer to the way we write mathematical expressions. When the mathematical concept of arithmetic makes sense, for example, [Group theory](<https://en.wikipedia.org/wiki/Group_(mathematics)>), operator overloading can be particularly useful.

## Beyond Arithmatic

The set of operators offered by some programming languages is not limited to arithmetic operators. It includes bitwise operators and beyond. C++ is famous for using shift operators with streams.

```cpp
std:cout << "Hello world!" << std::endl;
```

While this approach looks okay to begin with, string formatting becomes increasingly complex.
```cpp
std:cout << "Hello world! I am a number" << num << std::endl;
```

Compare that with plain old `printf`:

```c
printf("Hello world! I'm a number %d\n", 10)
```

The `format` function used in the first example also better.

The Spray router in Scala (deprecated) takes operator overloading to another extreme and uses a bash of operators to [define HTTP server routes](https://spray.readthedocs.io/en/latest/documentation/spray-routing/):

```scala
trait LongerService extends HttpService with MyApp {
 val route = {
 path("orders") {
 authenticate(BasicAuth(realm = "admin area")) { user =>
 get {
 cache(simpleCache) {
 encodeResponse(Deflate) {
 complete {
 // ...
 }
 }
 }
 } ~
 } ~
 pathPrefix("order" / IntNumber) { orderId =>
 pathEnd {
 (put | parameter('method ! "put")) {
 // ...
 }
 }
 }
 }
 }
}
```

Actor-based programming languages like Erlang support a dedicated operator `!` to send a message to an actor:

```erlang
%send message Message to the process with pid Pid
Pid ! Message
```

Haskell uses a dedicated operator `>>=` for the Monad `flatMap` operation. For [example](https://play.haskell.org/saved/lvSs9d56):

```haskell
duplicate:: a -> [a]
duplicate input = [input, input]

main :: IO ()
main = do
 let result = [1,2] >>= duplicate
 print result
 -- [1,1,2,2]
```

Monads are an interesting concept and are subject to lots of [memes](https://antousias.com/content/images/2020/11/44b0bd758f8ee5c81362923f0d5c8e017c9ddf623925e60c29a4c015b89fbb45.jpg). I hope to explore them from a mathematical perspective one day.

In all these examples, operators are used to express domain-specific concepts, which might take some time to get used to. In the Spray router example, it is likely to scare a first-time viewer!

## No Operator Overloading

Many languages do not support operator overloading, including Java and JavaScript. In these languages, libraries expose methods instead of operators. For example, let's consider calculating the likelihood of sample data under a normal distribution $\mathcal{N}(\mu, \sigma)$. Mathematically,

$$
p = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}
$$

A python program written with `numpy` can look like:

```python
import numpy as np

mu = 1
sigma = 0.5
x = np.array([0, -2, 1, 2.5, 3])

p = 1 / (np.sqrt(2 * np.pi * sigma ** 2)) * np.exp(-(x - mu) ** 2 / (2 * sigma ** 2))

print(p)
# [0.10798193, 0.00000001, 0.79788456, 0.0088637 , 0.00026766]
```

If we were to write the same expression in NodeJS with Tensorflow.JS (there are not much tensor libraries for JS, a topic for another day):

```ts
import tf from '@tensorflow/tfjs-node'

const mu = 1
const sigma = 0.5

const x = tf.tensor1d([0, -2, 1, 2.5, 3])
const p = x
 .sub(mu)
 .pow(2)
 .mul(-1)
 .div(2 * sigma ** 2)
 .exp()
 .div(tf.sqrt(2 * Math.PI * sigma ** 2))

console.log(p.toString())
// [0.1079819, 0, 0.7978846, 0.0088637, 0.0002677]
```

### Which one of these is better?

The answer is subjective: the Python version follows the mathematical structure and preserves its pattern, while the JavaScript version adds verbosity but makes the operation linear.

In fact, there is more than one way to convert a mathematical expression to a function-chaining format. This way, the programmer can explicitly choose the order of operations, as opposed to relying on precedence and grouping to decide for us. The chain explicitly spells out the computation graph.

### So, is operator overloading good or bad?

As always, it depends. Operator overloading is one of the powerful tools that make a programmer’s life easier. However, it has the potential to be misused, creating esoteric and often unergonomic patterns. When used in the right places, it’s more beneficial than harmful.

On the other hand, the unavailability of operator overloading forces library developers to expose chainable interfaces, which naturally lead into computation graphs—though at the cost of readability.

When an operator can have multiple meanings based on the context...

Operators Overloaded


Search has become an integral part of our daily lives. With major players like Google and Bing, as well as mission-oriented options like [Ecosia](https://www.ecosia.org/) and [DuckDuckGo](https://duckduckgo.com/), a variety of search engines have tackled the challenge of navigating the vast expanse of the internet.

However, these search engines cannot solve the search problem on a smaller, customizable dataset, such as an e-commerce catalog. In such cases, we have access to product and user data. This data can be used to derive personalized search results. In this article, we shall explore the technology choices we can make when building a multi-modal, conversational search experience.

## Conversational Search

Traditionally, search is thought of as typing something into a text box and expecting results. Several applications have [invented their own query languages](https://rachit.pl/post/you-have-built-a-compiler/) to enhance the capabilities of what cannot be achieved with a full-text search. The rise of LLMs has promoted search experiences like [Perplexity.AI](https://perplexity.ai) (recently, OpenAI also released their own version), which can tolerate imprecise inputs and provide context that encourages natural exploration.

The APIs of these LLMs are now capable of [Function Calling](https://platform.openai.com/docs/guides/function-calling), which allows us to expose custom business APIs to the LLMs and provide a complete in-chat experience.

## Multimodal Search

Words are never sufficient to explain something, which is where adding an image input into the mix can vastly enhance UX and reduce frustration. We can take it a step further by using [live audio and video capabilities](https://openai.com/index/hello-gpt-4o/) to remove text input completely from the equation. For the sake of this discussion, we are limiting ourselves to image inputs only.

## Technology Choices

Based on our requirements above, we have a few technology choices to make:

- Which LLM and LLM provider to use to power the conversational interface?
- How do we make our data accessible for multimodal search?
- What backend and UI frameworks to use?

### LLM Provider and APIs

We have a lot of choices for LLM providers in today's market:

- **Giants like [OpenAI](https://openai.com/api/pricing/), [Anthropic](https://www.anthropic.com/pricing), and [Gemini](https://ai.google.dev/pricing)**: These are the big players in the market and offer a variety of models to choose from. Since they operate at a large scale, their costs tend to be lower for low-volume applications. Gemini recently started [providing OpenAI-compatible APIs](https://ai.google.dev/gemini-api/docs/openai), making it easy to switch between providers.
- **Self-hosted models like LLaMa and Qwen**: Many open-weight (and some open-source) models are available on Hugging Face. You can evaluate, fine-tune, and self-host them. For self-hosting, since we need GPU VMs, the added cost and maintenance factors only make sense if you have a high load.
- **Open models on third-party hosting**: There are many third-party LLM hosting solutions, including Hugging Face, which offer competitive pricing and API compatibility options. They are a perfect tradeoff between giants and self-hosting.

> **A note on interoperability**: While OpenAI-compatible APIs and libraries like Vercel's AI SDK can make it seem like switching between LLMs is easy, it often isn't. I've seen GPT-3.5 and GPT-4 behave vastly differently on the same prompt. This might be due to the way the models are trained and fine-tuned. But this is definitely something to watch out for.

We can use the following generic criteria to choose the LLM and provider best suited for our task at hand:

- **Streaming vs Non-Streaming APIs**: LLMs are essentially next-word prediction models. Some APIs generate an entire paragraph as a response, while others return a streaming output. For an interactive experience, streaming output tends to provide better UX, as users do not feel stuck.
- **Context Window Size**: An LLM's ability to reference past text is limited by the length of its context window. If an LLM has a small context window, it might forget what the user said at the beginning, leading to frustration. We also need to consider dropping unimportant messages from the context window once it is full.
- **Speed**: LLM token-generation speed can vary based on its size and the infrastructure it is hosted on. The faster the LLM, the better the experience. However, overly fast LLMs can cause a flickering UI that may artificially need to be slowed down. Since our custom function calls also add delay, a fast model suits our purpose.
- **Modality**: Some LLMs are text-based, while others are multi-modal. While our image search use case works better with multi-modal LLMs (if they can understand the image and generate text input with an accurate description), text-based models can also be used—with image search being a separate pipeline.
- **Visibility**: It is important to understand how LLMs behave when they are running in the real world. While one can build a monitoring solution to review random chats, out-of-the-box solutions like [OpenAI Threads](https://platform.openai.com/docs/api-reference/threads) can be very helpful.

### Data Preparation and Storage

We need to make our product data accessible for both natural language and image search. For this task, we can leverage multimodal embeddings such as [CLIP](https://huggingface.co/docs/transformers/en/model_doc/clip) to generate text and image embeddings, which can then be used during the search.

There are many vector databases like [Milvus](https://milvus.io/) and [Qdrant](https://qdrant.tech/) that tackle the problem of returning the nearest vector given an input vector. Since we also want our search to include conventional filters (such as product category, age group, etc.), a plain vector database will not suit our use case. We are addressing the Hybrid Search problem, where both vector search and conventional search are used together. Fortunately, many conventional databases now offer vector search extensions:

- **Postgres with [pgvector](https://github.com/pgvector/pgvector)**: If you already have your main business data in Postgres, pgvector is an easy choice.
- **Purpose-built Hybrid search databases like [MyScale](https://myscale.com/)**: MyScale is a Clickhouse fork that adds vector indexing and vector search functionality. They also support native functions for full-text search.
- Almost every database now ships with vector functions and vector indexing.

We can evaluate the database choices using a few key points:

- **Online vs Offline Indexing**: Can our vector index support real-time insertions and updates, or do we need to run reindex operations periodically? While running a massive indexing operation may seem easy at first, it can quickly become unmanageable as data volume increases. Online indexing is always better if you have constantly changing or growing data.
- **Vector Functions**: We are focused on _distance_ functions that return how similar two input vectors are. These functions can either be used in the `ORDER BY` clause to get the nearest matches, or in the `WHERE` clause to filter the dataset into records of interest. What distance functions a database supports and how it limits their usage is an important factor. For example, a database that supports distance functions only in the `ORDER BY` clause is limited in that it can only strictly select a subset of the data where the distance to the source vector is greater than `0.8`.
- **Speed**: Since our search is conversational and the database queries are invoked by the LLM (which has its own delays), we need to ensure that data retrieval is fast.
- **In-database Embedding Functions**: Some databases offer functions that allow the caller to specify an embedding model and the input in order to generate the embeddings. While this eliminates the need for the application to generate query embeddings, it can potentially lead to inaccuracies if the embedding model for query embedding and data embedding are different.
- **Ease of Data Migration**: If the primary business database and the search database are different, we need to consider whether the original database supports Change Data Capture (CDC) events, which can be used to detect changes and insert them into the search database with embeddings computed. Otherwise, a periodic data sync job might be a good option.

### Backend and UI frameworks

Python is the go-to language for all ML-related activities due to the abundance of ML libraries and plug-and-play models, thanks to [Hugging Face Pipelines](https://huggingface.co/docs/transformers/en/main_classes/pipelines). [LangChain](https://www.langchain.com/) also provides primitives and extensions to wrap around LLMs, but it comes with a [reputation for being unnecessarily complex](https://minimaxir.com/2023/07/langchain-problem/).

On the other side, JavaScript is catching up with the state of things with ML libraries like [TensorFlow.js](https://www.tensorflow.org/js) and [Xenova's port of Hugging Face pipeline](https://www.npmjs.com/package/@xenova/transformers). Vercel's AI SDK ([`npm i ai`](https://www.npmjs.com/package/ai)) is also a zero-cost abstraction over multiple LLM providers, offering both backend streaming patterns and frontend hooks for popular frameworks like React, Svelte, and Vue.

At the end of the day, technology choices should be made to provide the best user experience, with frictionless developer experience coming second.

## The Hybrid Search

Now that you’ve made technology choices, prepared the data, stored it, and are ready to take user input, let's say the user is querying for "Shirts to wear for a Hawaii vacation." The LLM, detecting the user's intent, responds with a message indicating that the user should wait while it searches for those shirts, and invokes our `find` function with the following parameters:

- `category=shirt`
- `tags=['vacation']`
- `query='casual, breathable shirt for vacation in Hawaii'`

Now, based on the context in the request (possibly from a cookie), we know the user ID, which allows us to infer:

- `gender=Male`
- `pastPurchases=[...]`

We can generate a text embedding and query the product database as follows:

```sql
select title, price, image_url, distance(embedding, queryEmbedding) as distance
from products
where gender = 'male'
  and category = 'shirts'
  and tags in ('vacation')
order by distance desc
limit 100
```

Let’s also assume that we have two additional scoring algorithms:

- `imageSimilarity(input_image, target)` that returns how similar the user input image is to a given image.
- `likelihood(pastPurchases, product)` that returns how likely a given product is to be purchased based on past transactions.

Running these functions on our DB results, we end up with three scores for each of the filtered products. We need to combine these scores and re-rank the results to show the most appropriate results for the user. Although these scores are numerical and might fall within the same range, they cannot be combined arithmetically since they come from different sources and have different meanings. For example, a likelihood score of `0.5` is vastly different from an embedding distance of `0.5`. Even among different embedding algorithms, the same score can represent different levels of similarity.

### Rank Fusion

In order to solve this problem, we need to convert these scores to have the same meaning. Converting _scores_ to _ranks_ is a common method employed in these scenarios. We order our dataset by these scores so that the nearest entry appears first, and then take the position of the entry as its rank. In our example, we will create three sets of products ordered by embedding distance, image similarity, and likelihood, respectively.

For every product in the set, we can now assign a score as follows:

$$
\mathrm{score} = \sum_{i \in \mathrm{Rankset}} \frac{w_i}{b_i + \mathrm{rank}}
$$

This expression gives more weight to the highest ranks, while giving rapidly decreasing weights to later ranks. The constant $b_i$ dampens the effect of early ranks contributing more weight, whereas the constant $w_i$ acts as the weight for the rank set. These constants have to be set according to the business requirements. This technique is called [**Reciprocal Rank Fusion**](https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking) and is one of the basic techniques to combine ranks. There are other more sophisticated methods that can take user feedback into consideration as well.

After the rank fusion, we get a score for each product that we can use to sort the product set and present it to the user. This product set is obtained based on both regular and multimodal search inputs, along with the personalization factor added in, thus enhancing the UX.

## Feedback loop and Maintenance

No system will be perfect on day one. We need to keep an eye on our conversational search experience and measure its impact, for example, using conversion metrics. We can also gather explicit user feedback by asking them to vote on LLM responses and recommendations. Additionally, we should monitor for possible abuse of the conversational experience and detect attacks or jailbreaks by randomly auditing PII-removed user interactions.


Techology choices that we can make while building a conversational, multimodal search experience

Building a Search Engine

## A Good Morning

One fine day, we woke up to a page at 7AM that one of our non-critical services are returning more 500s than the usual. In fact, 500s constituted more than 50% of the responses, and that's why we were paged. This service ran smoothly for more than a month now, and we had just enabled message publishing in that service last evening. Suspecting this would be the cause, we quickly disabled the functionality by turning a feature flag off and the 500s gradually came down until the alert automatically turned itself off.

Once we were completely awake and caffeinated, we started looking into the logs of the service and a few things stood out:

1. The entries with `level=ERROR` had a spike in the duration of increased 500s. The errors were 502 actually, since the service was quite unavailable during that time.
2. Those log entries were specifically coming from the MQ client - we use self-hosted RabbitMQ
3. The logs were repeatedly saying
 - MQ sending us `ECONNRESET`
 - Our connection retry mechanism kicking in, and becoming successful sometimes, failing other times
 - A seemingly crpyic error message - `connection closed: FRAME_ERROR - type 65, all octets = <<>>: {frame_too_large,1342177289,4088} (501)`

Our service is written in NodeJS + TS, and we use [`@cloudamqp/amqp-client`](https://www.npmjs.com/package/@cloudamqp/amqp-client) to communicate with MQ. We subscribe to [`AMQPChannel`](https://cloudamqp.github.io/amqp-client.js/classes/AMQPChannel)'s [`onError`](https://cloudamqp.github.io/amqp-client.js/classes/AMQPChannel.html#onerror) method to kick in our connection retry mechanism. Our retry mechanism is quite raw and simple and does not take of messages that would be sent during the retry window - they would be lost. The retry mechanism keeps track of a counter, and backs off exponentially. Once the retry is successful, the counter is reset and everything is back normal.

While the retry mechanism seemed to work fine in this case (***spoiler**: it wasn't*), which we were happy about, the number `1342177289` worried us. We have configured the maximum message size as `4088` bytes, which is more than sufficient in almost all the cases. What we were seeing instead was a message of around 1GB, which would be close to impossible! I joked that I'm 99.99% confident that 1GB message would not be the case, but left the remaining probability for me to be amused.

We scoured the logs of the service and its historical data and nothing could come to as close as a GB, or even a few 100KBs. Turning back to logs, We noticed that same number `1342177289` was repeated multiple times. It is highly improbable that we get multiple 1GB ish messages with the same byte count - we never retry publishes as well.

Having gone through articles of magic numbers appearing during crashes, I became particularly interested in the number `1342177289` - would it be the case it was a legit packet size and _somehow_ one or more bits got corrupted to become this humongous big number? I started looking into the bits then:

```
0101 0000 0000 0000 0000 0000 0000 1001
```

Weird - there are more zeros than ones - yes. But 2 bits at Most Significant side should not haven been set accidentally. Meanwhile, we have pasted the number into Google and [it only brought up results **ONLY** related to RabbitMQ!](https://www.google.com/search?q=1342177289)

## What had happened?

[John Eckersberg makes a good job at explaning what this number means](https://john.eckersberg.com/debugging-rabbitmq-frame_too_large-error.html). To make sense of this number, we need to include `type 65` into the picture as well -- `65` is ASCII `A`. Then there is channel number `19793`, which was not included in this log. These three numbers combined together makes out the string `AMQP` followed by the numbers `009`. For reference, the AMQP init packet expects `AMQP0091` in the header (`0091` being the version and represented as a raw number). We can see parts of it in our number:

```bash
1342177289
= 0101 0000 0000 0000 0000 0000 0000 1001
= 0x50 0x00 0x00 0x09
= "P" 0 0 9
```

This means that we were sending initialization packet after the connection was initialized. For reference, here is how our retry looked like:

```ts
class MqClient {
 private readonly client: AMQPClient
 // We keep a single connection
 private readonly connection: AMQPConnection | null = null
 private retryCount = 0

 public MqClient(url: string) {
 this.client = new AMQPClient(url)
 }

 // Users SHOULD call this
 public async init() {
 this.connection = await client.connection()
 this.connection.onEror(this.retry)
 this.retryCount = 0
 }

 private async retry() {
 this.retryCount += 1
 // Exponential backoff
 await sleep(2 ** this.retryCount * 1000 /*ms*/)
 await this.init()
 }
}
```

While this code looks good at the surface, **if an error happens during retry, since the retry handler is already registered, it is called again - now multiple retry sequence are running concurrently**. And it is entirely possible that a retry happens on already initialized connection - and that's what has happened.

## The Fix

The fix is simple, clear connection and associated handlers before attempting a retry. This is akin how React's `useEffect` wants us to [clean up all the subscriptions, callbacks etc when the component is unmounting](https://react.dev/learn/synchronizing-with-effects#how-to-handle-the-effect-firing-twice-in-development).

However, that's not the end of the story. We observed that MQ connections were being closed when the traffic to our service was mere 20% more than usual. We were running 3 pods in Kubernetes and observed our pods were running over limit on CPU utilization. Scaling them to 16 did bring things back to normal at 100% CPU utilization. We need to further investigate why an act of publishing the message uses up the resources so much. Until then, we plan to use [`HorizontalPodScalar`](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) on CPU utilization to scale up the service on load.

The day when we received a 1GB message on our RabbitMQ (did we?) and how we handled it

1342177289


We have a microservice at our workplace that handles Data Ingestion. That
microservice also comes with a UI to configure and tune the Ingestion process.
Eventually, we added a few features to the same UI such as a dashboard showing
derived metrics from the data. We host this microservice in Kubernetes and
expose it to the users through a GKE Ingress.

## The Problem

Everything was working fine until we saw all pods under high load during bursts
of incoming data. Our main data receiver services put the messages into
RabbitMQ, which our microservice in question consumes. Our microservice is
written in NodeJS and we do not have the luxury of multi-threading (we have
Worker Threads, but we wanted to keep things simple). Thus, we saturate the CPU
during the loads to keep things optimized, resulting in near 100% utilization. 

There is a solution for this problem in k8s - Horizontal Pod Autoscalar (HPA),
which watches a deployment and scales it up and down to keep a target metric
constant. We configured our HPA for our microservice with the target of keeping
target CPU usage at 50% (which never happens on the load, it will always be
saturated), and thought the problem was solved. Alas, the problem started
there!

We soon started hearing our platform engineers getting paged and woken up at
midnight as our service was throwing 500s to the users. It was quite mysterious
to begin with as the UI and the APIs serving it were working fine. For the
first few days, the solution to clear the page was to wake up, visit the UI in
the browser, and, hit refresh 5-6 times! This problem had to be solved.

## Digging Deep 

Since we knew that the 500s problem started after we configured the HPA, we
knew it had to do something with it. We began digging it up in a lower
environment.

The first thing that we had to do was delete the HPA and take the reins of
scaling into our control. Fortunately, k9s provide an easy UI to scale a
deployment up and down. We started with scaling our pod to 30, which took some
time to get created. The scale triggered a new node allocation as well, as we
run our lower environments as minimally as possible. When the deployment was
scaling, we kept repeatedly hitting the UI in the browser and saw nothing
suspicious. After this, we scaled down the deployment and saw nothing special
as well. We thought hitting the UI manually was not enough and wrote a cURL
script to do the task for us with a sleep of 0.1s.

Then we realized that our lower environment does not utilize an ingress, rather
than a nginx proxy to keep things simple. We pin the traffic to go to one pod
in the proxy. This is not an ideal production setup but sufficed for lower
environments as we mostly run only one replica there.

Thus, we decided to create an Ingress in the lower environment to make it
similar to the production one. Then, we repeated the exercise of scale-ups and
scale-downs. We started seeing 504s and 503s in the script output both when
scaling up and scaling down. Finally, we were able to reproduce the issue.

## Health Checks

Google has good documentation on why 5xx errors happen during scale downs; it
is due to the delay in the sync between the ingress and Kubernetes cluster.
However, it does not explain the errors during scale-ups. Then, we had to
carefully inspect our deployment yamls and found that we have defined the
`tcpSocket` health check for our service.

While `tcpSocket` is a good check for the service which becomes ready to accept
connections as soon as it can start up, it was not ideal for our service, as it
has to do some initial service discoveries. We use Consul to do service
discovery and run it in the pod along with the service container. And, it is
sometimes slower to start than our service itself. This makes our application
crash if the Consul is too late to start. Eventually, our service becomes
healthy after several crashes when the Consul is fully up.

This meant that we had to implement an API endpoint returning the status of
consul, and use that in the `readinessProbe` through `httpGet`. We used the
same API for `livenessProbe` as well, since we did not have other requirements
for it. If our NodeJS process is struck (maybe doing a long-running CPU-bound
task), it will not respond to the liveness probe, and eventually be killed by
the k8s. We tested new health checks and verified that they were working well.
It was the time for a new rapid scale-up and scale-down test.

And, to our surprise, we did not see any changes, the errors continued coming
up despite setting the same health check on Ingress `BackendConfig` as well!

## A deviation

This time, we wanted to do the test from inside the cluster using cluster local
DNS. We jumped into one of our containers (again, k9s makes it very easy) and
ran the script. This time, no errors while scaling up and down!

This could mean one thing - there is something fishy going on with GKE's
Ingress. We have to report the issue with an MWE yet, but we wanted a solution
for this problem immediately as the users were complaining about the issues a
lot.

## The Solution

We knew that this service supported two sets of use cases - processing the data
and serving the front end and APIs. The scaling was caused by data processing
while the frontend traffic is predictable and steady. Thus, we can have two
sets of pods

1. To serve frontend traffic - with a constant number of replicas and behind a
service and exposed through Ingress
2. To serve data processing needs - with a HPA, not exposed to the public

This was an easy solution for us as all our data processing inputs came through
MQ, and we had a `listenToMqMessages` function to set up the listeners. If we
ever had an API to handle data ingestion, we would simply write a message to MQ
in the API. 

In Kubernetes, we can't have a single deployment serving both needs. So, we
created a `service-noweb` deployment that sets an environment variable
`NO_WEB=true`. Our application checks for the value of this variable and turns
on listening to MQ messages. The `NO_WEB=false` deployments can still write to
MQ, but can't listen to any messages.

Now, we have two deployments of the same application hosted for different
needs. Even in the case of heavy data processing loads, the UI and associated
APIs are not impacted as the data processing happens independently.

## Thoughts

This solution is simple and feels hacky. Shouldn't the scaling have happened
smoothly and have protected us from doing all these? - Ideally yes. But the
world is not as simple as that. Google recommends setting a hook before pod
termination just to buy them time to unregister the pod from the Ingress. I
feel that is a hacky thing to do - as it is not obvious from just seeing the
configuration why it is there.

This exercise resulted in copy-pasting the entire deployment spec, just to
change the names and add a new environment variable - this is not DRY. We are
considering moving this to FluxCD with kustomization templates to solve this.

However, there might be a real case where we want to scale our web frontends up
one day in the future. This solution does not take care of that at all assuming
the traffic is constant and a pre-determined number of replicas are sufficient
to handle them. When that would be the case, I think GKE Ingress would have a
fix in place.

We had another microservice facing a similar issue. But it was hard to split it
into critical and non-critical paths as we did in the case of this
microservice. We are exploring running haproxy pods behind ingress (they
reconcile faster than the Ingress controller), which will then route the
traffic to this microservice - This is more hackier than splitting the
microservice. We would not have needed all of this if the GKE platform
implementation of the Ingress controller was perfect.

## Moving beyond

The exercise of splitting the microservice into two deployments made me think
about Modular Monoliths for a bit. We already host all our NodeJS applications
under a monorepo and greatly benefit from code sharing across projects without
having to set up package publishing infrastructure. Instead of monorepo, what
if we had a monolith with some rules about code organization, that generates
several microservice bundles? I will be processing this thought for some time
and will come up with an article soon .... Stay tuned!



We had a microservice hosted in Google Kubernetes Engine that was giving us 5xx errors during scaling. We explore the reasons and solution in this post.

Splitting a Microservice

Cache is one of the important building blocks in any application. When we
started to build projects in NodeJS, we had to come up with our own
implementation of the cache that abstracted the underlying data store. We had
the following goals in mind when starting to build a cache:

- **Ergonomic:** The cache should be easy to use, with minimal changes to
existing code base
- **Performant:** The cache operations should not slow down the system
- **Easy to Invalidate:** The cache should support easy partial/full
invalidations that can be triggered by the application
- **Type Safe:** The cache API along with the invalidation API should be type
safe in TypeScript.

We had built an in-memory cache earlier as it was easy to implement (you know?
software developers are lazy...) But we soon ran into inconsistency problems
when we scaled our pods to more than one. It was the time to build a truly
distributed cache.

## The API design

To make the cache ergonomic, the API should be really simple, seamless and
type-safe. While languages such as Java and Python provide `@` decorators that
can be used to wrap functions and classes, JS supports such decorators only for
classes and class methods. We do not use classes mostly, resorting to functions
instead. So, decorators were not an option for us.

The natural option that remains after that is the Higher Order Functions. We
defined a `cached` function, that takes another function as the input and adds
caching capabilities to it. So, if we have a function named `getStudentInfo`,
adding a cache to is as simple as `const cachedGetStudentInfo =
cached(getStudentInfo)`.

We want the `cached(f)` to be type-safe -- we only support passing functions as
arguments. We further restricted ourselves to support pass in only the `async`
functions that makes external network calls. So, we came up with the following
type for `f`:

```typescript
type Cacheable = (...args: any[]) => Promise<any>
```

Now, `cached` function does not modify the function signature in any way, the
caller still continues to call it like the original function. So, its signature
looks like:

```typescript
const cached = <T extends Cacheable>(f: T) => T
```

## Implementation

We outsource the responsibility of storage, distribution and fault tolerance to
Redis. We store the cached values in the plain key value space of Redis. We
derive the key by concatenating unique ID of the function with the JSON
representation of the arguments. We store the resulting value as a JSON as
well. 

> This adds a serious limitation to the functions that we can cache, for their
arguments to be serializable and deserializable. Since our function arguments
are plain values and objects at the most, this is not a problem for us. 
>
> Whenever the situation demanded us to cache a non-serializable value, we
ended up using in-memory cache without any distribution.

The key implementation detail comes in how we want to define the cache key. For
example, we can do

```typescript
const getCacheKey = (...args: any[]) => JSON.stringify(args)
```

or, 

```typescript
const getCacheKey = (...args: any[]) => args.map(arg =>
JSON.stringify(arg)).join("/")
```

While both ways can derive a valid string key, the latter preserves the
position of the arguments and naturally forms a tree structure. To illustrate
it further, suppose we have a function `getStudentInfo = (university: string,
year: number, course: string) => Student[]`, and we derive cache keys using the
second function. They might look like:

```
"uni-1"/2022/"CS-101"
"uni-1"/2023/"CS-101"
"uni-2"/2023/"CS-Intro"
"uni-2"/2023/"CS-DBMS"
"uni-1"/2023/"CS-201"
```

Note that quotes in the keys come from the JSON representation itself. We
observe that different universities follow different patterns while naming
courses as well. This key representation comes powerful when we want to
invalidate the cache.

## Invalidation

> There are only [two hard
things](https://martinfowler.com/bliki/TwoHardThings.html) in Computer Science:
cache invalidation and naming things.
>
> -- Phil Karlton

Well, this is one of the places the things get to start interesting. When using
an in-memory cache, we had an `invalidate` function that busted the entire
cache by replacing the cache storage with `{}`, which was an `O(1)` operation.
If we try to do the same thing for the tree cache, we have to delete `n` keys,
making it `O(n)` -- A tradeoff .

Often, we do not need to bust the WHOLE cache, but rather some keys of it.
Since our cache implementation wraps the functions and we derive cache keys
using function arguments, it makes sense to invalidate the cache using a
_prefix_ of the cache keys, which prunes a branch of the tree. In the above
example, we can choose to invalidate all entries starting with `"uni1"`, or
`"uni2"/2023`, depending on the use case. While the invalidation is still
`O(n)`, we have got more utility.

Suppose we have to write `invalidate` function to achieve this thing, and
return it as a property of the cached function. The big question remains as how
to make this function type-safe? For our example function `getStudentInfo`, the
`invalidate` call might look like any of the below:

- `invalidate()` - Invalidates all entries in the cache
- `invalidate(university: string)` - Invalidates all entries beginning with
`university`
- `invalidate(university: string, year: number)` - Invalidates all entries
beginning with `university` and `year`
- `invalidate(university: string, year: number, course: string)` - Invalidates
a specific entry that has the values matching all of `university`, `year` and
`course: string`

Fortunately, TS has a `Parameters<T extends Function>` type helper can get us
half way there - Given a function type `T`, it returns its parameter types as
an array. For example:

```typescript
const getStudentInfo = (university: string, year: number, course: string) => {}

type Parms = Parameters<typeof getStudentInfo>
// type Params = [university: string, year: number, course: string]
```

Note that the return type preserves the names of the arguments as labels. But
these names can not be used in any ways and just serve to help developers. In
essence, the returned type is equivalent to `[string, number, string]`.

Now we have to define a type helper `Prefixes<T extends any[]>` that generates
all possible prefixes of `T`, starting with an empty array. As usual, we can
invoke the magic of pattern matching and recursion to solve this.

```typescript
type Prefixes<T extends any[]> = T extends [infer First, ...infer Rest] //
First we take destructure the elements of the array
 ?
 | [] // An empty array is a prefix of all arrays
 | [First, ...Prefixes<Rest>] // Then we derive other prefixes by
combining the First element with all the prefixes of Rest elements
 : [] // If input is not an arry, there are no prefixes

type Result = Prefixes<['a', 'b', 'c', 'd']>
// type Result = [] | ["a", "b", "c", "d"] | ["a"] | ["a", "b"] | ["a", "b",
"c"]
```

With the `Prefixes` in our hand, we can now define our `cached` function
completely, using in memory storage.	

```typescript
const cached = <T extends Cacheable>(f: T) => {
 const cache: Record<string, any> = {}
 const getKey = (args: any[]) =>
 args.map((arg) => JSON.stringify(arg)).join('/')

 const wrapped = (async (...args: Parameters<T>) => {
 const key = getKey(args)
 if (!(key in cache)) {
 cache[key] = await f(args)
 }
 return cache[key]!
 }) as T

 const invalidate = async (...args: Prefixes<Parameters<T>>) => {
 const prefix = `${getKey(args)}/`
 const existingKeys = Object.keys(cache)
 existingKeys.forEach((key) => {
 if (key.startsWith(prefix)) delete cache[key]
 })
 }

 return Object.assign(wrapped, { invalidate })
}
```

Redis provides convenient commands to iterate over the keys with pattern
matching, thus saving us from looping over all the key space.

## The other hard thing

Our `cached` function referenced Redis client and Cache TTL values from global
scope variables. When we wanted to use this utility across different projects,
we have to remove these dependencies from `cached`. Our method of dependency
injection was through a Higher Order Function that looked like `(redisClient:
RedisClient, ttl: number) => <T extends Cacheable>(f: T) => T`. Now the hard
problem was to name that HOF!

We iterated over the names `getCached` - absurd, `getCacheFunction` - more
absurd! At the end, we ended up naming it `cacheFactory` - albeit it was a
`Factory`!

> Naming things has an impact on code readability and ease of debugging. Naming
Higher Order Function is more hard. Few days back, I was finding a bug during
data processing where a validation was failing for some data. After the
validation, the data was supposed to be processed and ingested.
>
> When I was going throgh the code (written by me, btw), I found the calls to
`validate` and `process` were in sequence, and the way our systems worked, all
calls to `validate` should have been failed; but mysteriously, some records
were passing.
>
> At the end, It turned out to be `validate` and `process` were Higher Order
Functions which returned the functions for validation and processing
respectively. If we knew this earlier, we could have saved around 1 day of
speculation why the bug was there at all. And we learnt a less to name things
properly (yeah, we know that is hard) to help our future selves.

How we implemented a type safe distributed cache in TS that supported partial invalidations

Tree Cache

As a software engineer, I helped build a few systems and products at work this year. Here I present a high level overview of my learnings and thoughts without specificities to the products I built.

2023 - Year In Review

Suppose we have an object, and one of its fields an array. Taking every array
elements and adding the rest of the objects keys is _exploding_ the object
w.r.t that array element. For example, if we have an object `{university: "XYZ", batch: 2019, students: [{name: "Alice"}, {name: "Bob"}]}`, exploding it
w.r.t `students` results in `[{university: "XYZ", batch: 2019, name: "Alice"}, {university: "XYZ", batch: 2019, name: "Bob"}]`.

We can encounter the need for the explosion in many cases. For example, let's
have an e-commerce order object, where it contains the individual items in an
array. When we want to process individual items with the order details,
explosion is the suitable operation.

With understanding the _explode_ operation, let's proceed to define our goals
clearly w.r.t to the result and type-safety

## The Goals

- We want to have a function `explode(object, key)` which does the explosion
- The `key` not being present in `object` should result in type error
- The `key` not being array should result in type error
- The returned array must be appropriately typed

We are resorting to type errors in most of the cases as we can be confident
about the function inputs when we run the code inside our app. If we want to
ship this function in a library, we need to do runtime checks as well, as there
is a possibility of this code called in plain JS without any typechecking. In
this post, for the sake of breivity, We are omiting the runtime checks.

## Solution without Types

The problem of explosion is easy and straightforward - we extract the value at
the key, map over the array and add rest of the properties to every object in
it.

```javascript
const explode = (object, key) => {
 const { [key]: array, ...rest } = object
 return array.map((item) => ({
 ...item,
 ...rest,
 }))
}
```

## Moving to the Type Realm

Now, let's add the types to the `explode` function, starting with the basics to
match the JS schematics:

```typescript
const explode = (object: object, key: string) => {
 const { [key]: array, ...rest } = object
 return array.map((item) => ({
 ...item,
 ...rest,
 }))
}
```

TS yells us at about two problems here:

1. We don't know if the `key` is indeed a key of the `object`

2. By virtue of that, `array` has type `any`, which makes `item` implicitly
 `any`.

Let's first restrict `key` to be a valid key of `object`. Since our parameters
have dependency now, we need to make our function generic. If the generic
parameter is `T`, we can either mark `object` or `key` as `T`. For the reasons
that will become evident later, let's mark the `object` as `T`. With that,
let's also add generic _constraint_ on `T` to be a subtype of `object`.

```typescript
const explode = <T extends object>(object: T, key: keyof T) => {
 const { [key]: array, ...rest } = object
 return array.map((item) => ({
 ...item,
 ...rest,
 }))
}
```

With this, we restrict `key` to be a valid key of `T`, not any string. This
solves the problem #1. We still need to solve problem #2. Before doing that,
let's first restrict `key` to only the keys where value is an array.

## Hello, Key Mapping!

Let's first understand how `keyof T` looks like before filtering it:

```typescript
type UniversityRecord = {
 university: string
 batch: number
 students: { name: string }[]
}

type Keys = keyof UniversityRecord
// "university" | "batch" | "students"
```

Thus, `Keys` is a union of strings, and it does not recursively descend into
the object structure, which is as expected. Now, can we loop over the union?
**Unfortunately not**. To understand why, remember there is no _order_ in
unions - `a | b | c` as `b | c | a` and etc. To loop over the keys and filter
them out, we can use [_key
remapping_](https://www.typescriptlang.org/docs/handbook/2/mapped-types.html#key-remapping-via-as)

```typescript
type ArrayKeys<T extends object> = keyof {
 [K in keyof T as T[K] extends any[] ? K : never]: unknown
}
```

There are so many things going on here even if the code is deceptivly small.
Let's break down the things happening here:

- We are looping over every key `K` in `keyof T` in a mapping context (inside
 `[]`). Remember that such loop was not possible with unions.
- We are remapping the key using `as` keyword. If we map the key to `never`, it
 will be excluded
- Remapping supports conditions too - we check if `T[K]` is an array. If so, we
 keep `K` as it is; otherwise, we will exclude it using `never`.

Using this type helper, we can now define `explode` function as:

```typescript
const explode = <T extends object>(object: T, key: ArrayKeys<T>) => {
 const { [key]: array, ...rest } = object
 return array.map((item) => ({
 ...item,
 ...rest,
 }))
}
```

Now, we have to solve one problem: TS _still_ does not know that `object[key]`
is an array. While we can do further type manipulations, we can use runtime
functions to solve this.

```typescript
const explode = <T extends object>(object: T, key: ArrayKeys<T>) => {
 const { [key]: array, ...rest } = object
 return Array.isArray(array) ? array.map((item) => ({ ...item, ...rest })) : []
}
```

By adding `Array.isArray`, the `array` is treated as `any[]`. TS defines
`isArray` as a [_type
predicate_](https://www.typescriptlang.org/docs/handbook/2/narrowing.html#using-type-predicates)
with the following declaration:

```typescript
function isArray(arg: any): arg is any[]
```

When `isArray` returns `true`, TS knows that the `arg` is `any[]`, so we can
access `map` and `item` is infered as `any`. Let's try our new function
definition to check if argument types work properly:

```typescript
const res1 = explode(
 {
 university: 'XYZ',
 batch: 2019,
 students: [{ name: 'Alice' }, { name: 'Bob' }],
 },
 'university'
) // type error

const res2 = explode(
 {
 university: 'XYZ',
 batch: 2019,
 students: [{ name: 'Alice' }, { name: 'Bob' }],
 },
 'students'
) // no error
```

Well, so far so good. But if we check the return type of `explode`, it is
`any[]`, which makes it less useful in practice. Let's solve that by adding a
return type to it.

## Enter Type Gymnastics

We have following information about `explode(object, key)`:

- It returns an array (well, TS is intelligent to know that already, as it
 infers `any[]` as the return type)
- Every element of the array is an object.
- The object is derived by concatenating two objects:
 1. `object` with `key` removed
 2. the type of elements of `object[key]`

With this information, we can write a type helper `Explode`:

```typescript
type Explode<T extends object, K extends keyof T> = T[K] extends any[]
 ? (Omit<T, K> & T[K][number])[]
 : never
```

Few interesting things are happening here:

- We constrint `K` to be `keyof T`, not `ArrayKeys<T>`. Using `ArrayKeys` here
 does not add any useful information as TS does not get to know `T[K]` is an
 array. Thus, we choose a simpler path.
- We do the check `T[K] extends any[]` again, due to the reason explained
 above. If it does not, we return `never`. This case _never_ happens if `K
extends ArrayKeys<T>`
- If the check passes, we _omit_ `K` from `T` and intersect with the
 constituent type of `T[K]`
- To get the constituent type of `T[K]`, we index it on `number`, as arrays are
 indexed based on number

Now, we can proceed to type the `explode` function completly:

```typescript
const explode = <T extends object, K extends ArrayKeys<T>>(
 object: T,
 key: K
) => {
 const { [key]: array, ...rest } = object
 return (
 Array.isArray(array)
 ? array.map((item) => ({
 ...item,
 ...rest,
 }))
 : []
 ) as Explode<T, K>
}
```

It looks little bit different from our above definition. Apart from the `as
Explode<T,K>` cast, we typed `key` with `K` where `K extends ArrayKeys<T>`.
Typing `key` as `ArrayKeys<T>` and passing `typeof key` to `Explode` will keep
the generic argument to `Explode` as `ArrayKeys<T>`, not the specific literal
type that is passed in `key`. With this, we can proceed to _explode_!

## Testing

Let's call `explode` few times to check if everything works well. For testing,
let's assume we have an e-commerce order object:

```typescript
const order = {
 type: 'Order',
 customer: {
 name: 'Alice',
 address: '123 st.',
 },
 items: [
 {
 id: '111',
 name: 'Long sleeve T Shirt',
 price: 15,
 currency: 'USD',
 },
 {
 id: '232',
 name: 'Jeans',
 price: 12,
 currency: 'USD',
 },
 ],
 offers: [
 {
 code: 'FIRST-CUSTOMER',
 discountPercentage: '20',
 },
 {
 code: 'REFFERAL',
 discountPercentage: '10',
 },
 ],
}
```

As per our expectations, `explode` on this object should work for `items` and
`offers`, but not on other keys:

```typescript
const result1 = explode(order, 'type') // type error
const result2 = explode(order, 'items') // works
/*
const result2 = [
 {
 id: "111",
 name: "Long sleeve T Shirt",
 price: 15,
 currency: "USD",
 type: "Order",
 customer: {
 name: "Alice",
 address: "123 st.",
 },
 offers: [
 {
 code: "FIRST-CUSTOMER",
 discountPercentage: "20",
 },
 {
 code: "REFFERAL",
 discountPercentage: "10",
 },
 ],
 },
 {
 id: "232",
 name: "Jeans",
 price: 12,
 currency: "USD",
 type: "Order",
 customer: {
 name: "Alice",
 address: "123 st.",
 },
 offers: [
 {
 code: "FIRST-CUSTOMER",
 discountPercentage: "20",
 },
 {
 code: "REFFERAL",
 discountPercentage: "10",
 },
 ],
 },
]
*/
const result3 = explode(order, 'offers') // works
/*
const result3 = [
 {
 code: "FIRST-CUSTOMER",
 discountPercentage: "20",
 type: "Order",
 customer: {
 name: "Alice",
 address: "123 st.",
 },
 items: [
 {
 id: "111",
 name: "Long sleeve T Shirt",
 price: 15,
 currency: "USD",
 },
 {
 id: "232",
 name: "Jeans",
 price: 12,
 currency: "USD",
 },
 ],
 },
 {
 code: "REFFERAL",
 discountPercentage: "10",
 type: "Order",
 customer: {
 name: "Alice",
 address: "123 st.",
 },
 items: [
 {
 id: "111",
 name: "Long sleeve T Shirt",
 price: 15,
 currency: "USD",
 },
 {
 id: "232",
 name: "Jeans",
 price: 12,
 currency: "USD",
 },
 ],
 },
]
*/
```

We can also check the output types of `result2` and `result3`

```typescript
type Simplify<T> = T extends object
 ? {
 [K in keyof T]: Simplify<T[K]>
 }
 : T

type Result2 = Simplify<typeof result2>

/*
type Result2 = {
 type: string
 customer: {
 name: string
 address: string
 }
 offers: {
 code: string
 discountPercentage: string
 }[]
 id: string
 name: string
 price: number
 currency: string
}[]
*/

type Result3 = Simplify<typeof result3>

/*
type Result3 = {
 type: string
 customer: {
 name: string
 address: string
 }
 items: {
 id: string
 name: string
 price: number
 currency: string
 }[]
 code: string
 discountPercentage: string
}[]
*/
```

We can see that the type result matches the type of execution result, which
made `result2` and `result3` strongly typed, and, caught invalid key in
`result1`, making our `explode` function robust in type domain.

We explore the operation of exploding an object in this post, and make it robust by adding strong typing for its inputs and output

Exploding with Type Safety

**Also Read:** [Recursive Types](/blog/recursive-types)

TypeScript has a very solid type system that makes few things possible, which
are not possible in other typed languages. To show the powers of TS type
system, we will go through a practical problem in this post and sprinkle types
to it to make it robust.

## The Problem

APIs and Databases normally use differnet styles of naming variables. While API
fields are `camelCase` most of the times, in DB, we often name columns in
`snake_case` often. 

Snake case names in DB are reasonable, as SQL is case-insensitvive. A variable
in camelcase, say, `jsonPayload` is converted to `jsonpayload` unless quoted.
Quoting is often inconvinent and often missed when working in a SQL console
directly.

While ORMs can be configured to handle these conversions in majority of the
cases, when we need to deal with db directly, we need to handle case
conversion. The problem is to handle the conversion in a type-safe way. We will
solidify our expectations soon.

## The Beginnings 

Let's start with a JS function to convert camel case strings to snake case, and
iteratively add features later on. Converting a camel case string to snake case
is easy -- we iterate through the characters of the word, if we find an
uppercase letter, we emit an underscore (`_`) and the corresponding uppercase
letter. Otherwise, we will just emit the current letter. Let's assume that word
does not contain any special characters (it makes sense, as we can't use
special characters while naming identifiers and DB columns).

```javascript
const isUpperCase = (letter) =>
 letter === letter.toUpperCase()

const toSnakeCase = (word) =>
 word
 .split("")
 .map((letter) =>
 isUpperCase(letter)
 ? ["_", letter.toLowerCase()]
 : [letter],
 )
 .flat()
 .join("")

const result1 = toSnakeCase("helloWorld") // "hello_world"
const result2 = toSnakeCase("helloWorldTest") // "hello_world_test"
```

That was simple. Let's write a function to convert keys of an object to snake
case while perserving the values as-is. Think of the input object as whatever
we got in API body, and the output as what we want to write to DB.

```javascript
const convertKeysToSnakeCase = (object) =>
 Object.fromEntries(
 Object.entries(object).map(([key, value]) => [
 toSnakeCase(key),
 value,
 ]),
 )

const result = convertKeysToSnakeCase({
 apiVersion: "2023-02",
 jsonPayload: "{hello: 2}",
})
// { api_version: "2023-02", json_payload: "{hello: 2}" }
```

## Sprinkle some Types!

Let's start by tyoing adding types to `isUpperCase` and `toSnakeCase` functions

```typescript
const isUpperCase = (letter: string) =>
 letter === letter.toUpperCase()

const toSnakeCase = (word: string) =>
 word
 .split("")
 .map((letter) =>
 isUpperCase(letter)
 ? ["_", letter.toLowerCase()]
 : [letter],
 )
 .flat()
 .join("")
```

Note that the return types are infered as `string` in both the cases, so we
don't need to type it explicitly. Type Inference is a powerful tool in TS which
eliminates the need of type annotation in most of the cases. I have explored a
few examples in [my last post](/blog/type-inference), go check it out too!

Now, it is time to type `convertKeysToSnakeCase` too. We can start as:

```typescript
const convertKeysToSnakeCase = (object: object) =>
 Object.fromEntries(
 Object.entries(object).map(([key, value]) => [
 toSnakeCase(key),
 value,
 ]),
 )
```

Typing `object` param as `object` (sorry for the repitition!) makes sense, and
output type is infered as `{[k: string]: any}`, which is an object with only
string keys.

## Setting Expectations

When we pass an object through `convertKeysToSnakeCase`, we lose all type
information and we can index the result with any key. Ideally, the keys should
get converted to snake case in type and all other key accesses should result in
type error. Put it in code,

```typescript
const result = convertKeysToSnakeCase({
 apiVersion: "2023-02",
 jsonPayload: "{hello: 2}",
})
result.abc // Should result in a type error saying that the key does not exist
result.json_payload // Should be no error, type should be string
result.jsonPayload // Should result in a type error saying that the key does
not exist
```

With the expectations set, we can move on to solving the problem. The solution
is exactly as [Literal Magic](/blog/literal-magic), but presented in a
differnet way.

## Enter Generics

We want a type `ConvertKeysToSnakeCase` which accpets a generic type `T` which
should be an object, and convert its keys to snake case. Let's start by
ensuring that the type only accepts objects as input:

```typescript
type ConvertKeysToSnakeCase<T extends object> = T

type Result1 = ConvertKeysToSnakeCase<42> // Error
type Result2 = ConvertKeysToSnakeCase<{
 apiVersion: string
 jsonPayload: string
}> // Success
```

We added a generic _constraint_ that `T` should be a subtype of `object`, which
is making sure that we don't accidentally pass in other types. Now, somehow, we
have to loop over the keys of the object and transform them into snake case.
Let's assume that there is a helper `ToSnakeCase` to convert individual keys to
snake case.

```typescript
type ToSnakeCase<T extends string> = T // It does nothing now

type ConvertKeysToSnakeCase<T extends object> = {
 [Key in keyof T as ToSnakeCase<
 Key & string
 >]: T[Key]
}
```

In `Key in keyof T as ToSnakeCase<Key & string>`, we are looping over all keys
of T (got by `keyof T`), with indivual key as `Key`. Then, we are mapping it
using `as` keyword to `ToSnakeCase<Key>`. We need `Key & string` in the
argument as key of an object may be `string | number | symbol`, and we want
eliminate `number | symbol` from the loop. 

Now, let's move to the fun part of implementing `ToSnakeCase` itself!

## Hi Recursion!

To implement the type `ToSnakeCase<T extends string>`, we can't iterate over
indivual items of `string` unfortunately as we did in `toSnakeCase` function
(However, TS supports iterating over the keys of an object using `keyof` as we
just saw). TS supports recursion with template literal types. Before going to
TS, let's try solving the problem in JS recursively.

```typescript
const isUpperCase = (letter: string) =>
 letter === letter.toUpperCase()

const toSnakeCase = (word: string): string => {
 const [first, ...rest] = word
 if (!first) return "" // Base case
 const restString = rest.join("")
 return isUpperCase(first)
 ? `_${first.toLowerCase()}${toSnakeCase(
 restString,
 )}`
 : `${first}${toSnakeCase(restString)}`
}

const result1 = toSnakeCase("helloWorld") // "hello_world"
```

Let's break down how this is working:

- We first split the `word` into `first` and `rest`, where `first` is `string |
undefined`, and `rest` is `string[]`using [JS spread
syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax)
- `first` will be `undefined` whenever `word=""`, so we return `""` in that
case (by equality, we could have returned `first` too, but returning `""` gives
clarity to the reader)
- We convert `rest` to a string to prepare it to be called recursively
- We check if `first` is an uppercase, similar to iterative case. If it is the
case then emit an underscore and corresponding lowercase letter; otherwise, we
don't change `first`
- We append the result of the recursion to the first character using [JS
Template
Strings](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals).

Note that:

- We had to explicitly type the return value of `toSnakeCase` as `string` as TS
infered `any` as the return type of recursive function
- While this code works fine in native JS, in order for it to work in TS, we
need to set `"downlevelIteration": true`

Splitting an iterable to its _head_ and _body_ is a common operation in many
functional programming languages, which enables an operation on _head_ and
recursion on _body_. Fortunately, TS provides such functionality too, which we
can leverage. Let's explore the functionality:

```typescript
type ListComponents<T> = T extends [
 infer First,
 ...infer Rest,
]
 ? { head: First; body: Rest }
 : never

type StringComponents<T> =
 T extends `${infer First}${infer Rest}`
 ? { head: First; body: Rest }
 : never

type ListResult1 = ListComponents<[]>
// type ListResult1 = never
type ListResult2 = ListComponents<[2]>
/*
type ListResult2 = {
 head: 2;
 body: [];
}
*/
type ListResult3 = ListComponents<[1, 2, 3, 4]>
/*
type ListResult3 = {
 head: 1;
 body: [2, 3, 4];
}
*/
type ListResult4 = ListComponents<"hello">
// type ListResult4 = never

type StringResult1 = StringComponents<"">
// type StringResult1 = never
type StringResult2 = StringComponents<"a">
/*
type StringResult2 = {
 head: "a";
 body: "";
}
*/
type StringResult3 = StringComponents<"abc">
/*
type StringResult3 = {
 head: "a";
 body: "bc";
}
*/
type StringResult4 = StringComponents<["a", "b"]>
// type StringResult4 = never
```

TS is doing _Pattern Matching_ here - `[infer First, ...(infer Rest)]` and
`${infer First}${infer Rest}` are the patterns. If the input _matches_ the
pattern, the type variables `First` and `Rest` will get their respective
values. In the first pattern, we needed `...` to explicitly tell that `Rest`
should be an array. In second pattern, `Rest` is automatically resolved to rest
of the characters.

With the power of pattern matching in hand, we can easily convert the recursive
version of `toSnakeCase` into a type.

```typescript
type ToSnakeCase<T extends string> =
 T extends `${infer First}${infer Rest}`
 ? First extends Uppercase<First>
 ? `_${Lowercase<First>}${ToSnakeCase<Rest>}`
 : `${First}${ToSnakeCase<Rest>}`
 : T

type Result1 = ToSnakeCase<"helloWorldTest"> // hello_world_test
type Result2 = ToSnakeCase<"JsonPayload"> // _json_payload
```

This code looks very similar to the function code we have written, except that
it is working in _Type Domain_, meaning it does not do any runtime action. Note
that the `Uppercase` and `Lowercase` helpers are provided by TS. What if we
could have same code across Type Domain and runtime?! - well, providing a
runtime type system is not one of the TS goals.

With `ToSnakeCase` in our hand, we can revisit `ConvertKeysToSnakeCase` now.

```typescript
type ConvertKeysToSnakeCase<T extends object> = {
 [Key in keyof T as ToSnakeCase<
 Key & string
 >]: T[Key]
}

type ObjectResult = ConvertKeysToSnakeCase<{
 apiVersion: string
 jsonPayload: string
}>
/*
{
 api_version: string;
 json_payload: string;
}
*/
```

And there we are! Now we can add types to `convertKeysToSnakeCase` function

```typescript
const convertKeysToSnakeCase = <T extends object>(
 object: T,
) =>
 Object.fromEntries(
 Object.entries(object).map(([key, value]) => [
 toSnakeCase(key),
 value,
 ]),
 ) as ConvertKeysToSnakeCase<T>
```

With that,

```typescript
const result = convertKeysToSnakeCase({
 apiVersion: "2023-02",
 jsonPayload: "{hello: 2}",
})
result.abc // Error: Property 'abc' does not exist...
result.json_payload // Works
result.jsonPayload // Error: Property 'jsonPayload' does not exist...
```

And we have successfully achieved the expectations! Note that few edge cases
exist in our implementation, for a comprehensive treatment on this usecase,
please look into [my earlier post](/blog/literal-magic).

## More Recursion!

Having solved camel case to snake case conversion in both runtime and type
domain, let's do the same of reverse.

```typescript
// Iterative version
const toCamelCase = (snakeCaseString: string) =>
 snakeCaseString.replaceAll(/_./g, (match) =>
 match.at(1).toUpperCase(),
 )
```

Here, we find all underscores and the next character, and replace it with the
uppercase version of the next character.

> **Bug Alert**: Both `toSnakeCase` and `toCamelCase` miss a few edge cases.
Ideally, `toSnakeCase(toCamelCase(s)) === s` and ` toCamelCase(toSnakeCase(s))
=== s`, in both runtime and type domain, which is not the case here. I will
leave it as an exercise to the reader to fix them to handle the edge cases.

We can also achive the same thing in recursion:

```typescript
const capitalize = (word: string) => {
 const [first, ...rest] = word
 if (!first) return ""
 return `${first.toUpperCase()}${rest.join("")}`
}

const toCamelCase = (snakeCaseString: string) => {
 const [first, ...rest] = snakeCaseString
 if (!first) return ""
 const restString = rest.join("")
 return first === "_"
 ? capitalize(toCamelCase(restString))
 : `${first}${toCamelCase(restString)}`
}
```

The recursive version is similar to `toSnakeCase` w.r.t splitting and the base
case, but has a different logic during recursion.

Now, we can convert this to type domain.

```typescript
type ToCamelCase<T extends string> =
 T extends `${infer First}${infer Rest}`
 ? First extends "_"
 ? Capitalize<ToCamelCase<Rest>>
 : `${First}${ToCamelCase<Rest>}`
 : T
```

We can go one step forward and use TS pattern matching to our advantage:

```typescript
type ToCamelCase<T extends string> =
 T extends `${infer First}_${infer Rest}`
 ? `${First}${Capitalize<ToCamelCase<Rest>>}`
 : T
```

Here we added underscore to the pattern itself to simplify the recursive case.
Note the we can't to this type of pattern matching in JS
[_yet_](https://github.com/tc39/proposal-pattern-matching)!

## Concluding Remarks

Recursion is a powerful tool. It is the method of solving a chunk of a problem
at a time, and then repeating the same on one or more subproblems. Recursion
also helps to bring out the mathematical structure of the problem. While
recursion is generally not optimized in many procedural languages, they are
heavily optimized in functional languages and are fun to use!

An exploration of recursive logic in the type domain

Recursive Types

There is a big portion of people who hate TypeScript because they think they
need to type _everything_ and it makes the code ugly and unreadable. While
typing everything does take the ergonomics away, TypeScript **does not** force
you to type everything - if you type the key parts of your code and logic (most
of the time, they are libraries), it can _infer_ the type in the rest of the
places. In this article, I will take you through an example where we need to
type the core and reap the benefits of inference everywhere else.

## The Problem

We want to define an MQ listener, that

- Could validate the incoming messages to be a JSON
- Could validate the JSON to follow a certain schema
- Execute the async handler function with the parsed object
- The handler returns a boolean indicating whether the execution of the message
was successful 

For the sake of simplicity, let's assume that the `listener` is a Higher-order
function (i.e., which takes another function as input) taking the `handler`
function as an input along with the `schema`. For parsing schema, we will use
`zod`.

For the solution to be valid, the `listener` should **only** accept the
`handler`s that take the type specified by `schema` as the input. If this is
not the case, it should be a TypeScript error, not a runtime error.

Based on the requirements above, we know that the `listener` function would
look something like this:

```typescript
const listener = (schema, handler) => {/*...*/}
```

The fun part is to type the function now -- what should be the type of `schema`
and `handler`? Before doing that, let's familiarise ourselves with `zod` and
the utilities that it provides.

## Zod 101

[Zod](https://github.com/colinhacks/zod) is a schema definition and object
parsing library. First, we define a schema using the primitives that `zod`
provides, and then we can access the `parse` method of the schema, which
validates the passed-in object and throws a `ZodError` in case parsing fails
(for those you fear exceptions - I do, in most of the cases - there is a
`safeParse` method which returns a `{success: true, data: T} | {success: false,
error: ZodError}`).

The type of the schema is `ZodSchema` most of the time, if you do some
post-parsing calculations, it will be `ZodEffects`. They have a very good
utility `infer` to get a plain type from `ZodSchema` and `ZodEffects`. Let's
get our hands dirty and see some code in action!

```typescript
import { z } from "zod"

const pageViewEventSchema = z.object({
 type: z.literal("pageView"),
 title: z.string().min(1),
 url: z.string().min(1),
 additionalInfo: z.optional(z.record(z.string(), z.any()))
})
```

The schema definition is very much like the type definition, but with
additional flexibility to define further validations like `min(1)` (which says
that the string must be at least of length 1). Using this schema is
straightforward:

```typescript
const unparsedOject = {
 /* This can be anything that comes from outside of the system
 * For example - user inputs, API call body, messages etc
 */
}

try {
 const pageViewEvent = pageViewEventSchema.parse(unparsedObject)
 // Now do something with pageViewEventSchema
} catch (e) {
 if (e instanceof ZodError) {/* handle the parsing error */}
}
```

If you are scared of expectations like me, you can use `safeParse`

```typescript
const parseResult = pageViewEventSchema.safeParse(unparsedObject)
if (parseResult.success) {
 const pageViewEvent = parseResult.data 
 // Now do something with pageViewEventSchema
} else {
 const e = parseResult.error
 // Handle the parsing error
}
```

In both cases, the constant `pageViewEvent` would be of the following type:

```typescript
{
 type: "pageView", // Note that this is a literal, not a string
 title: string,
 url: string,
 additionalInfo?: Record<string, any> // Record is a TS-inbuilt type 
}
```

To derive the type automatically, we can use `infer` utility:

```typescript
type PageViewEvent = z.infer<typeof pageViewEventSchema>
```

Note that the `infer` works for `ZodEffect`s as well. Following is the example
of an effect, where we set `additonalInfo` to an empty object if we do not
receive it (i.e., it is `undefined`).

```typescript
const pageViewEventSchemaWithEffect = z.object({
 type: z.literal("pageView"),
 title: z.string().min(1),
 url: z.string().min(1),
 additionalInfo: z.optional(z.record(z.string(), z.any()))
}).transform(
 // This is executed after the successful parsing with the parsed object as
input
 (result) => ({ ...result, additionalInfo: result.additionalInfo || {} })
);

type PageViewEventWithEffect = z.infer<typeof pageViewEventSchemaWithEffect>
/*
{
 type: "pageView",
 title: string,
 url: string,
 additionalInfo: Record<string, any> // Note that this is no more optional
}
*/
```

Equipped with the knowledge of `ZodSchema`, `ZodEffects` and `infer`, let's try
to solve the problem at hand:

## Attempt 1

In `const listener = (schema, handler) => {/*...*/}`, we can start typing as
follows:

- Let's assume that the type `T` is the type we are defining schema for
- Then, `schema` is of type `ZodSchema<T>` or `ZodEffects<T>`
- `handler` is a function taking an input `T` to `Promise<boolean>`

```typescript
import { ZodSchema, ZodEffects } from "zod";

type HandlerFunction<T> = (input: T) => Promise<boolean>

const listener = <T>(schema: ZodSchema<T>|ZodEffects<T>, handler:
HandlerFunction<T>) => {}
```

Here is a problem - `ZodEffects<T>` does not accept any `T`. `T` must be a
subtype of `ZodTypeAny`.

```typescript
import { ZodSchema, ZodEffects, ZodTypeAny } from "zod";

type HandlerFunction<T> = (input: T) => Promise<boolean>

const listener = <T extends ZodTypeAny>(schema: ZodSchema<T>|ZodEffects<T>,
handler: HandlerFunction<T>) => {}
```

With this setup, we can experiment with the following:

- Keep `async message => true` as the second argument. Note that we need
`async` as `HandlerFunction` is expected to return a `Promise`
- Try to pass in `pageViewEventSchema` as the first argument, TS complains that
it does not satisfy `ZodTypeAny`
- Try to pass in `pageViewEventSchemaWithEffect` as the first argument, TS
does not error out. But `message` is not the plain type
`PageViewEventWithEffect`!

We can go down the path of exploring `zod` internal types to get our work done.
But there is a good way out by changing our assumption of what `T` should be by
taking the key fact that `z.infer` accepts both `ZodObject` and `ZodEffects`.
Let's try that once.

## Attempt 2

In `const listener = (schema, handler) => {/*...*/}`, we type as follows:

- Let's assume that the type `T` is the type of the Zod schema
- Then, `schema` is of type T
- `handler` is a function taking an input `z.infer<T>` to `Promise<boolean>`

Let's try to implement it:

```typescript
type HandlerFunction<T> = (input: T) => Promise<boolean>

const listener = <T>(schema: T, handler: HandlerFunction<z.infer<T>>) => {}
```

There is a slight problem with this, as `infer` does not work with any `T`, but
with only `ZodType`s (we get a good TS error explaining this, so we can adapt).

```typescript
type HandlerFunction<T> = (input: T) => Promise<boolean>

const listener = <T extends ZodType>(schema: T, handler:
HandlerFunction<z.infer<T>>) => {}
```

Now, TS is all fine. Let's experiment with the same cases as the previous
attempt.

- Keep `async message => true` as the second argument. Note that we need
`async` as `HandlerFunction` is expected to return a `Promise`
- Try to pass in `pageViewEventSchema` as the first argument. There is no error
and we can see that the type of the `message` is the same as `PageViewEvent`,
we get nice autocompletion!
- Try to pass in `pageViewEventSchemaWithEffect` as the first argument. There
is no error and we can see that the type of the `message` is the same as
`PageViewEventWithEffect`

So, all the test cases passed. We can go a further step ahead and make sure
that the `listener` function does not take an invalid `schema` and `handler`
combination as the input - it must be a TS error.

```typescript
const handler = async (message: PageViewEventWithEffect) => true

listener(pageViewEventSchema, handler)
```

There is a TS error now, which reads in the end that `Types of property
'additionalInfo' are incompatible.`, which is indeed the case.

Thus, we have finally solved the `listener` matching the `schema` and `handler`
types. Let's take a zoomed-out picture of this and recap what we have learnt
the way.

## Learnings

TS generics is a powerful tool - it can make your life easy or hard depending
on how to use it. Choosing a generic parameter to parametrize the function when
the parameters are interdependent is hard - even if making `T` the simple type
that directly corresponds to the object that we wanted to work with was
intuitive, it made the problem of deriving the type for another parameter
harder. With the utility to convert zod schema to a plain type, we were able to
solve the problem easily. Thus, it is always worth thinking about solving a
problem from multiple directions!

## How does it help to remove the type boilerplate?!!

In the example that we have seen above, we extracted lots of types and
sprinkled them everywhere to understand the problem on a deep level. Let's see
how we can get rid of all the boilerplate and arrive at a much cleaner code.

```typescript
import { z } from "zod";

// Library
const queue = {
 // Dummy queue
 subscribe: (handler: (message: string) => Promise<void>) => {}
};

type HandlerFunction<T> = (input: T) => Promise<boolean>;

const listener = <T extends z.ZodType>(
 schema: T,
 handler: HandlerFunction<z.infer<T>>
) => {
 queue.subscribe(async (message) => {
 try {
 const messageData = JSON.parse(message);
 const parsedMessage = schema.parse(messageData);
 const result = await handler(parsedMessage);
 if (!result) {
 console.error("Handler returned false");
 }
 } catch (e) {
 console.error(e);
 }
 });
};

// Application code
const pageViewEventSchema = z.object({
 type: z.literal("pageView"),
 title: z.string().min(1),
 url: z.string().min(1),
 additionalInfo: z.optional(z.record(z.string(), z.any()))
});

listener(pageViewEventSchema, async (event) => {
 console.log("Got pageView event")
 console.log(event.title)
 console.log(event.url)
 console.log(event.additionalInfo)
 return true
})
```

While our library code here does pretty much type-lifting, our application code
is a plain JS code with the added benefits of type checking - who doesn't want
it at no cost? If you want to type this code manually, it will be very much
ugly as we need to provide a `zod` type as a generic argument `T`.

## Playground

You can play with these examples at
https://codesandbox.io/s/admiring-fog-93l6pp?file=/src/clean.ts:0-1016

Making Types Work for You

In my last article [Lazy is easy](/blog/lazy-is-easy), I implemented lazy
loading the properties of a JS object, with full type-safety. There was one
caveat, though -- we named the properties as variable names (`name`), when they
were the getters. Ideally, we could have named them `getName`, and `Proxy`
convert the access to the `name` to call to `getName`. This, however, adds
couple of interesting challenges from the TS domain, which we shall go through
in this article.

## Enforcing the Property Names as `get<Something>`

This is one of the first requirements of the problem. Let's go ahead and modify
the `LazyInput` type from the previous article to support this.

```diff
 type LazyInput = {
- [key: string]: () => any
+ [key: GetterName]: () => any
 }
```

The `GetterName` type enforces that the string should start with `get`,
followed by an uppercase character and any string.

```typescript
type GetterName = `get${UppercaseLetters}${string}`
```

We can define `UppercaseLetters` as an union of all uppercase letter literals,
but I'm lazy to do that :sweat_smile:. Instead, let's define a constant string
containing all lowercase characters, and split it somehow to a union of
uppercase characters.

```typescript
const letters = "abcdefghijklmnopqrstuvwxyz"

type Split<T extends string> =
 T extends `${infer First}${infer Rest}`
 ? First | Split<Rest>
 : never

type LowercaseLetters = Split<typeof letters>

type UppercaseLetters = Uppercase<LowercaseLetters>
```

Here, type `Split` is recursively splitting input to union until it is empty.
We then use TS-inbuilt type `Uppercase` to get the desired union.

## Deriving the property name from `GetterName`

In the Proxied lazy object, we want to _access properties_, not _call getters_.
Let's create a type to convert `GetterName` to property names. For example,
this type would convert `getPropertyName` to `propertyName`. Here, we are

- removing the `get` from the begining of the string
- converting the next letter to lowercase
- keeping the reamining part as it is.

```typescript
type PropertyName<T extends GetterName> =
 T extends `get${infer FirstLetter}${infer Name}`
 ? `${Lowercase<FirstLetter>}${Name}`
 : never
```

Let's try it in action.

```typescript
type t1 = PropertyName<"getPropertyName"> // = propertyName
type t2 = PropertyName<"setPropertyName"> // Error
type t3 = PropertyName<"getpropertyName"> // Error 
```

Now that we get it working, let's implement the return type of our new `lazy`
function.

## The new `Lazy`

Armed with the type `PropertyName`, we can now define the `Lazy` type as:

```typescript
type Lazy<T extends LazyInput> = {
 [K in keyof T as PropertyName<
 K & GetterName
 >]: ReturnType<T[K & GetterName]>;
}
```

There are several differences here from previous article:

- We are _renaming_ the keys here using `as` operator to `PropertyName`
- While doing so, we are intersecting the key type `K` with `GetterName` to
ensure we pass in the right types.

Let's check if that works:

```typescript
type t1 = Lazy<{}> // = {}
type t2 = Lazy<{
 getX: () => number,
 getLongProperty: () => string
}> // = {x: number, longProperty: string}
type t3 = Lazy<{
 hello: number
}> // = {}
```

Cool, our type _seems_ to work, except on `t3`, where I thought TS should give
us an error. Let's dig deep and understand why it is not happening.

## Enter the TS Rabbithole

Let's look closely at the definition of `Lazy<T>`, where we defined a
_constraint_ on `T` that it should _extend_ `LazyInput`. Let's check if the
type `{hello: number}` satisfies that constraint.

```typescript
type test = {hello: number} extends LazyInput ? true: false // = true
```

Woah, seems like TS is doing something wrong here :open_mouth:, although we
specify the key format in `LazyInput`, TS is not respecting that at all! To
investigate this further, let's supply a `GetterName` key, but with an invalid
value (say `number`)

```typescript
type test = {getHello: number} extends LazyInput ? true: false // = false
```

Ah, now it is `false` and TS seems back alive. Does it mean that TS is
validating the type of the values, but not the keys:thinking:? Btw, what does
the `extends` is checking, exactly?

### The `extends` Keyword in Type Expressions

Let's think of extending a class in object oriented languages. The extended
class (a.k.a _subtype_) has all the properties (and methods) of its parent
class (a.k.a _supertype_) along with some of its own (also overrides). In TS
also, inheritance works in the same way. Let's go back to some examples now.
Here, I'm using `interface` to make use of `extends` keyword; we can achieve
the same things with `type` and intersection using `&`.

```typescript
interface Person {
 name: string,
 age: number
}

interface PersonWithPhone extends Person {
 phoneNumber: string
}

interface PersonWithHealthInfo extends Person {
 bloodGroup: string
}

type t1 = PersonWithHealthInfo extends Person ? true: false // = true
type t2 = PersonWithPhone extends Person? true: false // = true
type t3 = {} extends Person? true: false // = false
type t4 = Person extends Person? true: false // = true
type t5 = {name: string} extends Person? true: false // = false
```

Here, the `extends` keyword is both used to create subtypes and check if a type
is subtype of another, and the results are as what we expect -- if a type
contains all properties of `Person`, the `extends` check returns `true`.
Keeping this model in mind, let's go beyond pre-defined properties to explore
what happens in case of keyed properties.

Let's consider back our `LazyInput` type, with some simplifications. 

```typescript
type LazyInputSimplified = {
 [key: GetterName]: number
}
```

Let's look at some valid and invalid _values_ for this type:

```typescript
const e1: LazyInputSimplified = {} // valid
const e2: LazyInputSimplified = {"hello": 1, "world": 2} // invalid
const e3: LazyInputSimplified = {2: 5} // invalid
const e4: LazyInputSimplified = {"getA": 3} // valid
```

This is also as expected. Note that the empty object is also a valid value.
Let's think what values a subtype (say `T`) of `LazyInputSimplified` can have
of. By definition of subtyping,

- It can have all the values that `LazyInput` can have, along with
- **It can define its own values.**

The point that `T` can define extra properties is the key here. Since empty
object is also a valid `LazyInputSimplified` value, we are free to add _any_
keys to it as long as we don't touch `GetterName` and provide it with an
invalid value (other than `number`). To make it clear:

```typescript
type t1 = {} extends LazyInputSimplified ? true: false // = true
type t2 = {hi: "hello"} extends LazyInputSimplified? true: false // = true
type t3 = {hi: "hello", getA: boolean} extends LazyInputSimplified? true: false
// = false
type t4 = {hi: "hello", getA: number, getB: number} extends
LazyInputSimplified? true: false // = true
```

- `t1` is `true`, as empty object type is a valid `LazyInputSimplified`
- `t2` is `true`, as it defined extra keys than `LazyInputSimplified`
- `t3` is `false`, as it violated `LazyInputSimplified` constraint in `{getA:
boolean}`
- `t4` is `true`, as it defines extra keys than `{getA: 3, getB: 4}` , a valid
`LazyInputSimplified`

Now we know why `type test = {hello: number} extends LazyInput ? true: false //
= true` was true. It did extend `LazyInput` by defining an extra key `hello`.
We did not define in `LazyInput` what extra keys should look like. Let's look
how we can disallow extra keys altogether!

### Restricting the Generic Parameter

Now, we can frame our problem statement more abstractly. We have two types `T`
and `U`, where `T` is a subtype of `U` (that is, `T extends U`). Remember that
`T` can define extra properties than `U`, now we want to get rid of them. To do
that, we should:

- Loop over all keys `k` of `T`
- If `k` is a key of `U` also, we proceed to compare the value types
 - If `U[k]` is a subtype of `T[k]`, we keep it as it is
 - otherwise, we set `T[k]` to be `never` so that we disallow any values there
- Otherwise, we set the value to `never` so that we disallow all the values to
key `k`.

This way, we keep all the keys `k`, but set the values to `never`, so that any
assignment to that key results in a type error. The type implementing this
logic now looks as:

```typescript
type Restrict<T extends U, U> = {
 [Key in keyof T]: Key extends keyof U
 ? U[Key] extends T[Key]
 ? T[Key]
 : never
 : never
}
```

With this type, let's validate our assumptions on `LazyInputSimplified`:

```typescript
type t1 = Restrict<{}, LazyInputSimplified>
// type t1 = {}
type t2 = Restrict<{hi: "hello"}, LazyInputSimplified>
// type t2 = {hi: never}
type t3 = Restrict<{hi: "hello", getA: false}, LazyInputSimplified>
// Error, violates generic constraint
type t4 = Restrict<{hi: "hello", getA: number, getB: number},
LazyInputSimplified>
// type t4 = { hi: never; getA: number; getB: number; }

```

As we see from the above examples, we get `never` on incompatible properties,
thus preventing them from assigning. Now, we can redefine `Lazy` using new
`Restrict` utility:

```typescript
type Lazy<T extends LazyInput> = T extends Restrict<
 T,
 LazyInput
>
 ? {
 [K in keyof T as PropertyName<
 K & GetterName
 >]: ReturnType<T[K & GetterName]>
 }
 : never
```

The core part of the type is still the same. On top of that, we check if `T`
contains any restricted keys, returning `never` if so. With that, our first
example becomes,

```typescript
type t1 = Lazy<{}> // = {}
type t2 = Lazy<{
 getX: () => number,
 getLongProperty: () => string
}> // = {x: number, longProperty: string}
type t3 = Lazy<{
 hello: number
}> // = never
```

Which is now cool, as it says we can't have incompatible keys on the `Lazy`
type. However, we still have to enforce it at the compile time, which we now do
at the function definition part. Finally, we are out of the TS rabbithole
:tada:.

## The new `lazy()`

Now, let's begin the fun part of implementing the actual `lazy` method using
Proxies. Rememember, we need to

- Take a property name
- Convert it to getter name
- Call the getter function, cache the result and return it.

To do this, let's start by implementing a function to get the getter name given
the property name.

```typescript
const getterName = (prop: string) => {
 const [first, ...rest] = prop.split("")
 return ["get", first!.toUpperCase(), ...rest].join(
 ""
 ) as GetterName
}
```

Then, we can proceed to implement the `lazy()` method as,

```typescript
const lazy = <T extends Restrict<T, LazyInput>>(
 object: T
) => {
 const cache = {} as any
 return new Proxy(object, {
 get(target, prop) {
 if (!(prop in cache)) {
 // Forward symbol | number to original object
 if (typeof prop !== "string")
 cache[prop] = target[prop as keyof T]
 else {
 const getter = (target as LazyInput)[getterName(prop)]
 cache[prop] = getter ? getter() : undefined
 }
 }
 return cache[prop]
 }
 }) as any as Lazy<T>
}

```

Some notes here:

- We can not return `Proxy` as `Lazy` as we did earlier, as the keys are also
different. Since we are sure about this, we do `as any as Lazy<T>`
- We also type `cache` as `any` where we could have used `Lazy<T>`. This is
simply to save ourselves from lots of type castings throughout!
- We choose to forward `number | symbol` accesses to original object as it is,
while doing so, we have to convert `prop` to `keyof T` as we are not doing any
modifications to key here.
- To get the type-safe version of `getter`, we are first casting `target` to
`LazyInput` (which is `Restrict<T, LazyInput>`)

Phew, that's lots of TS manipulations even when we are writing the logic part.
Anyway, let's go ahead and test the `lazy()` method we just implemented:

```typescript
const logAndReturn = <T>(arg: T) => {
 console.log("Logging", arg)
 return arg
}

const object = {
 getBoolean: () => logAndReturn(true),
 getString: () => logAndReturn("hello world"),
 getNumber: () => logAndReturn(21),
}

const lazyObject = lazy(object)
console.log(lazyObject.boolean) // Logging true, true
console.log(lazyObject.boolean) // true

console.log({ ...lazyObject }) // {getBoolean: undefined, ...}
```

Our results are bittersweet:

- We got successful type inference for the type of the `lazyObject`
- Caching is also working, we are seeing the `logAndReturn` call for only the
first access
- _However_, destructuring and logging the object does not seem to get hang of
new property names yet...

Let's go ahead and fix the problem for good.

## Enter the JS Rabbithole

When we are overriding `get` method in the `Proxy` handler, we want to somehow
tell the JS that we have new keys. Fortunately, we can also override the
`ownKeys` method to return the new keys. To do that, we

- First, get the key names from underlying object. Remember, they are getters.
- Map them to property names, let's write an utility function for that.

```typescript
const propertyName = <T extends GetterName>(
 getterName: T
): PropertyName<T> => {
 const [first, ...r] = getterName.slice(3).split("")
 return [first!.toLowerCase(), ...r].join(
 ""
 ) as PropertyName<T>
}
```

Let's override the `ownKeys` method also:

```typescript
const lazy = <T extends Restrict<T, LazyInput>>(
 object: T
) => {
 // same as before
 return new Proxy(object, {
 // same as before
 ownKeys(target) {
 return Object.keys(target)
 .map((getter) => propertyName(getter as GetterName))
 },
 })
}
```

Then, in the driver,

```typescript
const object = // same as before
const lazyObject = // same as before
console.log(Object.keys(lazyObject)) // []
```

Wow, it is still `[]` :cry:. Without `ownKeys` override though, it was
`["getBoolean", "getString", "getNumber"]`. We are still better than that, but
have to make our new keys visible to JS. Turns out that we have to make them
`enumerable` and `configurable`. To do this, we can override the
`getOwnPropertyDescriptor` of the proxy handler:

```typescript
const lazy = <T extends Restrict<T, LazyInput>>(
 object: T
) => {
 // same as before
 return new Proxy(object, {
 // same as before
 getOwnPropertyDescriptor() {
 return {
 enumerable: true,
 configurable: true,
 }
 },
 })
}
```

With this, let's go to the driver again:

```typescript
const object = // same as before
const lazyObject = // same as before
console.log(Object.keys(lazyObject)) // ["boolean", "string", "number"]
// Destruct to force evaluate all keys
console.log({ ...lazyObject }) // {boolean: true, string: "hello world",
number: 21}

```

Nice, we are seeing the expected things now :smile:.

## Closing thoughts

With all the back and forth with the TS and JS Proxies, we were able to
implement a version of `lazy` that does lazy evaluation and caching along with
key mapping. If we pass in an non-matching key to the argument of `lazy`, it is
an runtime error as well as compile time one.

```typescript
const object = {
 getBoolean: () => logAndReturn(true),
 getString: () => logAndReturn("hello world"),
 getNumber: () => logAndReturn(21),
 x: 33,
}

// error here: Type 'number' is not assignable to type 'never'.
const lazyObject = lazy(object)
console.log(lazyObject.boolean)
console.log(lazyObject.boolean)
// Runtime error here, calling ownKeys, error in propertyName
console.log(Object.keys(lazyObject))

console.log({ ...lazyObject })
```

The logic in type domain and the runtime is tightly coupled. This is not
without limitations, though. Not passing an object literal to `lazy` might not
give you all the benifits, and type errors are hard to debug many times.
However, this shows the power of what TS can do at type level. As usual, I
dread writing of `PropertyName` type and `propertyName` function having same
logic -- this is unavoidable as of now, until we have (can we?) a TS runtime
with access to types.

## Get your hands Dirty!

Try this code sample in
[CodeSandbox](https://codesandbox.io/s/optimistic-elbakyan-fkjo9t?file=/src/index.ts).

## Caveats

- For a `Record<string, string>`, although we have specified string keys, we
can have number keys. This is due to JS internally converting number index
access to string access.

 - ```typescript
 const x: Record<string, string> = {2: "Hi"} // valid
 ```

- The `Restrict<>` type seems incomplete, for example, a call to `lazy(2)` is
still valid, though I didn't understand why this works yet. Maybe a TS Bug?
:thinking:

- Anything else, you tell me!

## Credits

- [Type Level Programming in
TypeScript](https://www.youtube.com/watch?v=vGVvJuazs84&ab_channel=GabrielVergnaud)
-- an excellent talk where I got ideas about thinking about types as sets, and
constructs on them
- [Type
Compatibility](https://www.typescriptlang.org/docs/handbook/type-compatibility.html)
-- the docs page where I'm _still_ figuring out the assignability

## PS

There can be some typos in this writing, I was too lazy to spell check the
article since writing it took an entire day. Hope you understand me
:upside_down_face:

Implementing Lazy access in TS with Key Mapping

Lazy is hard

Recently, I came across a use case in TypeScript where the properties of an
object were expensive to calculate, and they were not used that frequently. In
summary, the function looked like this:

```typescript
const doHeavyComputation = (args) => {
 // do something with args
 return value
}

const aFunctionReturningObject = () => {
 // Do some preparation
 return {
 property1: doHeavyComputation(args1),
 property2: doHeavyComputation(args2),
 // other properties
 }
}
```

And, its usage looked somewhat like this

```typescript
const user = () => {
 const returnedObject = aFunctionReturningObject()
 if (returnedObject.property1 === someValue) {
 // do Something with that value
 return
 } else if (
 returnedObject.property2 === otherValue ||
 returnedObject.property3
 ) {
 // do some other computation
 return
 }
}
```

As you can see, calculating `property2` and `property3` is unnecessary if the
`property1` equals `someValue`. However, we know this only at the time of
usage, which does not help us in any way to optimize during the computation. In
this short article, let's see how we can approach this problem with the
constraints of:

- Doing minimal modifications at the computation side
(`aFunctionReturningObject`)
- Doing **no** modification at the site of usage
- Keeping the usages type-safe and syntactically clean (ideally, same as what
we have written earlier)

## Become Lazy

Turns out that this problem is pretty common in Computer Science and its
solution is called **[Lazy
Evaluation](https://en.wikipedia.org/wiki/Lazy_evaluation#:~:text=In%20programming%20language%20theory%2C%20lazy,avoids%20repeated%20evaluations%20(sharing).)**.
Lazy Evaluation is the method of evaluating an expression at the site of its
usage, rather than the declaration. In our function `aFunctionReturningObject`,
we are calculating `property1`, `property2`, ... during its declaration. With
lazy evaluation in place, they would be calculated when they are used (Here, in
`user`, during `returnedObject.property1`).

While Haskell uses Lazy Evaluation by default, and some languages like Kotlin
and Scala provide the `lazy` keyword to achieve the same, we don't have such an
option in JS. However, instead of _returning a value_, if we _return a function
that returns the value_, the value would not get calculated until the function
is called. In our case, it would look something like this:

```typescript
const aFunctionReturningObject = () => {
 // Do some preparation
 return {
 property1: () => doHeavyComputation(args1),
 property2: () => doHeavyComputation(args2),
 // other properties
 }
}

const user = () => {
 const returnedObject = aFunctionReturningObject()
 if (returnedObject.property1() === someValue) {
 // do Something with that value
 return
 } else if (
 returnedObject.property2() === otherValue ||
 returnedObject.property3()
 ) {
 // do some other computation
 return
 }
}
```

However, this solution has a few problems:

- The property access is now a function call, which requires modifications at
the user side
- Evaluating the same property twice, would require evaluating
`doHeavyComputation()` twice, which might not be ideal if the heavy computation
is returning the same values every time

## Lazy Pro

Let's address these problems by writing a function `lazy`, that would take in
an object with values in the format `() => any`, and make the property access a
regular one instead of a function call. It will also take care of caching the
evaluations.

To address the cases of intercepting the property access of an object and
redefining its behavior, JS provides a mechanism named
[Proxy](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy).
We can wrap our original object with the proxy, along with handlers to redefine
its behavior. Let's look into the implementation of `lazy` using `Proxy`.

```typescript
const lazy = <T extends {}>(object: T) => {
 const cache = {} as T
 return new Proxy(object, {
 get(target, prop) {
 const key = prop as keyof T
 if (!(key in cache)) {
 const value = target[key]
 cache[key] = value ? value() : undefined
 }
 return cache[key]
 },
 })
}
```

The logic is straightforward albeit with an error:

- We take in a type `T` that extends `{}`, as Proxy requires an object as its
input
- In `Proxy`, we intercept the `get` method, which corresponds to all property
accesses
- The `get` method takes `target` (in our case, same as `object`) and the
`prop`, the property being accessed as parameters
- `prop` can be any `string | symbol`, we cast it to `keyof T` as `key`.
- We check if `key` is the `cache`, if it isn't, we access it from `target`,
check if it is not undefined, and evaluate it
- Finally, we return `cache[key]`

TS screams us with an error that we can not use `value()`, as it is `unknown`.
This is true, as we don't know the values of the type `{}` in advance. We need
to restrict ourselves to a subtype of `{}`, that has `() => any` as values.

```typescript
type LazyInput = {
 [Key: string]: () => any
}
```

Then, we can modify the signature of `lazy` as,

```typescript
const lazy = <T extends LazyInput>(object: T) => {
 // same as before
}
```

Cool! the `lazy` function is free of type errors now. Let's check it in action.
Before that, to check if caching is working, let's define a `logAndReturn`
function that does exactly what it is named :smile:

```typescript
const logAndReturn = <T>(value: T) => {
 console.log("Logging", value)
 return value
}
```

Then, let's prepare input for `lazy`

```typescript
const input = {
 intValue: () => logAndReturn(2),
 stringValue: () => logAndReturn("string")
}
```

Let's call `lazy` now!

```typescript
const lazyOutput = lazy(input)
```

Let's log its values

```typescript
console.log(lazyOutput.intValue) // "Logging 2", 2
console.log(lazyOutput.intValue) // 2
console.log(lazyOutput.intValue) // 2
console.log(lazyOutput.stringValue) // "Logging string", string
console.log(lazyOutput.stringValue) // string
```

The log looks nice, on the first property access, it called `logAndReturn` and
subsequent accesses used the value from the cache. However, most of the use
cases would not be to log the value, rather than to use it. Let's try that as
well:

```typescript
const addOne = (x: number) => x + 1
const addPrefix = (prefix: string, value: string) =>
 `${prefix}${value}`

const intResult = addOne(lazyOutput.intValue)
const stringResult = addPrefix(
 "result:",
 lazyOutput.stringValue
)

console.log({ intResult, stringResult })
```

Ah, now we see an error in calls to `addOne`!

```
Argument of type '() => number' is not assignable to parameter of type 'number'.
Argument of type '() => string' is not assignable to parameter of type 'string'.
```

It turns out that the type of `lazyOutput` is still

```typescript
type LazyOutput = typeof lazyOutput
/*
type LazyOutput = {
 intValue: () => number;
 stringValue: () => string;
}
*/
```

with the values being functions. But, we want their return types instead! We
can fix that by creating a type `Lazy<T>`

```typescript
type Lazy<T extends LazyInput> = {
 [Key in keyof T]: ReturnType<T[Key]>
}
```

Note that `ReturnType` is a TS inbuilt type, that returns the return type of a
function. Now, we modify the `lazy` function as:

```typescript
const lazy = <T extends LazyInput>(object: T) => {
 const cache = {} as T
 return new Proxy(object, {
 // same as before
 }) as Lazy<T>
}
```

With this, the errors in the call of `addOne` are now gone, and the logic in
the type domain matches that of the execution domain. The `LazyOutput` type is
now,

```typescript
type LazyOutput = typeof lazyOutput
/*
type LazyOutput = {
 intValue: number;
 stringValue: string;
}
*/
```

Which is what we expected. Now, we can return to our initial example and can
rewrite it as:

```typescript
const aFunctionReturningObject = () => {
 // Do some preparation
 return lazy({
 property1: () => doHeavyComputation(args1),
 property2: () => doHeavyComputation(args2),
 // other properties
 })
}

const user = () => {
 const returnedObject = aFunctionReturningObject()
 if (returnedObject.property1 === someValue) {
 // do Something with that value
 return
 } else if (
 returnedObject.property2 === otherValue ||
 returnedObject.property3
 ) {
 // do some other computation
 return
 }
}
```

In this code, we wrapped the return value with `lazy` and deferred the
computations of the values by putting them inside a function. The `lazy`
function did not require us to change the `user` function at all, which is very
good, as we don't know from how many places `aFunctionReturningObject` would be
called, especially if it is in a library.

## Try it Yourself!

The code for this article is available on
[CodeSandbox](https://codesandbox.io/s/little-grass-8qbn70). Feel free to fork
it out and play with it.

## Lazy Pro Max

Some of you might object to the key naming in the argument of `lazy`. Since we
now have a function as a value, it would be rather good to name `property1` as
`getProperty1`, which would make things more clear. We can hand off the
responsibility of key renaming to `Proxy`, which would convert the access of
`property1` to `getProperty1` call.

This is a valid objection. However, implementing these rules out that _Lazy is
Easy_, thus, I will cover it in another article _Lazy is Hard_ :sweat_smile:.

Evaluating lazy helps to save computation power. But, how we can implement one?

Lazy is Easy

The variable naming conventions differ across tools and programming languages.
While the database column names mostly use `snake_case`, the APIs and data
models prefer to use `camelCase`. Most of the ORMs such as Hibernate and
TypeORM do this conversion automatically, making the process seamless. However,
I recently came across a use case in TypeScript where this conversion was
manual, we had to retain the type information.

> **The problem**: Given an object having the keys in `camelCase` format,
convert them to `snake_case` **retaining Type Information after the
conversion**. 

While it is easy to convert a string to `snake_case`, retaining the type
information is the key here. Let's start with a simple, recursive (you will
know in a short while why we are doing recursion :smile:) function to convert a
string from `camelCase` to `snake_case`.

```typescript
const isUpperCase = (str: string) => str === str.toUpperCase()

const stringToSnakeCase = (str: string): string => {
 const [first, ...rest] = str
 const restString = rest.join("")
 return str
 ? isUpperCase(first)
 ? `_${first.toLowerCase()}${stringToSnakeCase(
 restString
 )}`
 : `${first}${stringToSnakeCase(restString)}`
 : str
}

```

The function is pretty simple:

- First, we destruct the string into `first` and `restString` (`rest` is an
array, we join it to get the string)
- If the first character is an upper case character, we insert `_` in the
beginning, convert it into lower case and proceed to convert the rest of the
string. 
- Otherwise, we keep the first letter as it is, and convert the rest of the
string. 
- At the end, the string would be empty, and we return it.

> **Note**: We have to manually add the return type to the function as
`string`, since [TS can not infer return types of the recursions
yet](https://github.com/Microsoft/TypeScript/issues/3336).

Using this function, we can write a function to convert the keys of an object
to `snake_case` recursively.

```typescript
type Value = object | string | number | boolean

const _snakify = (obj: Value): Value =>
 typeof obj === "object"
 ? Object.fromEntries(
 Object.entries(obj).map(([key, value]) => [
 stringToSnakeCase(key),
 _snakify(value),
 ])
 )
 : obj
```

Let's add a wrapper to this function to take in and return an `object`.

```typescript
const snakify = (obj: object): object => _snakify(obj)
```

To test this function, we declare a constant value and check if the function
converts its keys to the snake case correctly.

```typescript
const obj = {
 hiWorld: 2,
 helloWorld: {
 hiTest: "test",
 hiTestTwo: {
 threeFour: false,
 },
 },
}

const snakified = snakify(obj)

console.log(snakified)
```

The output is,

```typescript
{
 hi_world: 2,
 hello_world: { hi_test: "test", hi_test_two: { three_four: false } }
}
```

This looks fantastic. However, if we want to access any property of the
`snakified`, TypeScript gives us an error, as it had become the plain `object`.

## Adding Types

Now it is the fun part -- we want to add types to the `snakifyObject` function
to retain the information about the keys. In the end, we should be able to
access `snakifiedObject.hi_world`, for example, with no errors.

To begin with, let's define a `type` to convert a `string` to `snake_case`. 

```typescript
type StringToSnakeCase<Str extends string> =
 Str extends `${infer First}${infer Rest}`
 ? First extends Uppercase<First>
 ? `_${Lowercase<First>}${StringToSnakeCase<Rest>}`
 : `${First}${StringToSnakeCase<Rest>}`
 : Str

```

This type definition looks similar to the definition of our `stringToSnakeCase`
function, pretty much the same thing is happening here but at the type level.
Let's break it down:

- First, we destruct the string type `T` into `First` and `Rest` using string
literal syntax and `infer` keyword.
- TypeScript resolves `First` to be a character and `Rest` to be a string
(btw, both are type `string` in TypeScript).
- Then, we do conditional to check if `First` is uppercase, and add `_` to the
string based on that.
- If such destruction can not happen (when `Str` is empty), we return `Str`
itself.

Now, we can use this type as,

```typescript
type Test1 = StringToSnakeCase<"helloWorld"> 
// type Test1 = "hello_world"

type Test2 = StringToSnakeCase<"LongListOfWords"> 
// type Test2 == "_long_list_of_words"

type Test3 = StringToSnakeCase<3.21>
// Error: Type 'number' does not satisfy the constraint 'string'
```

We can proceed to define the type `Snakify<T>` to convert the keys of the
object to `snake_case`.

```typescript
type Snakify<T> = T extends object
 ? {
 [K in keyof T as StringToSnakeCase<
 K & string
 >]: Snakify<T[K]>
 }
 : T
```

Here, we check if `T` is a subtype of `object` -- if it is, we map its `string`
keys using `StringToSnakeCase`, and recursively `Snakify` its values; else, we
return the type `T` itself (this will happen at the end of recursion).

Using this type, we can now annotate `snakify` function as,

```typescript
const snakifyObject = <T extends object>(obj: T) =>
 _snakify(obj) as Snakify<T>

```

Now, the return types of the `snakified` object in the above example are
inferred correctly as :tada:

```typescript
type Result = typeof snakified
/*
type Result = {
 hi_world: number;
 hello_world: {
 hi_test: string;
 hi_test_two: {
 three_four: boolean;
 };
 };
}
*/
```

## Caveats

If there is an `_` already present in the key, `isUpperCase()` in the code and
`T extends UpperCase<T>` test in the type will pass and we get another `_`. For
example,

```typescript
const obj = {
 hiWorld: 2,
 hi_world: "Hello"
}

const snakified = snakify(obj)

type Result = typeof snakified
/*
type Result = {
 hi_world: number;
 hi__world: string;
}
*/

console.log(snakified)
```

The output would be,

```typescript
{ hi_world: 2, hi__world: "Hello" }
```

While we can fix the code by updating `isUpperCase` function as,

```typescript
const isUpperCase = (str: string) =>
 str === str.toUpperCase() &&
 str !== str.toLocaleLowerCase()
```

to fix the type, first, we need to define a type `IsAlpha<T>` returning `T` if
`T` is an alphabet. We can do this as,

```typescript
type IsAlpha<T extends string> = T extends Uppercase<T> &
 Lowercase<T>
 ? never
 : T
```

The key point here is that, for a non-alphabet, its uppercase and lowercase
would be equal to itself, returning the `never` type. Now, we can add this chek
to `StringToSnakeCase` type as,

```typescript
type StringToSnakeCase<T extends string> =
 T extends `${infer First}${infer Rest}`
 ? First extends IsAlpha<First> & Uppercase<First>
 ? `_${Lowercase<First>}${StringToSnakeCase<Rest>}`
 : `${First}${StringToSnakeCase<Rest>}`
 : T

```

Then, the `type Result` would be equal to,

```typescript
type Result = {
 hi_world: string | number;
}
```

which fixes the issue.

## Conclusion and Thoughts

I never thought that TypeScript's string literal types and pattern matching are
useful in real-world scenarios until I came across this use-case :sweat_smile:.
They are undoubtedly powerful type tools in the TypeScript programmer's
arsenal. In a similar spirit to this use case, we can write utilities and
associated types to convert strings across various conventions.

One thing I really thought about was code-duplication. Across this example, we
wrote similar logic both in code and type, achieving the same things. Since the
type information does not exist at all during runtime, unfortunately, this
seems to be the only way. I'm really curious about how problems similar to
this, that involve type manipulation and mapping are handled in other
programming languages; please let me know if you come across such situations
before!

How I solved typing object key mapping  in TypeScript

The Literal Magic


While exploring Scala, one of the features that caught my eyes was that
functions can accept two sets of arguments. Typically, we write `function
name(argumentList)` in most languages. Scala goes one step ahead and allows
`function name(argumentList)(implicitArgumentList)`. In this article, we shall
explore implicit arguments from a Scala-newbie perspective. meaning that we
won't dig deep into architectural patterns and theoretical implications,
rather, we explore them from a syntax and syntax-sugar perspective.

## A First Dive

Let's write a function to concatenate a list of strings with a separator. As a
good first step, we write this as a function accepting two arguments and a
return value

```scala
def concatStrings(strings: Seq[String], separator: String): String = {
  strings.foldRight("") {_ + separator + _}
}
```

Several things are going on here, let's break them down one by one:

- Scala uses `[]` to denote generics; thus, `Seq[String]` is a `Seq` of
`String`s

- The return value is the value of the last line of the function; writing
`return` is optional

- `foldRight` is similar to `reduce` function in JS, its first argument is the
initial value `""`

- `foldRight` takes a reducer function as a
[_curried_](https://docs.scala-lang.org/tour/multiple-parameter-lists.html)
argument

- The reducer function is of type `(String, B) => B`, The type `B` comes from
the initial value, which is also `String` now

- The funny `_`s are a shorthand to access function arguments; the First `_` is
of type `String`, and the second `_` is of type `B` (= `String`). You can't
write a third `_`,  that's an error :sweat_smile:

- `{...}` is a shorthand for the funciton body, otherwise you would have
written `strings.foldRight("")((a, b) => a + separator + b)`; Notice two set of
arguments `foldRight()()`, don't worry, we will address it right away :smile:

Then, we invoke this function and get a return value from anywhere.

```scala
def main(args: Array[String]): Unit = {
  val strings = Seq("Hello", "World")
  val result = concatStrings(strings, "+")
  assert(result == "Hello+World+") // The extra + is because of foldRight
}
```

## Hello Implicit!

Let's make our function `concatStrings` to accept an implicit `separator`
argument. The function body looks similar to the previous case, excpet the
argument list:

```scala
def concatStrings(strings: Seq[String])(implicit separator: String): String = {
  strings.foldRight("")((a, b) => a + separator + b)
}
```

The second set of arguments begins with the `implicit` keyword and marks them
as _implicit_. The implicit arguments are inferred from the scope at
compile-time and passed to the function as an argument. To make it clear, let's
look at the invocation of our new `concatStrings` function.

```scala
def main(args: Array[String]): Unit = {
  val strings = Seq("Hello", "World")
  implicit val delimiter: String = "+"
  val result = concatStrings(strings)
  assert(result == "Hello+World+")
}
```

Here, we are creating a constant named `delimiter` of type `String`, which is
also declared as `implicit`. When the compiler encounters the call to
`concatStrings`, it looks for values of type `String` in the scope, which is
marked as `implicit`. In this case, it is the constant `delimiter`. Then, it is
passed as an argument to the function.

Let's play with implicit arguments by modifying the code.

### Missing Implicit

If the compiler can not find `implicit` values in the scope, it is an error.

```scala
def main(args: Array[String]): Unit = {
  val strings = Seq("Hello", "World")
  val result = concatStrings(strings)
  assert(result == "Hello+World+")
}
```

Here, the compiler complains with a readable, understandable error

```
could not find implicit value for parameter separator: String
	val result = concatStrings(strings)
```

### Multiple Implicits

If we have multiple `implicit` values of the same type, that is also an error.

```scala
def main(args: Array[String]): Unit = {
  val strings = Seq("Hello", "World")
  implicit val delimiter: String = "+"
  implicit val anotherDelimiter: String = "+"
  val result = concatStrings(strings)
  assert(result == "Hello+World+")
}
```

The error is

```
ambiguous implicit values:
 both value anotherDelimiter of type String
 and value delimiter of type String
 match expected type String
    val result = concatStrings(strings)
```

### Explicitly Implicit

We can also explicitly pass the `implicit` argument, as we did in `foldRight()`

```scala
def main(args: Array[String]): Unit = {
  val strings = Seq("Hello", "World")
  val result = concatStrings(strings)("+") // Notice the second argument list
  assert(result == "Hello+World+")
}
```

### Implicit from Upper Scope

Implicit values can be from upper scope as well:

```scala
object ImplicitHelper {
  def concatStrings(): String = {
    // Same as before
  }
}

object Main {
  implicit val delimiter: String = "+"
  
  def main(args: Array[String]): Unit = {
    val strings = Seq("Hello", "World")
    val result = ImplicitHelper.concatStrings(strings)
    assert(result == "Hello+World+")
  }
}
```

Here, the `object ...` is a singleton object, which can define methods as well.
Think of it as a `static` method. When we have implicit values present in
multiple scopes, the closest one is picked first (as expected :grin:).

```scala
object ImplicitHelper {} // Same as before

object Main {
  implicit val delimiter: String = "+"

  def main(args: Array[String]): Unit = {
    val strings = Seq("Hello", "World")
    implicit val delimiter: String = "_" // This one is picked
    val result = ImplicitHelper.concatStrings(strings)
    assert(result == "Hello_World_")
  }
}
```

Implicits can flow in from `import` as well, suppose we have an `object` named
`Config` in the file `Config`

```scala
// Config.scala
object Config {
  implicit val delimiter: String = ","
}
```

In our `Main.scala` (in which we write the `main` function),

```scala
// Main.scala
import Config._

object ImplicitHelper {} // Same as before

object Main {
  def main(args: Array[String]): Unit = {
    val strings = Seq("Hello", "World")
    val result = ImplicitHelper.concatStrings(strings)
    assert(result == "Hello,World,")
  }
}
```

Since we imported every member of the object `Config`, the compiler can match
the `delimiter` to the argument of `contactStrings`

## Viewing Implicit Arguments

Since the implicit arguments can be inferred directly from the code without
analyzing all the imports in all scopes, debugging others' code can be hard.
Fortunately, IntelliJ has an option to _explicitly_ show the `implicit`
variables :tada:. Just enable it from `View -> Show Implicit Hints` (you might
need Scala Plugin to do this). With this option enabled, the editor window
looks like this,

![IntelliJ Idea showing implicit
argument](https://media.graphassets.com/GSdBIKACSz2g9SW9r5ai)

Now you can follow the definition of the value as usual.

## Implicit Conversions

Since functions are the first-class citizens of Scala, they can be an
`implicit` argument as well, here is a sample:

```scala
import scala.language.implicitConversions

object ImplicitHelper {
  def toInt[A](argument: A)(implicit function: (A) => Int): Int =
function(argument)
}

object Main {
  implicit def length(string: String): Int = string.length

  def main(args: Array[String]): Unit = {
    val string = "Hii"
    val result = ImplicitHelper.toInt(string)
    assert(result == 3)
  }
}
```

Here, the object method `Main.length` is the implicit parameter to the `toInt`
function. Note that we also needed to import
`scala.language.implicitConversions`. To quote the [Scala
Docs](https://docs.scala-lang.org/tour/implicit-conversions.html), we need
this,

> Because implicit conversions can have pitfalls if used indiscriminately the
compiler warns when compiling the implicit conversion definition.

So, we are making it explicit that we are using implicit conversions :smile:
(BTW, conversion = function).

## Implicitly Implicit :sunglasses:

We have already seen that we can bring in `implicit` values in the scope using
`import`. If we can have multiple `implicit` values defined in the imported
classes, we have some interesting effects.

Let's take the earlier example of `concatStrings()`. We have a `DefaultConfig`
object setting delimiter to `","`, and a `SpecialConfig` object setting it to
`"+"`. Let's also have these files:

1. `ImplicitHelper.scala` -- Defines `concatStrings` method
2. `Main.scala` -- Invokes the `contactStrings` method
3. `DefaultConfig.scala`, and
4. `SpecialConfig.scala` -- as defined earlier

Now, 

```scala
// DefaultConfig.scala
object DefaultConfig {
  implicit val delimiter: String = ","
}
```

And,

```scala
// SpecialConfig.scala
object SpecialConfig {
  implicit val delimiter: String = "+"
}
```

And,

```scala
// ImplicitHelper.scala
object ImplicitHelper {} // As before
```

At last,

```scala
import ImplicitHelper._
import DefaultConfig._

object Main {
  def main(args: Array[String]): Unit = {
    val strings = Seq("Hello", "World")
    val result = concatStrings(strings)
    assert(result == "Hello,World,")
  }
}
```

Since we `import DefaultConfig._`, the delimiter is now `","`. We can quickly
swap that for `SpecialConfig._` to have a different delimiter.

```scala
import ImplicitHelper._
import SpecialConfig._

object Main {
  def main(args: Array[String]): Unit = {
    val strings = Seq("Hello", "World")
    val result = concatStrings(strings)
    assert(result == "Hello+World+")
  }
}	
```

  And, the import order does not matter.

## Concluding Thoughts

The `implicit` arguments, especially through `import` provide a nice way to
implement composability and dependency injection patterns. Thanks to the
tooling, we still get the benefits of following arguments to their definitions.
Sometimes, like in implicit conversions, we still need to be careful not to
misuse them.


Scala's functions are different since they accept two set of arguments as opposed to one in many programming languages. What does it serve? Let's look into it...

The Implicit Arguments

Static Site Generation (SSG) is great. It allows faster response time to users
and enhanced SEO capabilities. I use SSG for all my websites, including an
[e-commerce](https://benakastores.com) store that [I built](/blog/no-code-code)
for our family business. The website uses [NextJS](https://nextjs.org/) and
[Hygraph](https://hygraph.com/). While everything ran great with SSG, the
ongoing inflation triggered some problems by frequently varying prices. To
update the price of an item, this was my workflow:

1. Update the price in Hygraph
2. Publish the content
3. Trigger static site rebuild on Vercel

Since the price change only affected a few pages, rebuilding the entire website
is an unnecessary task. Fortunately, [Incremental Static Regeneration
(ISR)](https://nextjs.org/docs/basic-features/data-fetching/incremental-static-regeneration)
comes to our rescue. ISR can be triggered either periodically, or, on-demand.
In this article, I will go through my idea of the solution and how I
implemented it.

## The Idea

My idea was to do every step of the process in a single place. It would be
Hygraph, as the price editing happens there. I imagined the following workflow:

1. Update the price in Hygraph
2. Publish the content
3. Click a button somewhere in the Hygraph content editor to trigger ISR on the
pages affected

Thanks to the Hygraph [UI Extensions](https://hygraph.com/docs/ui-extensions)
(UIX, currently in preview), which enhances its UI functionality, this
workflow was possible. I decided to use them with the following mechanism:

- The website provides a `POST` endpoint (say `/api/revalidate`) to trigger ISR
- The UIX provides a button, on clicking which, calls the endpoint with the
current object data in the body

### Building the Hygraph UIX

Before building the UIX, I doubted whether it can access the current object
being edited.
[@cadudecastroalves](https://www.linkedin.com/in/cadudecastroalves/) from
Hygraph Community cleared my doubt and pointed me to the documentation.

Hygraph UIX is essentially a React application, which gets rendered inside the
Hygraph editor UI. I initially bootstrapped the project with
[Vite](https://vitejs.dev/) and found that Hygraph could not connect to the
extension in the development mode. Since it was a known problem, I decided to
go with [CRA](https://create-react-app.dev/).

To get started, I just had to install
[@hygraph/uix-react-sdk](https://www.npmjs.com/package/@hygraph/uix-react-sdk)
package and import `Wrapper` and `useFormExtension` from it. Using them, the
UIX skeleton would look like,

```tsx
import {
 Wrapper,
 useFormSidebarExtension,
 FormSidebarExtensionDeclaration,
} from "@hygraph/uix-react-sdk"

const declaration: FormSidebarExtensionDeclaration = {
 name: "Name your UIX here",
 description: "Describe what your UIX does",
 extensionType: "formSidebar", // This extenstion will appear at the sidebar
 config: {
 // Declare any user-defined values that you are going to need in your UIX
 },
}

function SidebarComponent() {
 // form contains the state of the current object being edited in the Hygraph
 // extenstion contains UIX metadata, along with config values
 // model contains the information about the model of which current object is
an instance of
 const { form, extension, model } =
 useFormSidebarExtension()
 // Do something useful using the form and extension variables
 // You can use useEffect() with form as a dependency to listen to changes
 return Your UI goes here!
}

function App() {
 return (
 <Wrapper declaration={declaration}>
 <SidebarComponent />
 </Wrapper>
 )
}

export default App
```

For my use case, I had the following config variables:

- `ENDPOINT` -- The endpoint on which to trigger `POST`
- `API_KEY` -- The secret to be added in the `Authorization` header

I also added several state variables to the skeleton:

- A boolean state variable `dirty` to check if the form had been modified

- An object state variable `values` to store the value of the current object
being edited in Hygraph

- An effect depending on the `form` to update `dirty` and `values`

- A callback to call `ENDPOINT`

- A boolean state variable `loading` set true if an API call is in progress

- A string state variable `message` to notify the user of the result of the API
call.

The UI was very simple, containing a button to trigger the callback, and a
paragraph to display the `message`. In the end, the UIX config looked like
this:


![image-20220522165055642.png](https://media.graphassets.com/AAI8aBp6S3mbfIzUJ5rf)

The UIX UI was simple:


![image-20220522165125043.png](https://media.graphassets.com/8ZxAKJWgSdazkDgFORAO)

The UIX source is available at
[GitHub](https://github.com/vinayakakv/hygraph-uix-trigger-api); you can also
install the extension available at
[hygraph-uix-trigger-api.vercel.app](https://hygraph-uix-trigger-api.vercel.app/)
from Hygraph settings.

### The ISR Handler

With the extension in place, it was time to implement the ISR handler! In
NextJS, the ISR handler is a normal API handler that triggers the`
unstable_revalidate()` function on the response object. This function takes a
path as the input and triggers ISR. For security purposes, it is good to use a
secret as the authentication mechanism, to control the triggers of the API.

Since the UIX sends the whole object being edited, it was necessary to type (at
least a part) of them in the ISR handler. At Hygraph, I had `Product` and
`Category` schemas, each containing a `slug` field and were related two-way. I
also had the `Info` schema, as a key-value pair. The typings looked like this:

```typescript
type HasSlug = { slug: string }

// This is how Hygraph defines relationships
type HygraphReference<T extends string, K> = {
 [key in T]: {
 connect: K[]
 disconnect: K[]
 }
}

type ProductRevalidateData = HasSlug &
 HygraphReference<"categories", HasSlug>

type CategoryRevalidateData = HasSlug &
 HygraphReference<"products", HasSlug>

type RequestData =
 | ({ model: "Product" } & ProductRevalidateData)
 | ({ model: "Category" } & CategoryRevalidateData)
 | { model: "Info" }

```

I also had a `getPaths(req: RequestData): string[]` method to return a list of
all paths to be revalidated. The method would calculate the paths based on the
request data being sent from the Hygraph. With this method in place, the ISR
handler was as simple as:

```typescript
export default async function handler(
 req: NextApiRequest,
 res: NextApiResponse<ResponseData>
) {
 if (req.method === "POST") {
 const authToken =
 req.headers["authorization"]?.split(" ")[1]
 if (authToken !== process.env.REVALIDATION_TOKEN) {
 res.status(401).end()
 return
 }
 const request: RequestData = JSON.parse(req.body)
 const paths = getPaths(request)
 await Promise.all(
 paths.map(path => res.unstable_revalidate(path))
 )
 res.status(200).send({ success: true })
 } else {
 res.status(204).end()
 }
}

```

While testing this code, I spent around half an hour debugging the ISR not
triggering. It was due to missing `await Promise.all()` around
`res.unstable_revalidate()` calls :sweat_smile:.

## Caveats and Possible Improvements

The Hygraph UIX calls the API by using `fetch()`. While doing the API call, the
`Origin` header will be set to the domain at which the UIX is hosted. If the
CORS is not enabled at the API server, the browser will block the request. To
handle this, I had to set CORS headers in the API response as:

```typescript
res
 .setHeader("Access-Control-Allow-Origin", "*")
 .setHeader("Access-Control-Allow-Methods", [
 "POST",
 "OPTIONS",
 ])
 .setHeader("Access-Control-Allow-Headers", [
 "authorization",
 "content-type",
 ])

```

This will set the headers for all HTTP methods, even though the browser will
call only `POST` and `OPTIONS`. In case modifying the server configuration to
enable CORS is not possible, the UIX can use `mode: "no-cors"` in the
`fetch()`, at a cost of making the response
[opaque](https://fetch.spec.whatwg.org/#concept-filtered-response-opaque).

When the Hygraph UIX supports Vite during the development, I plan to migrate
the project to Vite, for a nice DX. Apart from that, the UIX can also have an
option to specify resource paths, when adding to the sidebar of a model. Feel
free to open a PR if that is the use case that you need!

## Thanks

[@cadudecastroalves](https://www.linkedin.com/in/cadudecastroalves/) for
helping me to get started on Hygraph UIX

How I used NextJS' Incremental Static Regeneration in our static e-commerce website and built a GraphCMS extension to trigger it

Exploring NextJS ISR

B2B companies are often faced with the problem of data migraion during customer
onboarding. Whilst the company has a standard data models defined, they can not
expect the customers to bring in the data with the same format. To handle this
problem, one can use a one-time script to transform the data. However, this
solution has a lot of drawbacks:

- The script is not reusable, as it is built for a specific customer
- With every new customer coming in, there is a lot of engineering work to do
to create a new script for them
- If the data format changes in between, the script may require a lot of changes
- The script has to contain the data validatation logic and must be able to
handle API failures. Otherwise, the program would crash somewhere in between,
only to fix the problems and run it again

In this article, we shall explore how I built a simple solution to handle these
problems, with a scope to easily extend it.

## Go-To Solutions

Before building software to solve a problem ourselves, it is always good to
look at the available options in the market, and check if we can trade money
off for the engineering efforts. Since B2B customer onboarding is a frequent
problem, there were already a few established SaaS companies providing a
solution for it.

- [FlatFile](https://flatfile.com/) is an excellent tool to onboard business
customers. Their offerings range from providing an interactive UI to
collaborate with the customers to custom NodeJS to transform the fields of the
data.

- [OneSchema](https://www.oneschema.co/) operates in the same business area as
the FlatFile. Their solution offers an easy way to clean the data and export
the rows with errors and share them with the customer. On another end, they
also support transforming the data and handling ingestion and incremental
updates.
- [CSVBox](https://csvbox.io/): CSVBox is a simple solution to map the columns
in the input spreadsheet to a defined schema. Although googling "Flatfile
alternative" brings them to the top, they are not quite there yet!

After exploring some other solutions such as [Dromo](https://www.dromo.io/), I
decided to go with a custom solution.

## The Solution

The solution was named _Morpheus_ after a ["Matrix"
character](https://en.wikipedia.org/wiki/Morpheus_(The_Matrix)) and the [Greek
god of dreams](https://en.wikipedia.org/wiki/Morpheus). The interesting part of
the word was "morph" -- to transform, which the project would do the
spreadsheet input data.

The project consisted of two parts -- one to import the JSON data that is
already in the format required by the POST APIs, and another to convert the
spreadsheet data coming into the standard JSON format. A validator was placed
in between these steps to ensure that data was in the format needed by API.

### Importer

The importer tool is responsible for uploading the data present in the JSON
files to the APIs. The challenge comes here with the references between the
entities. To give a concrete example, let's assume the following scenario:

- We have `Student` and `Course` resources, where multiple `Student`s can take
a single `Course`
- The backend generates UUIDs for the `Student` and `Course` resources and uses
them as the primary keys
- The endpoint `POST /api/course`, takes a JSON body field `students`, an array
of UUIDs corresponding to `Student` resources.
- The endpoint `GET /api/student` has a query parameter `search`, which takes a
string of `[field1: value1][field2: value2]` format, returning the matching
resources.

We do not know the UUID of the `Student` resource until it is present in the
backend; thus, it JSON input file has to contain some other _unique_ field of
the student. Then, the code must call the `GET` endpoint, and return the UUID
of the resource. 

To handle such scenarios, I created a `Model` class in the code, responsible
for keeping track of dependencies and resolving them. The class looked like,

```typescript
type Reference = {
 model: Model
 field: string
 oneToMany: boolean
}

type SearchParams = Record<string, string>

class Model {
 constructor(
 public name: string,
 private args: {
 getEndpoint: string
 postEndpoint: string
 references: Reference[]
 }
 ) {
 // initialize all properties
 }

 async getOne(params: SearchParams): Promise<string> {
 // return the ID of the object matching the search criteria
 }

 async createOne(obj: {}): Promise<string> {
 // create one object and returns its ID
 }

 async resolveDependencies(obj: {}): Promise<{}> {
 // takes in a value from JSON, returns the object with resolved dependencies
 // the returned object can be used in `createOne()`
 }
}

```

The `getOne()` method converts `SearchParams` into a string in the format of
the `search` parameter and calls the `GET` API, handling the case of multiple
objects and no objects returned.

Similarly, the `createOne()` method calls the `POST` API to persist the object
and returns its ID once it is created.

The method of interest is `resolveDependencies()` here. It walks through all
`references`, reading in the value from the `field`, and replacing it with IDs
from the `getOne()` method. For this method to work, the resources have to be
imported in the order of their resolved dependency (Here, `Course` must be
imported after importing `Student`). I used
[`jsonpath`](https://www.npmjs.com/package/jsonpath) to give extra flexibility
while accessing the `field`. Using the `Model` class, we can define `Student`
and `Course` models as:

```typescript
const student = new Model("Student", {
 getEndpoint: "https://exmaple.com/api/student",
 postEndpoint: "https://example.com/api/student",
 references: [],
})

const course = new Model("Course", {
 getEndpoint: "https://example.com/api/course",
 postEndpoint: "https://example.com/api/course",
 references: [
 {
 model: student,
 field: "$.students",
 oneToMany: true,
 },
 ],
})

```

Now, we can start importing the objects. Let's say we have the following data.

```typescript
const students = [
 {
 email: "alex@uni.edu",
 name: "Alex Jones",
 },
 {
 email: "bob@uni.edu",
 name: "Bob Harris",
 },
 {
 email: "chris@uni.edu",
 name: "Chris Muller",
 },
]

const courses = [
 {
 name: "Intro to CS",
 id: "CS101",
 students: [
 {
 email: "alex@uni.edu",
 },
 {
 email: "bob@uni.edu",
 },
 ],
 },
 {
 name: "Fundamentals of Programming",
 id: "CS102",
 students: [
 {
 email: "chris@uni.edu",
 },
 ],
 },
]

```

Creating the students is straightforward. We call `student.createOne()` for 
all student objects.

```typescript
await Promise.all(
 students.map(studentObject =>
 student.createOne(studentObject)
 )
)
```

Before creating the courses, however, we need to resolve student IDs.

```typescript
const resolvedCourses = await Promise.all(
 courses.map(courseObject =>
 course.resolveDependencies(courseObject)
 )
)
```

The `resolvedCourses` looks like

```typescript
[
 {
 name: "Intro to CS",
 id: "CS101",
 students: ["uuid1", "uuid2"],
 },
 {
 name: "Fundamentals of Programming",
 id: "CS102",
 students: ["uuid3"],
 },
]
```

Now, we can import the `resolvedCourses` as usual.

```typescript
await Promise.all(
 resolvedCourses.map(courseObject =>
 course.createOne(courseObject)
 )
)
```

> **Note**: The `Model` class takes only care of handling references, creating
and getting objects. It is _still_ your responsibility to ensure that other
fields are valid.

### Transformer

Once the importer tool was ready, it was time to build the Transformer tool.
Assuming that we will have student names in `student.xlsx` with sheet `names`
and an API to query their email based on the names (which is a bad example,
please bear with me :sweat_smile:), the Transformer would look as follows:

```typescript
const studentTransformer = new Transformer()
 .field("name", data => data["Student Name"])
 .field("email", async data =>
 fetch(`https://uni.edu/api/email?name=${data["name"]}`)
 .then(response => response.json())
 .then(res => res.name)
 )
 .input("student.xlsx", { sheet: "names" })
 .output("student.json")

await studentTransformer.transform()
```

The `field(name: string, action: FieldAction)` function is defining the JSON
field name and action to compute the value at the field. It is possible to
return objects and arrays from the action, thus making nested structures
possible. The action handlers support both async and sync operations, having
the following type:

```typescript
type DataRow = Record<string, string>

type Output<T> = T | Promise<T>

type FieldAction<T> = (dataRow: DataRow) => Output<T>

```

When working with real data, I came across the need of skipping some rows based
on the condition. This resulted in the addition of the `filter(predicate:
Predicate)` method into the fluent interface, with `Predicate` defined as:

```typescript
type Predicate = (dataRow: DataRow) => Output<boolean>
```

Continuing the example of `Student`s and `Course`s, let's assume `courses.xlsx`
contains a big list of Course IDs and Course Names duplicated for every student
taking the course. To generate `course.json`, we need a grouping, rather than a
1-to-1 mapping. To do this, I created `groupBy(action: GroupByAction,
aggregateFields: string[])`) function, with `GroupByAction` defined as:

```typescript
type GroupByAction = (dataRow: DataRow) => Output<string[]>
```

The `GroupByAction` would return a list of strings, ideally picking them from
the `dataRow`. The transformer would then keep track of the returned strings as
keys to group the results returned by `FieldAction` of fields in
`aggregateFields` into an array. The fields not in `aggregateFields` are only
computed once. With the `groupBy()` method in place, we can write the
transformer for `Course`:

```typescript
const courseTransformer = new Transformer()
 .groupBy(data => [data["Course ID"]], ["students"])
 .field("name", data => data["Course Name"])
 .field("id", data => data["Course ID"])
 .field("students", async data => ({
 email: await fetch(
 `https://uni.edu/api/email?name=${data["name"]}`
 )
 .then(response => response.json())
 .then(res => res.name),
 }))
 .input("courses.xlsx", { sheet: "course" })
 .output("courses.json")

await courseTransformer.transform()
```

In the above example, the `FieldAction` for `the "id"` field will be run only
once for a course, and, the results from the `FieldAction` of the `"students"`
would be aggregated as an array.

### Validator

Once the transformer outputs the JSON file, before importing it, it was
necessary to check whether the objects follow the specification of the API,
since the importer does only the job of resolving references. I chose
[`zod`](https://github.com/colinhacks/zod), which I had come across GitHub
trending, to define the schema and validate the JSON file. With `zod`, schema
definition for our `Student` and `Course` would look like:

```typescript
import z from "zod"

const student = z.object({
 name: z.string(),
 email: z.string().email(),
})

const course = z.object({
 name: z.string(),
 id: z.string().regex(/CS\d{3}/),
 students: z.array(student.partial()).nonempty(),
})
```

And we can do validation as:

```typescript
const result = student.safeParse(obj)
```

## The Interface

The project _Morpheus_ had a CLI interface, built using
[`commander`](https://www.npmjs.com/package/commander).

For the Import tool, I defined all models in the backend in `models.ts` and
exported all of them in a `Map<string, Model>`. The same was repeated with
transformers, with `transformer.ts` exporting a `Map<string, Transformer>`. A
separate `transformer.ts` is required for a separate customer. The maps
contained the name of the model as the key. This helped to choose a model name
in the CLI, and get the corresponding transformer or importer.

During processing, if some error happens, the program would not halt; instead,
the program writes the error-causing object and error message to an error file
(specified in the Command Line) and continue working.

Once the process has finished, one can inspect the error file, fix the errors,
and get the objects back using the `clean` subcommand. These objects can be
further fed into the tool, or, sent back to the customer for clarifications.

## Impacts

This project had a huge impact on customer onboarding. Specifically, it

- reduced the amount of work required to onboard a customer, from creating a
project to writing several functions
- decreased the customer onboarding time from weeks to days
- increased the visibility into the errors during processing and import

Apart from customer onboarding, I used the Import tool heavily in combination
with [Mockaroo](https://www.mockaroo.com/) to create dummy data for product
demos to potential customers. Thanks to the Import tool, I created around 10
such environments, incrementally gathering the feedback from the business
:smile:.

## Learnings

Since I handled this project from initialization to the first milestone, I
learned how to run a project, what features to prioritize based on the business
requirements, and communicate them with all the stakeholders. Apart from
coding, I was involved with documenting the project and gathering feedback from
other developers regarding code quality and interface usability, from which I
was able to understand the project from different perspectives.

How I built a fluent interface to transform spreadsheet data into JSON that can be imported via the APIs.

Project Morpheous

I was working with [Pulumi](https://www.pulumi.com) to migrate AWS cloud
resources of a company to IaC. In this article, we shall explore why I chose
Pulumi, how the migration was, and the standards and processes defined along
the way.

## Looking for an IaC solution

Terraform is the first IaC solution that comes to anyone's mind. Although it is
widely used in the industry, I found it a little intimidating to use HCL, as it
reinvented some of the basic programming constructs such as loops. I will take
an example straightly from the [Terraform
Docs](https://www.terraform.io/language/expressions/for#filtering-elements) to
illustrate my point.

```hcl
variable "users" {
 type = map(object({
 is_admin = bool
 }))
}

locals {
 admin_users = {
 for name, user in var.users : name => user
 if user.is_admin
 }
 regular_users = {
 for name, user in var.users : name => user
 if !user.is_admin
 }
}
```

In a general-purpose programming language, say TypeScript, we can write this
code as:

```typescript
type User = {
 is_admin: boolean,
}

let users: Map<string, User> = {} // The value is coming from somewhere, say a
config or input

let admin_users = new Map(
 [...users].filter(
 ([name, user]) => user.is_admin
 )
)

let regular_users = new Map(
 [...users].filter(
 ([name, user]) => !user.is_admin
 )
)

```

While anyone comfortable with JavaScript or TypeScript can understand the below
code, it takes some time to understand HCL code and get used to its syntax. I
preferred general-purpose programming languages for IaC for a few reasons:

1. Developers can write IaC in the language that they are used to write
applications, this eliminates the need to learn a new language
2. A developer with sufficient knowledge can maintain the IaC codebase; no need
to have dedicated personnel for this
3. One can use the _best practices_ of the programming language to organize the
code, such as refactoring into modules, etc.
4. In some cases, debugging IaC is also possible by using common debuggers by
stepping into the code and inspecting variables!

When I saw a [Fireship
video](https://www.youtube.com/watch?v=tomUWcQ0P3k&lc=UgxQFl_FF5yB0bKhXF14AaABAg&ab_channel=Fireship)
about Terraform, a comment mentioned that he should make a video of Pulumi as
well; that's how I got to know about Pulumi! Going through their
[website](https://www.pulumi.com/) and [docs](https://www.pulumi.com/docs/), it
seemed they offered the exact needs that I wanted. After I read the article
[_"Imperatively
Declarative"_](https://chris.nunciato.org/blog/how-pulumi-is-declarative/) by
Christian Nunciato, I had a clear idea of what Pulumi does, and why it is
flexible over other solutions.

While Pulumi offers support for many languages (**latest**: [Pulumi now
supports JVM languages and YAML as
well!](https://www.pulumi.com/blog/pulumi-universal-iac)), I chose to go ahead
with TypeScript + NodeJS for Pulumi for a few reasons:

- Project management with `npm` is easy, it just takes doing `npm install` to
set up the project on a new machine
- At that time, Pulumi's [`awsx`](https://www.npmjs.com/package/@pulumi/awsx)
and [`eks`](https://www.npmjs.com/package/@pulumi/eks) packages were only
available on `npm`; more about them later!
- Going with TypeScript over JavaScript is natural, due to the benefits offered
by static type checking. Plus, Pulumi's packages are also in TypeScript,
providing excellent typing support.

## The Highlights

In the following sections, I will pick some interesting points about how Pulumi
helped write a clean IaC project. Let's use the following AWS architecture for
illustration purposes.

- VPC with public and private subnets, replicated for production and staging
- An EC2 jumpbox VM in the public subnet of staging VPC, which is in its own
security group
- VPC peering connection between production and staging VPCs
- An RDS instance placed in the private subnet of the VPC, with connections
allowed from the jumpbox machine, replicated for production and staging

### Stacks, Outputs, and References

In Pulumi, a _stack_ consists of a group of cloud resources. From one codebase,
we can create as many stacks as we wish. One typical use case of the stacks
would be to have `staging` and `production` environments, with nearly identical
resources. Apart from the environments, the stacks also help to break down the
infrastructure into several units using [Stack Reference
Architecture](https://www.pulumi.com/registry/packages/aws/how-to-guides/aws-ts-stackreference-architecture/).

We can break down our architecture into `network` and `database` units, with
`stage` and `prod` environments. We shall use `<unit>.<env>` syntax to name the
stacks. Each unit sits in its own repository and has a separate stack file. 

Let's focus on the jumpbox in the `network.stage` stack. Since we want the RDS
to be accessible from the jumobox, we export its security group ID as the stack
output.

```typescript
// network/index.ts
import * as aws from "@pulumi/aws"

function createJumpbox(name: string) {
 const securityGroup = new aws.ec2.SecurityGroup(
 `${name}-sg`,
 {
 ingress: [
 {
 cidrBlocks: ["0.0.0.0/0"],
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 description: "Allow SSH connection",
 },
 ],
 egress: [
 {
 cidrBlocks: ["0.0.0.0/0"],
 fromPort: 0,
 toPort: 0,
 protocol: "all",
 description: "Allow all outgoing connections",
 },
 ],
 }
 )
 const vm = new aws.ec2.Instance(name, {
 ami: "<add-some-ami-here>",
 instanceType: "t3.medium",
 vpcSecurityGroupIds: [securityGroup.id],
 // Add other options here, such as SSH Key, subnets
 })
 return { securityGroup, vm }
}

const { securityGroup, vm } = createJumpbox("jumpbox")

// Export the security group ID as the stack output
export const jumpboxSecurityGroupId = securityGroup.id

```

On the `database` unit, to allow _only_ the connections from jumpbox at port
`5432` (which is the port at which PostgreSQL listens), we can do it this way:

```typescript
// database/index.ts

import * as pulumi from "@pulumi/pulumi"
import * as aws from "@pulumi/aws"

// If we are in the stack `db.stage`, environment is `stage`
const getEnvironment = () => pulumi
 .getStack()
 .split(".")
 .slice(-1)[0]

const environment = getEnvironment()
// Create a Stack Reference to the `network` stack in the same environment
const networkStack = new pulumi.StackReference(
 `network.${environment}`
)
// Get the jumpbox security group ID
const jumpboxSecurityGroupId = networkStack.requireOutput(
 "jumpboxSecurityGroupId"
) as pulumi.Output<string>
// Create Security group and allow incoming connection from jumpbox security
group
const securityGroup = new aws.ec2.SecurityGroup("db-sg", {
 ingress: [
 {
 securityGroups: [jumpboxSecurityGroupId],
 fromPort: 5432,
 toPort: 5432,
 protocol: "tcp",
 description:
 "Allow incoming connection from jumpbox at port 5432",
 },
 ],
})
// Create RDS Instance
const rdsInstance = new aws.rds.Instance("db", {
 instanceClass: "db.m3.medium",
 vpcSecurityGroupIds: [securityGroup.id],
 // Associate subnet group etc
})

```

The `requireOutput()` method of the `StackReference` class is the key here.
Using the stack reference pattern, one can easily split a large infrastructure
into smaller _units_ and link them.

> **Note**: When using the stack reference, if the output of a stack changes,
we have to update other stacks manually by selecting the stack `pulumi stack
select <stack name>` and `pulumi update.
>
> Sometimes, not doing so might break the things elsewhere.

The stack reference can also be used to refer to the stacks in the same project
as well. As an example, in `network.prod`, we create [VPC
Peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html)
connection to `network.stage` and export `jumpboxSecurityGroupId` reusing it
from `network.stage`

```typescript
// network/index.ts

import * as pulumi from "@pulumi/pulumi"
import * as aws from "@pulumi/aws"

// Get the current environment, getEnvironment() defined earlier
const environment = getEnvironment()

const isStaging = environment === "stage"

let securityGroupId = null

if (isStaging) {
 // Function createJumpbox() defined in earlier example
 const { securityGroup: jumpboxSecurityGroupId, vm } =
 createJumpbox("jumpbox")
 securityGroupId = jumpboxSecurityGroupId.id
} else {
 const stagingEnvironment = new pulumi.StackReference(
 "network.stage"
 )
 // Create VPC peering
 securityGroupId = stagingEnvironment.requireOutput(
 "jumpboxSecurityGroupId"
 )
}

export const jumpboxSecurityGroupId = securityGroupId

```



### Config File, Secrets

In the above examples, you may have seen that we have hardcoded the EC2
instnace type in the code. Whenever we want to change the instance type,
editing the code does not seem to be a good idea. Instead of that, Pulumi
offers us configs, which can be different for different stacks in the same
project. In the `network` project in the above example, we can refactor the
instance type to `Pulumi.network.<env>.yaml`, using `pulumi` CLI.

```sh
pulumi config set jumpboxInstanceType t3.medium
```

We can also check the config file to verify that the value has been added.

```yaml
# Pulumi.test.stage.yaml
config:
 test:jumpboxInstanceType: t3.medium
```

Then, in the code, we can access config values using the `Config` class.

```typescript
import * as pulumi from "@pulumi/pulumi"
import * as aws from "@pulumi/aws"

const config = new pulumi.Config()

const instanceType = config.require("jumpboxInstanceType")

function createJumpbox(
 name: string,
 args: { instanceType: string }
) {
 const { instanceType } = args
 // same as above
 // const securityGroup =
 const vm = new aws.ec2.Instance(name, {
 ami: "<add-some-ami-here>",
 instanceType,
 vpcSecurityGroupIds: [securityGroup.id],
 // Add other options here, such as SSH Key, subnets
 })
 return { vm, securityGroup }
}
// Create jumpbox instance
const { securityGroup, vm } = createJumpbox("jumpbox", { instanceType })
```

Pulumi can also store sensitive values such as API keys and passwords in the
config via secrets. We can pass the `--secret` flag to `pulumi config set`
command to store a secret.

```sh
pulumi config set --secret secretTest someValue
```

It will add the encrypted `"someValue"` to the YAML file. Note that the
encrypted value will be different based on your key.

```yaml
# Pulumi.test.stage.yaml
config:
 stack-test:secretTest:
 secure: v1:ZweLtoIrBL/w8QjD:EGbnlgUGg1RcOb+lrQK7o7yXsE0oAh2Hqw==
```

To access the decrypted value, we use `Config.requireSecret()` function in the
code.

```typescript
import * as pulumi from "@pulumi/pulumi"

const config = new pulumi.Config()

const secretValue = config.requireSecret("secretTest")

// secretValue is pulumi.Output<any>, so we log it once we have a value for it
secretValue.apply(value => {
 console.log(value)
 pulumi.log.debug(value)
})

export const secretOutput = secretValue
```

While the `secretValue` in the above code is accessible by other resources as
the plain string, it is not printable. The log statements above just print
`[secret]`, instead of giving out the value. This value is stored encrypted
even in the state file!

> **Note**: Although Pulumi ensures that the secret value is not stored in
plaintext from its side, it is still your responsibility to store them securely
on the cloud side.
>
> As an example, let's say we are using an API key as an environment variable
to AWS Lambda. While we can store the API key as a secret in Pulumi config, it
is still viewable plaintext at Lambda-side. 

### Crosswalk Packages

The [Pulumi CrossWalk](https://www.pulumi.com/docs/guides/crosswalk/aws/)
packages encapsulate commonly used infrastructure patterns as a
`ComponentResource`. While we can create VPC using `@pulumi/aws`, by using
`Vpc` and `Subnet` classes along with `RouteTable` class to connect the
subnets, and `NatGateway` and `InternetGateway` to configure the network
access, the creation of the same using `@pulumi/awsx` package is as simple as,

```typescript
// network/index.ts
import * as awsx from "@pulumi/awsx";

type SubnetArgs = awsx.ec2.VpcSubnetArgs;

const publicSubnetCount = 2;
const privateSubnetCount = 2;

// Returns array with n elements
const range = (n: number) => [...Array(n).keys()];

const publicSubnets: SubnetArgs[] = range(
 publicSubnetCount
).map(() => ({ type: "public" }));
const privateSubnets: SubnetArgs[] = range(
 privateSubnetCount
).map(() => ({ type: "private" }));

const vpc = new awsx.ec2.Vpc("vpc", {
 cidrBlock: "10.0.0.0/16",
 subnets: [...publicSubnets, ...privateSubnets],
 numberOfNatGateways: 1,
});

```

### Lambda Magic

When we write the IaC in TypeScript, we are in the realm of code. Pulumi offers
excellent ways to utilize this realm and provides some helper functions with
callbacks, which are then [magically converted into AWS
Lambdas](https://www.pulumi.com/docs/guides/crosswalk/aws/lambda/). As a
concrete example, we can listen to an event on an S3 bucket by writing a
callback function in the IaC.

```typescript
import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("media-bucket");

bucket.onObjectCreated("bucketEvent", (args) => {
 console.log("Got ObjectCreated event");
 // You can do something useful with args
 // This will be executed in the Lamdba
});

```

Such functionality is available for a variety of event sources. I implemented
this functionality for MQ by overriding the source code in the project. I plan
to test this functionality thoroughly using LocalStack and open a PR near soon.

## Defining the Standards

Pulumi does not enforce any restrictions on project organization. While nothing
is wrong with creating every resource of the stack in the `index.ts` file,
keeping the code organized helps to maintain it. Here are some of the standards
I defined and followed while writing the IaC:

- Stack names should follow the `projectName.environment` syntax.
 - `orgName/projectName` is not supported in the S3 backend, thus this naming
convention.
 - Thanks [@briggsl](https://twitter.com/briggsl) for pointing that out!

- Every source file should be under the `src/` directory. By default, Pulumi's
`aws-typescript` template creates `index.ts` in the project root.
- No resource should be directly created in `index.ts`, which is not directly
relevant to the main resource that the stack creates. They should be rather
refactored into `resource.ts` exporting `createResource()` function.
 - If the resource creation takes only one call, it can still be in the
`index.ts`, as there is no point in wrapping it with other function.
 - Mostly, these were related to IAM; Creating roles and policies were
refactored into a separate file.
- No function should decide the resource name on its own, rather it should be
passed top-down from `index.ts`.
 - This makes it easy to follow the path from which a resource is coming.
 - The functions can prepend some identifier to the name while passing it to
sub-functions
 - The name should be the first argument of the function, for example,
`createResource(name, args)`. This is in line with Pulumi's convention of
creating resources.
- No hard-coded values in the code, they should be rather in the config.
 - The config values are stored under an object, whose type is declared in the
`index.ts` as `ProjectConfig`.
 - Then, one can use `config.requireObject<ProjectConfig>("projectConfig")` to
retrieve the config.
 - No other file than `index.ts` should read the config! This would make it
easy to track the flow of the config values.
- No call to `Config.getX()` and `StackReference.getOutput()` methods!
 - These methods will silently return an empty value if the key is not present
 - The `require` methods are rather strict and will throw an exception if the
key is not present, thus, eliminating a few errors

## CI/CD for the Infra

While [Pulumi Service](https://www.pulumi.com/product/pulumi-service/) provides
a nice way to preview PR changes and look at the current state of the stack, I
went with S3 to store the state file, and used GitLab CI/CD pipelines for
deploying infrastructure changes. The rules of the pipeline looked as:

- Pipeline contains two stages:
 - `preview`, to preview infrastructure changes
 - `deploy`, to deploy the changes
- `deploy` is only available for the `main` branch, and contains manually
triggered jobs
- After `deploy` is complete, a Slack notification is triggered
- There can be multiple jobs in the `preview` and `deploy` stages, depending on
the environments

Then, the developer workflow would be:

- Create a branch for proposed infrastructure changes
- Commit and push the code, verify if changes look OK
- Start a PR, the reviewer looks at the code _along with_ the `preview` job
output to ensure nothing is breaking
- Once the PR is accepted and merged, a maintainer can trigger `deploy`
- Team is notified on Slack about new changes :tada:

## Overall Experience

Pulumi was my first experience with IaC, and it helped me to get started with
AWS more easily, thanks to the nice APIs in the code. Whenever I was struck
with something, I would go to the [Pulumi Slack
Community](https://slack.pulumi.com/), and people are very helpful there.

Pulumi brought IaC to General Purpose Languages, which also can benefit from
existing good practices regarding reusability, as I experienced it with
`TypeScript`. I used Pulumi at a beginner level, and there is still a lot to
learn regarding `ComponentResource`, stack functions, and Policy as a Code.

Experience and learnings during project that migrated the infra of a company to IaC using Pulumi

Pulumification

Our family owns a spices and local products shop, which was forced to shut down
during the COVID-19 pandemic, as all the public activities came to a halt. This
forced us to think of new ways of selling products, without us shipping our
products to other places previously. I, being a Computer Science student,
decided to solve this by creating an [e-commerce
website](https://benakastores.com), showcasing all of our products. In this
post, we shall look into how I built the website in the first version, its
functioning, and, the building of its second version which is currently up now.

## The Beginnings

In the beginning, I decided to go with a no-code SaaS platform, as I was busy
working on my Master's research project
[AndMal-Detect](https://github.com/vinayakakv/android-malware-detection). There
are a lot of platforms to choose from, such as
[Shopify](https://www.shopify.in/), [GoDaddy](https://www.godaddy.com/) and
[JustDial](https://yourownwebsite.justdial.com/). I was focusing on the cost
initially since I was unsure how the business would go online. Attracted by
[Wix](https://www.wix.com/)'s cheap plan, which included a free domain for a
year, I went ahead with it.



The setup was pretty simple, I photographed all our products with Nikon D7200
DSLR I had, and wrote the description for some of them. The Wix UI provided a
nice option to create these products, although the work was repetitive. Then,
using the template builder, I created a home page, which itself showcased all
the products. Within a matter of weeks, the website was up and running!

## Life with No-Code

We got started on e-commerce very well. Initially, I created social accounts
with the website and kept them very active. Soon, my interest in social media
[got drying up](https://vinayakakv.com/writings/i-built-the-blog), and soon I
ended deleting up every social media account except Instagram. Thanks to the
nice SEO capabilities provided by Wix, we still kept getting orders. 



On average, we got around 5-6 orders a month. While the numbers seem pretty
low, we got an incredible community of customers, who are always supportive and
keen to give feedback on the website and the business, in general. With this
feedback, we were able to figure out some pain points of them.

### The Pain Points -- Customers' and Owner's

- **Slow** -- The website was _tooooo_ slow! We often heard complaints that
several parts of the website were not loading even on a fairly good internet
connection. We ended up getting orders on WhatsApp in such cases.
- **Glitchy** -- Sometimes, clicking on buttons would not work. Worst of all,
when coupled with the website being slow, there was no state to indicate
transitions between the pages, leaving users in a state of confusion.
- **Unnecessary Features** -- At checkout, by default, the option to enter
promo codes shows up. While our customers are very familiar with them, we could
not afford to roll out promo codes and discount offers. This led to churn in
some cases as well!
- **Less flexible Shipping Options** -- We calculate our shipping charges by
weight. While Wix allows this method, the weight slabs are limited in number,
mostly up to 7 or 8. When the cart weight exceeds the slab, the shipping price
is capped at the maximum, although it could be higher in the real world. It was
hard to convince our customers about the change in the shipping charge.
- **Less Room for Customizability** -- As a website owner, I wanted to enjoy
the full _ownership_ of the website, customizing it to the lowest details.
Unfortunately, I was limited to the features that Wix offered.
- **The Cost!** -- While the initial cost of Wix was attractive and affordable,
it grew over the years. For example, the cost of the 3rd year (which
was supposed to be this year) is 10x of 1st year!

## Choices of the Migration

While I was experiencing these issues for a quite long time, I sprung into
action after I was confident in website development using React by building a
[website for myself](https://vinayakakv.com) using
[Gatsby](https://www.gatsbyjs.com/). The video about [Mantine UI by
Fireship](https://www.youtube.com/watch?v=ouncVBiye_M) further encouraged me to
take the project up and complete it within some weeks. I found several
alternative approaches:

1. Use [NextJS Commerce Starter](https://nextjs.org/commerce) and a headless
E-Commerce CMS
2. Go with Shopify or Wix ADI which seem to be fast and promise good [core web
vitals scores](https://web.dev/vitals/)
3. Build something on my own!

### NextJS Commerce Starter

The NextJS Commerce Starter template can seamlessly blend with several headless
CMSes tailored for e-commerce. Their demo with the BigCommerce backend seemed
pretty fast and I liked the performance of the website. When it came to the
choice of the CMS, [BigCommerce](https://www.bigcommerce.com/) seemed quite
costly. The story was the same with other CMSes like
[Swell](https://www.swell.is/). However, [Saleor](https://saleor.io/), an
open-source CMS seemed to be quite promising.

Saleor is fully feature-packed and extendible through plugins. It offers both
self-hosted and cloud options, and the cost of the cloud exceeded that of Wix.
I was looking for some cheap cloud VM and found Oracle cloud providing a good
ARM machine in the free tier. Unfortunately, it did not accept any credit cards
to sign up for the trial!

### Another SaaS No-Code/Low-Code solution

Choosing another SaaS No-code solution could solve some of the pain points, but
in any case, it would not be fully customizable. Thus, I decided not to go with
this option and move ahead with investing some time into developing the website
from the scratch.

## The Migration to Code

I was pretty confident with React+TypeScript by working on enterprise projects
at that time. Randomly, I came across [Hygraph E-Commerce
Starter](https://github.com/hygraph/hygraph-commerce-starter) while browsing
through my GitHub feed (thanks GitHub!). Although I had built my blog with
Hygraph, I never thought of sourcing an e-commerce website using Hygraph, as I
always imagined a fully-functional backend server listening to new orders. The
starter template gave me a good start.

### Tech Stack

I decided on the following stack for the website.

- **[Hygraph](https://hygraph.com/)** for the content layer. Hygraph has
excellent data modeling capabilities like two-way typed relationships and
provides a GraphQL API to consume the data.
- **[NextJS](https://nextjs.org/)** as the React framework. The Static Site
Generation is particularly useful to pull in the data from Hygraph during the
build. The API routes can be used for order placement.
- **[Mantine UI](https://mantine.dev/)** as the CSS framework. Mantine UI is
nice-looking, modern, and sleek; although I plan to introduce a design system
and write a custom component library.
- **[MongoDB](https://www.mongodb.com/)** for storing order and customer data.
Hygraph could have been used for this; more on that later!

In the upcoming sections, I will pick up some interesting and challenging parts
of the migration process and discuss them in some detail.

### Bootstrapping the Project

Although Hygraph e-commerce starter provided a nice frontend with
NextJS+TailwindCSS, I decided to start barefoot using the official NextJS
starter. On the Hygraph side, I cloned their template and sample data into a
new project.

After setting up prettier and path aliases in `tsconfig.json`, to avoid the
long train of `../..` in the imports, I made the first push and set up CI/CD
using [Vercel](https://vercel.com/). For every commit pushed and the PR
created, Vercel would provide me with a deployment, which I can test and share
with friends for feedback.

### Assembling the UI pieces

I didn't go with a design-first approach, as I'm not an expert designer myself.
I had the following idea of the website layout in my mind:

- The website would contain a homepage, product page, and category page along
with the checkout page
- The homepage would consist of a hero, along with the product grid
- Product page will display the product image, description, and a CTA to add
the item to the cart
- Category page would show a product grid consisting of the products in that
category
- Shipping page would take the user through a flow, asking for address details
and order confirmation

While looking for UI components to achieve this, I came across [Mantine
UI](https://ui.mantine.dev/), a collection of UI components built using
Mantine. I copy-pasted some of them into my codebase and tweaked them according
to the needs. It was only after I finished implementing the functionalities, I
relooked at the code and optimized it.

### The GraphQL Interface

The core part of the website is to fetch products from Hygraph and populate the
static pages. Since Hygraph provides a GraphQL interface, I needed to use a
GraphQL client. I went with
[`graphql-request`](https://www.npmjs.com/package/graphql-request), as it was
minimal.

The fun part is to _type_ the GraphQL responses to get meaningful suggestions
in TypeScript. First, I created schema definitions in the `schema/` folder for
every Hygraph schema. An example schema looks like this:

```typescript
// schema/product.ts
import { Variant } from "./varient"
import { Category } from "./category"

export type Product = {
 name: string
 slug: string
 description: string
 variants: Variant[]
 categories: Category[]
}
```

I relied on querying multiple items from Hygraph, even though it supports
querying a single item based on a condition. Since I knew what I would be
querying, I created a type to represent the GraphQL response.

```typescript
// schema/response.ts
export type Keys =
 | "products"
 | "productVariants"
 | "collections"
// More here in the OR 

export type Response<T, key extends Keys> = {
 [keyName in Keys & key]: T
}
```

The type `Response<T, key>` is the crucial part here. It can accept _only_ the
strings defined in type `Keys` for its second parameter. For example, the type
`Response<T, "products">` resolves to `{product: T}`. I had to spend half a day
figuring out the intersection `[keyName in Keys & key]`!

Then, everything was straightforward. I wrapped the `request` method of
`GraphQLClient` with my method, passing in the authentication tokens, and
directly returning the `Response` type. 

```typescript
// helpers/graphql.ts
import type { RequestDocument } from "graphql-request"
import type { Keys, Response } from "@schema"
import { GraphQLClient } from "graphql-request"

const client = new GraphQLClient(process.env.GRAPHCMS_ENDPOINT!)

export const request = async <T, K extends Keys>(args: {
 query: RequestDocument
 variables?: { [key: string]: string | number | boolean }
 isMutation?: boolean
}): Promise<Response<T, K>> => {
 const { query, variables = {}, isMutation = false } = args
 return client.request<Response<T, K>>(query, variables, {
 authorization: `Bearer ${
 isMutation
 ? process.env.GRAPHCMS_MUTATION_TOKEN!
 : process.env.GRAPHCMS_READ_TOKEN!
 }`,
 })
}
```

With the wrapper method in place, querying in `getStaticProps` was as easy as 

```typescript
const { products } = await request<
 Pick<Product, "name" | "variants" | "slug">[],
 "products"
 >({
 query: graphql`
 {
 products {
 slug
 name
 variants {
 id
 title
 price
 weight
 product {
 name
 }
 images {
 url
 }
 isPrimary
 }
 }
 }
 `,
 })
```

While I sometimes used `Pick<T, keys>` to mention the keys being chosen in the
GraphQL query explicitly, I was lazy enough to just use `T` in some other cases
:sweat_smile:! I am still looking for a way to better type GraphQL in
TypeScript, please let me know if you know of other ways! 

### Fixing the Initial Load Glitch

After some days into the development, when the website was feature-ready, I
started to observe an initial glitch during the website's loading. The glitch
was prominent on slow internet connections, and it appeared as if the website
was rendered without initial CSS. You can check this yourself at this [preview
link](https://benaka-malnad-stores-jmo09opcu-vinayakakv.vercel.app/).

![No CSS on initial page
load](https://media.graphassets.com/yIkXNCKpSuO4y6x0W2uQ)

This was a big headache, and after trying every possible way to fix this
problem, I was in no luck. I even felt that going with UI frameworks was a bad
idea. When I experimented with other websites built with Mantine, this was not
the case. I was checking the Mantine versions in `package.json` and thought I
could do `npm update` to check this. Fortunately, it solved the issue :tada: by
upgrading `@mantine/core` from `v4.0.9` to `v4.1.0`.

### `Stepper` with Sandwiched Content

The checkout page required a stepper to take the user through the flow. While
Mantine provides [`Stepper` component](https://mantine.dev/core/stepper/), it
does not allow content to be in-between the steps when
`orientation="vertical"`. Either, I could use a sidebar-content layout with the
stepper in the sidebar (which was not obvious on smaller screens), or, override
the stepper myself. Inspired by the [MUI
Stepper](https://mui.com/material-ui/react-stepper/#vertical-stepper), I chose
to override the stepper myself, and it was easy. I plan to start a PR in
Mantine with this improvement. Following is the result of the override.

![Stepper with content
in-between](https://media.graphassets.com/pNy2FlmOQ4SAsU4KT22w)

### The Backend

Vercel functions can nicely facilitate the needs of an API. In the Hygraph
e-commerce starter template, the schema contained `Order` and `OrderItem`, and
it was integrated with an external payment gateway. In our case, we did not
want a payment gateway, as we collect payment by contacting the customers
manually. Apart from that, I needed to get a notification whenever an order is
placed. While Hygraph webhooks can be used for this, I noticed a few
difficulties:

- Hygraph mutation to create order was slow, and Vercel Functions were timing
out. Later I fixed this by providing separate APIs for mutating `Customer` and
`Order`; however, there were still occasional timeouts
- In the notification, I wanted customer details and order details. Since the
webhook only provided the IDs of the nested fields, I still needed to fetch
them separately.

Thus, I decided to go with MongoDB, which provides 25GB of storage in their
free trial. Instead of calling Hygraph mutations, the APIs now directly wrote
to MongoDB. Automagically, this also reduced the timeouts. On the MongoDB side,
I used triggers to send an email to the customer once an order is placed,
keeping me in `bcc`. Thus, both problems were solved with a single solution.

Apart from writing to the database, the backend API also handles calculating
shipping charges. Finally, I was free from the limitations and was able to
write a very simple code to calculate shipping charges.

```typescript
import { NextApiRequest, NextApiResponse } from "next"
import type { ShippingProvider } from "@schema"

type RequestData = {
 pincode: string
}

type ResponseData = {
 shippingProviders: ShippingProvider[]
}

function getShippingProviders(pincode: string): ShippingProvider[] {
 console.log(
 JSON.stringify({
 path: `/shippingProvider`,
 pincode,
 })
 )
 const isBangaloreRegion = pincode.startsWith("5600")
 const isKarnatakaRegion = ["56", "57", "58", "59"].some(digit =>
 pincode.startsWith(digit)
 )
 const shippingProviders: ShippingProvider[] = [
 {
 name: "Indiapost Parcel",
 key: "INDIA_POST",
 ratePerKg: 45,
 },
 ]
 if (isBangaloreRegion) {
 shippingProviders.push({
 name: "DTDC Courier",
 key: "DTDC",
 ratePerKg: 40,
 })
 } else if (isKarnatakaRegion) {
 shippingProviders.push({
 name: "DTDC Courier",
 key: "DTDC",
 ratePerKg: 60,
 })
 }
 return shippingProviders
}

export default function handler(
 req: NextApiRequest,
 res: NextApiResponse<ResponseData>
) {
 const { pincode }: RequestData = JSON.parse(req.body)
 const shippingProviders = getShippingProviders(pincode)
 res.status(200).json({ shippingProviders })
}
```



### Data Migration

Once the website was functional, it was time to migrate the data from Wix to
Hygraph. Fortunately, Wix allowed me to export the products as a CSV file,
which I downloaded and cleaned in Google Sheets. Then, I converted it into a
JSON file. Since I wanted the product images to be exactly `800x800` dimension,
I used [`sharp`](https://www.npmjs.com/package/sharp) to resize images, and
[`node-fetch`](https://www.npmjs.com/package/node-fetch) to upload them into
Hygraph media endpoint. Once I got the uploaded image IDs, I created the
objects in Hygraph using GraphQL mutations. The data migration took about 2-3
hours, as I was fixing the bugs, and manually writing the retry logic! 

### Redirects and SEO

Once the data was migrated, the website was production-ready. I added the meta
tags using [`next-seo`](https://www.npmjs.com/package/next-seo). Since the
paths of the pages were substantially changed, I had to configure `308
Permanent Redirect` for all such pages, to make new pages appear in Google. I
had to manually write the redirect map in `next.config.js`
:smiling_face_with_tear:.

## The Deployment :rocket:

One day, when I was still developing the website, I got a Whatsapp message from
a customer that the website is not working. I immediately contacted them and
got to know that the website had become too much glitchy that they were unable
to place orders. The time to deploy the website to production has come!

With the redirects ready, I proceeded to detach the domain from the Wix website
and deleted it entirely (I deleted it fearing the duplicate content would
affect SEO, I never took a screenshot to compare the website with the new one
:cry:!). Then, in the Vercel dashboard, I assigned the domain
`benakastores.com` to the website, and, it was up and running!

The next day, we got an order from the new website, which was amazing. I
submitted the new sitemap to Google, and the new paths were visible after some
two weeks. Now, the website loads much faster, shipping logic is much simpler,
no unnecessary components and it is fully customizable :tada:. It solved all
the old problems, with scope for improvements on the horizon.

## Scope for Improvements

There is a lot to improve now, both from the business side and the website
side. Some of them are:

- Currently, we observe that the customers enter the website through Google,
shop for one particular item, and check out. Although I have added
recommendations at checkout, customers are not bothering to go through them.
- We can create a brand by custom-designing the website components and making
the experience unique. It is planned for the v3 of the website, though!
- When we want to update a product, currently, we have to rebuild the entire
website. I'm now looking into [NextJS' Incremental Static
Regenration](https://nextjs.org/docs/basic-features/data-fetching/incremental-static-regeneration)
to handle such situations.
- If you find something, please feel free to reach out to me!

## Thanks

- Our customers for the feedback!
- [@sourabhdaroji](https://www.linkedin.com/in/sourabhdaroji/) for design
suggestions
- [@chaithalibt](https://www.linkedin.com/in/chaithali-b-t-73ab63123/),
[@vinyasns](https://www.linkedin.com/in/vinyasns/), and,
[@ikshudhanva](https://www.linkedin.com/in/ikshudhanva/) for testing the
website from an early stage

How I migrated our E-Commerce shop from No-Code tool to NextJS

Blog