
Storage is a critical component of any software system. Some applications use storage to persist their data, while others, such as databases, rely on it exclusively.

The operating system provides an abstraction of storage through files and folders, which are managed by filesystems and supported by one or more physical disks. When the underlying disk becomes full, the OS and applications may reject requests, potentially leading to downtime. In this article, we will explore the "disk full" issues we encountered, how we resolved them, and the steps we took to ensure they do not occur again.

## Access Denied

One morning, while attempting to log in to our corporate VPN — through which we access our development and production servers — we were met with timeouts. The issue was widespread, and no one could connect to the VPN.

Fortunately, we were able to access our cloud provider's VM management page and confirmed that the VPN host was up and running. Eager to investigate further, we allowed SSH access to the machine, restricting it to our machine's public IPs. This required us to manually create and edit firewall rules through the UI, as our Terraform setup was inaccessible within the VPN.

Once inside, we `tail`ed the logs of our VPN server and found messages indicating that the storage was full. Running `df -h` confirmed that the root storage was 100% occupied. We used `du -h` to analyze the suspected culprits, and sure enough, logs were consuming around 70% of the storage. The VPN server was logging access attempts and other activities into the log files along with its internal SQLite database for audit purposes. Fortunately, this meant we could delete _all_ the old logs without any impact.

Unfortunately, due to a misconfiguration, we couldn't gain root access to the machine. To resolve this, we had to shut down the machine (after all, the VPN was down anyway!), detach the boot disk, attach it to another VM where we had root access, and delete the logs. Afterward, we reattached the disk back to the VPN VM, and voilà — everything was back to normal.

The first step in our long-term fix was to ensure the right users could gain root access. Next, we configured a cron job to automatically delete old log files using `ansible.builtin.cron`. May that problem never repeat!

## Database Overloaded

One night, in the middle of sweet REM sleep, we were paged because one of our production-critical services was responding with an elevated rate of 500 errors. Debugging revealed that one of the database nodes (we use a ScyllaDB cluster) was down _and_ the application was using that node's IP as its single contact point.

ScyllaDB supports passing a list of node IPs as contact points to establish the initial connection. Once connected, the client becomes cluster-aware — meaning that even if the initial node used for the connection goes down, the connection remains unaffected.

On that fateful day, one node went down, and a service using that node's IP as its contact point was restarted. As a result, the service was unable to connect to the cluster and became unhealthy. Unfortunately, our [`deploylib`](/blog/you-have-built-kubernetes-sir) did not catch this issue and allowed traffic to flow to these unhealthy service instances.

The first thing we did was update the contact point to another node's address and restart the affected service to restore its health. Even though one node was down, the cluster remained healthy — thanks to replication! Afterward, we began investigating the root cause.

Looking at the database logs revealed that there was no space left on the device. ScyllaDB [recommends keeping at least 20% of the disk free](https://opensource.docs.scylladb.com/stable/getting-started/system-requirements.html#disk-space) to support computations and other maintenance operations. Additionally, we realized we had forgotten to configure storage space alerts 😅.

We self-host ScyllaDB using their [Kubernetes operator](https://github.com/scylladb/scylla-operator), which provides a `ScyllaCluster` custom resource to create and manage Scylla clusters. To bring the affected node back online, we attempted the following actions:

1. **Increase storage in the node configuration:**  
   The operator complained that it could not perform this action. This is a known limitation of the operator.
2. **Scale down nodes in the rack to zero, then increase storage:**  
   The pod did not respond to Kubernetes' shutdown signals. The `scylla` process kept it busy, requiring manual termination. Despite this, the operator refused to increase storage because the persistent volume was still attached.
3. **Delete the rack and recreate it with increased storage:**  
   This approach worked but triggered resharding, which took a few hours. Adding the rack back initiated another round of resharding. These operations consumed significant CPU resources at times, impacting production performance. As a result, we had to schedule them during off-hours. [Tablets](https://opensource.docs.scylladb.com/stable/architecture/tablets.html) may provide a way to make cluster scaling less costly in the future. Scylla's enterprise version also reportedly offers the ability to reduce the priority of maintenance operations.

At the end, we had a stable cluster, albeit with asymmetric storage capacity — the newly brought-up node had more storage than the others. We needed to increase the storage capacity of the other nodes as well. However, it was clear that approach #3 was not the ideal solution!

One of our [engineers](https://www.linkedin.com/in/gregdaydevops/) devised an intelligent way to handle this scaling:

1. **Prevent the Scylla operator from making any changes to the cluster:**  
   Scale the operator deployment to zero.
2. **Orphan-delete the replica set of the Scylla rack:**  
   This is necessary because Persistent Volumes (PVs) are managed by Kubernetes, but Kubernetes [does not allow PV capacity to be changed](https://github.com/helm/helm/issues/7998) after the Replica Sets are created.
3. **Manually increase PV size by adjusting the `PersistentVolumeClaim` size:**  
   The updated PVs should automatically reattach to the pods.
4. **Edit the `ScyllaCluster` resource to specify the new storage size:**  
   Bring the Scylla operator back up. It should automatically recreate the Replica Sets. At this point, there is no configuration drift.

This procedure is also documented in one of the [PRs](https://github.com/scylladb/scylla-operator/pull/2243) on the Scylla-operator GitHub repository. We applied this procedure to all the remaining nodes and successfully made the cluster storage symmetric.

Eventually, we updated our applications to accept an array of IPs as contact points, ensuring better resilience. Additionally, we set up monitoring on all nodes for critical metrics, including storage space. Now, we can sleep comfortably, knowing we’ll be alerted to any potential problems well in advance.


When the storage runs out, the troubles begin...

Storage Full


Time moves so fast; it feels like it has been just days since I wrote [2023 in Review](/blog/2023-review), and now 2024 is wrapping up. It was a superb year, and in this article, I will touch upon the things I worked on and learned this year.

## 1. Hybrid Search

Search is a ubiquitous experience that spans across our daily lives. With the touch of conversational AIs, we can make search more personalized and accessible, with the LLM acting as an empathetic guide to the topics we are searching for. Using a combination of image and text embedding models, Vector DB, and rank fusion techniques, we built a conversational search experience. The details can be found in the article linked below:

<BlogCard slug="hybrid-search" title="Hybrid Search" />

## 2. MQ Shenanigans

We have a high-traffic service in production to monitor real-time changes on product pages, backed by a Bloom filter to deduplicate requests at scale. We wanted to route the data captured to another service to commit it to a database, and we chose RabbitMQ for this task. All was fine until one day when we observed an increase in 5xx errors with traffic and encountered a strange error message: the broker received a message with a size of 1,342,177,289 bytes.

Debugging this issue was a fun experience, and I wrote more about it in a separate article:

<BlogCard slug="1342177289" title="1342177289" />

In the end, we had to switch to [rabbitmq/client](https://www.npmjs.com/package/rabbitmq-client), which implemented automatic retries and provided a useful abstraction. With that library, our server code looks more organized than ever — `publisher.ts`, `consumer.ts`, `init.ts`, `cleanup.ts`, `server.ts`, and so on! Additionally, the [@cloudamqp/amqp-client](https://www.npmjs.com/package/@cloudamqp/amqp-client) fixed [the issue](https://github.com/cloudamqp/amqp-client.js/pull/123)!

## 3. DB Admin

While Postgres is praised as an all-purpose database, cloud-managed versions often restrict its full potential by limiting the plugins that can be installed. Additionally, specialized databases come with their own benefits and performance optimizations tailored to their intended use cases. This year, I worked with a few databases, focusing on self-hosting and managing them.

### ClickHouse

While researching a vector storage engine for our Hybrid Search Engine, we wanted a SQL-compatible database that could store and index vector data alongside other types of data. [ClickHouse](https://clickhouse.com/) turned out to be a great choice for us, as it is optimized for fast analytics queries, and its vector search [is impressively fast even without indices](https://news.ycombinator.com/item?id=35637860).

We self-hosted ClickHouse using [Bitnami Helm Charts](https://bitnami.com/stack/clickhouse/helm). While the setup was straightforward, I struggled to get data distribution working until I discovered the [`CREATE DISTRIBUTED TABLE`](https://clickhouse.com/docs/en/engines/table-engines/special/distributed) DDL, which made distribution work flawlessly.

ClickHouse uses a [ZooKeeper](https://en.wikipedia.org/wiki/Apache_ZooKeeper)-compatible application called Keeper to manage synchronization among nodes. This is an interesting approach that offloads synchronization from the core database.

### ScyllaDB

To meet production SLAs, we often need a fast-read database where consistency is not critical. Several NoSQL databases excel in this aspect. [ScyllaDB](https://www.scylladb.com/) is a Cassandra-compatible database that offers excellent performance at a lower operating cost.

We deployed a self-managed ScyllaDB cluster in Kubernetes using their [operator](https://github.com/scylladb/scylla-operator). However, ScyllaDB recommends using bare-metal VMs for direct hardware access to achieve even greater performance. A limitation of the operator is that it does not allow storage expansion for nodes once the cluster is created, even though Kubernetes persistent volumes (PVs) can be scaled without data loss. We had to follow [several steps](https://github.com/scylladb/scylla-operator/pull/2243/files) to bypass the operator and perform these operations manually.

One of the strengths of ScyllaDB’s architecture is its fault tolerance. During various management operations, we experienced one or two nodes going down without impacting production. Additionally, the ScyllaDB client is cluster-aware and automatically routes queries to the appropriate nodes. Buoyed by the success of this deployment, we are now considering moving our self-hosted Redis to a cluster architecture as well.

## 4. Bashing Bash

There are many ways to deploy an application and manage its lifecycle — some managed by cloud providers, some abstracted by Kubernetes, and others entirely managed by internal tools. I worked with a set of internal tools written in Bash to migrate applications managed by them to Kubernetes. What started as a simple set of scripts to automate deployments had evolved into a massive codebase, for which Bash was far from ideal. I detailed this experience in the article below:

<BlogCard
  slug="you-have-built-kubernetes-sir"
  title="Dear Sir, You Have Built Kubernetes"
/>

## 5. Data Normalization Using LLMs

Unstructured data in the form of long-form texts, images, and audio is everywhere. One of the key challenges businesses face is converting such data, specific to their domain, into a structured format suited to their needs. Before LLMs, these operations were mostly manual and time-consuming.

With multimodal LLMs and their ability to output structured data, the problem becomes much simpler to solve, provided we craft a [simple prompt](https://kenkantzer.com/lessons-after-a-half-billion-gpt-tokens/) and supply the right schema for the desired output. Thankfully, [Vercel's AI SDK](https://sdk.vercel.ai/docs/introduction) makes this process easy. We are running these pipelines in production, asking the LLM to flag outputs where it is not confident. We also validate the outputs using business logic and flag certain cases for manual review.

We are using Google's Gemini 1.5 and OpenAI's GPT-4o for different use cases, both of which are multimodal. The results show significant improvements in accuracy and processing time. Overall, it has been a game changer.

## 6. Split Microservices

We had a consumer-facing microservice that also handled some background tasks. We observed that the consumer-facing APIs were throwing 5xx errors when the service was autoscaling due to an increase in the load of background tasks. Upon investigation, we identified that the issue was with the ingress, which was routing traffic to pods scheduled for termination during downscaling.

While GKE [suggests](https://cloud.google.com/kubernetes-engine/docs/troubleshooting/load-balancing#500-series-errors) adding a sleep before termination, we opted to split the microservice to separately handle consumer-facing UIs and background tasks, using a single flag to manage the configuration. You can read more about this approach in the article below:

<BlogCard slug="splitting-microservice" title="Splitting Microservice" />

## 7. Tree Cache

Caching enhances application performance by providing fast access to frequently accessed information. We have a function, `cached`, that wraps a function and caches its results in Redis. However, when the underlying data changes, we need an intelligent way to invalidate this cache.

Should we invalidate all the cache entries for this function? While this approach sounds straightforward, it often results in excessive invalidation. What about cases where the function has multiple parameters, and the parameter order reflects the domain model? Additionally, how can we make the `cached` function and its invalidations type-safe in TypeScript?

The article below dives deep into these challenges and presents a solution:

<BlogCard slug="tree-cache" title="Tree Cache" />

## 8. Tensor Math in JS

JavaScript began as the language of the web, but today, its scope has expanded far beyond that. JavaScript now powers backends through runtimes like Node.js, Deno, and Bun. As [Atwood's Law](https://www.laws-of-software.com/laws/atwood/) states:

> Any application that can be written in JavaScript, will eventually be written in JavaScript.

That said, the field of machine learning seems to lag behind in this regard. While there are excellent libraries like [Xenova's Transformers](https://github.com/huggingface/transformers.js) for running Hugging Face pipelines in the browser and backend, and runtimes like [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/) make it easy to execute pretrained models, a noticeable gap exists. Specifically, JavaScript lacks an accelerated matrix library like `numpy` or `jax`, as well as a comprehensive suite of scientific algorithms like `scipy`.

Fortunately, TensorFlow's JavaScript wrapper, [`tensorflow.js`](https://www.tensorflow.org/js), provides mathematical and neural network operations, offering a glimpse of hope for this gap. However, the absence of operator overloading in JavaScript introduces unique and interesting patterns in code, which I discussed in detail in the article below:

<BlogCard slug="operator-overloaded" title="Operators Overloaded" />

## 9. Akshara Tools

Language began as a medium for communication between individuals. Soon, writing followed, allowing knowledge to be passed across generations. Poetry and singing became mediums to express emotions, often evoking feelings through rhythmic patterns. The Indian subcontinent is home to numerous languages and writing systems, most of which descend from the _Brahmi_ family. I worked on a set of tools to parse and identify patterns in letters (known as _Akshara_, literally "the one that does not decay"):

- **Tokenizer**: Brahmic letters are not as atomic as Latin letters. A single letter may consist of a vowel indicator, consonant component, and additional sounds. This tool identifies the various components of a letter and displays them.
- **Prastara**: Reading a passage often involves distinguishing between long and short notes. A pattern of these notes constitutes a poetical _metre_. This tool identifies the long and short notes in a text and attempts to determine the metre (_Chandas_ in Sanskrit) in which the poetry is written.
- **Katapayadi Decoder**: Ancient Indian mathematicians devised numerous ways to encode large numbers in memorable forms. _Katapayadi_ is one such scheme where consonants map to numbers on a 0-9 scale. This tool decodes numbers from an input sentence.
- **Transliterate**: All of the above tools are implemented for the [Kannada Unicode Block](https://en.m.wikipedia.org/wiki/Kannada_(Unicode_block)), as Kannada is my native language. Since not all users are familiar with this script, there is a need to transliterate between the preferred script and Kannada. The [Aksharamukha](https://www.aksharamukha.com/) Python library provides such functionality, which I run in the browser using [Pyodide](https://pyodide.org/en/stable/). I also bypassed a few incompatible dependencies. This tool serves as a debug interface for the `aksharamukha` package.

These tools are hosted at [akshara.vinayakakv.com](https://akshara.vinayakakv.com) and are [open source](https://github.com/vinayakakv/akshara), complete with examples to explore. Go give them a try!

## 10. Self-Hosting

We have grown accustomed to cloud-based experiences that make our lives easier. However, these conveniences often come at the cost of our data being _owned_ by these entities, which might choose to monetize it, potentially compromising our privacy.

Recently, I explored self-hosting my photo library using [Immich](https://immich.app/). I delve deeper into this experience in the article below:

<BlogCard slug="self-hosting-with-pi" title="Self-Hosting with Pi" />

## 11. Year of AI

The year 2024 witnessed the rise of AI in the form of generative text and image models. The advent of intelligent chat interfaces like ChatGPT and Anthropic brought AI into our daily lives. While I, like many [others](https://www.joshwcomeau.com/blog/the-end-of-frontend-development/), believe that AI is not going to replace software engineers anytime soon, it has already become an invaluable addition to our toolbox.

Here are a few personal experiences where AI significantly helped achieve a goal, effectively turning engineers into "2x engineers" (not quite 10x yet!):

- I built the **entire** interface of _Akshara Tools_ using Vercel's [v0](https://v0.dev/). I first asked it to generate the skeleton and then iteratively requested several components. While the code output from the AI was not final — I had to edit CSS classes to replace `margin`s with `gap`s, among other tweaks — it provided a solid head start. [Here](https://v0.dev/chat/ggbkmcN2pBA?b=b_JkLpq0375IA) is the chat snippet.
- ChatGPT is extremely helpful for trivial code generation tasks, such as converting SQL DDL to a Zod schema, generating an example object from a Zod schema, or implementing methods for a Java `interface` using the same pattern as an initial manual implementation.
- One of my coworkers used ChatGPT for process-level debugging to understand why a process was stuck inside a container. It analyzed `ps aux` outputs, suggested using `lsof` to examine open files and network connections, and more. While we had to filter out some less relevant suggestions based on our knowledge, it was a great assistant.
- Another coworker successfully set up an entire Kubernetes (k8s) environment using Flux with significant help from ChatGPT, and without external assistance! Flux was configured to watch a folder and create resources in the cluster, while manually managed resources like `Secret`s were created with YAML files and commands suggested by ChatGPT.
- I migrated this blog from remote CMS-hosted articles to MDX source files with the help of [Cursor's IDE](https://www.cursor.com/). The AI wrote one-off scripts for downloading the data and converting it into the required format. However, I couldn’t achieve satisfactory results while attempting to migrate the blog from Gatsby to Astro.
- Cursor's IDE also helped me create a simple script to resize and watermark photos for [my personal blog](https://vinayakakv.substack.com/). I guided it to use `sharp` for image processing and the `bun` runtime. This demonstrates how we can decide the tools and let AI handle one-off program generation. In such cases, the quality of the code doesn't matter much as long as it works.
- We explored estimating the parameters of a normal distribution from a noisy sample dataset using ChatGPT's code execution capabilities. While we carefully reviewed the code, the analysis proved very helpful for our implementation.
- ChatGPT's ability to browse the web and perform searches has made it a powerful tool for information discovery. For example, I was searching for a music lecture using Google’s advanced operators but couldn’t find it. A few interactions with ChatGPT helped me accurately identify the presenter and the lecture. This kind of experience genuinely elevates quality of life. [Here](https://chatgpt.com/share/676bb325-1a74-8002-9bb5-3496f3f725a1) is the chat snippet of the search.
- I often use ChatGPT to spell-check and correct the grammar of my articles nowadays.

## 12. Move Fast, Break Things, Fix Fast, Don't Burn Out

Working in a startup often means dealing with uncertain requirements due to an ever-changing business landscape. Last year, we delivered a project to bring TrueFit to Shopify in a record time of three months. This year, we maintained the same pace, thanks to a commitment to [keeping things simple](https://en.wikipedia.org/wiki/KISS_principle), making small pull requests and merging them quickly, and having a hassle-free release experience with Flux and Kubernetes.

Moving at this speed often results in occasional bugs in production. However, that’s not a problem, as we can fix them quickly given our fast-paced workflow. The attitude of directing all attention to solving a production-critical issue, inspired by [_Kaizen_](https://en.wikipedia.org/wiki/Kaizen), also plays a crucial role. Once the issue is resolved, we return to our rapid development pace.

An important aspect of this model is avoiding stress from deliverables. This requires brutal prioritization of tasks to stay focused on what truly matters. Taking rest whenever necessary is essential for preventing burnout. The key is being self-aware enough to recognize early signs of burnout and actively taking steps to recover.

## Wrapping Up...

As evident from the above paragraphs, 2024 was a fantastic year. There are a few things I might have missed mentioning and some I wish to elaborate on, which I plan to do in the coming days. None of this would have been possible without the unwavering support of my family and friends, a collaborative and trust-based workplace, incredible coworkers, and the tranquility of the place I call home.

With that, I conclude this article with an optimistic outlook for 2025. It’s easy to be pessimistic or neutral, but by being an [optimist](https://www.youtube.com/cleoabram), you can help craft the world you imagine.

Happy New Year, y’all!


2024 - Year In Review


In addition to being a Software Engineer, I'm also a [Photographer](https://vinayakakv.substack.com/). I take a lot of photos, and as a result, I ran out of free storage space on Google Photos. Rather than paying for a recurring subscription to Google for additional storage, I decided to self-host my photos. I was inspired by a [friend](https://www.linkedin.com/in/vinyasns/) who was already doing this.

## The Software

There are many self-hosted photo solutions, each with its own niche. Here are a few that I know of:

- [PhotoPrism](https://www.photoprism.app/)
- [Ente](https://ente.io/): Focused on secure storage of photos, with encryption at rest.
- [Immich](https://immich.app/): Offers a Google Photos-like UI and experience, along with an app.

I decided to go with Immich because the setup is straightforward—just pull the `docker-compose.yml`, configure the `.env` file, and run it!

## Takeout and the First Backup

Google Photos provides an option to export all your photo data, but there's a catch—the EXIF metadata for the photos is stored in a separate `.json` file. Thankfully, there are tools available to merge this information back into the EXIF metadata. I used [GooglePhotosTakeoutHelper](https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper) for this task.

Once the photos were prepared, the upload process was straightforward. Immich provides a CLI, which can also be run inside Docker, to batch upload the photos. The CLI handles deduplication automatically. Given the large volume of photos I had, the first upload took most of the day to complete.

## Daily Backup

Most of my photos are taken on my smartphone, and the Immich app can automatically recognize and back them up to the server. However, when the server and the smartphone client are not on the same network, communication becomes challenging.

To address this, the server needs either a static IP or a dynamic DNS setup to be accessible from outside the local network. However, exposing your server to the public internet comes with risks. Without proper firewall configurations, it can become a potential security vulnerability.

Fortunately, [Tailscale](https://tailscale.com/) comes to the rescue in such cases. It creates a VPN where devices can be manually added by installing a daemon. Once on the VPN, the devices can communicate as if they were on the same local network—all **without requiring a central VPN server!** Their [technical architecture](https://tailscale.com/blog/how-tailscale-works) document provides an excellent summary of how this is achieved.

With the server and smartphone on the same Tailscale network, the backup process becomes seamless. Thanks to Tailscale's efficient architecture, communication remains fast even over a mobile 5G network.

## To the Pi

Initially, I hosted the server on my laptop, which I turned on only when backups were needed. However, this quickly became inconvenient when I wanted to access photos remotely—it meant I had to keep the laptop on and awake at all times.

Soon, I decided to upgrade the setup to a [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/) with 8GB of RAM. While it doesn't match the performance of Apple Silicon for image processing, it is sufficient for my needs. I exported the PostgreSQL database from the laptop, which dumps SQL `INSERT` statements, and imported it into the Pi's database. The external hard disk storing the photos required no changes.

> The ease with which I migrated the application from one host to another highlights the beauty of abstraction, where an application, along with its dependencies and runtime, is bundled as a _container_. These containers are the foundation of modern cloud applications, enabling effortless scheduling and scaling.

## Pi Shenanigans

When I purchased the Raspberry Pi, I didn’t have a micro-HDMI to HDMI adapter, which meant I couldn’t see any error messages displayed on the screen. Initially, I attempted to boot the Pi from a USB pendrive instead of a MicroSD card due to past experiences with overheated and damaged memory cards. While I successfully burned the image onto the pendrive, the Pi refused to boot.

After deciding to abandon the pendrive idea, I opted to use a MicroSD card instead. However, I quickly discovered that I didn’t have a MicroSD adapter to connect the card to the SDXC port on my MacBook. Luckily, I found an old adapter lying around—but it was missing the lock switch. This caused the card to mount in read-only mode, leaving me worried that the card itself might be defective.

Curious, I opened the adapter and found that the connections inside were mostly 1:1, with no complex circuitry for the lock mechanism. I learned that [any firm obstruction in place of the missing lock switch could force it into the "unlocked" state](https://www.youtube.com/watch?v=oKrn-0jXZCU). With this simple fix, I successfully burned the image and finally booted up the Pi.

## Storage Architecture

While it's possible to format a plain hard disk and use it to store photos, migrating to another disk can become quite challenging, especially when dealing with a large number of small files. Achieving fault tolerance would typically require hardware-level RAID, which is not cost-effective at a smaller scale. Additionally, extending the storage by adding another disk and exposing it as a single volume to applications is often not straightforward with this approach.

[Logical Volume Manager (LVM)](https://wiki.archlinux.org/title/LVM) offers a solution to the challenges mentioned above. I learned about LVM from [a friend](https://www.linkedin.com/in/gregdaydevops/), and it immediately resonated with me. LVM allows for software-level RAID, the ability to extend volumes by adding additional disks, and even growing or shrinking partitions online (subject to the underlying file system). These features make it an incredibly flexible and powerful tool for managing storage.

For now, I use a simple setup with a single disk, a single volume, and a single BTRFS partition. BTRFS also provides storage-level data compression, which is a bonus. If needed, additional storage can be added easily, and RAID configurations can be set up for fault tolerance in the future.

## Is it Worth It?

While paying Google for storage might seem like an easy option, it comes at the cost of relinquishing ownership of your data. In some cases, Google can [lock you out of your data](https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html). Photo data is highly valuable and can [reveal a lot of personal information](https://theyseeyourphotos.com/), which could even have financial implications.

On the other hand, self-hosting has its own downsides, such as maintenance. You need to treat your system as if it were a production-critical setup, with regular database backups, data replication, and hardware checks. However, the level of maintenance is entirely up to you — you’re free to do more or less depending on your comfort and requirements. The key trade-off is that you’re the owner, and with that ownership comes full responsibility.


How I self host a Photo server with Raspberry Pi

Self-Hosting with Pi


It is the early 2010s, and you have a bunch of Java applications to deploy. There are good servers like [Tomcat](https://tomcat.apache.org/) and [Jetty](https://jetty.org/index.html). You procure a VM in the cloud, set up the OS and servers, deploy the application, and expose it to the public. All is good, until...

## There is More Traffic!!

Congrats! You have grown your customer base, and your application is experiencing traffic that cannot be handled by a single VM. Now, you decide to horizontally scale the system by adding more VMs. You do not want to keep writing the same commands to configure and deploy applications to every machine, so you decide to write a bunch of Ansible and Shell scripts to handle these tasks for you. Thus, `deploylib` is born.

Now, you don't want to be running these scripts from your machine and want to give control back to the developers. So, you also create a Jenkins VM with appropriate permissions, write a `Deploy` job with suitable parameters that `ssh`es into every machine and handles the deployment. Cool, we’ve solved the problem, right? Not quite yet...

## Rolling Deployment

During a deployment, you don’t want all machines to go down together when updating to a new version of the application. You want the process to happen in groups so that you can abort the deployment if you notice anything going wrong in the earlier groups. To achieve this, you logically define the groups, update your `deploylib` to handle it, and break the `Deploy` job into phases to ensure everything runs smoothly.

Whenever there is a new version of the application, you run the `Deploy` job, and the apps go to production _automagically_ with no downtime. A phenomenal achievement! But one day, you wake up to a midnight page that one of the production-critical applications is down...

## Hanged State Machines

The particular version of the app was running smoothly in production for 15 days—until it decided not to. You check the application logs, run `htop`, and observe that there is a memory leak. The engineering team is busy with other business priorities, and fixing the memory leak is not one of them. Memory leaks, after all, require dedicated effort to identify and fix.

A bright engineer comes up with a workaround — just restart the applications every day, but outside business hours! All applications are essentially giant state machines, and a restart is often an easy way to fix them, as they return to their initial state upon restarting. Since deployments already restart the apps, you configure Jenkins to schedule the `Deploy` job daily, using the last deployed version.

Nice! At least now you can sleep without interruptions.

## Growth Brings New Challenges

More growth is always good news since it brings in more revenue. Confident that the system you’ve built can be easily scaled to handle new traffic, you decide to add two more groups. However, as soon as the new groups are added, a database alert — its CPU utilization is way too high.

The architecture you designed is horizontally scalable, yes — but your poor PostgreSQL instance is not. To address this issue, you decide to take the following steps:

1. Vertically scale the PostgreSQL instance for now, or better yet, enable the cloud provider’s autoscaling feature.
2. Move business-critical objects to a horizontally scalable NoSQL database, such as MongoDB.
3. Break down your monolith into microservices, with each microservice owning its own database, ensuring non-critical apps don’t interfere with critical ones.

All great! You implement #1 immediately. Points #2 and #3 take some time, but they are eventually completed.

## Microservice Heaven

You split your monolith app into multiple microservices, carefully defining the responsibilities of each microservice and exposing APIs to facilitate communication between them. To handle asynchronous communication, you deploy a message broker. Each microservice now has its own database, and fortunately, one or two production-critical microservices receive the most database resources.

However, not everything is shiny in the world of microservices. To sync certain business _facts_ across them, you end up with some data duplication. You decide this is a worthwhile tradeoff for the functionality it brings—after all, every engineering decision involves tradeoffs.

To deploy these microservices, you provision a VM for each one and scale it based on its needs. Naturally, your `Deploy` job requires updates—along with managing _groups_, you now need to handle each microservice individually. Taking the time to address this, you parameterize the job to accept the microservice as an input while carefully accounting for the rolling deployment process.

At this stage, `deploylib`—your set of shell scripts for automating deployment — is growing significantly. It now includes the deployment logic, microservice configurations, and more, all centralized in one place.

## Welcome, Kubernetes!

While you have automated most of the deployment setup, it’s still not completely free from human intervention. Services occasionally fail in the middle of the night, despite running the `Deploy` job every 12 hours now. The job has grown so complex that it sometimes catastrophically fails, leaving the system in a half-new, half-old state.

You decide to migrate the applications to Kubernetes (k8s) to take advantage of its automated recovery and autoscaling functionalities. While the microservices you’ve built should theoretically be easy to containerize, several challenges arise due to the way the apps were architected:

- The application runtime doesn’t work well in containers, as it relies on direct access to hardware resources.
- The build step generates numerous artifacts, requiring careful orchestration to produce a container image.
- You need to maintain backward compatibility — apps migrated to k8s must still be able to communicate with apps outside of it, and vice versa.
- The apps have a significant startup time because they load all application-critical data from the database into an in-memory cache layer to meet SLA requirements. The database access itself is slow, as it is highly normalized and requires around 17 joins for even simple queries!

While these challenges are addressable, you still have an affection for your `deploylib` framework and decide to repurpose it for gradually deploying new k8s apps. By leveraging a combination of tools such as `envsubst`, `kubectl apply -f`, `kubectl wait`, `kubectl get`, and others, you integrate k8s deployments into `deploylib`.

Although Kubernetes offers native rolling deployments, your architecture requires a different approach. To accommodate this, you create `Service` sets exposed behind a single `Ingress` and modify your `Deploy` job to handle rolling deployments. This results in Kubernetes performing rolling deployments within your `Deploy` job's rolling deployments — a "double roll"! 🙈

## Was it Really Needed?

Yes, the circumstances of the time demanded it, and you believed a simple set of deploy scripts would make life easier. However, over time, `deploylib` evolved into a [36k+ LOC shell script monster](https://bsky.app/profile/kelseyhightower.com/post/3l7m23hnv242k), with utilities written for almost everything. Even after Kubernetes was introduced, you stuck with the framework due to business requirements for making the migration _fast_. Ironically, the work ultimately slowed down because `deploylib` became a bottleneck that tied you up!

Modern GitOps-based Kubernetes deployment tools like [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) and [FluxCD](https://fluxcd.io/) can greatly simplify the deployment process. Additionally, tools like [external-dns](https://kubernetes-sigs.github.io/external-dns/v0.14.0/) and [external-secrets](https://external-secrets.io/latest/) can make Kubernetes deployments fully self-service, eliminating the need for a dedicated ops person.

After careful technical consideration and review, you decide to adapt these modern tools and finally retire `deploylib`, hoping that this new system doesn’t turn into the next `deploylib`.

## Is Shell Really Good for Large Projects?

While shell scripts can simplify and automate a few workflows, any large collection of shell scripts becomes difficult to debug due to several factors, including:

- **Variable scoping:** A variable can exist before the script runs and might persist after the script ends. While `local` can help, there still needs to be a robust way to communicate between different scripts.
- **Many ways to do the same thing:** For example, should we use `[ ]` (a.k.a. `test`) or `[[ ]]` (a shell built-in) for a condition? To check if a variable is empty, is `-z` preferable, or should we use `$var == ""`?
- **Syntax hacks:** Constructs like `${var/find/replace}`, `${var%substr}`, `${var:-default}`, and `${var##substr}` can be confusing and make life hard for readers unfamiliar with these conventions.

In such cases, a dedicated programming language with optional static typing can be a better choice, as long as the program has a consistent CLI. After all, [`kubectl`](https://github.com/kubernetes/kubectl) is a `Go` project, not a collection of shell scripts! [Kamal](https://github.com/basecamp/kamal) also deserves an honorable mention in the deployment world as a project of this kind.

Some tools, like [zx](https://github.com/google/zx), bring shell primitives to higher-level programming languages, offering the best of both worlds—for example, you can use features like `Promise.race()` to manage processes efficiently!

> This article is inspired by [Dear Sir, You Have Built a Compiler](https://rachit.pl/post/you-have-built-a-compiler/)


A tale of how a simple deployment script turned into a monster that could rival Kubernetes.

Dear Sir, You Have Built Kubernetes


Operators are a fundamental part of a programming language. They allow expressing expressions, mathematical or otherwise, elegantly and reduce the cognitive burden on the reader. Some programming languages take the concept of operators one step further and allow them to have different meanings depending on the context in which they are used, making them even more powerful. This approach is popularly known as "**Operator Overloading**." As with many things in software engineering, any flexibility comes with its own set of trade-offs. In this article, we will explore a few cases where operator overloading can be helpful or problematic.

## The Ways of Operator Overloading

Every language has its own way of defining operator overloads. I was introduced to the concept in C++ while following the book by [Daniel Liang](https://www.amazon.in/C-Programming-Language-Y-Daniel-Liang/s?rh=n%3A14447041031%2Cp_27%3AY.+Daniel+Liang). In C++, operators are special functions whose names begin with the keyword `operator`. For example, an addition operation for complex numbers can look like this:

```cpp
// test.cpp
// compile with `clang++ -std=c++20 test.cpp`
#include <iostream>

#include <format>

using namespace std;

class Complex {
  private: float real, imaginary;

  public: Complex(float real, float imaginary) {
    this -> real = real;
    this -> imaginary = imaginary;
  }

  Complex operator + (Complex & another) {
    return Complex(this -> real + another.real, this -> imaginary + another.imaginary);
  }

  string toString() {
    return format("{} + {} i", this -> real, this -> imaginary);
  }
};

int main() {
  Complex c1(2, 3), c2(1, 0);
  Complex c3 = c1 + c2; // This works
  cout << c3.toString() << endl; // 3 + 3 i
}
```

Apart from the `operator+` function, we also see the `<<` operator used for stream output, which can also be used for bit shifts. The arrow operator `->` is used for pointer member access (i.e., `a->b` and `(*a).b` are equivalent) since `this` is a pointer. C++ allows all of these operators to be overloaded as well, but it does not allow changing their precedence.

Other languages like Python and Kotlin require the implementation of special methods as part of the class in order for them to be exposed as operators. For example, we can achieve the above result in [Kotlin](https://pl.kotl.in/XkrlBsH0F):

```kotlin
data class Complex(val real: Double, val imaginary: Double) {
  operator fun plus(other: Complex): Complex {
    return Complex(this.real + other.real, this.imaginary + other.imaginary)
  }
}

fun main() {
  val c1 = Complex(2.0,3.0)
  val c2 = Complex(1.0,0.0)
  val c3 = c1 + c2
  println(c3)
}
```

As we can see from the above examples, operator overloading makes the interface (API, if you will) of the class more _natural_, and the expressions more readable, making them feel closer to the way we write mathematical expressions. When the mathematical concept of arithmetic makes sense, for example, [Group theory](<https://en.wikipedia.org/wiki/Group_(mathematics)>), operator overloading can be particularly useful.

## Beyond Arithmatic

The set of operators offered by some programming languages is not limited to arithmetic operators. It includes bitwise operators and beyond. C++ is famous for using shift operators with streams.

```cpp
std:cout << "Hello world!" << std::endl;
```

While this approach looks okay to begin with, string formatting becomes increasingly complex.
```cpp
std:cout << "Hello world! I am a number" << num << std::endl;
```

Compare that with plain old `printf`:

```c
printf("Hello world! I'm a number %d\n", 10)
```

The `format` function used in the first example also better.

The Spray router in Scala (deprecated) takes operator overloading to another extreme and uses a bash of operators to [define HTTP server routes](https://spray.readthedocs.io/en/latest/documentation/spray-routing/):

```scala
trait LongerService extends HttpService with MyApp {
  val route = {
    path("orders") {
      authenticate(BasicAuth(realm = "admin area")) { user =>
        get {
          cache(simpleCache) {
            encodeResponse(Deflate) {
              complete {
                // ...
              }
            }
          }
        } ~
    } ~
    pathPrefix("order" / IntNumber) { orderId =>
      pathEnd {
        (put | parameter('method ! "put")) {
           // ...
        }
      }
    }
  }
 }
}
```

Actor-based programming languages like Erlang support a dedicated operator `!` to send a message to an actor:

```erlang
%send message Message to the process with pid Pid
Pid ! Message
```

Haskell uses a dedicated operator `>>=` for the Monad `flatMap` operation. For [example](https://play.haskell.org/saved/lvSs9d56):

```haskell
duplicate:: a -> [a]
duplicate input = [input, input]

main :: IO ()
main = do
  let result = [1,2] >>= duplicate
  print result
  -- [1,1,2,2]
```

Monads are an interesting concept and are subject to lots of [memes](https://antousias.com/content/images/2020/11/44b0bd758f8ee5c81362923f0d5c8e017c9ddf623925e60c29a4c015b89fbb45.jpg). I hope to explore them from a mathematical perspective one day.

In all these examples, operators are used to express domain-specific concepts, which might take some time to get used to. In the Spray router example, it is likely to scare a first-time viewer!

## No Operator Overloading

Many languages do not support operator overloading, including Java and JavaScript. In these languages, libraries expose methods instead of operators. For example, let's consider calculating the likelihood of sample data under a normal distribution $\mathcal{N}(\mu, \sigma)$. Mathematically,

$$
p = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}
$$

A python program written with `numpy` can look like:

```python
import numpy as np

mu = 1
sigma = 0.5
x = np.array([0, -2, 1, 2.5, 3])

p = 1 / (np.sqrt(2 * np.pi * sigma ** 2)) * np.exp(-(x - mu) ** 2 / (2 * sigma ** 2))

print(p)
# [0.10798193, 0.00000001, 0.79788456, 0.0088637 , 0.00026766]
```

If we were to write the same expression in NodeJS with Tensorflow.JS (there are not much tensor libraries for JS, a topic for another day):

```ts
import tf from '@tensorflow/tfjs-node'

const mu = 1
const sigma = 0.5

const x = tf.tensor1d([0, -2, 1, 2.5, 3])
const p = x
  .sub(mu)
  .pow(2)
  .mul(-1)
  .div(2 * sigma ** 2)
  .exp()
  .div(tf.sqrt(2 * Math.PI * sigma ** 2))

console.log(p.toString())
// [0.1079819, 0, 0.7978846, 0.0088637, 0.0002677]
```

### Which one of these is better?

The answer is subjective: the Python version follows the mathematical structure and preserves its pattern, while the JavaScript version adds verbosity but makes the operation linear.

In fact, there is more than one way to convert a mathematical expression to a function-chaining format. This way, the programmer can explicitly choose the order of operations, as opposed to relying on precedence and grouping to decide for us. The chain explicitly spells out the computation graph.

### So, is operator overloading good or bad?

As always, it depends. Operator overloading is one of the powerful tools that make a programmer’s life easier. However, it has the potential to be misused, creating esoteric and often unergonomic patterns. When used in the right places, it’s more beneficial than harmful.

On the other hand, the unavailability of operator overloading forces library developers to expose chainable interfaces, which naturally lead into computation graphs—though at the cost of readability.


When an operator can have multiple meanings based on the context...

Operators Overloaded


Search has become an integral part of our daily lives. With major players like Google and Bing, as well as mission-oriented options like [Ecosia](https://www.ecosia.org/) and [DuckDuckGo](https://duckduckgo.com/), a variety of search engines have tackled the challenge of navigating the vast expanse of the internet.

However, these search engines cannot solve the search problem on a smaller, customizable dataset, such as an e-commerce catalog. In such cases, we have access to product and user data. This data can be used to derive personalized search results. In this article, we shall explore the technology choices we can make when building a multi-modal, conversational search experience.

## Conversational Search

Traditionally, search is thought of as typing something into a text box and expecting results. Several applications have [invented their own query languages](https://rachit.pl/post/you-have-built-a-compiler/) to enhance the capabilities of what cannot be achieved with a full-text search. The rise of LLMs has promoted search experiences like [Perplexity.AI](https://perplexity.ai) (recently, OpenAI also released their own version), which can tolerate imprecise inputs and provide context that encourages natural exploration.

The APIs of these LLMs are now capable of [Function Calling](https://platform.openai.com/docs/guides/function-calling), which allows us to expose custom business APIs to the LLMs and provide a complete in-chat experience.

## Multimodal Search

Words are never sufficient to explain something, which is where adding an image input into the mix can vastly enhance UX and reduce frustration. We can take it a step further by using [live audio and video capabilities](https://openai.com/index/hello-gpt-4o/) to remove text input completely from the equation. For the sake of this discussion, we are limiting ourselves to image inputs only.

## Technology Choices

Based on our requirements above, we have a few technology choices to make:

- Which LLM and LLM provider to use to power the conversational interface?
- How do we make our data accessible for multimodal search?
- What backend and UI frameworks to use?

### LLM Provider and APIs

We have a lot of choices for LLM providers in today's market:

- **Giants like [OpenAI](https://openai.com/api/pricing/), [Anthropic](https://www.anthropic.com/pricing), and [Gemini](https://ai.google.dev/pricing)**: These are the big players in the market and offer a variety of models to choose from. Since they operate at a large scale, their costs tend to be lower for low-volume applications. Gemini recently started [providing OpenAI-compatible APIs](https://ai.google.dev/gemini-api/docs/openai), making it easy to switch between providers.
- **Self-hosted models like LLaMa and Qwen**: Many open-weight (and some open-source) models are available on Hugging Face. You can evaluate, fine-tune, and self-host them. For self-hosting, since we need GPU VMs, the added cost and maintenance factors only make sense if you have a high load.
- **Open models on third-party hosting**: There are many third-party LLM hosting solutions, including Hugging Face, which offer competitive pricing and API compatibility options. They are a perfect tradeoff between giants and self-hosting.

> **A note on interoperability**: While OpenAI-compatible APIs and libraries like Vercel's AI SDK can make it seem like switching between LLMs is easy, it often isn't. I've seen GPT-3.5 and GPT-4 behave vastly differently on the same prompt. This might be due to the way the models are trained and fine-tuned. But this is definitely something to watch out for.

We can use the following generic criteria to choose the LLM and provider best suited for our task at hand:

- **Streaming vs Non-Streaming APIs**: LLMs are essentially next-word prediction models. Some APIs generate an entire paragraph as a response, while others return a streaming output. For an interactive experience, streaming output tends to provide better UX, as users do not feel stuck.
- **Context Window Size**: An LLM's ability to reference past text is limited by the length of its context window. If an LLM has a small context window, it might forget what the user said at the beginning, leading to frustration. We also need to consider dropping unimportant messages from the context window once it is full.
- **Speed**: LLM token-generation speed can vary based on its size and the infrastructure it is hosted on. The faster the LLM, the better the experience. However, overly fast LLMs can cause a flickering UI that may artificially need to be slowed down. Since our custom function calls also add delay, a fast model suits our purpose.
- **Modality**: Some LLMs are text-based, while others are multi-modal. While our image search use case works better with multi-modal LLMs (if they can understand the image and generate text input with an accurate description), text-based models can also be used—with image search being a separate pipeline.
- **Visibility**: It is important to understand how LLMs behave when they are running in the real world. While one can build a monitoring solution to review random chats, out-of-the-box solutions like [OpenAI Threads](https://platform.openai.com/docs/api-reference/threads) can be very helpful.

### Data Preparation and Storage

We need to make our product data accessible for both natural language and image search. For this task, we can leverage multimodal embeddings such as [CLIP](https://huggingface.co/docs/transformers/en/model_doc/clip) to generate text and image embeddings, which can then be used during the search.

There are many vector databases like [Milvus](https://milvus.io/) and [Qdrant](https://qdrant.tech/) that tackle the problem of returning the nearest vector given an input vector. Since we also want our search to include conventional filters (such as product category, age group, etc.), a plain vector database will not suit our use case. We are addressing the Hybrid Search problem, where both vector search and conventional search are used together. Fortunately, many conventional databases now offer vector search extensions:

- **Postgres with [pgvector](https://github.com/pgvector/pgvector)**: If you already have your main business data in Postgres, pgvector is an easy choice.
- **Purpose-built Hybrid search databases like [MyScale](https://myscale.com/)**: MyScale is a Clickhouse fork that adds vector indexing and vector search functionality. They also support native functions for full-text search.
- Almost every database now ships with vector functions and vector indexing.

We can evaluate the database choices using a few key points:

- **Online vs Offline Indexing**: Can our vector index support real-time insertions and updates, or do we need to run reindex operations periodically? While running a massive indexing operation may seem easy at first, it can quickly become unmanageable as data volume increases. Online indexing is always better if you have constantly changing or growing data.
- **Vector Functions**: We are focused on _distance_ functions that return how similar two input vectors are. These functions can either be used in the `ORDER BY` clause to get the nearest matches, or in the `WHERE` clause to filter the dataset into records of interest. What distance functions a database supports and how it limits their usage is an important factor. For example, a database that supports distance functions only in the `ORDER BY` clause is limited in that it can only strictly select a subset of the data where the distance to the source vector is greater than `0.8`.
- **Speed**: Since our search is conversational and the database queries are invoked by the LLM (which has its own delays), we need to ensure that data retrieval is fast.
- **In-database Embedding Functions**: Some databases offer functions that allow the caller to specify an embedding model and the input in order to generate the embeddings. While this eliminates the need for the application to generate query embeddings, it can potentially lead to inaccuracies if the embedding model for query embedding and data embedding are different.
- **Ease of Data Migration**: If the primary business database and the search database are different, we need to consider whether the original database supports Change Data Capture (CDC) events, which can be used to detect changes and insert them into the search database with embeddings computed. Otherwise, a periodic data sync job might be a good option.

### Backend and UI frameworks

Python is the go-to language for all ML-related activities due to the abundance of ML libraries and plug-and-play models, thanks to [Hugging Face Pipelines](https://huggingface.co/docs/transformers/en/main_classes/pipelines). [LangChain](https://www.langchain.com/) also provides primitives and extensions to wrap around LLMs, but it comes with a [reputation for being unnecessarily complex](https://minimaxir.com/2023/07/langchain-problem/).

On the other side, JavaScript is catching up with the state of things with ML libraries like [TensorFlow.js](https://www.tensorflow.org/js) and [Xenova's port of Hugging Face pipeline](https://www.npmjs.com/package/@xenova/transformers). Vercel's AI SDK ([`npm i ai`](https://www.npmjs.com/package/ai)) is also a zero-cost abstraction over multiple LLM providers, offering both backend streaming patterns and frontend hooks for popular frameworks like React, Svelte, and Vue.

At the end of the day, technology choices should be made to provide the best user experience, with frictionless developer experience coming second.

## The Hybrid Search

Now that you’ve made technology choices, prepared the data, stored it, and are ready to take user input, let's say the user is querying for "Shirts to wear for a Hawaii vacation." The LLM, detecting the user's intent, responds with a message indicating that the user should wait while it searches for those shirts, and invokes our `find` function with the following parameters:

- `category=shirt`
- `tags=['vacation']`
- `query='casual, breathable shirt for vacation in Hawaii'`

Now, based on the context in the request (possibly from a cookie), we know the user ID, which allows us to infer:

- `gender=Male`
- `pastPurchases=[...]`

We can generate a text embedding and query the product database as follows:

```sql
select title, price, image_url, distance(embedding, queryEmbedding) as distance
from products
where gender = 'male'
  and category = 'shirts'
  and tags in ('vacation')
order by distance desc
limit 100
```

Let’s also assume that we have two additional scoring algorithms:

- `imageSimilarity(input_image, target)` that returns how similar the user input image is to a given image.
- `likelihood(pastPurchases, product)` that returns how likely a given product is to be purchased based on past transactions.

Running these functions on our DB results, we end up with three scores for each of the filtered products. We need to combine these scores and re-rank the results to show the most appropriate results for the user. Although these scores are numerical and might fall within the same range, they cannot be combined arithmetically since they come from different sources and have different meanings. For example, a likelihood score of `0.5` is vastly different from an embedding distance of `0.5`. Even among different embedding algorithms, the same score can represent different levels of similarity.

### Rank Fusion

In order to solve this problem, we need to convert these scores to have the same meaning. Converting _scores_ to _ranks_ is a common method employed in these scenarios. We order our dataset by these scores so that the nearest entry appears first, and then take the position of the entry as its rank. In our example, we will create three sets of products ordered by embedding distance, image similarity, and likelihood, respectively.

For every product in the set, we can now assign a score as follows:

$$
\mathrm{score} = \sum_{i \in \mathrm{Rankset}} \frac{w_i}{b_i + \mathrm{rank}}
$$

This expression gives more weight to the highest ranks, while giving rapidly decreasing weights to later ranks. The constant $b_i$ dampens the effect of early ranks contributing more weight, whereas the constant $w_i$ acts as the weight for the rank set. These constants have to be set according to the business requirements. This technique is called [**Reciprocal Rank Fusion**](https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking) and is one of the basic techniques to combine ranks. There are other more sophisticated methods that can take user feedback into consideration as well.

After the rank fusion, we get a score for each product that we can use to sort the product set and present it to the user. This product set is obtained based on both regular and multimodal search inputs, along with the personalization factor added in, thus enhancing the UX.

## Feedback loop and Maintenance

No system will be perfect on day one. We need to keep an eye on our conversational search experience and measure its impact, for example, using conversion metrics. We can also gather explicit user feedback by asking them to vote on LLM responses and recommendations. Additionally, we should monitor for possible abuse of the conversational experience and detect attacks or jailbreaks by randomly auditing PII-removed user interactions.


Techology choices that we can make while building a conversational, multimodal search experience

Hi there, Welcome to Vinayaka's space!

Blog

Storage Full

2024 - Year In Review

Self-Hosting with Pi

Dear Sir, You Have Built Kubernetes

Operators Overloaded

Building a Search Engine