Abhishek Sah — Blog

Executable Onboarding: From Docs to Agents

Wed, 20 May 2026 10:00:00 GMT

Most backend services of any real size are painful to bring up locally, and once they are running, testing your changes by calling the actual APIs and walking through real flows is just as tedious. With the rise of LLMs, the better answer is not a longer README. It is a small agent. That agent can run the setup itself, remember what it has done, and follow instructions in plain English, both for bringing the service up and for driving the tests on top of it. This post is about one such agent, built for the service I work on day to day.

These days at work, I am mostly building on Frontier, an open source identity and billing platform from Raystack. On the identity side, it owns the org, project, and service-account model, and it manages roles and permissions through an authorization graph backed by SpiceDB. On the billing side, it tracks plans, subscriptions, features, usage, and invoices, and it integrates with a payment provider so the same organization model handles both “who can do what” and “who pays for what.” In practice, a B2B SaaS team using Frontier does not have to rebuild the usual primitives from scratch, like tenant isolation, role-based access, SSO, service accounts for API traffic, plans wired to feature entitlements, metered usage, and invoicing. All of those come out of the box, with one consistent org model underneath. That is what a mature platform here is supposed to do: take the parts every enterprise product re-implements and turn them into something the application team can configure rather than write.

That breadth is what makes Frontier useful, and it is also what makes it expensive to test locally. A single change can travel through identity, authz, and billing in the same request path. This post is about a Claude Code plugin I wrote to close that gap.

The Cost of a Real Local Test

Frontier needs PostgreSQL (two databases, its own and SpiceDB’s), a SpiceDB instance, migrations applied, and the server built from source against the right config. Once the server is running, you still have to authenticate, and there is more than one way to do it depending on who you are calling as: mail OTP for regular users, headers for service users, and a separate super-admin path tied to the config file. Each path gives you a different session, and each session can only call a specific subset of RPCs.

To test a change touching project role assignment end to end, you bring up the stack, run the OTP flow, capture the cookie, create an org, a project, a service user, and a few users with different roles, and only then can you call the RPC the change is actually about. On a good day, an hour is gone before you send the first meaningful request. On a bad day (a fresh laptop, a stale config, a SpiceDB version mismatch, a forgotten migration), it has been known to take a couple of days before someone gets the stack working end to end.

So most engineers do not do this. They write a unit test, watch CI go green, and ship. But a unit test does not catch the kinds of bugs a change like this usually causes: stale SpiceDB relations after a role update, a service user RPC that breaks on a missing header, a role assignment that reads correctly in code but leaves incomplete relations in the authz graph. These bugs are usually small once you find them. The painful part is that you only find them after the change has shipped.

The stakes are higher here than they are for most services. For an identity and billing platform, correctness and security are the whole point of the product. A small bug in authorization means someone sees data they should not see. A small bug in billing means a customer gets an invoice that is wrong. Neither of those is something you can fix quietly in the next release. They end up in incident channels and on support tickets. Catching this kind of bug locally, before it ships, is much cheaper than catching it after.

The Plugin

frontier-sandbox is a Claude Code plugin, published in a small marketplace repo. It does two things that are usually treated separately: it takes care of the local setup, and it drives the actual RPC testing on top.

Setup. One command brings up the stack: it detects an existing session and reuses it, otherwise it provisions PostgreSQL and SpiceDB (Docker by default, local install as a fallback), builds Frontier, runs migrations, and starts the server. rebuild restarts the binary without dropping data.

Testing. Authentication is where the plugin saves the most time. Frontier’s config already supports test identities with a hardcoded OTP (useful for local and CI, switched off in production), and the plugin knows how to take advantage of that. It works with a small, predictable set of users covering the three identity types you usually need: regular users, service users, and super admins. When you ask it to issue an RPC as one of them, it runs the OTP flow against that hardcoded value, persists the cookie, and reuses the session across subsequent calls. It also reads Frontier’s proto definitions, so it can resolve request and response shapes inline and emit a curl equivalent on demand.

Beyond auth, the plugin takes care of the rest of the tedious testing work too. Without it, you would be hand-crafting curl calls or building Postman requests, filling in dummy payloads, looking up UUIDs you just created, reading JSON responses line by line, and then repeating the whole exercise for every edge case the API has, like invalid inputs, wrong-scope arguments, missing fields, and the occasional monkey test where you throw something deliberately strange at the endpoint to see how it reacts.

The plugin handles each of those: it shapes the request from the proto, fills in identifiers it already knows about, summarizes the response, and walks through the edge cases as it goes. You spend your time deciding what to test, not what to type.

Docker Gives State, the Plugin Gives Memory

You can get most of the way there with a docker-compose file and a seed script. That is in fact what the plugin uses for its Docker mode under the hood. What that setup does not give you is memory across a working session: which identity you logged in as, which cookie is still valid, which RPC needs the super-admin path, or which entities you just created and might want to call against next.

That continuity is the actual product here. You say “create an org as super admin,” then “now add user2 as a viewer on the project we just created,” and the skill figures out which project you mean, picks the identity with the right scope, and shapes the request accordingly. Docker compose can hand you a running service, but the plugin goes one step further: it hands you a running service that also keeps track of what you have already done with it in the same session.

What This Changed

A recent PR I shipped, #1481, touched project role assignment. Without the plugin, I would have run two or three scenarios by hand. With the plugin handling both the setup and the actual test calls, twenty-nine scenarios got covered before I opened the PR. I prompted and reviewed; the plugin did the running. The breakdown was nine on the happy path, seven on authorization (org owner, org admin without project role, project viewer who should be denied, unauthenticated caller, and the obvious adjacent cases), and thirteen on validation (invalid UUIDs, non-existent principals, wrong-scope roles, empty bodies). I posted the full table of what was tested and what passed as a comment on the PR, so the reviewer could see at a glance what had been covered without having to ask.

Each scenario ran against a clean database with the right identity, against a binary I had just rebuilt. The actual runs took seconds. The LLM proposed most of the cases itself, given the diff and the proto definitions. My job was to review the list, fix any issue that turned up during a run, ask the plugin to rebuild and restart the server, and retest. The bootstrap and the manual testing work that used to eat up that time are now effectively free.

The same run also caught a real bug. In earlier local testing, some of the authorization cases had been passing when they should have failed, because SpiceDB still had permission relations left over from previous runs. The code path doing the role update only ever added new relations and never cleaned up the old ones, so the leftover state was hiding the problem. Once every run started from a clean database, the false passes went away and I could finally tell whether the fix actually worked.

The Broader Lesson

Most backend services of any real size have a version of this problem. The people who built it know in their head how to bring it up locally, and the people who joined later usually do not. There are several ways to authenticate, all obvious to the original team and confusing to everyone else, and there are dozens of small decisions about which identity should issue which call. Newer engineers and people who only contribute now and then pay for that complexity on every change, and the README, if there is one, has usually drifted from how things actually work.

Turning the setup into something that actually runs the steps, rather than something that only describes them, works for a few simple reasons. It checks what is already installed instead of assuming. It can make sensible choices during setup, like which port to use or which dependency manager is available on the machine. It remembers what you have done in the current session, so the next step can build on it. And it lets you ask for things in plain language, instead of remembering exact flags and arguments. For a service where skipping end-to-end tests turns into production incidents, that is a small investment with a quick payoff.

Future Improvements

Seed data is fixed-shape today. The skill creates a small canned set of orgs, users, and projects. I would like to be able to describe the shape I need in plain English, something like “an org with twenty members, three projects, and a mix of viewer, editor, and owner roles”, and have the skill build it out, instead of writing each step by hand every time I need a new variation.

The interaction is too plain-text right now. When the skill needs an input from the user (Docker or local, which port to use, which user to log in as), it asks in regular chat text and waits for a typed reply. I would like those moments to show up as proper selectable choices inside Claude Code, the way Claude’s own prompts do, so the user can just pick an option instead of reading a paragraph and typing the answer back.

No browser-side testing yet. The current plugin understands Frontier’s proto layer and can call any RPC on demand, but Frontier also ships an Admin UI and a JavaScript SDK with a demo app. Anyone working on the JS SDK still has to test their changes by clicking through the UI by hand. I want to extend the plugin so it can drive the browser as well, simulating logins, button clicks, and form submissions, so SDK changes can be tested as easily as RPC changes already are.

If your service takes more than a few minutes to bring up locally, the highest-leverage version of this is probably the one you do not write a doc for. Instead, you write the executable equivalent of the doc and let your team invoke it. Your first version will be rough, but the time it saves your team is real, and you get that time back very quickly.

The plugin lives at whoAbhishekSah/claude-marketplace.

Managing Infrastructure at Scale With Terraform

Fri, 13 Mar 2026 22:12:03 GMT

What this post covers:

Why for_each exists and how it avoids the index-shifting problem that count has
How sets, maps, and maps of objects work with for_each, and why lists need toset()
The flatten-then-map pattern for turning nested data into flat maps
Dynamic blocks, the known-values constraint, and other gotchas

When your Terraform codebase manages a handful of resources, almost any approach works. But as infrastructure grows — more services, more environments, more accounts — you start needing ten, fifty, or a hundred copies of the same resource with slightly different config. At that point, how you loop over resources decides whether adding a new service takes one line or a hundred, and whether removing one accidentally destroys something else.

Terraform gives you two ways to create multiple resources: count and for_each. This post is about for_each — how it works, where it helps at scale, and where it gets tricky.

Why Not `count`

The count argument identifies resources by their position in a list:

variable "bucket_names" {
  default = ["logs", "data", "backups"]
}

resource "aws_s3_bucket" "this" {
  count  = length(var.bucket_names)
  bucket = "mycompany-${var.bucket_names[count.index]}"
}

In state, these resources are tracked as aws_s3_bucket.this[0], aws_s3_bucket.this[1], and aws_s3_bucket.this[2].

If you remove “data” from the middle of the list, “backups” shifts from index 2 to index 1. Terraform sees that index 1 has changed and that index 2 has disappeared — so it plans a destroy and recreate of the backups bucket. In production, that could mean deleting a bucket that holds customer data.

How `for_each` fixes this

Instead of tracking resources by position, for_each identifies each one by a name:

resource "aws_s3_bucket" "this" {
  for_each = toset(["logs", "data", "backups"])
  bucket   = "mycompany-${each.value}"
}

In state, these become aws_s3_bucket.this["logs"], aws_s3_bucket.this["data"], and aws_s3_bucket.this["backups"]. If you remove “data”, only that resource gets destroyed, and the other two stay exactly as they are.

The for_each argument accepts either a set or a map, but it does not accept a list directly. You can convert a list with toset(). When you use a set, each.key and each.value refer to the same thing. When you use a map, each.key gives you the key and each.value gives you the value.

Maps

Maps are where for_each gets more useful, because you can attach different configuration to each item:

variable "buckets" {
  default = {
    "logs"    = "us-east-1"
    "data"    = "us-west-2"
    "backups" = "eu-west-1"
  }
}

resource "aws_s3_bucket" "this" {
  for_each = var.buckets
  bucket   = "mycompany-${each.key}"

  tags = {
    Region = each.value
  }
}

To add a bucket, you add one line to the map. To remove one, you delete a line. Terraform figures out the rest.

Maps of Objects

In most cases, each resource needs more than a single value. You can handle that with a map of objects:

variable "services" {
  type = map(object({
    port     = number
    replicas = number
    memory   = optional(string, "256Mi")
  }))

  default = {
    "api"    = { port = 8080, replicas = 3, memory = "512Mi" }
    "worker" = { port = 9090, replicas = 2 }
  }
}

resource "kubernetes_deployment" "this" {
  for_each = var.services

  metadata {
    name = each.key
  }

  spec {
    replicas = each.value.replicas

    template {
      spec {
        container {
          name   = each.key
          image  = "myregistry/${each.key}:latest"
          port   = each.value.port
          memory = each.value.memory
        }
      }
    }
  }
}

Caveat: Every object in the map must have the same shape. If “api” has a port field, then “worker” needs one too. You can use optional() with a default value for fields that not every entry needs.

Referencing `for_each` Resources

To reference a specific instance, you use the key in square brackets:

output "api_name" {
  value = kubernetes_deployment.this["api"].metadata[0].name
}

To reference all instances at once, you use a for expression to build a new map:

output "all_names" {
  value = { for k, v in kubernetes_deployment.this : k => v.metadata[0].name }
}

Caveat: You cannot pass kubernetes_deployment.this directly as an output. Terraform expects you to transform it with a for expression. The splat syntax ([*]) that works with count does not apply to for_each resources.

“Known Values” Constraint

The keys you pass to for_each must be known at plan time, which means they cannot come from the output of another resource. For example, this fails:

resource "aws_route_table_association" "this" {
  for_each  = { for s in aws_subnet.this : s.id => s }
  subnet_id = each.key
}

The subnet IDs do not exist until Terraform actually creates them, so it cannot determine the keys during the plan phase. The fix is to drive both resources from the same input variable:

variable "subnet_config" {
  default = {
    "subnet-a" = { az = "us-east-1a", cidr = "10.0.1.0/24" }
    "subnet-b" = { az = "us-east-1b", cidr = "10.0.2.0/24" }
  }
}

resource "aws_subnet" "this" {
  for_each          = var.subnet_config
  availability_zone = each.value.az
  cidr_block        = each.value.cidr
}

resource "aws_route_table_association" "this" {
  for_each       = var.subnet_config
  subnet_id      = aws_subnet.this[each.key].id
  route_table_id = aws_route_table.main.id
}

Since both resources share the same map, the keys come from a variable and are known at plan time. This is worth doing on purpose — when one map drives multiple resources, the code becomes easier to follow and less likely to break.

Flattening Nested Data

When your data is nested, you need to turn it into a flat map before for_each can work with it. For example, say you have multiple users and each of them needs several IAM policy attachments:

variable "user_policies" {
  default = {
    "alice" = ["arn:aws:iam::policy/ReadOnly", "arn:aws:iam::policy/S3Full"]
    "bob"   = ["arn:aws:iam::policy/ReadOnly"]
  }
}

The transformation happens in three steps. First, nested for loops produce a list of lists. Then, flatten collapses them into a single flat list. Finally, a for expression converts that list into a map with composite keys:

locals {
  user_policy_pairs = flatten([
    for user, policies in var.user_policies : [
      for policy in policies : {
        user   = user
        policy = policy
      }
    ]
  ])

  user_policy_map = {
    for pair in local.user_policy_pairs :
    "${pair.user}-${basename(pair.policy)}" => pair
  }
}

resource "aws_iam_user_policy_attachment" "this" {
  for_each   = local.user_policy_map
  user       = each.value.user
  policy_arn = each.value.policy
}

Caveat: The composite key must be unique across all entries. If you change the key format later, Terraform will treat every affected resource as a destroy-and-create.

Caveat: Make sure you carry the outer key (like the user name) into the flattened object. If you drop it during flattening, you will not be able to reference it in the resource block later.

Dynamic Blocks Inside `for_each`

There are situations where you need to loop across resources and also loop within a single resource. The outer for_each handles the first part, and a dynamic block handles the second:

variable "security_groups" {
  default = {
    "web" = {
      ingress_rules = [
        { port = 80, cidr = "0.0.0.0/0" },
        { port = 443, cidr = "0.0.0.0/0" },
      ]
    }
    "api" = {
      ingress_rules = [
        { port = 8080, cidr = "10.0.0.0/8" },
      ]
    }
  }
}

resource "aws_security_group" "this" {
  for_each = var.security_groups
  name     = each.key

  dynamic "ingress" {
    for_each = each.value.ingress_rules
    content {
      from_port   = ingress.value.port
      to_port     = ingress.value.port
      protocol    = "tcp"
      cidr_blocks = [ingress.value.cidr]
    }
  }
}

Caveat: Inside the dynamic block, the iterator is called ingress.value rather than each.value. It takes its name from the block label. The each variable still refers to the outer for_each on the resource itself.

Putting It Together: Multi-Environment EKS Node Groups

This pattern is close to what you would see in a real codebase where a team runs Kubernetes across staging and production. A single variable drives all the node groups:

variable "environments" {
  type = map(object({
    cluster_name = string
    node_groups = map(object({
      instance_type = string
      min_size      = number
      max_size      = number
      desired_size  = number
      disk_size     = number
      labels        = map(string)
      taints = list(object({
        key    = string
        value  = string
        effect = string
      }))
    }))
  }))
}

locals {
  all_node_groups = flatten([
    for env_name, env in var.environments : [
      for ng_name, ng in env.node_groups : {
        env_name     = env_name
        cluster_name = env.cluster_name
        ng_name      = ng_name
        ng_config    = ng
      }
    ]
  ])

  node_group_map = {
    for ng in local.all_node_groups :
    "${ng.env_name}-${ng.ng_name}" => ng
  }
}

resource "aws_eks_node_group" "this" {
  for_each        = local.node_group_map
  cluster_name    = each.value.cluster_name
  node_group_name = each.value.ng_name
  instance_types  = [each.value.ng_config.instance_type]
  disk_size       = each.value.ng_config.disk_size

  scaling_config {
    min_size     = each.value.ng_config.min_size
    max_size     = each.value.ng_config.max_size
    desired_size = each.value.ng_config.desired_size
  }

  labels = merge(each.value.ng_config.labels, {
    environment = each.value.env_name
    node_group  = each.value.ng_name
  })

  dynamic "taint" {
    for_each = each.value.ng_config.taints
    content {
      key    = taint.value.key
      value  = taint.value.value
      effect = taint.value.effect
    }
  }
}

To add an environment or a node group, you add an entry to the map. The flatten-then-map step produces keys like “production-observability” and “staging-general”, and Terraform manages each one independently in state.

Summary

count vs for_each: You should use count only for conditional creation (count = var.enabled ? 1 : 0), and reach for for_each whenever you need multiple instances.

Input types: The for_each argument takes a set or a map. If you have a list, you can convert it with toset().

Known values: All keys must be known at plan time, so you should drive them from variables rather than resource outputs.

Flattening: Nested data needs to be flattened into a map with composite keys, and you should carry context from the outer loops into the flattened objects so it is available in the resource block.

State keys are identity: If you change a key, Terraform will destroy and recreate that resource. Choose your keys carefully, and use terraform state mv when you need to rename one.

Dynamic blocks: The iterator inside a dynamic block is named after the block label, not each, so keep track of which variable refers to which loop.

Go Contexts: What I Learned by Getting Things Wrong

Sun, 01 Mar 2026 22:12:03 GMT

If you write Go and work on distributed systems, you will use context in almost every function you write. It shows up in HTTP handlers, database calls, gRPC methods, and Kubernetes controllers. Most of us learn the basics early — pass a context, check for cancellation, move on. But the subtle parts? Those show up when things break in production and you spend a Thursday night staring at goroutine dumps.

I recently spent time studying Go contexts from the ground up. Not the documentation kind of study, but the kind where you write code, make mistakes, and then figure out why things went wrong. This post captures what I learned, including the parts that surprised me.

1. Cancelling Work from the Outside

The simplest use of context is cancellation. You have a function doing work, and you want the caller to be able to say “stop.” The function itself does not decide when to stop. The caller does.

ctx, cancel := context.WithCancelCause(context.Background())

This gives you a context and a cancel function. When you call cancel(someError), the context’s Done() channel closes, and anything watching it knows to wrap up. The error you pass in is the cause, and you can retrieve it later with context.Cause(ctx).

Here is a small worker I built while learning. It wraps a function and runs it in a loop, with support for cancellation and cleanup callbacks:

type Worker struct {
    fn         func() error
    ctx        context.Context
    cancel     func(cause error)
    afterFuncs []func()
}

func NewWorker(fn func() error) *Worker {
    return &Worker{fn: fn}
}

func (w *Worker) Start() {
    if w.ctx != nil {
        return
    }
    w.ctx, w.cancel = context.WithCancelCause(context.Background())
    for _, fn := range w.afterFuncs {
        context.AfterFunc(w.ctx, fn)
    }
    go w.work()
}

func (w *Worker) AfterStop(fn func()) {
    if w.ctx != nil {
        return
    }
    w.afterFuncs = append(w.afterFuncs, fn)
}

func (w *Worker) Stop() {
    if w.ctx == nil {
        return
    }
    w.cancel(ErrManual)
}

func (w *Worker) Err() error {
    return context.Cause(w.ctx)
}

The idea is simple. The caller creates a worker with a function, registers any cleanup callbacks with AfterStop, and then calls Start. The worker creates its own context and cancel function, registers the cleanup callbacks using context.AfterFunc, and launches the work loop in a goroutine. When someone calls Stop, it cancels the context with ErrManual. If the function itself fails, the loop cancels the context with ErrFailed. Either way, the registered AfterFunc callbacks run after cancellation.

The work loop looks like this:

func (w *Worker) work() {
    for {
        select {
        case <-w.ctx.Done():
            return
        default:
            err := w.fn()
            if err != nil {
                w.cancel(ErrFailed)
                return
            }
        }
    }
}

The worker checks ctx.Done() at the top of each loop. If the context is cancelled, it returns. Otherwise, it runs the function.

This works, but there is a catch.

Catch: The first cancel wins

If you call cancel more than once with different causes, only the first one counts. The rest are silently ignored.

ctx, cancel := context.WithCancelCause(context.Background())
cancel(errors.New("first"))
cancel(errors.New("second"))
fmt.Println(context.Cause(ctx)) // prints: first

This matters when you have multiple reasons a piece of work might stop. Say your worker fails internally with ErrFailed, and then someone also calls Stop() with ErrManual. The cause will be ErrFailed, because that happened first. If you are debugging a failure and checking the cause, you need to know that the cause reflects whoever got there first, not whoever you expected.

2. Cancellation Is Cooperative, Not Preemptive

Cancellation in Go is often misunderstood as something that happens automatically. It does not. Go has no mechanism to kill a goroutine from the outside. When you cancel a context, all you are doing is closing a channel. If the code running inside the goroutine never checks that channel, it will keep running as if nothing happened.

Lets take a look at this worker loop again:

select {
case <-w.ctx.Done():
    return
default:
    err := w.fn()
}

If w.fn() takes 30 seconds to run, and you cancel the context at second 2, the goroutine does not exit at second 2. It exits after 30 seconds, when fn returns and the loop gets back to the select statement. The cancellation was sitting there in the Done() channel the whole time. Nobody was listening.

This is a fundamental design decision in Go. The language gives you the tools to signal cancellation, but it is your responsibility to check for it. If your function does not look at the context, it will not respond to cancellation.

The fix: pass the context down

If your function does something slow — an HTTP call, a database query, a file transfer — it should accept a context.Context and pass it to whatever it calls.

// Instead of this:
fn func() error

// Do this:
fn func(ctx context.Context) error

Now, inside fn, if you are making an HTTP request, you use http.NewRequestWithContext(ctx, ...). If the context is cancelled, the HTTP client aborts the request. The cancellation propagates through the entire call chain, from the top-level caller all the way down to the network socket. That is the whole point of context — it flows downward through your program.

Catch: Do not spawn goroutines just to wrap cancellation

You might think: “I will launch fn in a goroutine, and if the context is cancelled, I will just abandon it.” The problem is that the goroutine is still running. Nobody is waiting for its result. It is leaked. You traded one problem (slow cancellation) for another (resource leaks). The right approach is always to make the function itself aware of the context.

3. Sequential Operations and Context in Kubernetes Operators

If you have written a Kubernetes operator, you have seen this pattern. The controller-runtime framework calls your Reconcile function whenever a custom resource changes. It passes you a context. That context gets cancelled if the operator pod is shutting down — during a rolling deploy, a node drain, or a leader election change.

Say your reconciler does three things in sequence:

func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    meta, err := r.fetchMetadata(ctx, req.Name)
    if err != nil { return ctrl.Result{}, err }

    params, err := r.validateParams(ctx, meta)
    if err != nil { return ctrl.Result{}, err }

    err = r.pushConfig(ctx, params)
    if err != nil { return ctrl.Result{}, err }

    return ctrl.Result{}, nil
}

Each call takes 2 to 5 seconds. Now your operator pod gets a SIGTERM during a rolling update. You are halfway through validateParams.

What happens is straightforward once you understand cooperative cancellation. The context gets cancelled by the framework. But validateParams is already running. It does not stop mid-execution just because the context flipped. It finishes, returns an error (assuming it checks the context internally), and then the error check prevents pushConfig from running. If validateParams does not check the context, it runs to completion, returns a result, and then pushConfig gets called with a cancelled context.

The important thing here is that you must check errors between each call. If you ignore the error from fetchMetadata and proceed to validateParams, you are doing work with potentially invalid data on a cancelled context. The context cancellation only helps if two things are true: the function you called respects the context, and you check the error it returns.

Catch: Partial completion and idempotency

What if metadata was fetched and validation passed, but pushConfig failed because the context was cancelled? The next reconcile starts from scratch. Is that a problem?

In most operators, no. The standard approach is to make reconciliation idempotent. You store progress in the resource’s status subresource. Each reconcile checks current state against desired state and does only what is needed. If metadata is already stored in status, the next reconcile skips that step. The status acts as your checkpoint.

This is why Kubernetes operators lean on status subresources so heavily. They are not just for reporting — they are the mechanism that makes retries cheap and safe.

4. Parent-Child Context Chains and Cause Propagation

Contexts form a tree. You start with context.Background(), then create children from it, and children from those children. When a parent is cancelled, all of its children are cancelled too.

But here is the part that caught me off guard: the cause propagates from parent to child.

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

childCtx, childCancel := context.WithCancelCause(ctx)
defer childCancel(errors.New("cleanup"))

When the 5-second timeout fires, the parent is cancelled with context.DeadlineExceeded. The child gets cancelled too, because its parent is done. Now, what does context.Cause(childCtx) return?

I expected it to return "cleanup", because that is what I passed to childCancel. But it returns context.DeadlineExceeded. The parent’s timeout cancelled the child before childCancel was ever called. And since the first cancel wins, the deferred childCancel(errors.New("cleanup")) is a no-op — the cause was already set by the parent.

Catch: A child’s cancel function only matters if the child is cancelled before its parent

If you are relying on context.Cause to distinguish between “we timed out” and “we were explicitly stopped,” you need to make sure the explicit stop happens first. Otherwise, the parent’s cause will already be set.

5. `context.Background()` Is the Root

A quick note that seems obvious but is worth stating: context.Background() can never be cancelled. It has no deadline, no cancel function, and no values. It is the starting point of every context tree. Same goes for context.TODO().

ctx := context.Background()
fmt.Println(ctx.Err())          // <nil>
fmt.Println(context.Cause(ctx)) // <nil>

You create all other contexts from this root. Nothing cancels it.

6. Fire-and-Forget Goroutines and Context Lifetimes

This one shows up in almost every HTTP service I have seen. You have a handler that does some work and then kicks off a background task:

func handler(w http.ResponseWriter, r *http.Request) {
    data, err := fetchFromDB(r.Context(), query)
    if err != nil { return }

    go sendToKafka(r.Context(), data)  // fire and forget

    w.Write(response)
}

The bug: when the handler returns, the request context is cancelled. Your Kafka goroutine is using that same context. The Kafka write races against context cancellation, and most of the time, it loses. Your audit log silently drops messages, and nobody notices until someone asks “why are there gaps in the logs?”

The fix is to use a different context for work that must outlive the request:

bgCtx, bgCancel := context.WithTimeout(context.Background(), 10*time.Second)
go func() {
    defer bgCancel()
    sendToKafka(bgCtx, data)
}()

This context is not tied to the request. It has its own timeout, and it will not be cancelled when the handler returns.

Catch: Request contexts die when the handler returns

Any goroutine that needs to outlive the HTTP request must not use the request’s context. Create a new one from context.Background() with its own timeout.

7. Graceful Shutdown: Structuring the Context Hierarchy

When your service receives a SIGTERM — during a deploy, a node drain, or a scaling event — you need every subsystem to stop cleanly. Contexts give you the structure for this.

Say your service has three subsystems: an HTTP server, a Kafka consumer, and a background reconciler. Here is how you wire them up:

func main() {
    parentCtx, cancel := context.WithCancelCause(context.Background())

    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
    go func() {
        sig := <-sigCh
        cancel(fmt.Errorf("received signal: %v", sig))
    }()

    g, gCtx := errgroup.WithContext(parentCtx)
    g.Go(func() error { return httpServer.Run(gCtx) })
    g.Go(func() error { return kafkaConsumer.Run(gCtx) })
    g.Go(func() error { return reconciler.Run(gCtx) })

    if err := g.Wait(); err != nil {
        log.Printf("shutdown: %v", err)
    }
}

When SIGTERM arrives, the parent context is cancelled, which propagates to all three subsystems. The errgroup waits for all of them to finish. You do not exit the process until every subsystem has confirmed it is done.

Catch 1: Signal handling catches everything if you are not specific

signal.Notify(ch) with no signal arguments will catch every signal your process receives, including harmless ones like SIGCHLD and SIGURG. Always be explicit about which signals you care about.

Catch 2: Cleanup operations need a live context

When your Kafka consumer’s context is cancelled, it needs to commit the offset of the last message it processed. But if you use the cancelled context to commit, the commit will fail — the context is already done.

func (k *KafkaConsumer) Run(ctx context.Context) error {
    for {
        msg, err := k.reader.FetchMessage(ctx) // respects cancellation
        if err != nil {
            return err
        }
        k.process(msg)
        k.reader.CommitMessages(context.Background(), msg) // must succeed
    }
}

The fetch uses the cancellable context — that is how the consumer knows to stop pulling new messages. But the commit uses context.Background(), because the commit must go through regardless of whether the consumer is shutting down.

8. Fan-Out: Cancelling the Stragglers

Here is a pattern that shows up in API gateways and aggregation services. You call five downstream services in parallel. You need any three to succeed. Once you have three, cancel the remaining two.

The natural instinct is to cancel the parent context. But that is wrong. The parent context is your request context — cancelling it would also cancel your ability to write the response. Instead, create a child context specifically for the fan-out:

func aggregate(w http.ResponseWriter, r *http.Request) {
    parentCtx := r.Context()

    fanoutCtx, cancelFanout := context.WithCancel(parentCtx)
    defer cancelFanout()

    results := make(chan Result, 5) // buffered to prevent goroutine leaks

    for _, svc := range services {
        go func(s Service) {
            val, err := s.Call(fanoutCtx)
            results <- Result{val: val, err: err}
        }(svc)
    }

    var successes []Result
    for i := 0; i < 5; i++ {
        select {
        case <-parentCtx.Done():
            respond(w, successes) // return whatever we have
            return
        case r := <-results:
            if r.err == nil {
                successes = append(successes, r)
                if len(successes) == 3 {
                    cancelFanout() // cancel remaining, NOT the parent
                    respond(w, successes)
                    return
                }
            }
        }
    }
    respond(w, successes)
}

Catch 1: Buffer your channels

The channel must be buffered with capacity equal to the number of goroutines. After you cancel and return, the remaining goroutines will eventually finish and try to write their results. If the channel is unbuffered, they block forever. That is a goroutine leak. A buffered channel lets them write and exit, even though nobody reads those results.

Catch 2: Do not cancel the parent to stop children

Create a dedicated child context for the group of work you want to cancel. The parent context should stay alive for your own operations, like writing the HTTP response.

Contexts are a small API. Four functions, a couple of interfaces. But the patterns that emerge from them — cancellation chains, graceful shutdown, fan-out coordination — are the backbone of how Go services manage work. Getting them right is the difference between a service that shuts down cleanly and one that leaks goroutines, drops messages, and leaves you debugging at 2 AM.

Concurrency Controls in Golang

Thu, 19 Feb 2026 22:12:03 GMT

From Simple Data Pipelines to Token Buckets

In my day-to-day role as a Site Reliability Engineer, my primary focus naturally gravitates toward system stability, infrastructure scaling, and Kubernetes management. While I write a considerable amount of Go and Python to automate these operations, I rarely get the opportunity to build complex, highly concurrent backend patterns from scratch. Lately, I decided to look closer at how high-throughput systems actually process all that data, which naturally led me to study Go’s concurrency models. I wanted to step outside my usual operational boundaries and understand exactly how software engineers manage massive data streams efficiently without overwhelming their infrastructure.

I’ve always found that the best way to actually understand a complex topic is to try explaining it. I’m writing this post to solidify my own grasp on these architectural patterns, breaking them down from a basic data pipeline all the way up to a production-grade token bucket rate limiter

Level 1: The Basic Pipeline for Stream Processing

Imagine you need to parse a massive 50GB log file to extract specific error messages. If you attempt to load that entire file into memory at once, your application will inevitably crash due to memory exhaustion. The elegant solution to this problem is the Pipeline Pattern, which allows you to break the workload into distinct processing stages that run simultaneously. By passing data downstream through Go channels, your application only ever needs to hold a single line of the file in memory at any given time.

When designing a basic pipeline, the architecture generally flows through three primary components:

The Reader (Submitter): This stage reads the source file line-by-line and pushes each string into an outgoing channel.
The Transformer: This middle stage reads from the incoming channel, performs the necessary filtering or modifications, and pushes the successful results to the next channel.
The Sinker (Consumer): The final stage collects the processed results and handles the final output or database insertion.

func aggregateLog(ctx context.Context, filepath string) int {
	pending := reader(ctx, filepath) 
	collector := transformer(ctx, pending) 
	return sinker(collector) 
}

⚠️ Common Pitfall: The Blocking Trap

When writing these individual stages, it is incredibly easy to accidentally block a goroutine forever, which creates a silent memory leak. If a user cancels the operation or a system timeout occurs, your goroutines must be able to recognize that signal and exit immediately.

If you simply send data to a channel like pending <- line, and the downstream consumer has already stopped listening, your program will hang indefinitely. To prevent this, you should always wrap your channel sends and receives in a select statement alongside the context cancellation signal.

select {
case pending <- line:
    // The data was successfully sent downstream
case <-ctx.Done():
    return // The context was cancelled, so we shut down gracefully
}

In a real-world backend, requests frequently get cancelled, connections drop, or API timeouts expire. The context is how Go signals your application to stop working on a task that is no longer needed. If your goroutines ignore this cancellation signal while waiting to send or receive data on a channel, they will hang in the background forever. Over time, these stranded goroutines accumulate and cause silent memory leaks that degrade performance and eventually crash your application.

Level 2: Scaling Up with Fan-Out and Fan-In

While a linear pipeline is fantastic for memory management, it introduces a new bottleneck if your middle transformation stage is computationally expensive. If your transformer performs a heavy regex match or a slow database lookup that takes a full second, your entire program operates at a maximum speed of one line per second.

To resolve this bottleneck, we introduce the Fan-Out pattern. Instead of relying on a single transformer, we spawn a pool of worker goroutines that all read concurrently from the exact same input channel.

Managing multiple workers safely requires a sync.WaitGroup to track their progress. The most challenging aspect of this pattern is knowing exactly when to close the output channel. If you close the channel prematurely, your active workers will panic when they try to send data. Conversely, if you forget to close it entirely, your final sinker stage will wait forever.

func transformDispatcher(ctx context.Context, pending <-chan string) <-chan string {
	collector := make(chan string)
	wg := &sync.WaitGroup{}
    
	// Fan-Out: Spawn multiple parallel workers
	for i := 0; i < 3; i++ {
		wg.Add(1)
		go worker(ctx, pending, collector, wg)
	}
    
	// Fan-In: Wait for all workers to finish in a background routine, then close
	go func() {
		wg.Wait()
		close(collector)
	}()
    
	return collector
}

This design gives you some great built-in benefits: true parallelism, natural load balancing (since multiple goroutines are actively pulling from one shared channel), and an organized, ordered completion of tasks.

However, there is one major caveat you must keep in mind: output order is not preserved. Because the workers operate independently and process data at slightly different speeds, the results will arrive at the final sinker completely out of sequence. If your application strictly requires the output to match the original input order, this specific Fan-Out pattern is incorrect and you will need to reach for a different design.

⚠️ Common Pitfall: The “Busy Wait” Loop

When you write the infinite loop for your worker routines, it is tempting to include a default case in your select block to handle moments when the channel is empty. You must avoid doing this.

If you include a default case, you accidentally create a “busy wait.” Instead of pausing, the loop runs millions of times per second whenever there is no immediate data available, which will entirely consume your CPU resources.

Here is what that mistake looks like in our log transformer worker:

The Bad Way (Burns CPU):

for {
    select {
    case <-ctx.Done():
        return
    case checkStr := <-pending:
        // Process the log line
    default:
        // DANGER: If 'pending' is empty, the code instantly falls through to here.
        // The loop restarts immediately, spinning endlessly and burning 100% CPU.
    }
}

To fix this, you simply remove the default case. Without it, the select statement blocks naturally. The Go scheduler recognizes that the goroutine has nothing to do, so it puts the routine to sleep until new data arrives in the channel or the context cancels.

The Good Way (Zero CPU while waiting):

for {
    select {
    case <-ctx.Done():
        return
    case checkStr, ok := <-pending:
        // If the channel is closed, ok is false and checkStr is "".
        // Without this check, the loop would spin on zero-values
        // just like a busy wait — except harder to spot.
        if !ok {
            return
        }
        // If 'pending' is open but empty, the goroutine safely sleeps here.
        // It wakes up the moment a new log line arrives or the channel closes.
    }
}

Level 3: Controlling Flow with a Rate Limiter

Sometimes the architectural challenge isn’t moving data too slowly; it is moving data too quickly. If your pipeline calls a third-party API that strictly enforces a limit of five requests per second, executing your tasks instantly will result in blocked connections or banned IP addresses.

To mitigate this risk, we need to build a concurrency turnstile using time.Ticker to enforce a strict, mandatory delay between each operation. However, a common misconception here is that the upstream code simply keeps dumping data into the incoming channel at lightning speed while the rate limiter builds a massive internal queue. If that were true, your application would quickly exhaust its available memory by stockpiling thousands of unprocessed items.

Instead, Go channels naturally provide a powerful system design feature called backpressure.

Because our rate limiter explicitly pauses to wait for the next timer tick, it completely stops pulling new items out of the pending channel. Once that incoming channel is full, the upstream reader is forced to pause because it physically cannot send any more data. The slow speed of the rate limiter naturally pushes back on the fast reader, which forces the entire pipeline to slow down and synchronize to a safe, predictable pace.

Here is what that pipeline stage looks like when we wrap it in a proper function. Notice how the logic intentionally blocks reading the next item from the pending channel until the throttled channel successfully emits the current one alongside the timer tick.

// limiter takes a channel of pending requests and outputs them at a strict, steady pace.
func limiter(ctx context.Context, pending <-chan string, interval time.Duration) <-chan string {
    // The channel where we will send our rate-limited data downstream
    throttled := make(chan string)

    go func() {
        // Ensure we clean up the channel and ticker when we exit
        defer close(throttled)
        
        ticker := time.NewTicker(interval)
        defer ticker.Stop()

        for {
            select {
            case <-ctx.Done():
                return
                
            // Step 1: Read a single pending item. 
            // If we are waiting for a tick below, this read cannot happen, which creates backpressure!
            case item, ok := <-pending:
                if !ok {
                    return // Upstream closed the channel, meaning all work is done
                }
                
                // Step 2: We have the item. Now, we must wait for the green light.
                select {
                case <-ticker.C:
                    // Step 3: The tick arrived. Now we safely send the item downstream.
                    // We must check context again here to prevent a deadlock if downstream crashes!
                    select {
                    case throttled <- item:
                    case <-ctx.Done():
                        return
                    }
                case <-ctx.Done():
                    return
                }
            }
        }
    }()
    
    return throttled
}

Level 4: Token Buckets and Handling Bursts

While our basic rate limiter does the job, it is too rigid for most real-world applications. If a user hasn’t made a single request in an hour, the system still forces them to wait 200 milliseconds before processing their very first action. Real-world APIs like AWS or Stripe solve this by using the Token Bucket algorithm, which allows an initial burst of immediate traffic before the strict throttling takes over.

To build a burstable rate limiter, we decouple our timing mechanism from our consumer logic by utilizing a buffered channel as our token bucket.

This architecture requires three distinct components interacting together:

The Bucket: A buffered channel where the capacity represents our maximum allowed burst limit.
The Refiller: A background goroutine that adds a single token to the bucket every interval.
The Consumer: Our primary loop that simply attempts to read a token from the bucket before it processes the next incoming item.

Here is what this looks like when we wrap it into a proper pipeline stage:

// burstableLimiter allows an initial burst of traffic, then throttles to a steady interval.
func burstableLimiter(ctx context.Context, pending <-chan string, interval time.Duration, burstLimit int) <-chan string {
    throttled := make(chan string)
    
    // 1. The Bucket: A buffered channel to hold our available tokens
    bucket := make(chan struct{}, burstLimit)

    // Pre-fill the bucket so the initial burst executes immediately
    for i := 0; i < burstLimit; i++ {
        bucket <- struct{}{}
    }

    // 2. The Refiller: Runs in the background and adds tokens
    go func() {
        ticker := time.NewTicker(interval)
        defer ticker.Stop()

        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
                // We use 'default' here to make this send non-blocking.
                // If the bucket is full, we simply drop the new token and move on.
                select {
                case bucket <- struct{}{}:
                default:
                }
            }
        }
    }()

    // 3. The Consumer: Reads data and waits for tokens
    go func() {
        defer close(throttled)

        for {
            select {
            case <-ctx.Done():
                return
            case item, ok := <-pending:
                if !ok {
                    return // Upstream closed the channel
                }

                // Wait for a token. As long as the bucket is not empty, this is instant!
                select {
                case <-bucket:
                case <-ctx.Done():
                    return
                }

                // We have the item and the token. Safely send it downstream.
                select {
                case throttled <- item:
                case <-ctx.Done():
                    return
                }
            }
        }
    }()

    return throttled
}

Because we deliberately pre-filled the bucket during the initialization phase, the pipeline processes the first few items instantly. Once that initial burst depletes the bucket, the consumer naturally pauses and waits for the refiller goroutine to drop a new token into the bucket at our specified interval.

⚠️ The Strategic Exception: Why We Used default in the Refiller

This has confused me a lot. Earlier in this post, I explicitly called out the dangers of using a default case, noting how it creates a CPU-burning busy loop. However, you might have noticed that I intentionally used one inside the refiller goroutine. This isn’t a mistake; it’s a specific pattern called a non-blocking send.

If the system experiences a lull in traffic, the token bucket quickly fills up to its maximum burst capacity. When the next timer tick fires, the refiller attempts to push another token into that full channel. If we simply wrote bucket <- struct{}{}, the operation would block. The refiller goroutine would freeze, waiting for a consumer to take a token.

By wrapping that send operation in a select statement with an empty default case, we tell the Go runtime: “Try to drop a token into the bucket. If the bucket is full, simply discard the token and go back to sleep until the next tick.” It gracefully enforces the maximum burst limit without locking up our background process.

You might wonder why this doesn’t spike the CPU like the earlier example. In the Level 2 pitfall, the default case was a direct option in the main select block. Whenever the channel was empty, the select instantly fell through to the default case, which caused the outer for loop to spin continuously without pausing.

In our refiller, the default case sits inside a nested select block. The outer select still forces the goroutine to pause and wait for the ticker.C channel. The program only evaluates that nested default case once per tick. This ensures the loop only executes every 200 milliseconds, keeping CPU usage negligible.

Final Thoughts and Best Practices

Stepping away from my usual infrastructure and reliability work to study these backend patterns has definitely changed how I look at low level design. Writing highly concurrent code in Go yields incredibly powerful applications, but it requires strict operational discipline to keep those systems stable under heavy load.

As I worked through these iterations, I developed a practical checklist that I now use when reviewing any concurrent code.

Respect the Context to Prevent Goroutine Leaks: In a real-world backend, requests frequently get cancelled, connections drop, or API timeouts expire. The context is how Go signals your application to stop working on a task that is no longer needed. If your goroutines ignore this cancellation signal while waiting to send or receive data on a channel, they will hang in the background forever. Over time, these stranded goroutines accumulate and cause silent memory leaks that degrade performance and eventually crash your application. Always pair your blocking channel operations with a case <-ctx.Done():.
Embrace Backpressure: A well-designed pipeline does not rely on massive, bottomless queues that hoard memory. Instead, it leverages the natural blocking behavior of Go channels. If a downstream stage (like our rate limiter) slows down, let the channel fill up. That full channel forces the upstream reader to pause, which safely synchronizes the entire application to a sustainable pace.
Watch Out for the “Busy Wait” Trap: Be extremely careful when using a default case inside an infinite for loop. Normally, a select statement pauses your goroutine until a channel is ready to send or receive data. If you add a default case, the select becomes non-blocking. When the channels are empty, the code instantly executes the default case and restarts the loop. This causes the loop to spin millions of times per second, which completely consumes your CPU. You should only use default when you explicitly want to make a single channel operation non-blocking—like dropping a token when a bucket is full—and only if the outer loop is already paused by something else, like a timer tick.
Understand the Trade-offs of Parallelism: When you implement a Fan-Out pattern, you gain massive speed improvements and natural load balancing across multiple worker goroutines. However, because those workers operate independently, the final output order is not preserved. Always evaluate if your specific use case requires strict sequential processing before throwing a worker pool at the problem.

Documenting this progression helped me solidify my own understanding of these concurrency models. Hopefully, breaking down these concepts from a basic pipeline to a production-grade token bucket helps you build more resilient systems as well.

Terraform on CI - Part 2

Sat, 07 Jun 2025 22:12:03 GMT

This blog is part 2 of a series of articles. In the last blog, we saw the benefits of running Terraform on CI. The following section will provide details on how to set up Terraform. Let’s zoom in on what challenges one can face while setting up the Terraform run on CI.

Typically, Terraform is used to manage infrastructure in the Cloud. It can use a native provider such as AWS Provider. Terraform can also utilize other tools to perform various automation tasks, such as Helm for installing and managing resources in Kubernetes clusters, Ansible for VM orchestration, and others. You may require more providers depending on your infrastructure requirements. Each provider has its setup caveats. We will cover some of these provider setups.

Terraform Version

Let’s start with the Terraform version itself. Locking and standardizing a Terraform version for everyone is a crucial step to avoid provider version conflict issues. Terraform version changes can affect the modules. If we create a state file using a Terraform version other than the decided-upon locked version, we may encounter issues with state file compatibility in newer versions.

From the docs:

In general, Terraform will continue to work with a given state file across minor version updates. For major or minor releases, Terraform will update the state file version if required, and give an error if you attempt to run an older version of Terraform using an unsupported state file version.

State file issues are more nasty to deal with than other minor issues, such as some backend flags being deprecated: Example. Such issues need to be dealt with on a case-by-case basis. If you are able to run terraform init and terraform plan in all modules without any error, it means you have chosen the right version for the current IaC codebase.

Cloud provider access

Terraform will need to talk to the cloud providers via CI. Proper authentication mechanisms must be in place for the “Plan” and “Apply” commands to run without any access errors. To understand this in detail, I’ll take the Example of AWS, but the idea remains the same for other providers. I will use the example of GitHub Actions to demonstrate how to run Terraform on a CI/CD pipeline.

When an Infra admin runs Terraform from a local machine to make Infra changes, they typically use their identity (AWS SSO Profile, for example) to authenticate with AWS. AWS Terraform provider can get auth configuration from several sources with a pre-defined priority order. We can also use any listed auth mechanisms to authenticate terraform on CI.

Setup

The first setup involves creating an AWS directory in CI (~/.aws) and an AWS configuration file that the TF AWS Provider can use. Inside this file, we should mention a role that has access to create/destroy/change all kinds of infra in your AWS account.

This is how the role policy should look like:

{
  "Statement": [
    {
      "Action": "*",
      "Effect": "Allow",
      "Resource": "*",
      "Sid": ""
    }
  ],
  "Version": "2012-10-17"
}

We have given */* permissions to this role. However, we can also restrict access by allowing selective actions on specific resources, providing better control over infrastructure and costs.

Let’s call this IAM role core-tf-runner-role.

We assume this role in CI to provision infrastructure in our AWS account. We will need to allow the Github Runners to assume this role. We can achieve that via a trust relationship. AWS has a nice blog on how to achieve just that. You can refer to it here. It involves configuring an OIDC identity provider inside an AWS account. This setup enables the usage of the IAM role and short-term credentials. It will allow the GitHub runner to assume the core-tf-runner-role.

On CI, we need to create a configuration file in the format below to provide it with an AWS identity. The AWS Profile should assume the role we created above.

Example Config file:

[profile core_aws_account]
role_arn = arn:aws:iam::dev-acnt-id:role/core-tf-runner-role
credential_source = Environment
region = us-west-1

We have multiple ways to make his file available on CI, such as pre-baking in the CI image, importing as a GitHub action step, or using shell magic, among others. For the sake of simplicity, I assume that we are creating this file from a script. Here is a demonstrative GitHub Actions code block that describes the approach.

- name: create aws directory and a config file
  run: mkdir ~/.aws && touch ~/.aws/config

- name: Render AWS configs
  run: |
    <<your script to load the Config file inside ~/.aws/config

- name: Setup terraform
  uses: hashicorp/setup-terraform@v3
  with:
    terraform_version: '1.6.0' # locking the version

With this setup ready, we can run commands inside Terraform modules on the GitHub Actions CI job.

Below is an example Terraform module that uses the AWS profile we just set up. Here, the state file is in the same AWS account where we are creating other infrastructure resources.

//terraform.tf
terraform {
  required_version = "~> 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.59.0"
    }
  }

  backend "s3" {
    bucket              = "<<bucket-name>>"
    key                 = "<<state path>>"
    region              = "us-west-1"
    profile             = "core_aws_account"   //access provided via ~/.aws/config file
    use_legacy_workflow = false
  }
}

//provider.tf
provider "aws" {
  region  = var.region
  profile = "core_aws_account"
}

//main.tf
resource "aws_s3_bucket" "example" {
  bucket = "my-tf-test-bucket"

  tags = {
    Name        = "My bucket"
    Environment = "Dev"
  }
}

Multi AWS Account setup

Suppose your modules have multiple AWS account connectivity requirements. In that case, it’s best to create one tf-runner role per AWS account for better control and management, e.g., prod-tf-runner-role, Security-Tf-Runner-Role, dev-tf-runner-role. The config file expands correspondingly.

[profile dev_aws_account]
role_arn = arn:aws:iam::dev-acnt-id:role/dev-tf-runner-role
credential_source = Environment
region = us-west-1

[profile prod_aws_account]
role_arn = arn:aws:iam::prod-acnt-id:role/prod-tf-runner-role
credential_source = Environment
region = us-west-1

[profile security_aws_account]
role_arn = arn:aws:iam::prod-acnt-id:role/security-tf-runner-role
credential_source = Environment
region = us-west-1

We can establish the same trust relationship with Github in each role, but that can be tedious to manage. We can “DRY” it further by creating an intermediate role that can assume all these runner roles and can be assumed by Github Action runner. I’ll call this intermediate role “core-tf-runner-role” here.

Conceptually:

To achieve this flow, we will need to add a trust relationship between core-tf-runner-role and other runner roles.

// create this trust relationship with all account runner roles
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCNT_ID:role/core-tf-runner-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Additionally, we need to allow the core-tf-runner-role to assume other roles by attaching an appropriate policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": [
        "arn:aws:iam::dev-acnt-id:role/dev-tf-runner-role",
        "arn:aws:iam::prod-acnt-id:role/prod-tf-runner-role",
        "arn:aws:iam::security-acnt-id:role/security-tf-runner-role"
      ]
    }
  ]
}

With this setup, we can now use the AWS config file to handle the last leg of role chaining.

To recap, we have set the following:

Created runner roles per AWS Account with */* permissions.
Created core-tf-runner-role which can assume Runner roles in various accounts
Allowed Github Action to assume this core-tf-runner-role via OIDC IdP flow
Created an AWS Config file on CI to allow TF AWS Providers to authenticate

The GitHub action file is where the role chaining happens.

- name: create aws directory and a config file
  run: mkdir ~/.aws && touch ~/.aws/config

- name: Render AWS configs
  run: |
    <<your script to load the Config file inside ~/.aws/config

- name: Setup terraform
  uses: hashicorp/setup-terraform@v3
  with:
    terraform_version: '1.6.0' # locking the version

- name: Assume runner role
  uses: aws-actions/configure-aws-credentials@v2
  with:
    role-to-assume: arn:aws:iam::AWS_ACNT_ID:role/core-tf-runner-role
    audience: audString
    aws-region: us-west-1
# next steps will run plan / apply commands

CI Workflow Design

Now that we have authentication sorted out, let’s design the CI triggers. A well-designed workflow should:

Run the terraform plan on all pull requests
Run the terraform apply when the pull request is merged
Handle multiple Terraform module changes in a single pull request
Provide clear output for review in all actions

We will also need a runner with appropriate network connectivity in various AWS accounts and VPCs. Let’s keep the runner setup part out of scope for the sake of brevity in explaining the topic of this blog. Let’s consolidate all these in a GitHub Actions workflow file.

Terraform Plan on Pull Requests

name: Terraform Plan

on:
  pull_request: # run this wf on PRs on this repo
  push:
    branches:
      - main

jobs:
  terraform-plan:
    runs-on: internal-runner # A runner with proper networking setup
    steps:
      - uses: actions/checkout@v3

      - name: Create AWS config
        run: |
          mkdir -p ~/.aws
          cat > ~/.aws/config << EOF
          [profile dev_aws_account]
          role_arn = arn:aws:iam::dev-acnt-id:role/dev-tf-runner-role
          credential_source = Environment
          region = us-west-1

          [profile prod_aws_account]
          role_arn = arn:aws:iam::prod-acnt-id:role/prod-tf-runner-role
          credential_source = Environment
          region = us-west-1

          [profile security_aws_account]
          role_arn = arn:aws:iam::security-acnt-id:role/security-tf-runner-role
          credential_source = Environment
          region = us-west-1
          EOF

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: '1.6.0' # your desired version

      - name: Assume Terraform runner role
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::AWS_ACNT_ID:role/core-tf-runner-role # allowed to assume via OIDC IdP flow
          audience: audString
          aws-region: us-west-1

      - name: Get changed files # relative to main branch
        id: changed-files
        uses: tj-actions/changed-files@v40

      - name: Create directory to store plan out file
        run: |
          mkdir -p artifact

      - name: Run PLAN on changed leaf directories # custom script
        run: |
          python3 change_detection.py ${{ steps.changed-files.outputs.all_changed_files }}

      - name: Upload Artifact
        if: github.ref == 'refs/heads/main' # upload artifact only when branch is merged to main
        uses: actions/upload-artifact@v4
        with:
          name: terraform-pr-artifact
          path: artifact

This workflow does several important things:

Assumes the core tf runner role via OIDC IdP flow (short-term credentials used)
Setup terraform at a particular version
Find all changed files relative to the target branch(main)
Run a script that takes an input list of all the changed files.

1. detect the changed terraform modules in the PR from the list of changed files provided as input
2. for each changed module(here in my example, the leaf directories) in the IaC repo:
 - Runs `terraform init` utilising [chdir flag](https://developer.hashicorp.com/terraform/cli/commands#switching-working-directory-with-chdir)
 - Runs `terraform plan` and saves the plan out
3. Upload the plan-out file to GitHub Artifact to utilize it in a separate GitHub workflow.

Here is the change_detection.py script

import sys
import subprocess
import json
import utils


def main():
    tf_files_changed = utils.get_tf_files(sys.argv)
    changed_directories = utils.get_changed_directories(tf_files_changed)
    paths_references = {}

    # print the changed leaf
    for item in changed_directories:
        print(f"Detected changed directories: {item}")

    for item in changed_directories:
        print(
            f"\n===============================Processing: {item}===============================\n"
        )
        result = subprocess.run(["terraform", f"-chdir={item}", "init"],
                                stdout=subprocess.PIPE,
                                stderr=subprocess.STDOUT,
                                text=True)
        print(result.stdout)
        result.check_returncode()  # raises if non zero exit code

        plan_file_name = "_".join(item.split("/"))
        tf_plan_cmd = [
            "terraform", f"-chdir={item}", "plan", "-out", f"{plan_file_name}"
        ]
        result = subprocess.run(tf_plan_cmd,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.STDOUT,
                                text=True)
        print(result.stdout)
        result.check_returncode()  # raises if non zero exit code

        #move plan file to artifact directory
        result = subprocess.run(
            ["mv", f"{item}/{plan_file_name}", f"artifact/{plan_file_name}"],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True)
        print(result.stdout)
        result.check_returncode()  # raises if non zero exit code
        paths_references[plan_file_name] = item

    with open('artifact/path_ref.json', 'w') as fp:
        json.dump(paths_references, fp)


if __name__ == "__main__":
    main()

The code is self-explanatory. I have used some utils functions. The actual implementation will depend on how you have organized the IaC repository. We are saving the plan-out file in GitHub Artifact to apply the same changes in a later stage. Github Artifact is one way to save and use the plan file. You can use other methods to manage the plan output file.

Terraform Apply

The admins can decide how they want to trigger the Apply. It could be manual or automated. Here is an example using manual Github Workflow dispatch, taking the workflow run ID of the planning stage we showed earlier as input.

name: Terraform APPLY

on:
  workflow_dispatch:
    inputs:
      plan_workflow_id:
        type: string
        required: true
        description: PLAN workflow run id

jobs:
  terraform-apply:
    runs-on: internal-runner #A runner with proper networking setup
    steps:
      - uses: actions/checkout@v3

      - name: Create AWS config
        run: |
          mkdir -p ~/.aws
          cat > ~/.aws/config << EOF
          [profile dev_aws_account]
          role_arn = arn:aws:iam::dev-acnt-id:role/dev-tf-runner-role
          credential_source = Environment
          region = us-west-1

          [profile prod_aws_account]
          role_arn = arn:aws:iam::prod-acnt-id:role/prod-tf-runner-role
          credential_source = Environment
          region = us-west-1

          [profile security_aws_account]
          role_arn = arn:aws:iam::security-acnt-id:role/security-tf-runner-role
          credential_source = Environment
          region = us-west-1
          EOF

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: '1.6.0'

      - name: Assume Terraform runner role
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::AWS_ACNT_ID:role/core-tf-runner-role # allowed to assume via OIDC IdP flow
          audience: audString
          aws-region: us-west-1

      - name: APPLY plan out files
        id: terraform_apply
        run: |
          python3 apply.py

Here is a similar script running the apply command. It reads each plan file in the artifact. It moves those plan files to their respective modules and applies them one by one.

"""This module reads the artifact contents and moves
the plan files to proper leaf directories and applies
the plan out files one by one"""
import subprocess
import json

path_ref_filepath = 'artifact/path_ref.json'

def move_plan_out_to_proper_dir(plan_file_name, path):
    plan_mv_cmd = ["mv", f'artifact/{plan_file_name}', f'{path}/plan.out']
    result = subprocess.run(plan_mv_cmd,
                            stdout=subprocess.PIPE,
                            stderr=subprocess.STDOUT,
                            text=True)
    print(result.stdout)
    result.check_returncode()


def run_terraform_init(path):
    result = subprocess.run(["terraform", f"-chdir={path}", "init"],
                            stdout=subprocess.PIPE,
                            stderr=subprocess.STDOUT,
                            text=True)
    print(result.stdout)
    result.check_returncode()


def show_plan_file(path):
    result = subprocess.run(["terraform", "show", "plan.out"],
                            cwd=path,
                            stdout=subprocess.PIPE,
                            stderr=subprocess.STDOUT,
                            text=True)
    print(result.stdout)
    result.check_returncode()


def run_terraform_apply(path):
    result = subprocess.run(
        ["terraform", f"-chdir={path}", "apply", "plan.out"],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True)
    print(result.stdout)
    result.check_returncode()


def main():
    with open(path_ref_filepath) as file:
        file_contents = file.read()

    successful_apply = []
    path_ref = json.loads(file_contents)

    for plan_file_name in path_ref:
        path = path_ref[plan_file_name]
        print(
            f"\n===============================Processing: {path}===============================\n"
        )

        move_plan_out_to_proper_dir(plan_file_name=plan_file_name, path=path)
        run_terraform_init(path=path)
        show_plan_file(path=path)
        run_terraform_apply(path=path)

        successful_apply.append(path)

    print("apply successful in these directories:", successful_apply)


if __name__ == "__main__":
    main()

After this job is successful, the pipeline will have applied the changes to each module. The output logs will display the logs from the Terraform run that achieves our desired goal.

In conclusion, you can use multiple approaches to model your CI workflow; the above is one example. A centralized Terraform execution environment helps achieve a collaborative infrastructure workflow for engineering teams. The shared state files ensure that changes are applied serially to common resources and that everyone on the team is using the same tooling, thereby avoiding configuration drift. The GitOps model scales with your team. It also allows for faster collaboration on the infrastructure code.

Best Practices and Learnings

Based on our experience running Terraform on CI, here are some best practices and learnings from this workflow:

Module structure

Organize your Terraform code into small, logical modules that can be updated without affecting other modules (single reason to change). This separation enables the parallel execution of Plan and Apply on CI on multiple modules in different CI jobs, reducing the cascading effect of infrastructure changes.

terraform/
  ├── networking/         # VPC, subnets, etc.
  ├── compute/            # EC2, ASGs, etc.
  ├── database/           # RDS, DynamoDB, etc.
  ├── observability/      # CloudWatch, Grafana, etc.
  └── iam/                # IAM roles and policies

Handle Terraform state locking

When multiple jobs run in parallel and multiple developers work on infrastructure, state locking becomes essential. We store our terraform state on S3 buckets.

terraform {
  backend "s3" {
    bucket         = "terraform-state-bucket"
    key            = "path/to/state/file"
    region         = "us-west-1"
    profile        = "core_aws_account"
  }
}

Separate environments with workspaces or directories

For multi-environment setups, use directories and sub-directories. Keeping environments separate makes testing easy, brings confidence in changes, and facilitates easier debugging and rollback.

terraform/
  ├── dev/
  │   ├── networking/
  │   └── kubernetes/
  ├── prod/
  │   ├── networking/
  │   └── kubernetes/

Use variables for cross-account resource references

When resources in one account need to reference resources in another account, use variables and data sources to facilitate this connection. It reduces confusion in resource naming and clearly expresses intent.

# In account A
output "vpc_id" {
  value = aws_vpc.main.id
}

# In account B
variable "account_a_vpc_id" {
  description = "VPC ID from Account A"
}

data "aws_vpc" "from_account_a" {
  provider = aws.account_a
  id       = var.account_a_vpc_id
}

Security considerations

Limit the permissions of the CI role to only what’s necessary. */* should be used cautiously.
Consider manual approval for sensitive changes
Be cautious with terraform output

Multi Cloud Infra

In case your infrastructure spans multiple cloud providers, the approach of role chaining and change detection remains the same.
The module organization becomes a key aspect in the smooth functioning of the pipeline
You may need to repeat the similar role chaining respective to other cloud providers. It will become easier if the underlying cloud provider supports a similar short-term credentials approach.

Conclusion

Running Terraform on CI offers numerous benefits for infrastructure management:

Standardized environments for all terraform runs
Improved collaboration through PR reviews
Version control and history for all infrastructure changes
Reduced manual toil and human error
Better onboarding experience for new team members
Reduces knowledge silos
Gives a solid base for your IaC to scale as the team/company grows

You can create a robust infrastructure-as-code pipeline that scales with your organization by setting up proper authentication and workflow design and following best practices. This approach democratizes infrastructure changes while maintaining security and control over critical resources.

The journey to effective Terraform in CI may require an initial investment in setup. However, the long-term benefits of increased productivity, reliability, and team satisfaction make the effort worthwhile.

In part 3, we will explore how to use Ansible and Terraform to manage a fleet of Virtual machines.

Terraform on CI - Part 1

Thu, 07 Nov 2024 22:12:03 GMT

We at Pixxel heavily use Terraform to create, change and destroy infrastructure. Terraform allows you to declare infrastructure as code in a language called HCL. Once you have expressed your IaC in code, you can commit to a version control. You can get it reviewed collaboratively; if things go wrong, you can roll it back. It allows us to move fast, following principles similar to software engineering, including clean code practices. With that said, the following section will provide a basic introduction to Terraform and explain how we use Terraform to democratize infrastructure at Pixxel.

Typical Terraform Workflow

Terraform allows you to dry-run your changes before actually applying them. This is called a “Plan.” The result of running a plan is a human-readable diff of the changes.

Here is the most straightforward Terraform file. It uses the kreuzwerker/docker provider to pull an image nginx:latest and create an nginx container named psyduck.

# main.tf

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "3.0.2"
    }
  }
}

provider "docker" {
  host = "unix:///var/run/docker.sock"
}

resource "docker_image" "nginx" {
  name = "nginx:latest"
}

resource "docker_container" "nginx" {
  image = "nginx"
  name  = "psyduck"
  ports {
    internal = 80
    external = 80
  }
}

If you run terraform plan from the root directory of this file. You get the following output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # docker_container.nginx will be created
  + resource "docker_container" "nginx" {
      + attach                                      = false
      + bridge                                      = (known after apply)
      + command                                     = (known after apply)
      + container_logs                              = (known after apply)
      + container_read_refresh_timeout_milliseconds = 15000
      + entrypoint                                  = (known after apply)
      + env                                         = (known after apply)
      + exit_code                                   = (known after apply)
      + hostname                                    = (known after apply)
      + id                                          = (known after apply)
      + image                                       = "nginx"
      + init                                        = (known after apply)
      + ipc_mode                                    = (known after apply)
      + log_driver                                  = (known after apply)
      + logs                                        = false
      + must_run                                    = true
      + name                                        = "psyduck"
      + network_data                                = (known after apply)
      + read_only                                   = false
      + remove_volumes                              = true
      + restart                                     = "no"
      + rm                                          = false
      + runtime                                     = (known after apply)
      + security_opts                               = (known after apply)
      + shm_size                                    = (known after apply)
      + start                                       = true
      + stdin_open                                  = false
      + stop_signal                                 = (known after apply)
      + stop_timeout                                = (known after apply)
      + tty                                         = false
      + wait                                        = false
      + wait_timeout                                = 60

      + ports {
          + external = 80
          + internal = 80
          + ip       = "0.0.0.0"
          + protocol = "tcp"
        }
    }

  # docker_image.nginx will be created
  + resource "docker_image" "nginx" {
      + id          = (known after apply)
      + image_id    = (known after apply)
      + name        = "nginx:latest"
      + repo_digest = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

You can apply the plan by running terraform apply if the plan looks good. You will observe the following output:

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

docker_image.nginx: Creating...
docker_container.nginx: Creating...
docker_image.nginx: Still creating... [10s elapsed]
docker_container.nginx: Still creating... [10s elapsed]
docker_image.nginx: Creation complete after 10s [id=sha256:4b196525bd3cc6aa7a72ba63c6c2ae6d957b57edd603a7070c5e31f8e63c51f9nginx:latest]
docker_container.nginx: Creation complete after 10s [id=cf2b32c1c0c3b3aa53386949fe5ca35b19b753a648e34d582fd35b5f5a858cf1]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Terraform stores the current infrastructure state in a “state file” named “terraform.tfstate” extension. When you run the plan, it will show the diff between the current code and the state file. When you accept and apply the diff, Terraform will make the appropriate changes to update the infra and state file. The code and state remain in sync. Terraform uses the Providers to make the infra changes.

To run Terraform collaboratively, a team of SRE/DevOps engineers may choose to share the same state file with everyone in the team - by putting it in a shared bucket, e.g., an S3 bucket. This way, multiple people making changes to infra will not accidentally overwrite each other. When someone runs a plan/apply action, Terraform acquires a lock on the state file. When others try to access the state, they will see an error ensuring the serializability of infra changes. We can share the state file by mentioning the backend block inside the terraform file.

Example:

# main.tf

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "3.0.2"
    }
  }

  backend "s3" {
    bucket = "mybucket"
    key    = "path/to/my/key"
    region = "us-east-2"
  }
}

Terraform will try to connect to this bucket and proceed to it using the configured authentication mechanism. In later sections, we will explore authentication in depth.

Going one step beyond

Even with a collaborative workflow, the responsibility of making infra changes relies solely on individual DevOps/SRE team members. This can be a good or bad thing, depending on how one views it.

It is good because only a particular set of people have to do infrastructure work, giving other developers (such as Backend, ML, and AI engineers) time to focus on their core problems, which are shipping out new features and fixing bugs. It is bad because only a particular set of people can make infrastructure changes 😄, which creates a blocker for other teams, even for the simplest tasks.

Most of the time, the infra changes requested by the developer and other teams are ops/toil in nature, such as giving or removing access, increasing disk sizes, creating SSH users on servers, or setting up CI/CD for a new service. Infra teams can set up local automation scripts for such ops tasks. They can even share the scripts with other teammates.

The issue with local runs is that the environment can break easily. For example, provider version mismatch, unintended package updates, corrupted config files, and cloud authentication issues can break local automation environments due to regressions. These issues can also be a problem for Terraform modules. Such regressions can delay resolving these ops requests and lead to X-Y problem scenarios. The Engineer wanted to complete the ops task but found themselves debugging the correct config file values. Situations like these can cause ops fatigue if not appropriately dealt with. Introducing a new member to the team increases the problem many folds. An experienced teammate must personally teach the new member all the “tips” and “tricks” of running Terraform. The complexity of the infrastructure landscape can also add up to more regressions, such as many cloud providers in use and legacy infra code. It just adds up the time to resolve the ops task at hand.

That’s where terraform on CI can help.

We can run the terraform automation scripts on a CI pipeline. In that case, we can drastically reduce the average time to resolve ops tasks. Furthermore, developers can get unblocked by contributing to such automation since the code speaks for itself! It saves much time for everyone. Moreover, running Terraform on a CI is just like any other recurring software engineering task that needs to run in an automated way if appropriately designed. In the upcoming section, I will explain how we put Terraform on CI at Pixxel. But let’s look at how the terraform workflow stated in the previous section changes. Below are the updated steps:

You make the required code changes to the relevant terraform modules.
You raise a PR with such changes
The CI detects the changed modules against the main branch.
The CI runs the Terraform plan on these modules using the same shared state file.
Other team members can review the plan. You can also perform static code checks such as linting and formatting.
If the plan looks good, the PR can be merged.
Once the changes are in the main branch, the plan can be applied(automatically or with a manual trigger).

This approach has several benefits. The CI will reliably set up Terraform, its providers, and other supporting packages. It can carry out tasks repeatedly without drifting configurations. Engineers can make multiple infra changes can be made in parallel. It is collaborative, as the logs from the Terraform run are visible to people inside your org. People can review the diff, request changes, and suggest changes quickly. Moreover, you can roll back reliably.

This workflow increases everyone’s participation in infrastructure and builds confidence in changes. And let’s be honest, solving ops problems all the time is boring and soon becomes frustrating. If you can’t avoid it, best automate it. Once you have Terraform running on CI, you can “hopefully” ask dev teams to raise PRs — for simple ops changes. You can point them to how to do it from similar ops changes done in the past. Of course, this claim that developers will raise Infra PR depends heavily on the engineering culture of the org 😅. But I think it brings hope for collaboration, better than zero visibility into infrastructure by developer teams.

With the intent set, part two of this blog will discuss the technical aspects of setting up Terraform on CI. We will cover Authentication and Authorization with cloud providers, Terraform providers, and CI triggers. Stay tuned!

Deploying OTEL Collector as Daemonset

Sat, 02 Dec 2023 22:12:03 GMT

The open telemetry collector is one of the most popular CNCF projects that attempts to standardise telemetry data collection, processing, and export. At Pixxel, we use OTEL Collector to discover targets in a k8s cluster, scrape metrics from the configured endpoint and export the metrics to the Grafana Mimir server. This is a mix of both push and pull approaches. The pull part refers to discovering targets and scraping the endpoints. The push part refers to remote-writing the metrics to a Monitoring server.

An alternative way to achieve this setup is to run Prometheus in agent mode which can scrape targets and remotewrite to desired endpoint. However, a significant drawback is fault tolerance. One Prometheus pod per cluster(agent or normal mode) is a single point of failure. If that pod becomes unhealthy, you may lose metrics for the whole cluster. However, these limitations can be overcome by proper resource/limit configuration to signal the priority of the pod to the Kubernetes cluster or by correct retry/buffer configuration that presents you with more ops work. Moreover, you may need to revisit these configurations as and when the workload increases in the cluster. It feels more work than a setup that distributes the scraping workload to many replicas(daemon sets). The OTEL collector was a no-brainer choice since Prometheus doesn’t support a daemon set deployment. Although there are other collectors that support daemonset deployment: like Grafana Agent.

The following section describes configuring the OTEL collector as a daemon set. The fun part is leveraging Kubernetes pod service discovery with a node name filter. Since there weren’t many good tutorials available on this deployment mode, I decided to write one :P

Setup

Let’s use the OTEL Collector helm chart to deploy it as Daemonset. I’ll be using Minikube for the demonstration below. Let’s start a local Kubernetes cluster with two nodes.

minikube start --nodes 2 -p local-k8s-cluster

Add OpenTelemety repo to Helm:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

Service Discovery

We want to scrape only those pods running on the same node for each OTEL collector pod. We can achieve that in 2 steps:

Discover all pods using kubernetes_sd_config. We will need to provide cluster roles to our daemon set to discover pods.
Filter using nodeName spec. ref.

scrape_configs:
  - job_name: otel-daemon-scraping
    scheme: http
    kubernetes_sd_configs:
      - role: pod
        selectors:
          - role: pod
            # only scrape data from pods running on the same node as the collector
            # assuming KUBE_NODE_NAME env will be set in collector pods env
            field: 'spec.nodeName=${env:KUBE_NODE_NAME}'

With the nodeName field selector filter, we can pick only targets on the same node. The node name is read from the env variable(assuming it is set in the env of the pods during provisioning). Prometheus doesn’t support env variables in the config file, which limits its usage as a daemonset.

After the pods have been discovered per node, we need a way to find the port number and path on the pods to scrape metrics. We can leverage pod labels and annotations for this. OTEL Collector provides the pod label and annotations inside the metric label for that target, which we can read and take required action using relabeling(relabel_configs):

relabel_configs:
  # scrape pods annotated with "prometheus.io/scrape: true"
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    regex: true
    action: keep
  # read the port from "prometheus.io/port: <port>" annotation and update scraping address accordingly
  - source_labels:
      [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    # escaped $1:$2
    replacement: $$1:$$2

The first item in the relabel_configs array checks for the presence of the annotation prometheus.io/scrape: true on the discovered target. Only those targets that match the regex specified in the keep block are kept. keep and drop actions allow us to filter out targets and metrics based on whether our label values match the provided regex. The second item in the array configures the host and port of the target. The scheme is set to http by default. The path is set to /metrics by default. These configs can also be overridden by relabelling but we will go with the defaults for brevity.

So far, we have established the receiver configuration of the OTEL Collector. To complete the demonstration, end to end, I will run a Prometheus Server where our collector pods will write the metrics. We can set up a Grafana instance to query from this server afterwards.

Metric storage with Prometheus

Here is a Kubernetes manifest to create a Prometheus deployment enabled with remote write, exposed via a K8s Service.

# prometheus.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s
    scrape_configs:
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  labels:
    app: prometheus-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - '--storage.tsdb.retention.time=12h'
            - '--config.file=/etc/prometheus/prometheus.yml'
            - '--storage.tsdb.path=/prometheus/'
            - '--enable-feature=remote-write-receiver'
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  selector:
    app: prometheus-server
  ports:
    - protocol: TCP
      port: 9090
      targetPort: 9090

Apply the above helm override to our cluster.

kubectl apply -f prometheus.yml

After applying the above file to our cluster we will have a working Prometheus deployment ready to accept remote writes. It is exposed via a kubernetes service which can be accessed via this DNS:

http://prometheus.default.svc.cluster.local:9090

Collector config

Now is the time to push some data to Prometheus. Let’s deploy our OTEL collector with the proper configuration using the Helm override file below.

# otel-override.yaml
mode: 'daemonset'
extraEnvs:
  - name: KUBE_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

config:
  exporters:
    logging:
      verbosity: detailed
    prometheusremotewrite:
      endpoint: http://prometheus.default.svc.cluster.local:9090/api/v1/write
      external_labels:
        collector: otel-collector

  receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: metrics-exporter
            scheme: http
            kubernetes_sd_configs:
              - role: pod
                selectors:
                  - role: pod
                    # # only scrape data from pods running on the same node as prometheus
                    field: 'spec.nodeName=${env:KUBE_NODE_NAME}'
            relabel_configs:
              # scrape pods annotated with "prometheus.io/scrape: true"
              - source_labels:
                  [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                regex: true
                action: keep
              # read the port from "prometheus.io/port: <port>" annotation and update scraping address accordingly
              - source_labels:
                  [
                    __address__,
                    __meta_kubernetes_pod_annotation_prometheus_io_port
                  ]
                action: replace
                target_label: __address__
                regex: ([^:]+)(?::\d+)?;(\d+)
                # escaped $1:$2
                replacement: $$1:$$2

  service:
    pipelines:
      metrics:
        receivers: [prometheus]
        exporters: [prometheusremotewrite]

clusterRole:
  create: true
  rules:
    - apiGroups: ['']
      resources:
        - pods
      verbs: ['get', 'list', 'watch']

  clusterRoleBinding:
    name: 'otel-discoverer'

ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false
  metrics:
    enabled: false

This can be applied to cluster: helm install otel-collector open-telemetry/opentelemetry-collector -f otel-override.yaml

Let’s go through some important aspects of this Helm override:

We set the deployment to daemon set using mode and added an extra env var to capture the node name on each daemon pod of OTEL Collector.
The metrics pipeline is built using the prometheus receiver and prometheusremotewrite exporter defined in their respective sections. If you are unfamiliar with receivers and exporters, here is a reference.
The receiver has rules for pod discovery using pod annotations. It picks only those pods that are running on the same node using the nodeName field selector. The nodeName is specified in a templated way, spec.nodeName=${env:KUBE_NODE_NAME}, which will render to node name during runtime.
The exporter writes to the Prometheus server at its remote write endpoint. It also places a label collector: otel-collector in all the series. Such “global” labels can be crucial when running multiple Kubernetes clusters in the same/different cloud accounts(AWS Account/GCP project).
The required cluster role and binding are specified to discover pods.
We also don’t open the OTEL Container Ports that are not needed in the scope of this article.

Node Exporter workload

Let’s run some actual workload and scrape metrics from it. Node Exporter is an utility that exposes machine metrics. For our demonstration, we will run this inside Kubernetes cluster as deamonset. This will ensure there is one pod per minikube node. These pods can be scraped with our OTEL Collector pods.

We will install Node Exporter using the helm chart with all defailt with few additions. The pod annotations will be added for discovering these pods. Since NodeExporter by default exposes the metrics in http scheme and /metrics path, we won’t need to specify it explicitly as our collectors already know that.

Let’s first add the repo:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

For scraping the node exporter pod, we require proper pod annotation that can be specified using the helm override file.

# node-exporter-override.yml
podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9100"

Install NodeExporter using helm with above mentioned override file:

helm install node-exporter prometheus-community/prometheus-node-exporter -f node-exporter-override.yml

With all components(Node Exporter daemon, OTEL Collector daemon and prometheus pod) up and running, this is how the cluster should look like:

➜  kubectl get pods
NAME                                                 READY   STATUS    RESTARTS   AGE
node-exporter-prometheus-node-exporter-h2vxr         1/1     Running   0          7m
node-exporter-prometheus-node-exporter-kcds4         1/1     Running   0          7m
otel-collector-opentelemetry-collector-agent-9rxm7   1/1     Running   0          5m1s
otel-collector-opentelemetry-collector-agent-mbq6x   1/1     Running   0          5m1s
prometheus-5b54c7c696-4kzjj                          1/1     Running   0          6m4h

Querying data

If everything goes right, you should see the data from node_exporter daemons in Prometheus. The simplest way to query is to pop up the prometheus UI on your browser and query for some data.

You can access the Prometheus UI on your browser by port-forwarding the Prometheus Kubernetes service (make sure your port 9090 on your local machine is free to use)

kubectl port-forward svc/prometheus 9090:9090

Navigate to http://localhost:9090 and execute the query shown in the image below. Verify if you are getting a one-time series row per node_exporter.

We can also spin up a Grafana instance for better visualisation: Port-forward Prometheus service locally and add the localhost:port as Data Source(also called Connections) in this Grafana instance.

Debugging

If something doesn’t work as expected, you can use debug logs to see the errors.

OTEL collector debug log: Modify the “service” section to add additional debug config.

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheusremotewrite]

Prometheus debug logs can be enabled via the flag "--log.level=debug" in the spec.containers.arg list

To recap, we looked into how to deploy OTEL collectors as daemons on a Kubernetes cluster and discover pods local to the collector’s node. We also looked into how to scrape metrics from the discovered pods and export these metrics to a remote server. We ran node exporter pods and scraped metrics from them. We also visualised the metrics in Prometheus UI. In the end, we looked into how to debug in case things don’t work as expected. That’s all I had to document for OTEL Collector usage. Stay tuned for more such blogs!

cAdvisor high cardinality

Sat, 18 Nov 2023 22:12:03 GMT

cAdvisor is a popular utility that provides resource usage and performance characteristics of running containers. It collects, aggregates and exports metrics about running containers. It comes integrated inside the Kubelet binary inside Kubernetes clusters. We can collect the cAdvisor metrics from the Kubelet API Endpoint /metrics/cadvisor. cAdvisor metrics can be beneficial to look at the resource consumption of your workload to achieve optimal resource utilisation, debug issues with containers and usage trends. These metrics are represented in the Prometheus Exposition format.

Here is an example metric for CPU Utilisation:

# HELP container_cpu_system_seconds_total Cumulative system cpu time consumed in seconds.
# TYPE container_cpu_system_seconds_total counter
container_cpu_system_seconds_total{container="",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05b3ee07_a2bd_47e7_9599_7961201f20dd.slice",image="",name="",namespace="test",pod="test-workload-7756958c5b-qtkgc"} 22.65 1699453605805

Since cAdvisor or Kubelet doesn’t store these metrics, one can not query historical data, which means users should periodically scrape these metrics and store them in some observability server like Prometheus or Grafana Mimir, etc.

Scraping cAdvisor

Since one cAdvisor instance is running per node, we will need to scrape all of them to get the complete picture of the resource utilisation. At Pixxel, we deploy Open Telemetry collectors as Daemonset to scrape metrics from pods running on all the nodes. That means each node will have one OTEL collector pod scraping the cAdvisor metrics(along with other endpoints).

The following snippet describes how you can scrape cAdvisor metrics using OTEL pods:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: cadvisor
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: true
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - action: labelmap
              regex: __meta_kubernetes_node_label_(.+)
            - target_label: __address__
              replacement: kubernetes.default.svc.cluster.local:443
            - source_labels: [__meta_kubernetes_node_name]
              regex: (.+)
              target_label: __metrics_path__
              replacement: /api/v1/nodes/$$${1}/proxy/metrics/cadvisor

The above should look familiar if you have experience configuring Prometheus. OTEL collector can be a drop-in replacement for Prometheus for scraping with some added benefits - like env variable support. We are scraping the cAdvisor target by doing Kubernetes Node Service discovery. The kubernetes_sd_configs retrievs scrape targets from Kubernetes REST API. The role: node discovers one target per cluster node with the address defaulting to the Kubelet’s HTTP port. We are explicitly setting the address(host and port) and path of the scrape target to the local API Server. I’ll quote Kubernetes documentation on how this address is formed:

The API server’s in-cluster address is also published to a Service named kubernetes in the default namespace so that pods may reference kubernetes.default.svc as a DNS name for the local API server.

Pods running in the cluster should authenticate with the API server with service account credentials. An equivalent curl request would be:

curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"  [https://kubernetes.default.svc.cluster.local:443/api/v1/nodes/ip-10-1-81-147.us-east-2.compute.internal/proxy/metrics/cadvisor](https://kubernetes.default.svc.cluster.local/api/v1/nodes/ip-10-1-4-5.us-west-1.compute.internal/proxy/metrics/cadvisor)

PS: Kubernetes also provides a handy kubectl command to access the API Server, using which we can query cAdvisor metrics:

kubectl get --raw /api/v1/nodes/ip-10-1-4-5.us-west-1.compute.internal/proxy/metrics/cadvisor

Metrics Cardinality

The metrics prepared by cAdvisor sometimes exceed the label limit set by the observability server. For example, in Grafana Mimir, we see the max label for any series set to 30 (by default) via the config option: max_label_names_per_series ref

After observing OTEL Collector logs after a few hours of cAdvisor scrape setup, I found the Mimir server rejected multiple series because they had more labels than the acceptable limit. Some had 32 and 35. I wanted to see what labels were getting published in cAdvisor. For container_oom_events_total metric, I saw the following labels:

beta_kubernetes_io_arch
beta_kubernetes_io_instance_type
beta_kubernetes_io_os
eks_amazonaws_com_capacityType
eks_amazonaws_com_nodegroup
eks_amazonaws_com_nodegroup_image
eks_amazonaws_com_sourceLaunchTemplateld
eks_amazonaws_com_sourceLaunchTemplateVersion
failure_domain_beta_kubernetes_io_region
failure_domain_beta_kubernetes_io_zone
id
k8s_io_cloud_provider_aws
kubernetes_io_arch
kubernetes_io_hostname
kubernetes_io_os
node_kubernetes_io_instance_type
topology_ebs_csi_aws_com_zone
topology_kubernetes_io_region
topology_kubernetes_io_zone

cAdviosr added these 20 labels(and more) in every series in addition to the global demographic labels such as cluster name, AWS account, etc that we add explicitly. In some metrics, this count reached 25 or so. We quickly concluded that not all of these are required - so instead of changing server limits, we could delete these labels without harming anything. Luckily, Prometheus provides a way to rewrite labelset using metric_relabel_config. We decided to drop all eks_ prefixed labels since these value can be derived from hostname alone. The way to do that is:

relabel_configs: 
  ...
  ...

metric_relabel_configs:
  - action: labeldrop
    regex: 'eks.*'

This change brought down the label count to acceptable limits. Practically we may not need even the topology_ prefixed labels. But we did the least amount of label rewriting to get the system running. If needed in future, we can look into further reducing the labelset using the same approach.

References:

Metric Relabeling

Handling shutdowns, gracefully

Fri, 29 Sep 2023 22:13:03 GMT

Signals

In operating systems, signals are the messages OS sends to the programs to notify about specific events. They are commonly used to interrupt or kill a process. Linux supports multiple signals, each signifying some event. For example, when you press CTRL+C on a process in your terminal, a SIGINT signal is sent to the process by OS. It’s an interrupt signal to terminate the program. A more strong shutdown signal is SIGTERM. The most forceful of all signals is SIGKILL, an instruction to kill the program immediately. You can also send the same signals using the kill utility in Linux. This utility is useful when you cannot CTRL+C a process, like a daemon. Example command: kill -SIGINT pid where pid represents the Process ID.

Modern container orchestration applications like Docker and Kubernetes use signals widely. In the Kubernetes cluster, a user can submit a request to terminate any pod at any time. Kubelet sends the shutdown request to container runtime to stop the containers in the pod. The main process inside each container should handle this signal for a graceful termination. This signal also contains a grace timeout period and defaults to 30 seconds. Read this for more details on how Kubernetes handles pod termination requests made by users.

Importance of handling signals

For a graceful shutdown, the program developer should ideally handle the SIGINT or SIGTERM signals manually. Typically, this would mean freeing up resource use by the process, committing WALs, performing cleanup, etc. The steps performed in the graceful shutdown differ with the application. If the program fails to terminate with SIGTERM, a more forceful SIGKILL might come next and immediately terminate it. Depending on the type of application, this may cause data losses, partial or complete unavailability, deadlocks or inconsistency in metadata, which might fail to reboot the process the next time. I’ve encountered the unavailability and deadlock problems more prevalent in applications that run in some cluster(quorum) and fail to handle graceful shutdown.

For instance, Consider a scenario where we are running a cluster of 3 nodes. There is a Master node that keeps the status of the cluster, including the two secondary nodes. Suppose a secondary node terminates in the cluster and fails to inform the Primary node about it(i.e. ungraceful shutdown). In that case, the Primary node might stop taking write requests altogether in some configuration. One such configuration could be the cluster running in synchronous replication mode. For any Write request to be successful, all secondary nodes must acknowledge the Primary node that the Write was accepted. Now consider the case where a secondary node dies ungracefully without informing the Primary node. The Primary node will wait for an ack from both nodes for incoming Write requests. Still, one ack never arrives within the timeout, so the primary node rejects the Write requests, resulting in partial unavailability. The primary node can continue serving the read requests; hence, it’s not 100% unavailability. One can handle such scenarios in many ways - like changing the replication from synchronous to asynchronous or allowing the Primary node to poll for status, etc. We can also argue if the secondary node had sent the termination notice to the primary node, the primary node would have better knowledge of the cluster status and stop taking write requests altogether (if it wants to maintain a quorum) or reduce the quorum size to 2 nodes.

Golang example

We’ve seen the importance of explicitly handling signals to perform cleanup. The following section gives a demonstration of how to handle Signals in Golang. Golang has built-in signal handling packages os/signal.

There are two parts to handle graceful shutdown:

Notifying our program for the signal received
Performing cleanup/shutdown tasks

For step (1): Golang runtime can notify the program for signals received on a signal channel.

import "os/signal"

..
..

signalChannel := make(chan os.Signal, 1)
signal.Notify(signalChannel)

For step (2): You can listen for signals on this newly created channel forever and handle Signals in a separate function.

import (
    "os"
    "os/signal"
    "syscall"
)

// example handler function that ignores 
// all signals except SIGTERM
func handler(signal os.Signal) {
	if signal == syscall.SIGTERM {   //SIGTERM received
        //primaryNode.notifysecondaryTermination(secondaryNodeId)
		os.Exit(0)
	}
}

func main(){
    signalChannel := make(chan os.Signal, 1)
    signal.Notify(signalChannel)

    go func() {
		for {
			s := <-signalChannel
			handleSignal(s)
		}
	}()
}

To conclude, crafting good software means giving equal importance to failure scenarios. You can build resilience in your program if you care about the failure modes of your program and what happens after a failure is encountered. Signal handling is one such case developers should care about. That’s all for this blog - catch you in the next one. Stay tuned!

Architecting with AWS VPC

Fri, 29 Sep 2023 22:12:03 GMT

Computer networking is a fascinating concept. Networking is how you connect computers worldwide to let them communicate with each other. The computers talk to each other as per the networking rules applied to them. This article will take you through the administration of computer networks in AWS Cloud.

VPC

A Virtual Proud Cloud(VPC) is an isolated network you can create in your AWS account, similar to a physical computer network (like LAN, etc.). It is logically isolated from other virtual networks in the AWS Cloud.

There are three essential aspects to any VPC:

Name of the VPC
Region: A VPC spans all Availability zones within the region.
Size: A continuous block of IP address represented using CIDR notation. CIDR notation is a mathematical way to represent blocks of IP addresses. CIDR Ranges

With these three inputs, AWS will create a network of IP addresses. The network is a virtual boundary where AWS will deploy your resources.

By default, all these IP addresses are private, which means no one can access the resources from outside the network, but resources inside the boundary can access each other. The first usable IP address in this range is 192.168.0.1. We will look into architecting the VPC to create network topologies of our choice.

To allow internet connectivity to your VPC, you must create an internet gateway(IGw hereon). It’s a highly available and scalable utility that allows traffic from the Internet to reach resources in your VPC. Each IGw has a unique ID in AWS.

Subnet

A subnet is a network inside the VPC. Think of it like a chunk of the “VPC pie” shown above. It is a continuous block of networks contained inside the VPC.

For example, 192.168.0.1/26 represents a total of 64 IP addresses in the (usable)range 192.168.0.1 to 192.168.0.62

We use subnets to isolate and optimize network traffic. We can also use them to provide high availability and connectivity options for the resources. Specific privacy rules decide what traffic can get in and out of any subnet. We represent these rules as “Route Tables”.

In AWS, any subnet has three significant aspects:

The VPC it belongs to
The availability zone
CIDR block

Route Tables

External traffic reaches inside the VPC via an IGw. But for the traffic to get to the right resource within VPC, we must route it to the correct network(Subnet). That’s where Route tables come into the picture. Route tables decide the routing of traffic within a VPC. For example, to allow traffic from IGw to a subnet, a route must exist from the IGw to the Subnet.

We classify a subnet as public or private by the type of routing table linked to it. We can apply Route tables at the subnet level or VPC level. Hence, we should place the resources that need Internet connectivity inside a public subnet. Similarly, put all resources that don’t need Internet connectivity inside a private subnet.

Here is an example route table:

Destination	Target
0.0.0.0/0	igw-id
10.1.0.0/16	local

Any route table has no meaning unless it gets attached to a network (i.e. Subnet). Each row in this table is a route that determines where to direct traffic. The first column represents the source of the traffic, and the second column represents the medium.

So, inside the table shown above, the first row implies the following :

0.0.0.0/0 represent traffic from any source (meaning all IP address in the universe) via the internet gateway and should be routed inside the Subnet this route table is attached to.

The second row implies that only traffic in the 10.1.0.0/16 range should be routed to the Subnet this route table is attached to. local is a particular target that means traffic from other subnets within the same VPC.

AWS route tables support many targets such as NAT Gateway, Network Interface, Peering connection, outpost local gateway, virtual private gateway, etc.

Each VPC has a default route table, the Main Route Table attached to it. The following describes the content of the Main Route table. For our VPC shown in the image above, the Main Route Table will contain the following two rows:

Destination	Target
192.168.0.1/24	local
0.0.0.0/0	igw-id

AWS assumes you want traffic to move between your resources created in the VPC, making the first entry by default. The second entry specifies traffic from the Internet can reach the VPC.

Architecting VPCs

AWS creates VPC for us in every region by default. When you create a new AWS account, you get a default VPC. Default VPCs provide a way to access the EC2 instances over the Internet properly. For your usage, you can select the default VPC or build a custom VPC on your own. The custom VPC could be more secure and provide granular options as you can control the subnets’ configuration and route tables.

A default VPC has a public subnet in each Availability Zone, an internet gateway, and settings to enable DNS resolution. Therefore, you can immediately launch Amazon EC2 instances into a default VPC - designed to fast-track the simple use cases. The default VPC provisions the CIDR range, subnets and gateway for you. The following is a list of configuration AWS sets for your default VPC:

Creates VPC of size /16 IPv4 CIDR block (172.31.0.0/16). This block provides up to 65,536 private IPv4 addresses.
Creates default subnet of size /20 in each Availability Zone. This Subnet provides up to 4,096 addresses per Subnet. AWS reserves a few of them.
Creates an Internet gateway and connects it to default VPC - so your VPC has Internet access by default.
Adds a route to the main route table that points all traffic (0.0.0.0/0) to the internet gateway.
Adds a route to the main route that allows inter-subnet traffic

Once you create the VPC, you should create subnets that let you decide the visibility of the IP addresses to the outside world. In the following section, we will deploy a sample application to see all the components in action.

AWS Networking in action

Our application is a movie directory consisting of a frontend that shows a catalogue of movies in a web-based interface and a database with some movies stored along with relevant tags and metadata(such as director name, release year, genre, etc.). We want to deploy the frontend and backend on separate EC2 machines where only the frontend should be accessible from the public Internet. Let’s begin with a simple organization of resources inside AWS VPC.

At the topmost level - we have the US-east-2 region represented with two availability zones. Let’s go through what we’ve done in the above diagram:

We created a VPC denoted by 10.1.0.0/16 and added an Internet Gateway to it
The VPC is split into four equal-sized subnets. Each availability zone will have two subnets, one public and one private. The Public subnets will have a custom route table associated, allowing a route from 0.0.0.0/0 via the Internet Gateway. You should place resources that must be reachable from outside the VPC inside this Subnet.
We host two replicas of each instance(frontend, backend and database) for high availability. We disperse the replicas of each application in different availability zones. So, if an availability zone goes down, our application remains reachable to users or other services.
We want only frontend replicas to be reachable from the Internet. Hence, we put frontend replicas inside Public Subnets. We put Backend and Database replicas inside a Private subnet. Frontend replicas can reach resources in the private Subnet with the help of route table entry 10.1.0.0/16 via the local target.

That’s all for a basic introduction to AWS Networking. We will explore more AWS-related concepts in future blogs. Stay tuned!

Siren - Alert management at scale

Sun, 16 Jul 2023 22:14:03 GMT

This blog takes inspiration from a significant project I undertook at Gojek in 2021-22. It has been pending publication for a while, mainly due to my lack of diligence 🙈. However, I have now resolved to be more active in blogging, starting with the unpublished drafts I have accumulated. Without further ado, let’s dive into it.

Siren is an orchestrator for alerting, written in Golang and open-sourced under Raystack community developed at Gojek Data Platform team back in 2021.

The Gojek Data platform team provides various offerings to Gojek’s engineering teams in order to make their data journey hassle-free and self-serve. The team enables engineers to ingest, process, operate and consume data with just a few clicks of buttons. This is achieved via a suite of products, serving use cases in the data lifecycle viz: ingestion, loading, transformation, intelligence, analytics, and so on.

A snapshot of range of products that build the data-experience:

Among the various products available for supporting these use cases, Siren stands out in the operations domain. Siren enables developers to monitor the well-being of these products by placing alerts on their data pipelines. Now, let’s delve into the process of accomplishing this.

Introduction

Alerts are integral part of any observability pipeline. Once the automated system detects any anomaly in the system, it can trigger an alert, informing humans about it. Alerts can be defined on the metrics that developers care about like these scenarios below:

A Kafka broker is degraded e.g. CPU usage is high or disk filled etc.
High consumer lag can cause the degradation of other dependent services.
JobManager of some Flink cluster becomes unhealthy.
Sudden traffic increase on a service which demands resource resizing

Siren can be used to develop alerting pipelines quickly in a self-serve manner. The following will get into the details of how Siren achieves this, by first laying some foundation of how alerting works, in general.

With alerting, the aim is to notify via appropriate channels(Slack or Email or Pagerduty, etc.) when some event occurs(mostly bad events) so that developers are aware of the degradation and can take actions accordingly, well within time.

For that, you’re required to first collect the health data of the system under observation. This is typically done by some monitoring server(e.g. Prometheus, Influx, etc.). You also set alerts on top of the collected metrics. Siren can be used to configure alerts inside the monitoring servers. Monitoring Servers are referred to as Providers inside Siren. There are two ways of collecting metrics: push-based and pull-based.

Monitoring servers provide ways to define alert conditions on top of this data. Think of it as a continuous evaluation of a script over a window of data. One example script: Check if the CPU usage percentage is higher than 85% over a window of the past 10 minutes.

Once the data is stored by the monitoring server, and the alert scripts(also called rules) are set, they are evaluated periodically on this data by the monitoring server’s rule engine. If the specified conditions match, an alert notification is sent in the configured notification mediums(ex. Email, Slack etc). Below is a dummy architecture diagram of this flow.

Siren provides the abstraction for defining alert conditions via Rules and routing configuration via Subscriptions for a variety of monitoring providers. It lets users define alert conditions and also the routing configurations to dispatch the notifications when alerts trigger.

Siren Alert Configuration

Siren is written in Golang and open-sourced under Raystack. Currently, it supports alert management for Cortex metrics. We plan to add more monitoring providers e.g. Prometheus, Grafana Mimir, Thanos etc. (Your contributions are most welcome 🙏)

In order to create alerts in a monitoring provider(e.g. Cortexmetrics), you should define a Template first. The template has the alert body in the syntax as supported by the corresponding monitoring provider. In Prometheus compatible monitoring providers(Cortexmetrics, Thanos, Mimir etc) alert body is a yaml with 5 required keys: alert, expr, for, labels, and annotations. The template given below holds an array of templatized alerts of critical and warning severity.

Think of templates as a blueprint for your actual alerts. Here is an example template that can be used to create alerts on CPU usage.

apiVersion: v2
type: template
name: CPU
body:
  - alert: CPUWarning
    expr: avg by (host) (cpu_usage_user{cpu="cpu-total"}) > [[.warning]]
    for: '[[.for]]'
    labels:
      severity: WARNING
    annotations:
      description: CPU has been above [[.warning]] for last [[.for]] {{ $labels.host }}
  - alert: CPUCritical
    expr: avg by (host) (cpu_usage_user{cpu="cpu-total"}) > [[.critical]]
    for: '[[.for]]'
    labels:
      severity: CRITICAL
    annotations:
      description: CPU has been above [[.critical]] for last [[.for]] {{ $labels.host }}
variables:
  - name: for
    type: string
    default: 10m
    description: For eg 5m, 2h; Golang duration format
  - name: warning
    type: int
    default: 80
  - name: critical
    type: int
    default: 90
tags:
  - systems

This template is evaluated by go-templates with [[ ]] being the delimiter. As you can see, in the list of “variables”, for , warning and critical have been templatized. It denotes the default values and the data type as well. The user needs to give value to these “variables” during alert creation. After a valid request payload, an alert will be created by Siren. This alert can be further enabled/disabled using Siren APIs.

Templates provide a means of achieving reusability. A generic template (like the one shown above) can be used by multiple teams in multiple environments (i.e. dev, prod etc.) without any alert “configuration drift”. So based on the needs, teams can define their templates and create alerts from them using Siren APIs, instead of rewriting the same alert rule every other time.

Templates allow us to simplify alerting pipeline it to an extent that users don’t even need to know all the details of the alerting rule because most of the complexity is abstracted out by the Siren APIs. It is done by creating an alert marketplace of sorts, where people can read the description of the alert and provide just enough input to create the alert on their data pipeline. Here is a snap of this beauty:

After the Save button click, this user interface calls Siren Rules PUT API with the input provided by User. Siren identifies the appropriate provider and configures these alerts inside it. The same alerts can be edited again and again, like disabling and enabling with the thresholds preserved and changing thresholds.

With the primary use-case of alert management being solved, we looked for an elegant developer experience where we provide a CLI interface and GitOps model of alert management.

These interfaces are quite popular with teams that need alert management in bulk. The rules are uploaded in a git repo and a CI pipeline ensures corresponding alerts are set in the monitoring server in the exact same way. Read more about it here.

Siren Alert Routing Configuration

Siren also provides a way to define the routing of the triggered alerts in a similar fashion.

As we know, the monitoring server continuously evaluates the alerts and sends a notification when the alert fires. So there needs to be appropriate configurations set beforehand (with the required authorization) in order to send out those notifications.

For Prometheus compatible monitoring servers(like Cortexmetrics, Mimir etc.), these configs are set into Alertmanager. All of it can be done via Siren in a self-serve way using Siren. Here is an example of how we let users define alert routing in a UI.

Siren handles the internals to map these configs to the appropriate team, so that any alert owned by a team, reaches only their channels when alerts fire. This is achieved by Siren’s Subscriptions. You “subscribe” to alerts by defining “matchers” on receivers.

The “matchers” are a set of conditions(key-value pairs, kind of a label selectors of k8s) that must match on a given triggered alert for the subscription to be selected for alert routing. Receivers are an abstraction for actual notification medium with authorization support baked in. For example: a “slack” receiver holds the auth-token and workspace name such that alerts can be routed in the slack channels of that workspace on demand.

For example, the image above configures alert routing for a Firehose deployment called example-firehose. The matchers roughly translate to:

Whenever some alert has severity: critical, team: exampleTeam, deployment: example-firehose the alert will be routed to two receivers:

Slack with channel name as given in user input
Pagerduty with service key given in input.

Using subscriptions, you can define any matching condition and pick receivers to handle the routing. Siren subscriptions also abstract out the complexity of alert routing configuration for many providers(e.g. Cortexmetrics, Influx, etc.) via a single API interface.

To my knowledge, in production, it manages the configuration of approximately 1500 alerts and the routing of around 90 teams. By utilizing Siren, we ensure the overall health of services & products within the data platform.

That was a high-level overview of the functionality provided by Siren. We encourage you to share your comments and ask any questions you may have 🙌. Additionally, you can access the Siren documentation, which offers detailed explanations of concepts, terminologies, database schema, API contracts, usage guidelines, and various other informative guides.

Siren is currently in its early stages, and we are eagerly anticipating its potential impact on the open-source community. Our goals include expanding support for multiple monitoring providers and tackling challenging use cases, like enabling alert subscriptions without dependency on internal IAM policies.

If you’re interested in contributing, we invite you to explore the list of good first issues. available. Your contributions would be highly valued and appreciated. Thanks for reading!

Google IAP auth automation for CLI apps

Sat, 15 Jul 2023 22:14:03 GMT

This blog takes inspiration from a significant automation project I undertook at Gojek in 2021. It has been pending publication for a while, mainly due to my lack of diligence. However, I have now resolved to be more active in blogging, starting with the unpublished drafts I have accumulated. Without further ado, let’s dive into it.

Automating Google IAP Auth for Golang applications

At Gojek, various internal applications for Developers uses Google SSO. The Data Platform team also uses Google Authorization Server as the identity provider to identify the developer and enforce proper ACL. We use Google IAP to guard access to our applications.

Google IAP is widely used to control access to web based application running on browsers. The same applications had a CLI based client, where one can perform similar action using command line. Think of it like Google cloud console and gcloud CLI utility.

We wanted to enable our developers to perform the same actions using the CLI applications. The challenge was to authenticate our users and service accounts.

Google IAP performs authentication of the users using Google Single Sign-On and authorizes them against Cloud IAM policies before accessing the protected resource. Before diving deep into the solution, let’s see how it works behind the scene. Here we need to understand two terminologies: OAuth and OIDC.

OAuth 2.0

OAuth 2.0 is an industry-standard for allowing applications to perform some action on behalf of a user into some other application. For example, you can securely allow a 3rd party application to access your Google contact list without giving them your Google account username and password.

Here is how a typical OAuth 2.0 Flow works. You want to allow a 3rd party application (let’s call it 3PA, from hereon) to access your Google contacts list. The sequence of steps that happens under OAuth 2.0 flow is:

3PA prompts you to fetch your contacts list in the browser(ex: using some button prompt.)
3PA redirects your browser to the authorization server, Google in our case. In this request, 3PA will include these three things: client ID, redirect URL, scope(read-contacts).
The Google authorization server verifies your session or prompts you to log in (if not logged in already). Then, it shows you the consent form, asking you if you want the 3PA to access the data per the scope(read-contacts). You could choose to allow or deny.
If allowed by you, the Google authorization server redirects back to the 3PA(on redirect URL) with a temporary authorization code.
3PA then directly calls Google authorization server with the client ID, client secret and authorization code to get an access_token.
The 3PA can use this access token as a key to access users data. Such requests get authorized by the server which owns the user’s resources(contacts list).

One more thing to note here is that the authorization server issues client ID and the secret to the 3PA via a one-time registration flow. Read this to learn how to create access credentials.

OIDC

What’s missing in OAuth 2.0 protocol is that it doesn’t tell 3PA about who you are and anything about you; instead, it only grants 3PA to access your data on your behalf. Enters OIDC.

OIDC is a thin layer on top of OAuth 2.0, which adds functionality around login and profile information along with what OAuth 2.0 offers. So any client using OIDC protocol can establish a login session on behalf of the user. The authorization server supporting OIDC is providing the identity, also referred to as identity-provider. Remember the SSO login screens you see on various websites?

OIDC flow is very similar to OAuth 2.0 with a slight modification.

Here is how a typical OIDC flow differs:

In Step 2, 3PA redirects the user to the Google Authorization server, with the client ID, secret and a specific scope “OpenID”. So the authorization server knows that this will be an OIDC exchange.
In Step 5: The authorization server sent an access_token in exchange for the authorization code. With OIDC scope, the authorization server will also send one more token named id_token. Here, 3PA gets two tokens, access_token and id_token

The id_token is a JWT meaning 3PA can extract information(called claims) from it, such as name, email id, username, expiration etc.

Considering the previously mentioned sequence, we can explore the implementation of the aforementioned process in a Golang Applications.

Solution

We found a reasonably decent open-sourced Golang OIDC client library bwplotka/oidc to start from. With slight modifications as per the Google IAP contract, we made this library compatible to be used by our applications.

After using the modified library embedded in our application in production, we felt this solution could help others as well. So we extracted it out as a standalone library called Casper.

This is what Casper does:

Casper gives you an authenticated HTTP Golang Client for applications to make requests on behalf of users and service accounts.
It also cares about refreshing the token when expired and cache the token in the system keyring for future needs.

Example usage for a User authentication

import (
    casper "github.com/odpf/casper"
)

audience := "1234-xxx.apps.googleusercontent.com"

authenticatedHTTPclient, err := casper.GetAuthenticatedClient(audience)

if err != nil {
	fmt.Println("Error while getting authenticated HTTP client: ", err)
}

The client ID and secret can also be provided as Golang variables. If not provided, Casper falls back to ENV variables GOOGLE_OAUTH2_CLIENT_ID and GOOGLE_OAUTH2_CLIENT_SECRET

Example usage for a Service Account

import (
    casper "source.golabs.io/asgard/casper"
)

filename := "/path/to/service/account/key.json"
audience := "1234-xxx.apps.googleusercontent.com"

authenticatedHTTPclient, err := casper.GetAuthenticatedClient(
	audience, casper.WithServiceAccountCredentials(filename),
)
if err != nil {
	fmt.Println("Error while getting authenticated HTTP client: ", err)
}

The service account key is used to authenticate a Service account. Have a look at the Documentation to learn more about the usage ✌️.

Aftermath

Casper has removed the overhead for developers to automate Google IAP Auth flow in their Golang Applications, saving their time and effort so that they can focus on the actual problem at hand.

Docker Containers Read Only Mount

Sat, 10 Sep 2022 22:15:03 GMT

Docker containers are best suited for stateless application. But they also provide ways to manage state externally in an efficient and reliable manner. The developers need to be cautious while using external storage with containers. Here we discuss how to use external storage and some example best practices around it.

Docker Volumes

For application that rely on an external state, containers are not the best fit solution for deployment. If you store the state inside container, it will be lost once the container is stopped. Volumes are used for this specific reason. You can choose to store the state outside of your container, so that data remains intact even if container is stopped and removed.

Demo

Let’s use host machine’s permanent storage for storing some contents from inside a container. You can mount any directory from host machine inside the container machine.

Let’s create an directory demo on host machine and create a text file inside it.

$ cd ~
$ mkdir demo
$ cd demo
$ echo "foo bar" > hello.txt

Now let’s mount this directory in a docker container using the -v CLI option.

$ docker run --rm -ti -v ~/demo:/data: alpine:latest /bin/sh
/ # cat /data/hello.txt
foo bar
/ #

As you can see, we are able to access the contents of ~/demo which is in the host machine, mounted at /data inside our container.

The usage patterns are:

Any changes made inside ~/demo directory will be visible inside container at /data
Any changes made in /data from inside the container will be visible in ~/demo of the host machine.

Leveraging read-only mounts

It is also possible to mount directories as read only mode. You can do that by passing ro flag e.g.

docker run --rm -ti -v ~/demo:/data:ro alpine:latest /bin/sh

Extending on this idea, it is a good practice to mount your root volume of your container as read only mode, so that process within the container cannot write anything to the root filesystem of the container.

This way you can enforce all writes to go to the expected external storages mounted on different paths, so that the writes are not lost when container is stopped/removed.

Mounting root volume as read-only

The --read-only flag let’s you enforce just that.

docker run --rm -ti --read-only=true -v ~/demo:/data alpine:latest /bin/sh

In the above scenarios, when you run mount you get a output like this:

/ # mount
overlay on / type overlay (ro,relatime,lowerdir=/var/lib/doc...)
grpcfuse on /data type fuse.grpcfuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

The / is ro(read only) while directory mounted at /data is rw(read write).

In a nutshell

Mount root filesystem of container as read only permissions
Mount external volumes with read-write permissions

There are multiple benefits in doing this like this saves you from accidentally writing log-files(which the developer might be unaware of) in container’s internal filesystem and filling up the disk space allocated to container in production.

One should also be cautious while enforcing this as this is quite restrictive. It will not allow any files to be created in / by any process. Your applications must adjust to this restriction accordingly.

tmpfs

But there is a workaround to this as well. You can have /tmp as writeable directory and rest of the container as read only. Use --tmpfs argument to mount a tmpfs file system into the container. Although it is ephemeral i.e. the changes will be lost when container is stopped.

Here in the example below we are launching a 256MB RW temporary storage.

docker run --rm -ti --read-only=true -v ~/demo:/data --tmpfs /tmp:rw,noexec,nodev,nosuid,size=256M alpine:latest /bin/sh

PS: This article was inspired from the Docker Up and Running book

Reducing Docker Images size

Sat, 10 Sep 2022 22:13:03 GMT

Docker has become a defacto standard of building, distributing and running stateless softwares these days. Companies deploy large fleet of containers everyday via their CI/CD pipelines.

It would be interesting to look into the efficiency of docker builds mainly in terms of optimizing the size of container to make it even faster to release and download images.

docker history

We will talk about docker history command here.

Let’s look at a demo.

FROM fedora
RUN dnf install -y httpd
CMD ["/usr/sbin/httpd", "-DFOREGROUND"]

After building an image from this Dockerfile, the image size is ~466MB. In order to optimize the size, you should first look into each layers contribution in the final size.

docker history test will yield just that:

docker history test

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
b5b5ab1ead58   5 seconds ago   CMD ["/usr/sbin/httpd" "-DFOREGROUND"]          0B        buildkit.dockerfile.v0
<missing>      5 seconds ago   RUN /bin/sh -c dnf install -y httpd # buildk…   288MB     buildkit.dockerfile.v0
<missing>      4 months ago    /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      4 months ago    /bin/sh -c #(nop) ADD file:0ee69bd7b31ad9a58…   178MB
<missing>      5 months ago    /bin/sh -c #(nop)  ENV DISTTAG=f36container …   0B
<missing>      10 months ago   /bin/sh -c #(nop)  LABEL maintainer=Clement …   0B

The size of the image is 466MB. As you can see, surprisingly, fedora image was pulled and it was about 178MB, and the httpd took almost 290MB. It’s way higher than the whole fedora, definitely something is not right.

As it turns out, dnf and other popular package manager rely on a cache that store info about all the packages that are available for installation on that platform. We can easily reduce the size by getting rid of this huge cache.

FROM fedora
RUN dnf install -y httpd && \
    dnf clean all
CMD ["/usr/sbin/httpd", "-DFOREGROUND"]

On building the image again, after this cached repository metadata, we get this :

docker history test

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
8596a91ffcea   6 seconds ago   CMD ["/usr/sbin/httpd" "-DFOREGROUND"]          0B        buildkit.dockerfile.v0
<missing>      6 seconds ago   RUN /bin/sh -c dnf install -y httpd &&     d…   52.3MB    buildkit.dockerfile.v0
<missing>      4 months ago    /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      4 months ago    /bin/sh -c #(nop) ADD file:0ee69bd7b31ad9a58…   178MB
<missing>      5 months ago    /bin/sh -c #(nop)  ENV DISTTAG=f36container …   0B
<missing>      10 months ago   /bin/sh -c #(nop)  LABEL maintainer=Clement …   0B

The new size of the image is 231 MB. We were able to get drastic size optimization.

In conclusion, docker history is a powerful tool that can be used to inspect the layers and how much they add up to the final size of the image. Reducing docker images size has direct implication on application distribution and deployment time.

PS: This article was inspired from the Docker Up and Running book

Troubleshooting Docker build failure

Sat, 10 Sep 2022 22:12:03 GMT

When you encounter some error in docker build, there are efficient ways of debugging it like - “Using an intermediate container to debug next commands”.

For example: here is a failing build log:

Step 6/14 : ENV SCPATH /etc/supervisor/conf.d
 ---> Running in 9c0a385269cf
Removing intermediate container 9c0a385269cf
 ---> 8a773166616c
Step 7/14 : RUN apt-get -y update-all
 ---> Running in cd57fc47503d
E: Command line option 'y' [from -y] is not known.
The command '/bin/sh -c apt-get -y update-all' returned a non-zero code: 100

As you can see, the command on step 7 apt-get -y update-all has failed. We know that the command is malformed (since it’s just a demo), but instead editing the command and rerunning docker build, we can check the correctness of the command in an isolated environment itself.

Step 7 uses the container with id 8a773166616c to run the command.

We can use the same container to interactively debug what’s the issue:

$ docker run --rm -ti 8a773166616c /bin/bash
root@464e8e35c784:/#
root@464e8e35c784:/# apt-get -y update-all
E: Command line option 'y' [from -y] is not known.

root@464e8e35c784:/# apt-get update-all
E: Invalid operation update-all

root@464e8e35c784:/# apt-get -y update
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
...
Reading package lists... Done

Once you have found the issue with the command, you can fix it in the Dockerfile.

Golang WaitGroup

Sun, 06 Feb 2022 22:12:03 GMT

Golang provides a set of concurrency primitives which helps programmers write efficient concurrent code. WaitGroup is one of the constructs of Golang Concurrency. Let’s take a deep dive into how it works and how to best use it.

Why use WaitGroup

WaitGroups are an ideal way for you to wait for some set of concurrent operations to complete. Example: Wait for child processes to complete before parent process exits.

Let’s take a simple example.

package main

import "fmt"

func greet(){
	fmt.Println("Hello")
}

func main() {
	fmt.Println("Starting main...")
	go greet()
	fmt.Println("Exiting main...")
}

The output of the program comes to be:

$ go run main.go

Starting main...
Exiting main...

In the program above, the main function is not waiting for the goroutine greet to finish executing. So we do not see the result on stdout from the goroutine.

Usage

WaitGroup can be used here to wait for the concurrent function to complete before exiting the main.

In the modified version:

package main

import (
	"fmt"
	"sync"
)

var greetWaiter sync.WaitGroup

func greet() {
	defer greetWaiter.Done()
	fmt.Println("Hello")
}

func main() {
	fmt.Println("Starting main...")
	greetWaiter.Add(1)
	go greet()
	greetWaiter.Wait()
	fmt.Println("Exiting main...")
}

The output you get:

$ go run main.go

Starting main...
Hello
Exiting main...

In the modified snippet, the noteworthy changes are:

We are creating a WaitGroup object var greetWaiter sync.WaitGroup
When it is time to call the goroutine we increment the adder by 1: greetWaiter.Add(1)
Until we are done with the goroutine, we want to block the execution on main. So we wait till the goroutine finishes execution: greetWaiter.Wait()
From inside the goroutine, we signal done when the goroutine finishes: defer greetWaiter.Done()

Waitgroup Constructs

Add(n): Indicates to the WaitGroup, that n goroutines are beginning.
Done(): Indicates completion of the goroutine who calls it. This is typically used with defer, as a part of the various clean-up actions.
Wait(): Will block the function that has called wait(), until all the goroutines have indicated that they have exited.

Summary

In summary, we can think of WaitGroup as a thread-safe counter for tracking various goroutine’s execution and exit within a program.

Horizontally scaling up a Kafka cluster

Sat, 11 Sep 2021 22:12:03 GMT

I work in the Data Engineering team in Gojek, and my work involves managing lots of Kafka clusters, among other kinds of stuff. This blog is inspired by a recent outage in one of our Kafka cluster, causing degradation in the consumer applications. As a resort, we had to scale up the Kafka cluster horizontally. This blog will acquaint you with horizontally scaling up a Kafka cluster, spotlighting partition reassignment.

Due to a high load on the cluster, we were facing high CPU on few brokers. We had an option to either increase the computing power on each broker simply by replacing them with powerful machines, i.e. vertical scaling. Vertically scaling a Kafka cluster means replacing existing broker nodes with higher capacity nodes while keeping the same brokers.

But there are a few challenges when you do that. For example, it involves careful re-consideration of the broker configs to use the new compute power optimally. Over-provisioning of computing and storage resources to the cluster can be quite costly to the business. We didn’t want to put much effort into recalibrating our broker configurations as per new machine specs to save some time in a critical production support issue. So we decide to go for horizontally scaling the Kafka cluster.

Our Kafka brokers are usually standard Google Compute Engine instances. The broker configurations are defined in reusable IaC modules so that many teams can benefit from these well-thought configurations. All you need to do is provision your brokers using the IaC tool. In the Data Engineering team, we heavily use Terraform for IaC provisioning.

Process

Horizontally scaling the Kafka cluster involves these rough steps:

Create new machines provisioning storage, networking, and compute resources
Start the brokers with your chosen configurations and provisioned resources
Reassign partitions across the cluster so that the new brokers share the load and the cluster’s overall performance improves

Steps 1 and 2 can be challenging, time-consuming, error-prone and tricky when you don’t have a standard way of creating new machines and provisioning them with necessary software and configurations. The Data Engineering team has diligently put lots of effort into achieving these standards across any deployment landscape, be it containers or virtual machines. It gives us exponential benefits saving a lot of time, avoiding configuration drift and human errors. So here, the only thing left to figure out was reassignment of partitions.

Before scaling up, the cluster had six brokers and ~1600 topics. As you might know, Kafka topics are further split into partitions to load balance. Brokers store these partitions. Brokers replicate partitions as per the replication configuration. If you are interested in a refresher, I’ve written about these concepts here. When you add new brokers, you also need to manually rearrange the partitions across the cluster so that the new brokers can balance the load evenly across all brokers.

For each partition of a topic, one broker gets elected as leader of that partition, and a few brokers participate in replicating that partition. So, for example: for a Topic T with replication factor 3, a partition Pi is distributed among Broker 1 as leader and Broker 2 and 3 as followers. When you run reassignment on this topic, leaders and followers will get modified.

Let’s see a detailed example. The image below shows a topic test-topic with 12 partitions and a storage plan across nine brokers.

After running partition reassignment:

As you can see, the Kafka controller node readjusted the partitions storage to different brokers. Initially, the Kafka controller node stored Partition 0 on brokers 3, 4 and 5, with 4th as the leader. After the reassignment, it got stored on 1, 5 and 9 with the 5th broker as the leader.

When adding new brokers, the partition reassignment will ensure that the partitions of topics are balanced across the brokers, including the ones we added as part of scaling. We need to do this on all the topics for optimal load balance across newly added brokers.

Execution

Now let’s have a quick look at how to go about doing the reassignment.

The standard Kafka installation has the required script to run the reassignment. It can be found as bin/kafka-reassign-partitions.sh.

The reassignment is a 3 step process:

First, we put the list of topics we want to load balance in a JSON file.

{
  "topics": [
    {
      "topic": "test-topic"
    }
  ],
  "version": 1
}

We tell the Kafka cluster to generate a plan for us. It will log to console, the current partition replica assignment and proposed partition reassignment configuration

bin/kafka-reassign-partitions.sh \
--bootstrap-server "my-kafka.example.com" \
--broker-list "1,2,3,4,5,6,7,8,9" \
--topics-to-move-json-file "/root/topics.json" \
--zookeeper "my-zookeper.example.com" \
--generate

Put the proposed partition reassignment configuration in a JSON file.

We execute the assignment on the proposed plan.

bin/kafka-reassign-partitions.sh \
--bootstrap-server "my-kafka.example.com" \
--reassignment-json-file "/root/plan.json" \
--zookeeper "my-zookeper.example.com" \
--execute

The execution time depends on the size of the topic, and it happens asynchronously. You can query the state of this execution by verifying.

bin/kafka-reassign-partitions.sh \
--bootstrap-server "my-kafka.example.com" \
--reassignment-json-file "/root/plan.json" \
--zookeeper "my-zookeper.example.com" \
--verify

Next steps

Doing partition reassignment manually for a big cluster having thousands of topics is time-consuming. As a next step, we are looking to develop automation to do the partition reassignment with minimal intervention.

What makes Kafka awesome?

Thu, 07 May 2020 22:12:03 GMT

Kafka is one of those software that are in the foundations of most of the data-driven companies e.g. LinkedIn, Uber, Spotify, Slack, Gojek etc. Kafka solves the problem of high throughput real-time data streaming with minimal latency. It’s flexible and lean architecture has helped companies scale massively without worrying about the quality and latency of the data pipeline. In this blog, we will go through internals of mighty Kafka and will find out what makes Kafka so awesome.

Let’s begin with a subtle introduction.

What is Kafka

Kafka is a distributed streaming platform. To quote from the official website,

A streaming platform has three key capabilities:

Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
Store streams of records in a fault-tolerant durable way.
Process streams of records as they occur.

Kafka is used to build data pipelines, to move data among systems or applications. Kafka can store the data as well.

By “data”, we mean any form of raw data or event that needs to be sent across applications or services. For example, user activity events(page views, button clicks, logins, sign ups, likes, comments etc.) or operational metrics such as errors in applications, crashes, system usage etc.

These raw events are produced by users via their interaction with applications and kafka pipelines can be used to send(stream) them at desired places(applications/services). The consuming applications or services can decide to process this raw data in real time or store it for later processing.

Why do we need to stream these raw data/events in the first place?

Because this helps data-driven companies make efficient decisions. Companies can use this data to develop internal applications and microservices to improve their search relevance, recommendation systems, and target appropriate ads to customers etc.

As the use cases grow and companies scale, the user facing applications produce more and more data. Streaming that data to various internal applications and services without compromising scale is the one of the onus of data engineering.

This data can be of enormous volume. Think of the scale at which social media giants like Facebook, LinkedIn operate and storing all major events done by users on their website and mobile applications. This becomes billions of events every day.

Streaming such enormous amount of data maintaining minimum latency is no simple job. Kafka is very good at this(we will see how). Moreover, Kafka gives you the ability to perform processing on the stream. One example of such real-time processing is calculating surge pricing in apps(like Uber).

Surge pricing is calculated by taking into consideration the current demand and current supply. These two metrics are used in real-time to calculate how much surge price to add to the trip. For example, Surge price in Uber during rains in the evening.

The real-time usage of log data brings several challenges. One of which is high throughput. Kafka gives the ability to perform such real-time processing with minimal latency. Kafka is scalable, distributed, supports high throughput, supports real-time consumption of data and provides a rich set of APIs to make developers life easy.

Kafka Architecture

Kafka’s architecture is fairly simple. There are a few logical concepts that you need to understand in order to deep dive into Kafka. Kafka runs on Pub/Sub Model. Which means, there are entities which produce messages and put in some designated places and there are entities which subscribe to these places.

Topic: Stream of messages of a particular type is called a topic. For example, a BookingLog Topic for Uber will keep all messages of trip booking type made on the Uber app.

Producer: The entity which produces messages to a topic. The producer application constructs the messages and sends it to Kafka to store that message in a particular topic.

Broker: Brokers are servers where the published messages are stored.

Consumer: The entity which consumes data that was put in a topic(from the Brokers). Consumers consume messages inside a topic from Broker via a bridge called Message Streams.

Given these basic terminologies, you would have guessed the basic flow of data in Kafka.

Producers produce messages(data) in topics. Brokers store those data. Consumers can consume data from these topics.

Topics can be of very high throughput depending on the use cases. BookingsLog alone is a very high throughput topic if you think about Uber’s scale. So to balance the load, topics are further divided into Partitions.

Each broker stores one or more partitions of various topics.

Partitions of a topic are simply Log files. Think of all the partitions of a topic as a file on the storage system of the Brokers. Depending on the usage, this partition’s(Log file’s) size can vary. It can be very large for some high throughput topics. So it is split into multiple smaller files, called Segment Files.

A partition now logically looks like:

Partition P2 split into 4 segment FilesLet’s say Partition P2 is the set of segment files S1, S2, S3 and S4. Any new message produced to Partition P2 will be written to the last segment file(here, say, S4).

This is to be noted that, at first the segment file lives in the memory of a broker. After a predefined threshold is reached, then only the segment file is flushed to the disk i.e. written on the secondary storage. There are two such predefined thresholds:

A certain number of messages have been published from the publisher of that partition.
Certain time has elapsed since last published event.

The messages are only exposed to consumers when they are written to the disk. Messages residing in the memory of Brokers are never exposed.

Message format

Every message in Kafka is stored with an offset. This is so because the messages can be of varying size. So offset is decided by the size of the messages in the segment files.

For example, Let’s say for segment file S1, three messages came from the producer which are of size 20bytes, 40 bytes and 20 bytes.

Then the ordering of message in segment file with offset typically looks like the image below with offset of M1 as 0, offset of M2 as 20 and offset of M3 as 40.

Here you can see the offsets are in increasing order but they are not consecutive.

Traditionally, message queue systems (such as RabbitMQ) use message ID instead of offset. The message ID does the one-to-one mapping of Message’s location on disk. This allows the user to do random indexing of the messages. Random access to messages is a seek intensive operation. This is a costly IO operation affecting latency. Kafka avoids using message IDs.

Offsets are also used as consumers acknowledgements. If a consumer acknowledges an offset x, it is assumed that the consumer has consumed all messages with offset < x.

Managing partitions on the broker

It’s time to understand how brokers store and manage any partition. As we have understood, partitions are a set of segment files. Every segment file starts with some message. Every message is identified with a message offset.

Brokers store a sorted table in memory which basically keeps the first offset of each segment file.

Let’s see it this way. Let’s say for partition P1, there are 5 segment files(S1, S2, S3, S4 and S5). The first message in S1 has offset 0. Likewise, first messages of files S2, S3, S4 and S5 have offsets 1000, 2000, 2500 and 2700 respectively.

The broker will store an in-memory table which keeps the offset of the first message in each segment file. Here : 0, 1000, 2000, 2500, 2700

Table of starting offset of each segment fileAny new message which comes, if stored in S5, won’t affect this table. As soon as, a new segment file appears, this table will be updated with the offset of the first message in the new segment file.

This table is used to find the segment file which contains the offset asked by the consumer.

Consumption of messages

Consuming messages is fairly simple. The series of events are:

Consumers send a pull request to brokers. In this request, there are two things: Consumers specify the offset to begin with, and an acceptable number of bytes in the message.
Broker on receiving the offset to begin calculates which segment file has the message with asked offset. Broker searches the above table for the address of the segment file to which the message belongs.
Broker fetches the data from that segment file and sends to the consumer.
The consumer consumes this data.
The consumer sends the next offset in the new pull request.

Efficiency of Kafka

Kafka is considered the best real-time streaming platform out there. The efficiency speaks for itself in various benchmarking tests. This efficiency results from several design decisions(rather unconventional).

Storage Efficiency

The first one is the use of offsets instead of message-ids which saves the cost of seek intensive random accesses. On top of that, keeping an index of the offsets of starting message in each segment file makes consumption from a partition fairly efficient.

The second unconventional choice that we see in Kafka is avoiding any caching in Kafka Layer. Kafka avoids caching every message in memory at the process level. Instead, it relies on the underlying file system cache. This offers multiple benefits.

First, it avoids double buffering. Modern operating heavily use main memory for disk caching. A modern OS will happily divert all free memory to disk caching with little performance penalty when the memory is reclaimed. All disk reads and writes will go through this unified cache. So even if a process maintains an in-process cache of the data, this data will likely be duplicated in OS page cache, effectively storing everything twice. Although there could be a debate on when and how to process level caching is useful. Read this ACM article for a thorough analysis.

Secondly, this cache(the file system cache) will stay warm even if the broker service is restarted, whereas the in-process cache will need to be rebuilt in memory.

The third benefit is that no cache at process level means, less overhead in Garbage collection, giving efficiency in VM based languages.

The fourth benefit comes from the use of traditional caching heuristics like write-through cache. Since both consumer and producer access the segment files sequentially, normal caching heuristics present in most OS works fine.

Network Efficiency

Consumption of messages from the File system involves fetching message from secondary storage and sending all the way to network sockets. This typically involves reading data from the storage to page cache, copying page cache to application buffer, copying application buffer to kernel buffer and finally sending kernel buffer to a socket. This involves 4 data copying calls and 2 system calls.

Kafka uses the sendfile API available in Linux which does the same thing in 2 data copying calls and 1 system call. This optimized API results in the fast consumption of data.

Broker’s design efficiency

Kafka Brokers are stateless. It means that a broker doesn’t know which consumer has consumed till what offset. The consumer itself keeps track of the offset. It reduces overhead from the Broker. But there are cons as well. Deleting messages becomes difficult since the broker doesn’t know if all consumers have consumed that message. For this, Kafka gives a time-based SLA where retention policy can be configured for brokers.

Brokers being stateless gives a side benefit to consumers to deliberately choose to rewind back to an old offset and re-consume(if the messages have not exceeded retention period, of course). Though, it is a violation of queue but has proved as an essential feature. For example, if consumer application feeding on data from Kafka has some bugs, then the data can be replayed after the bug has been resolved.

Distributed Consensus

Let’s see how producers and consumers behave in a distributed setting.

The producers produce data to a topic stored in Brokers and consumers consumer from these topics. Producers need to decide which partition to produce data. A partition can be randomly selected by the producer or by using a partitioning function.

Consumers to a topic are put in a group called “Consumer Group”. Each message m of Topic T is delivered to only one consumer in the consumer group. At any time, all messages from one single partition are consumed by a single consumer. Different consumer groups independently consume messages from topics. No coordination is required among them. But within a consumer group, coordination is required.

The leader inside a consumer group is decided by a separate entity called Zookeeper. Kafka uses Zookeeper as a configuration store. Zookeeper does these 3 things for Kafka.

Detecting the addition and removal of brokers and consumers
Triggering re-balance process in each consumer when the above event occurs
Maintaining track of consumed offset of each partition

Re-balancing, whenever required, is taken care of by zookeeper by running a re-balancing algorithm.

Kafka uses Zookeeper to maintain 4 types of registries(config stores)for various purposes. Let’s quickly see what each registry stores.

Broker registry: Stores information about brokers. It stores the host and port of the broker and the set of topics and partitions stored in the broker.

Consumer Registry: Stores consumer group of each consumer and set of topics each consumer is subscribing.

Ownership Registry: Stores one path for every subscribed partition and the id of the consumer currently consuming for this partition. This consumer is said to own the partition.

Offset Registry: Stores for each partition, the offset of the last consumed message.

The first three registries are ephemeral i,e. if the creating client is gone, these path created in corresponding registries is removed from by zookeeper server. Offset registry is persistent.

Each consumer has a watcher on both Consumer registry and Broker registry. Whenever broker set changes or consumer group changes or the startup of a consumer happens the watcher notifies consumers to initiate a re-balancing algorithm process. This is to determine new subsets of partitions it should subscribe from. To learn more about the re-balancing algorithm, read this manual.

Delivery Guarantee

Kafka only guarantees at least once delivery. By design, they didn’t want to go for the exactly-once guarantee, because that would have retired efforts in two-phase commits which were not necessary requirement for Kafka.

Kafka also doesn’t guarantee to order of the messages. Since messages go to different partitions, can be consumed from different partitions as per the consumers want. So there is no inherent ordering of messages by design in Kafka. Consumers itself will have to take care of the ordering of the messages.

Kafka also takes care of the data corruption by giving Cyclic redundancy check at the message level. This allows you to check network error.

If you have been following up this far, thanks for reading. This blog is inspired by the Kafka white paper. I would suggest the readers explore this wiki to learn some more detailed concepts about Kafka.

Feel free to comment below your thoughts or connect with me on Twitter for any conversations on this blog.

No Silver Bullets indeed!

Sun, 08 Mar 2020 22:12:03 GMT

This week I spent some time finding and reading about classical textbooks and papers in software engineering so far. This blog post is about one of those highly esteemed papers written, titled “No Silver Bullet — Essence and Accident in Software Engineering” by Fredrick P. Brooks. This paper was written and published in 1986. Even after 34 years, this paper is widely discussed in the field of software engineering.

As we have known that in the field of hardware(electronics) there has been a two-fold increase in the number of transistors that can be put on an IC every two years. This is referred to as Moore’s law. Fredrick argues that there is no single development technique which by itself promises even one order-of-magnitude improvement within a decade in productivity or in reliability or in simplicity in software engineering. There is neither any management technique as well which promises such improvements. There are no silver bullets in software engineering.

But why is this case with software engineering? Fredrick analyses the intrinsic nature of the whole process that is “Software engineering” to find out why it is so. He finds two kinds of difficulties or complexities in software engineering. One is Essential difficulty and another is Accidental difficulty. Let’s go through each one by one.

Essential difficulties

Fredrick goes on writing that the very nature of software is the root cause that there is no silver bullet. And this only makes it unlikely that there will be any silver bullet in future. We cannot expect any silver bullets in this field of ours. The essence of complexity in software engineering lies in data sets, relationships among those data sets, algorithms and invocation of functions. But the hard part of building software lies in specification, design and testing. There will always be syntax errors and need for improvement in the code, but they are tiny things compared to the bigger things like unclear specification, design flaws and loose testing.

Software is perhaps much more complex than any other human construct because no two parts are alike. In this aspect, Software shows acute differences from hardware, automobiles or buildings etc. Scaling-up a Software entity is not merely a repetition of the same elements in a larger size rather it is an increase in the number of different elements. The elements, in most cases, interact with each other in some non-linear fashion, and the complexity of the whole Software increase in much more than linearly.

From this complexity, comes the difficulty of communication among team members, leading to product flaws, cost overrun, schedule delays, the difficulty of enumerating, lesser understanding of all the possible states of the program and from that comes the unreliability.

Changeability is one other kind of intrinsic difficulty with Software. All the successful software gets changed(evolved) as time passes. As software is found to be useful, people try it in new cases at the edge of, or beyond, the original domain. The vehicle on which software rides(the underlying hardware) is changing at a very fast pace. The software has to live beyond the life of the hardware.

Accidental difficulties

These are such difficulties which are not inherent to the domain but arise due to the interactions among the stakeholders and processes. There have been some developments in overcoming accidental difficulties in Software Engineering but these alone cannot promise order-of-magnitude improvements. Because they do not solve the inherent difficulties. Let’s go through some of the breakthroughs that solved accidental difficulties to some extent.

High-level languages are one such example. They have been the most powerful stroke for software productivity, simplicity, reliability and comprehensibility. High-level languages frees a program from much of its accidental complexity by providing abstractions over conceptual constructs. It eliminates a whole level of complexity that was never inherent in the program at all. But at some points, the high-level language becomes a burden, that increases the intellectual tasks of the user those esoteric constructs.

Unified programming environments like Unix have boosted productivity by integral factors. These integrated programming environments provide integrated libraries, unified file formats and piles and filters.

Hopes for Silver bullets

Fredrick, next analyses, technical developments that are most often advanced as potential silver bullets. Ada and other programming languages solved some accidental difficulties by its philosophy of modularization and abstract data types. But surely, it will not prove as the silver bullet, as it another high-level programming language. Object-oriented programming looks like one more hope for a silver bullet to Fredrick. But object-oriented programming removes all the accidental difficulties from the expression of design. The complexity of the design itself is essential. And an attack on essential difficulties makes no change whatever in that. Similarly, Fredrick goes on discarding Artificial Intelligence, Expert Systems, Automatic programming, Graphical Programming, Program Verification, Environment & Tools and workstations as silver bullets in Software Engineering.

Attacks on Conceptual Essence

Fredrick goes on analysing some promising attacks on the essential complexity of Software. Buying software is one such approach instead of building it, as more and more vendors offer more and better software products for a variety of applications. They generally come with better documentation and somewhat better maintained than homegrown software. Better software has always shown remarkable improvement in productivity, for example, personal computers with well-generalized writing, drawing file and spreadsheet programs.

Precisely deciding what to build is the hardest single part of building software. Most of the times client also doesn’t know what they want to bet built. So in planning any software activity, it is necessary to allow for an extensive iteration between the client and the designer as part of the system definition. It is really impossible for clients to specify completely, precisely and correctly the exact requirements of a modern software product before having built and tried some versions of the product they are specifying. Rapid prototyping has been a very promising attack on this essence.

Software systems must be grown, not built. It should be grown in incremental development. The system should first be made to run, even though it does nothing useful except call some dummy subprograms. Then bit-by-bit it should be evolved into complex modules and function calls. Iterative development has shown dramatic results in projects. This also helps in building morale. Enthusiasm jumps when there is a running system, even a simple one.

The last piece of this analysis concerns with people. Fredrick argues, that we can get good designs by following good practices instead of poor ones. Good design practices can be taught. Unix, Pascal, Smalltalk, Fortran are some examples of software that are the products of one or a few great designers. Fredrick advocates that the most important single effort we can mount is to develop ways to grow great designers. No software organisation can ignore this challenge. Good managers are scarce but no scarcer than good designers. Great designers and great managers are both rare. Organizations should spend equal effort in finding and developing great designers as they do on management prospects. For those are the people upon whom the technical excellence of the products will ultimately depend. Fredrick proposes few suggestions to grow great designers. Like systematically identifying top designers as early as possible. The best designers are often not the most experienced. Assign a career mentor to be responsible for the development of the prospect, device and maintain a career development plan for each prospect and provide opportunities for growing designers to interact with and stimulate each other.

Conclusion

Well, for starters, this paper convinces me of not searching for any silver bullets. Moreover, it gives a better perspective to me at how to look at software, in terms of Essential and Accidental difficulties. This mind shift change actually lowers the burden of thought process in a day to day basis. I shouldn’t fight or worry about the essence of Software. It happens most of the times to us that we get exhausted or frustrated with the problems while developing software. Thinking those difficulties in terms of essential and accidental is a huge thought process change. Complex/hazy requirements, non-linear interactions of different pieces are all inherent to Software. I think this is both the beauty and the curse of this field.

Inspired from the paper, next I have picked the Mythical Man-Month (anniversary edition). A detailed blog on the book would follow later. Thanks for reading. Feel free to connect with me on Twitter for any conversations on this blog.

Books I read in 2019

Tue, 14 Jan 2020 22:12:03 GMT

I wanted to talk about the books I have read in 2019. I feel like I have learnt some good things from books and I feel like I should share with everyone. So here we go!

It has been a good year since I started realizing the power of reading books. I spend most of my times with books related to the things I like viz: software development, astronomy, personal finances among many other things. A detailed list of books that I read in 2019 is here on Goodreads. This blog talks about the books I loved the most and recommend everyone to read once if interested in that genre.

I will mostly talk about the books of Personal Finances and Astronomy. Let’s start with Personal Finances first.

Personal Finances

Personal finances got my interest at the beginning of my senior year of college. I had taken a course called Financial Engineering and later on, I started liking the various concepts of finances(like options, trading, hedge fund market etc.). Later on, I read some books on personal finances. My personal favourite, on this topic, has been Rich Dad Poor Dad.

This book changed my vision of how I think about finances and expenses. This book teaches us the major difference between the Rich and Poor, which is the mindset (and money follows that). This book teaches you the mindset required to be rich. The author presents very good real-world examples which support his arguments. A simple example, taken from the book:

The poor and the middle class work for money. The rich have money work for them.

The book is filled with much such great advice which will change the way you look at your finances. Personally I agreed with almost all of the advice that was given in the book to be rich. But it is not an easy path to create wealth. The way we have been living our lives has conditioned us to the exact opposite(at least for me). So it requires a lot of discipline, and frequent self-introspection to ensure I am applying those learnings from this book.

I would highly recommend this book for anyone who has never cared about personal finances(literally, complete beginners) and who are struggling to get their financial situations right.

I was very keen to implement those learnings from Rich Dad Poor Dad, and so I started looking for books which teach what all financial instruments are available in India and their analysis. So the next book, I read was What Every Indian Should Know Before Investing. This is a great book for complete beginners who are not very familiar with the various financial instruments like Fixed Deposits(FD), Recurring Deposits(RD), Mutual Funds, Equities, and their tax implications. After gathering sufficient information, I started investing in multiple instruments. These books gave me enough exposure and confidence to take good care of my finances. Highly recommended!

Astronomy

Astronomy intrigues me on a very different level. I like to read about the discoveries and happening of the outer worlds. This subject is just so delightful to read about on any given day. Cosmos by Carl Sagan is such a treat for astronomy lovers. In a 384 pages long journey with Carl, you will discover many worlds like Venus, Mars, Jupiter, Saturn. You will find out what these worlds are made of, the possibility of intelligent lives on these worlds and their probable fate. I love the way he presents information about these world by logical inferences which are so simple to understand, they don’t require a very sound understanding of the Physical Sciences. After reading the books by Carl, I would say he is the best teacher I have got in astronomy. His books always spark curiosity in me to explore the unknown.

Apart from these two subjects, I read a few books targeted on the contemporary political scenarios of India. The free Voice by Ravish Kumar is a good example which presents various incidents happened around India which reflect how India as a nation is changing democratically and culturally.

So, overall it has been a good year. I learnt a lot of new things and applied them in real life. Looking forward to an even great year ahead!

Thanks for reading. Feel free to connect with me on Twitter for any conversations on this.

Object.assign() in JavaScript

Tue, 24 Dec 2019 22:12:03 GMT

Object.assign() is used to assign all enumerable properties of one or more source objects to some target object.

Object.assign() is a very useful and very frequently used method in JavaScript. It takes a target object and one or more source object. It modifies the target object with several usages listed below. Source objects are never modified.

Copying enumerable properties

var x = { a: 5 };
var y = {};

var z = Object.assign(y, x);
console.log(z, y, x);
//{ a: 5 } { a: 5 } { a: 5 }

Let’s see the usage of this method from some examples:

As we can see, the variable x has an enumerable property a which is equal to 5. By doing an Object.assign(y, x) those enumerable properties of x will be copied to y. The copied object will be returned by this method.

Overwriting common enumerable properties

Common properties of the target object will be overridden by values in source objects. Let’s see the usage of this method from some examples:

var x = { a: 5, b: 7 };
var y = { a: 4 };

var z = Object.assign(y, x);
console.log(z, y, x);
//{ a: 5, b: 7 } { a: 5, b: 7 } { a: 5, b: 7 }

As we can see, the variable x and y have common property a. By doing an Object.assign(y, x) the common enumerable properties of the target get overridden by that of the source.

Merging objects

This method can also be used to create new objects by merging two or more variables. For example:

var x = { a: 1 };
var y = { b: 2 };
var z = { c: 3 };
var v = { d: 4 };
var w = Object.assign(x, y, z, v);

console.log(w, x, y, z, v);
// { a: 1, b: 2, c: 3, d: 4 } { a: 1, b: 2, c: 3, d: 4 } { b: 2 } { c: 3 } { d: 4 }

Although there is nothing different in the working on this method in this example, if you see it from the perspective of utility, you can use this method for merging multiple objects in one object. Here we are merging y, z, v, into a target object x.

Object.assign() is very important method finding it’s usage at various places.

Core value of core engineering

Mon, 09 Sep 2019 22:12:03 GMT

At Gojek, fresh graduates go through a rigorous two months Bootcamp programme. The Bootcamp is structured in several modules. The first of these modules are core engineering. Here I will introduce you to some of the main themes of the core engineering module. Let’s dive in!

How to make decisions, quickly!

An engineer is supposed to make many decisions every day. Decisions can be of many aspects ranging from how to name a variable to what should be the road map of the team for the upcoming days and months. This is a problem that we face several times a day. You can’t just take any decisions. You have to have some reasoning behind it. You must do some analysis of alternatives. Sometimes you have to convince your teammates about your approaches to a problem(what decision you took)! Here come the issues when you have to explain everyone and convince them.

You need to develop many soft skills. No matter how smart you are or how great a programmer you are, but if you cannot communicate properly and if you cannot explain complex things to people in simple English, you are going to find it very hard to deal with people, every day.

Photo by Austin Distel on Unsplash

Time is the most important factor

Time is one of the most valuable resources for everyone in the office. So if you are making a conversation in your team, you need to be good enough to express your thoughts clearly and quickly. You need to learn how to speak clearly. In the Bootcamp, we go through a rigorous speaking drill where we practice how to be a good speaker and a good listener. Here I have learned how to convey your thoughts to your audience. An engineer has to deal with a wide range of audience working across different teams. For example, at Gojek, a product engineer may need to talk to Product Managers, Tech Leads, other engineers, business intelligence team, operations team, system engineers, peoples partner etc. There is a different level of engagement needed for different people. You need to put your words differently to be able to convey your message correctly and quickly.

Be fallacy free

Most of us fall in the trap of logical fallacies in our arguments. In the Bootcamp, we practice rigorously in speaking drill to make our arguments free from logical fallacies. We practice hard to inculcate the habit of making fallacy free arguments to sound convincing. Developing such speaking habits makes us reach an agreement in our arguments.

Of course, you will need your technical expertise in making decisions on a daily basis. But it’s not just about being technically sound, to be a great software engineer. It’s about those aspects of software development which are not taught in any university, the soft skills.

Baby Steps

The baby steps are about taking small steps to build the solution of a complex problem iteratively. Most of the time you don’t know prior to solving the problem what the solution is going to look like. In complex engineering areas, such as ours, most of the time we can’t say for sure for example how the service or app will look like in the end. There are many variables.

For example, the customer requirements are not clear initially or the scale is unknown or you are not sure about the approach etc. Baby steps help in these cases. In the core engineering Bootcamp, we develop the habit of taking baby steps in solving some problem because that way we will be able to solve some immediate problem which will, in turn, help you solve the bigger problem. This way you will do frequent commits. Baby steps should be visible on your commit history and not only on pen and paper. Test-driven development helps you take baby steps as you start seeing the bigger problem in smaller chunks of testable behaviours.

Being a team player

You are not going to work alone. You will always work in a team. We learn the principles of how to work in a team. In the Bootcamp, we also have a team of boot campers. We learn the process required to work effectively in a team. It introduces us to the most basic things in a team, for example, conventions, wiki, communication channel, stand up meetings, timings, objectives, limitations, approachability etc. What tools your team uses, what conventions you follow should be properly documented. The wiki should be the ultimate source of truth for every team member.

You won’t always write perfect code. No one writes perfect code. You will create technical debt and you cannot escape it. You take the decision that is best for the team and product. As an engineer, you have to decide what technical debts you can afford as a team. Some times you have to do the dirty work(e.g. not refactoring that code smell that you spotted) but you have to know how to repay as a team. By being a good team member is not only about being punctual or just doing your work. It is more than that. You should know what mistakes can you make and what mistakes you can afford. You need to be good at striking a balance between quality and delivering value.

All of this can be cultivated when you have a belief in the processes that your team follows.

Complex systems are such systems where output is not proportional to the input. Software is a complex system. The success of your product does not depend proportionally on the number of features you deliver. All of these above themes that I discussed above, help you chunk down the problem in smaller manageable batch sizes. In the hope that you deliver one feature correct and hopefully you will deliver the product right. These aspects help you make the correct software the correct way.

Thanks for reading. Feel free to connect with me on Twitter for any conversations on this.

Blockchain is revolutionary!

Sun, 11 Aug 2019 22:12:03 GMT

The problem with central ownership!

In today’s era, data is everything. Whoever owns the data has the ultimate power. The central authority to data has started bringing us all kinds of evils such as data theft, data manipulation and unauthorised usage. The centralized world is failing us on data privacy and control.

The big giants(such as Facebook, Google, Amazon, etc.) are controlling the data and using that data to target us with ads and recommendations. We have seen what are the drastic implications of data misuse because of such central authority. The recent such incident with the application FaceApp. In this article, the University of Sydney describes how Faceapp could be misuse user’s data.

How Blockchain can solve this!

I think blockchain can solve the problem of trust, security and ownership. At its heart, a blockchain is a record of transactions, like a traditional ledger. These transactions can be any movement of money, goods or secure data — a purchase at a supermarket, for example, or the assignment of a government ID number.

Blockchain is designed to store information in a way that makes it virtually impossible to add, remove or change data without being detected by other users. In the centralised internet, transactions are verified by a central authority. Blockchain applications could replace these centralized systems with decentralized ones, where verification comes from the consensus of multiple users. Blockchain usage algorithms based on cryptocurrency to solve these problems.

Many countries have started using blockchain-based application which provides peer to peer trust, safety, security, no single point of failure and no centralized authority over customer’s data. InstaDApp is a good example of how DApps are gaining popularity. Blockchain will be to businesses what the Internet was to communication. This is going to be the next big thing.

Thanks for reading. Feel free to connect with me on Twitter for any conversations on this.

REST vs HTTP

Sun, 11 Aug 2019 22:11:03 GMT

REST is a pretty common term that you will encounter now and then in web development. REST stands for Representational State Transfer. These are a set of core principles for architecting your software(mostly web services).

Initially introduced by Roy Fielding in his PhD dissertation, here I will try to explain in brief what those principles are, what it means to be RESTful, it’s different use cases and how HTTP comes into the picture.

Before getting into REST, we need to look at HTTP quickly. HTTP is an application layer data communication protocol on which(most of) the web works. This protocol defines how messages are formatted and transmitted and what actions web servers and browsers should take in response to various commands.

HTTP defines a set of methods to perform any action on a given resource. It has a pretty rich set of status codes which makes life way easier for a developer. There are various sophisticated features, also such as authentication, caching, cookies, etc.

So you see REST is a way of architecting your application and HTTP is the protocol which defines how your application(web services) should be built. One can think of RESTful web services as such services which follow the REST guidelines. HTTP is merely an instantiation of the REST guidelines. Your application built on top of HTTP might be perfectly RESTful, partially REST or kind of 60% REST.

It is very subjective how close is your software to being perfectly RESTful. Let’s see what should be the common characteristics in all RESTful applications.

URIs

Every resource(objects) must be uniquely identifiable by a URI. URI stands for Uniform Resource Identifier. Objects could be anything from multimedia objects(images, videos, gifs, etc.) to text files, database rows, etc. Anything that has merits to be identifiable gets a URI. There should be a single consistent naming of the resources. Here the developer should think what the right resources in your application are.

Linking URIs

These resources should contain links to other resources. Linking them is based on the fact that each has a unique ID. For example, you enter an address in your browser and get a web page(a resource) which links you to other resources. Representational resources link to each other.

Uniform Interface

REST says you should have the same set of methods for all resources(uniform interface). HTTP makes this specific by its methods GET, PUT, POST, DELETE, etc. So you see these methods define what action you could perform on the resources. REST defines these in a very general context. For example GET should be safe, idempotent and cacheable on all the resources. Similarly, there are semantics associated with other methods. It is possible that we could have more such methods, maybe in some other instantiation of REST. That point is REST doesn’t hold you back from having several methods on resources. But it should be consistent over the different type of resources.

Interacting with the resources

You interact with the resource through their representation only. For example if there is a customer resource identifiable by some URI. If you GET it in text/HTML format it should return you that format, or if you get it in application/XML format, that should also work. RESTful web services allow you to interact with the same resource in different representations.

Stateless communication

The server does not store any state about the client session. Let’s see it by an example of a shopping cart on an e-commerce website. You can add items to your cart, and you can delete items from your cart. A shopping cart can be implemented in many ways. In one implementation, the server could send a link to your shopping cart. Because it is a resource and identifiable by a URI and also it can be linked to other resources. In another implementation, the shopping cart could be an unnamed client-specific state(session state). So REST says you should go with the first one.

Why REST?

Well, you have other options also such as GraphQL. If you follow REST guidelines, then you can derive your application-specific interface from the REST specific interface. While doing this, the developer should take care of not violating the semantics associated with the HTTP methods.

Following REST lets you decouple your application also. Your client could be anything. It can be a browser, a terminal, another apache web server, proxy, etc. The client knows what result that its action is going to produce on the resources if the server is RESTful. For example, let’s say a client is searching for some resource on the server. If you have followed REST, then you have pretty robust GET on all the resources which return a readable representation of that resource. So that will let your client build fast and efficient search algorithms.

Thanks for reading. The SE-Radio Podcast with Stefan Tilkov inspires this blog. I tried to present the learnings in a brief, concise way. Feel free to connect with me on Twitter for any conversations on this.

Learnings in Gojek Bootcamp

Thu, 08 Aug 2019 22:12:03 GMT

This blog summarises what all things I have learned in the first week of GOJEK Bootcamp!

Communication is hard!

Until now, I was not even aware of this simple fact. In Bootcamp, we are learning how to speak clearly. The intent of the communication is that the person listening to you gets what you are trying to say. But different person perceives it differently. So it is your responsibility to make sure that you speak so rigidly yet simply that everyone in the room gets them exact same thing that you are trying to say. Speaking clearly and yet simplistically is hard. It comes with practice. You need to unlearn many things. You need to prepare your facts. You need to have a sound idea into your mind. Your arguments need to be logically consistent.

Never love your code!

This is also hard to do. But eventually, you will have to develop the habit of not loving your code. Because by doing so, you will lose the ability to evolve it. You will start taking the code reviews very personally.

A team has conventions!

We have learnt how a professional team functions. It functions on the basis of a structure backed by conventions and automation. The team decides the convention they will use in their projects. The team decides the tools that they will use. They write the conventions in the wiki and treat it as the source of truth. We boot campers are also a team. We are learning how to pick our conventions. How to logically argue and reach a conclusion whenever there is a conflict amongst teammates. To do that, you need to learn how to speak clearly and resolving conflicts without wasting yours and others time.

Any violation of conventions is not accepted. If code is violating any conventions it gets a rm -rf harshly. This thing also serves as a positive learning reinforcement. And also contributes to the fact that you mustn’t love your code.

To make sure you are not diverting from conventions, you use automation. Automation makes it easy to deal with rules in conventions. Otherwise, there will be inconsistency in the development process which will reflect in design, code and several other decisions.

Simple English Modelling

You should be able to express your intent in simple plain English. If you can do that, it will make it easier to make decisions quickly, argue rationally and will save everyone’s time. Most of the time we get diverted to other topics from the original topic in a discussion. Simple English modelling also help you not divert from the topic at hand.

So that’s the learning of week one. Most of this was centred around processes and communication in a professional setting. It is making more and more sense, as we go deeper into the Bootcamp.

Refactoring by Martin Fowler

Sun, 04 Aug 2019 22:15:03 GMT

Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure.

Why should you care about refactoring?

Design is a pretty sloppy heuristic. No design is final. You learn about your problem by actually solving it bit by bit. You improve the design as you get to know better about what you are trying to achieve. A poorly designed system is hard to change. Hard because it is hard to figure out where the changes are needed. If it is hard to figure out what to change, there is a strong chance that the programmer will make a mistake and introduce bugs. Refactoring will make sure your design is not decaying. It will make your program adaptable to upcoming changes.

When you find that yesterday’s decision doesn’t make sense today, you change the decision. Now you can do today’s work. Tomorrow, some of your understanding as of today will seem naive, so you’ll change that, too.

You refactor as you make more sense of your decisions or when the decisions are no longer applicable.

A word of caution

Before we start refactoring, we should have solid tests. Solid tests only can give you 100% assurance that you didn’t break anything unknowingly while refactoring. While refactoring you must not add any features. You should not add any tests unless you find you missed one earlier. Doing refactoring you only restructure your code. Refactoring mostly changes the interface, so you might have to change tests in order to cope with a change in the interface of the code.

When you are adding a feature to your program don’t think about refactoring. Don’t think about changing existing code. You should only write your functionalities and get tests to work.

Where can I refactor

Any place where you see a code smell, it is telling you to refactor. The phrase Code Smell was popularised by Kent Beck. Code smells are characteristic in the source code of a program that possibly indicates a deeper problem(broken window?)

So you identify the smells and fix them by the best practices of Refactoring. The author has dedicated seven chapters on what sort of smells are and what refactoring technique is best suited to handle them with proper reasoning. We will have a quick overview of smells and refactoring techniques.

Let’s Refactor!

Duplicated Code

This is the most common form of smell. You can apply Extract Method technique to make a separate method of the duplicated code. Here you take a clump of code and turns it into its own method. For example:

void printOwing(double amount) {
    printBanner();
    System.out.println ("name:" + _name);
    System.out.println ("amount" + amount);
}
void printBalance(double amount) {
    printBanner();
    System.out.println ("name:" + _name);
    System.out.println ("amount" + _balance - amount);
}

Applying the extract method technique to this would give us

void printOwing(double amount) {
    printBanner();
    printDetails(amount);
}
void printBalance(double amount) {
    printBanner();
    printDetails(_balance - amount);
}
void printDetails(double amount) {
    System.out.println ("name:" + _name);
    System.out.println ("amount" + amount);
}

There are many applications of the extract method technique. You can use it to make inline methods, replacing temporary variables with query methods.

Long Methods

A method is supposed to do one thing only. When you find a method that is doing more than one thing, it’s a smell. You need to decompose the method into several methods. You can use the above Extract Method technique or you can use Replace Temp With Query. Let’s see how Replace Temp with Query works.

double calculateTotal() {
  double basePrice = quantity * itemPrice;
  if (basePrice > 1000) {
    return basePrice * 0.95;
  }
  else {
    return basePrice * 0.98;
  }
}

Replacing the temporary variable basePrice with its query method with yield.

double calculateTotal() {
  if (basePrice() > 1000) {
    return basePrice() * 0.95;
  }
  else {
    return basePrice() * 0.98;
  }
}
double basePrice() {
  return quantity * itemPrice;
}

Large Class

When a class is trying to do too much, it often shows up as too many instance variables. When a class has too many instance variables, duplicated code cannot be far behind. We have seen how to deal with duplicated code. We can also split the class in two. Extract Class, Extract Subclass and Extract Interface are the refactoring techniques used when you encounter this type of smell.

Divergent change

Ever had a situation when a class has more than one reasons to change. This is a divergent change and a clear violation of SRP(Single Responsibility Principle). When you encounter this type of smell you need to split the class. The fundamental rule of thumb is to put things together that change together.

Shotgun Change

This is opposite of divergent change. When you make a kind of change(let’s say you changed your database from Postgres to MySQL) and you have to make a lot of little changes to a lot of different classes, then you have encountered this type of smell. Use Move Method, Move Field, Inline class techniques to refactor such broken windows.

Feature envy

Ever saw a method which calls so many methods from other classes. A method that seems more interested in a class other than the one it actually is in. Use Move Method, Extract Method, to place the method in the correct class.

Switch Statements

Switch statements on class types are a smell. Most times you see a switch statement you should consider polymorphism. Use Replace Type Code with Subclasses, Replace Type Code with State/Strategy or Replace Conditional with Polymorphism.

Middle Man

Methods that are only delegating tasks to other classes can be referred to as the middle man. Use remove Middle Man, Inline Method or Replace Delegation with Inheritance.

Data Class

These are classes that have fields, getting and setting methods for the fields, and nothing else. Such classes are dumb data holders and are almost certainly being manipulated in far too much. Data classes are like children. They are okay as a starting point, but to participate as a grownup object, they need to take some responsibility. Apply encapsulate field, encapsulate collection, remove setting method, move method, extract method.

There are a lot many refactoring examples in the book but I will conclude here. Thanks for reading. The book contains a lot more information on how to identify areas which need refactoring, how to refactor, what special care must be taken in particular techniques. Give it a read if you haven’t already.

Feel free to connect with me on Twitter for any conversations on this.

TDD by Kent Beck

Sun, 04 Aug 2019 22:14:03 GMT

Test-driven development (TDD) is a way of managing fear during programming.

To me, this is the best reason to use TDD. Fear makes you less certain which will clearly affect what sort of code you are going to write(production or test).

Fear makes you want to communicate less. What sort of effect would it have on your program when you have a constant fear and you can’t express your intent clearly.

Let’s get started with the introduction first!

TDD is a way of programming where you write tests first then you write the actual code. Let’s see this with an example.

You want to write a method which adds two numbers and returns the output. How will you write a code for this function?

Trick question. In TDD, you don’t think of code. You think of tests first. What set of tests when passed will demonstrate the presence of code we are confident will perform addition of two numbers. Once you have decided those test cases. You write the tests and run them. Obviously, they will fail(might not even compile) because the code is not there. Then you add the minimum amount of code required to make the tests pass. Once you do that you remove the duplication that you have introduced this way.

A TDD way of writing software looks kind of like this-

Add a little test
Run all tests and fail
Make a little change
Run the tests and succeed
Refactor to remove duplication

This is referred to as the Red-Green-Refactor cycle, the mantra of TDD!

To make the tests pass, you might have introduced duplication. Duplication is a symptom of a problem called dependency. Duplication most often takes the form of duplicate logic — the same conditional expression appearing in multiple places in the code. Objects are excellent for abstracting away the duplication of logic. By eliminating duplication before we go on to the next test, we maximize our chance of being able to get the next test running with one and only one change. We must get rid of the duplication between the test code and the working code.

When you are at RED, you quickly want to get to GREEN. How can you do that? Kent Beck tells 3 ways of getting to green quickly. The first two are fairly simple.

Fake It — return a constant from code and gradually replace constants with variables until you have the real code.
Obvious Implementation — type in real implementation.

When everything is going smoothly and you know what to type(Like the addition of two numbers), you put in obvious implementation after obvious implementation (running the tests all the time to ensure that what’s obvious to you is still obvious to the computer).

As soon as you to get an unexpected red bar(not everything is as obvious as an addition), you back up, shift to faking implementations and refactor to the right code. When your confidence is back, you go back to obvious implementations.

The third way of getting to green quickly is triangulation.

Triangulation — You Change certain important knowledge in the system and assert that the production code behaves in an accordingly expected manner. Use triangulation when you can’t see a way of eliminating duplication between code and tests. When you cannot think of what axes of variability are you trying to support in your design, make some of them vary and the answer may become clearer.

Doing clean code, you get the benefit of complete code coverage. You are confident about your design decisions. You move quickly. You don’t fear changes because you have tests that will tell you what you need to do next. You get the flexibility of easily refactoring your system because you have solid tests.

By practice, you learn to take small steps when you don’t know what is your code going to be. You take large steps when you know what your implementations are going to be. The size of your steps is dependent on the scope of the test.

TDD is very easy in many languages and testing framework such as Ruby with RSpec, even JAVA. With some langauges, TDD is tedious because of the whole Red-Green-Refactor cycle being large to handle in continuation. Experimentation also becomes tedious in such cases.

So that was a brief moment of ours with the book TDD by Kent Beck. Thanks for reading !!! Feel free to connect with me on Twitter for any conversations on this.

Clean code

Sun, 04 Aug 2019 22:13:03 GMT

Writing clean code is what you must do in order to call yourself a professional. There is no reasonable excuse for doing anything less than your best!

The best measurement of code quality is WTFs/minute!

So let’s get straight into actionable insights on how to reduce the above metric!

Meaningful Names

Naming is hard! Few things to take care while naming your variables and methods.

Use intention revealing names
Avoid disinformation
Make meaningful distinctions
Use pronounceable names
Use searchable names
Follow common conventions of the programming language you are using
Class names should be noun phrases
Method names should be verb phrases

Functions

Functions should be small. 4–5 lines is a very good metric. But it is very much specific to your choice of programming language. But the lesser, the better.
Functions should do one thing only.
There should be only one level of abstraction per function. For example, avoid doing tasks such as inserting employee object to a list and generating the employee payslip in a single function. Because those are a different level of abstractions. Also violates the doing one thing principle.
Avoid switch statements. Switch statements imply your function is doing more than one thing based on some condition. Think of a better way to express your thoughts(Polymorphism maybe?)
Use descriptive names
An optimal number of arguments to a function is zero. In the worst case, you can have one or two arguments. But more than two is a strict red alert. If a function has more than two arguments if possible make them a new class (if possible)!
Never use flag arguments. Flag arguments mean your function is doing two things. You need to split your function.
Don’t have side effects associated with calling your function. No hidden-changes. Clearly, do one thing!
Return value objects. Value objects are objects whose instance variables never change once they have been set in the constructor. Value objects biggest advantage is that they save you the trouble of nonorthogonality.
Prefer exception to return error codes. Use try-catch in place of if-else!
DRY — Don’t Repeat Yourself! Eliminate duplication at all cost.

Comments

Best comments are no comments — Comment show your inability to write clean code that expresses your intent.

Formatting

Indentation is most important.
There shouldn’t be any blank lines in between function. A blank line should mark a logical separation in methods, classes etc.
Instance variables should be placed at the starting of the class on top of methods.
The caller should be above the callee. (But language-specific!)
Focus on readability by giving appropriate vertical density.

Objects and Data Structure

What is the difference between Objects and Data Structure?

Objects hide their data behind abstractions and expose functions that operate on that data.
Objects do not expose data. Rather express their data in abstract terms.
Data Structures expose their data and have no meaningful functions.

Sometimes we need to use data structure, sometimes we need to objects and sometimes both. So we should be able to tell

With objects, we need to follow the Law of Demeter. Law of Demeter says that a method f of Class C should only call methods of

C
An object created by f
An object passed as an argument
An object held as an instance variable of C

Failing law of Demeter implies a function knows many things. Split that function into multiple functions. It could be split into several one line functions that delegate the task to other methods.

Error Handling

Use exceptions rather than error codes

Unit Test

TDD is encouraged to be followed.
Tests must be clean (Readability). Don’t treat tests as second class citizens. Your tests should also be written following all the clean code conventions. Test code is as important as production code.
Test coverage should be always high.
Try to have only one assertion per test. If not then try to minimize the number of assertions per test. There should be a single concept per test.

Follow the F.I.R.S.T acronym with unit tests.

F — Tests should run fast

I — Tests shouldn’t depend on each other

R — Tests should be repeatable in any environment

S — Tests should be self-validating (Test should have a boolean output)

T — Tests should be written Timely

Classes

Never have a public variable —this implies bad encapsulation. You should protect your classes privacy from the outer world.
Classes should be small. It should be smaller than that.
One class should have an only responsibility — SRP: Single Responsibility Principle. SRP means a class should have only one reason to change.
Class Name should describe what responsibility it fulfils.
A class should have high Cohesion. Maximum cohesion is achieved when each instance variable is used by each method.
When classes lose cohesion split them. It means SRP is being violated. Abstractions are getting mixed.

Emergence

Systems that are not testable are not verifiable if not verifiable then not deployable. RUN ALL THE TESTS
Refactoring is something which will help your system emerge. When refactoring you should remove duplication. When you refactor try to be more expressive with your intent. Impose SRP on your modules. Refactor them if SRP is being violated.

Few more tips!

Building your project(compilation, testing, packaging) should be easy. One should be able to build via a single command or a few commands only. It should bot involve various different checkpoints.
All test should run with one command.
Avoid multiple languages in one source file.
Avoid Surprises
Never overwrite Safety. This means to take care of your warnings. Simply suppressing your warnings could result in Chernobyl! A disaster.
Eliminate duplication. Follow DRY: Don’t Repeat yourself
Don’t Mix levels of Abstraction. Follow the law of Demeter.
Always remove dead code.
Always maintain consistency throughout your project.
Be as expressive as possible.
Don’t mix responsibilities.
Make logical dependencies physical.
Prefer Polymorphism to if-else, switch cases
Encapsulate conditionals, boundary conditions, edges cases

So that was a brief summary of writing clean code. Thanks for reading. Feel free to connect with me on Twitter for any conversations on this.

Pragmatic Programmer

Sun, 04 Aug 2019 22:12:03 GMT

A great book for anyone who wants to become a more effective and productive programmer! This book contains great tips on how to be one!

What is pragmatic, you may ask

Being pragmatic is all about being skilled in business, caring about your craft. Skill and craft come from experience. It comes from facing failures, learning things and not being afraid of them. Take this to the field of programming.

What makes a pragmatic programmer?

Pragmatic programmers are fast adopters. They are always inquisitive — asking lots and lots of questions. They are critical thinkers. They don’t operate on auto-pilot. They are the jack of all trades.

They take responsibilities. They don’t make lame excuses. When they make a mistake they accept it and provide options. They don’t play the blame game. They don’t live with broken windows(a bad design decision, poor code etc.)

They invest in their knowledge portfolio every day. They keep on diversifying the knowledge portfolio.

A Pragmatic approach

There are some ideas which apply to almost every type of software development. A set of ideas and processes which help you build better software.

Duplication

Duplication is the root cause of all evil. Duplicated logic is hard to deal with. It’s hard to refactor. It is a broken window. We fix all the broken windows. There could be various causes of duplication arising. It could be because of multiple representations of information, comments, language issues. It could have arisen from mistakes in design. Taking shortcuts because of lack of time mostly results in duplications. Remove duplications at all costs. Follow DRY: Don’t Repeat Yourself!

Orthogonality

Two or more things are orthogonal if a change in one doesn’t affect the other. We want to design components that are self-contained, independent and with a single well-defined purpose. You get a lot of benefits from orthogonality. By promoting orthogonality, you are promoting reusability, which in turn makes you productive. Orthogonality applies to code, to design and even teams. We should maximize orthogonality. Make your module easy to reuse.

Reversibility

One thing you should always keep in mind is that there are no final decisions. Make your software architecture flexible. Make it easy to accommodate changes. In an object-oriented way of programming, you can achieve this with low coupling, good encapsulation.

Tracer Bullets

Use tracer bullets to find you want to achieve. The tracer bullet method involves implementing a new application end-to-end to test every layer or component’s interaction. They are preferred to the labour of calculation. Using tracer bullets users get to something working very quickly. Developers build a structure to work in. You have an integration platform. You have something to demonstrate. You have a better feel for progress.

Prototype

Anything that you are not comfortable with could be prototyped. You can prototype architecture, new functionality, third party tools or UI design. Prototyping could be done in several ways. Prototyping is a learning experience. You build some disposable product, maybe using a totally different technology stack from the original product. You learn what you want to build using prototypes.

Basic Tools

Every craftsman needs a good set of tool to be productive and carry out the job effectively. For a programmer, these are some of the basic tools that you need to get comfortable with :

Up your shell game. Shell commands make you highly productive. Compare that to the GUI way of doing it and you will find out. Learn the shell commands and use them. The more you use them, the better you get at it. I have found stream editor(sed) magical. Exploiting the benefits of pipelines, redirection & composability, you can bring wonders through one single command.

Use a single editor well. Learn all the shortcuts and keymaps that you need to apply every moment. Vim vs VSCode should not be an argument. Whatever gets your job done and makes you productive, go for it.

Always use source code control. Git is the best(my opinion 😝 )!

Develop debugging psychology. Don’t panic when an error popups. While debugging, don’t assume something, rather prove it. This will save you from the trouble of blaming the system for a fault that is likely to be your own (select isn’t broken!).

Pragmatic Paranoia

Accept the fact that you can’t write perfect software. Because it doesn’t exist. No one writes perfect software. You should be convinced of the fact that you also don’t write perfect software. Pragmatic Programmers code in defences against their own mistakes. Here are some defensive measures that you can take.

Design by Contract

Bertrand Meyer came up with this elegant concept. Before a function starts, it may have some expectation of the state of the world, and it may be able to make a statement about the state of the world when it concludes. Its a contract between a function and any potential caller:

If all the routine’s preconditions are met by the caller, the routine shall guarantee that all postconditions and invariants will be true when it completes.

When you design by contract you be strict in what you will accept before you begin and promise as little as possible in return.

Crash Early

When you follow DBC(Design by Contract), it’s much easier to find and diagnose the problem by crashing early, at the site of the problem. You can easily found out where was the breach of contract. This way you will avoid any surprising results.

Assertive Programming

The idea is that “if it can’t happen, use assertions to ensure that it won’t.” Assertions check for things that should not happen. Assertions will save the day when you hit an error as silly as array being null, the string being empty, array not sorted etc.

Exception Handling

Use exceptions but don’t abuse your program in a messy unexceptional way. Use exceptions for exceptional problems. Something being exceptional depends on the context. A FileNotFound could qualify as an exception when you are expecting it should have been there. But when you are not sure whether the file should be there, raising an error seems an appropriate response.

Resource Handling

Many of the time we forget to close the file we opened to write. This could hit us in pretty serious problems when scaled. Always have a consistent plan to deal with resource allocation and deallocation.

Bend or Break

Your program should be flexible and adaptable in the face of an uncertain world. How can you achieve that?

Law of Demeter

Uncle Bob martin describes the law of Demeter as below:

A method f of Class C should only call methods of

C
An object created by f
An object passed as an argument
An object held as an instance variable of C

What happens when you don’t follow the law of Demeter?

You end up having lots of coupled classes. This will restrict your flexibility. Orthogonality will be compromised. So follow the law of Demeter. Organise your code into modules and limit the interaction between them.

Metaprogramming

The systems that we are trying to build should be highly configurable. Use metadata to describe configuration options for an application. Switching communication port should be better done via configuration than getting into the coding details. We should appreciate the values of metaprogramming.

Temporal coupling

There are various things in our day to day life which can be made concurrent. So are in software. Many of the tasks don’t depend on each other, so executing them concurrently can give a great deal of performance improvement. Temporal coupling is about analyzing workflow to find and improve concurrency.

While you are coding

Some advice on how you should code and how you should not!

Program by Coincidence

Ever faced the situation when you don’t know why the code is failing because you didn’t know why it worked in the first place. It seemed to work, given the limited “testing” that you did, but that was just a coincidence. I had many of these moments. So code deliberately. Proceed with a plan, always be aware of what you are doing and document your assumptions.

Estimating algorithm speed

Estimate the runtime of your algorithm. A Big-O analysis of your codebase would help you identify major areas of improvement in terms of computation.

Refactoring

Refactoring is a change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behaviour. We refactor to make sure our program is not rotting. You should refactor when you see a violation of the DRY principle. When you see nonorthogonality try to refactor to the best possible extent. Remove outdated knowledge from the codebase. Refactor early, refactor often.

Testing

A lot has been said and discussed on testing. You should write proper unit tests. Testing is more cultural than technical. It’s language and framework independent. Create a culture of testing.

Before the project

Most of the times when a project starts, requirements are not locked. Requirements are never fixed. They keep evolving as the customer and the developer get closer to understanding what they want to consume/build. Most of the times we need to dig for requirements by having one on one sessions with the consumer to identify the expectations rigidly. Promote abstractions. Talk in terms of abstractions because abstractions live longer than the implementations.

So that was a brief summary of what makes a pragmatic programmer. I have started to adopt these pieces of advice as I care more of my craft now than ever.

Thanks for reading !!! Feel free to connect with me on Twitter for any conversations on this.

Freya - The fondness for lambda architecture

Sat, 30 Mar 2019 22:12:03 GMT

How we went on building a product which can handle a huge amount of incoming data and process it online.

Hi Everyone,

I am here to talk about one of the projects that I did at my college. I took up this project during the final year of engineering at IIT J. This project is actually very different compared to other projects that I had done until then(which includes web development, NLP, Machine Learning etc.). We were a team of two. We had our fair share of difficulties and hurdles on the way. But that’s where the learnings come from 😃.

Problem Statement:

Handling inflow of huge amount of data and processing it online.

Let’s try to visualise the problem statement with a use case.

Consider an academic institution such as mine. When some student wants to perform data processing for her project on some huge amount of data, it is very tough to do it on her single computer/laptop due to the resource limit. We are talking data of terabytes and petabytes scale. So doing the processing on a single computer will take forever. We want to develop a solution for this problem. The commodity PCs at the computer laboratory can be used to distribute the task in hand. We want to develop a system so generic which will perform the desired job on the huge incoming data by processing it online. We also want it to be fault tolerant at all times. There are plenty of tools that do distribute computing but rarely there are such systems for this context.

So let’s say we have an incoming stream of images and the developer wants to run some classification algorithm over those images. The result of this classification should be written in a database for online queries. Running the desired job on each piece of data(here classification of images) linearly will take forever.

Can we distribute the classification task to more computers and aggregate the result when the job is done?

Lambda Architecture(LA) is one such solution. Nathan Marz described lambda architecture as a robust, scalable, generic, distributed system. Let’s talk about how LA handles this. Please bear with me. It will be worth it 😇.

What is Lambda Architecture

It all begins with data. Data of any type e.g. logs from a mobile application, location pings, images, audio, videos, banking transaction. The raw data is stored in the data centres. On top of that various operations are performed. LA makes two assumptions on the data. The assumption on this data is that it is online and it is immutable. The former online, here means that we are appending the new data at the end of existing data. The latter term immutable, means we are not changing any data at any point in time. Updates are replaced by new writes. I’ll justify these assumptions in a while.

Now you have queries on the data (the whole dataset). Literally running a function on the complete dataset has latency constraints as dataset size grows. For the time being let’s assume we are able to compute our function quickly on complete dataset. Talking about CAP theorem, partition tolerance is a necessity and not an option. So the tradeoff is between consistency and availability. Nathan Marz, proves in his work, that how mutable data and CAP theorem don’t play well together (the complexity of read-repairs). However, if we remove the updates by new writes there won’t be any divergent values at any point in time. So we will be able to achieve eventual consistency along with availability. CAP theorem has been beaten.

Talking about eventual consistency, we assumed that our function runs quickly on the entire dataset. This can be achieved by using a data system which can easily store large and constantly growing dataset and can compute the function in a scalable way.

To get results quicker, we can precompute the batch views and store in a database. This database can be indexed on the timestamp for quicker queries.

The only drawback that can arise here as the data grows is that the queries are out of date by a few hours. There will be eventual consistency. All that is left to do is to compensate for the last few hours of data. Nathan proposes a real-time system that computes the query on the last few hours of data for which batch view has not been calculated. This can be done quickly. So to finally answer a query on the whole dataset, we can merge the results from batch view and real-time view. The last few hours of data are again sent to batch layer for batch view precomputation. Let’s understand this approach with an example:

Let’s say you have 100 GB of images so far and the data is ever increasing. You compute you classification job on this whole dataset in let’s say ~5 hrs. You have written the result of this computation in a dataset indexed with the timestamp. Now say 10 MB of new images have arrived. Recomputing the classification job on 100GB+10MB will again take ~5 hrs. This will be really an inefficient way of handling the data and queries since you’ve precomputed the result on 100GB earlier.

How does Lambda Architecture work?

We can process the existing data in a batch processing framework(e.g Apache Hadoop), writing the result of the classification job in a database. So we have precomputed the batch view. The new incoming data can be processed inside a real-time processing framework(e.g. Apache Storm) and the result could again be written in a separate database. When a user queries about the job on the whole dataset, we can show results of the precomputed batch views aggregated with real-time views quickly. The new data can be sent again to the batch layer to be appended and job to be recomputed in the background on the new batch view. That way we have distributed the job among many computers and answering queries quickly.

Building Blocks of Freya

Freya extends this same idea in our context. You give it some computers with Ubuntu installed on them connected in a network. It will set up the batch layer and real-time layer along with a database gateway to query on the data. We used Apache Hadoop for the batch layer and Apache Storm for the real-time layer.

The batch layer of Freya is a master-slave Hadoop architecture. Master node runs the NameNode and the slave nodes act as DataNodes. Here is a diagram describing this architecture:

The real-time layer of Freya is also master-slave Storm architecture. Master node runs the Zookeeper server, Nimbus, UI and supervisor and the slave nodes act as supervisor nodes. Here is a diagram describing this architecture:

The database gateway is a Node.js app which exposes various API endpoints to query the databases for the result of the job performed on the whole dataset.

So overall we have an aggregation of these different components as the final product. Of course, it is not the best solution as it has its own limitations. For example, we have increased the number of point of failures. But given the optimization in terms of time that we are getting here is remarkable.

Merits and shortcomings of Freya

Some good things about Freya:

Since the input data is never modified, it makes huge map reduce jobs tractable. The jobs are easy to control and each stage in the job can be debugged easily.
The code keeps changing. The jobs keep changing. It may be because of bugs or requirement changes or other reasons. But Freya(since it is based on top of LA) is resilient to this change. Batch processing deals with this problem easily by re-computing.

There are some shortcomings as well.

The duplicative development effort in building hot(real-time) and cold(offline) paths of their processing pipeline. The additional overhead of reprocessing the data in the batch layer which were processed in the stream layer. The overhead of merging the results of both layer also counts as developer overhead.
Maintaining the code that produces the same result in every complex distributed framework. There are many points of failures.
Operational overhead is high for maintaining, running and debugging two systems simultaneously.

So why use Freya depends on the use case. In our case, lambda architecture is best-fit considering the academic use cases. So operational overhead is considerably low. Also, the nodes are physically located in one place. So it’s easy to debug and the infrastructure is less network error prone.

We have tried to achieve automation to the greatest extent possible. With Freya, your in-house LA will be up and running within minutes ✌️. Below is a demo.

Setting up Freya

The code for Freya is on Github. This project would not have been successful without the contributions of my project partner Rashi and guidance of our mentor Dr Sumit Kalra. I am thankful to them for their inputs. We are thinking of adding more features and bringing it to the usage of the common.

Thanks for reading !!! Feel free to connect with me on Twitter for any conversations.

JSON vs Protocol Buffer

Sat, 09 Jun 2018 22:12:03 GMT

Introduction:

Data is everywhere. Computer programs perform different operations on the data to make meaning out of them. There may be a need of data exchange among programs as well. For ex. say a mobile app(client) may ask data from an API endpoint(server) or a web server may query a database for data etc. This article is about the data exchange part.

So how can a program written in a language (say Ruby) exchange data with another program written in language (say JavaScript) over a network. We need some uniform format for data which all languages are compatible with, which all language can easily understand and parse efficiently in their data structures. We first encode the raw data in that format and serialize it, send it over the network or make it available to other programs over a network.

JSON

JSON is one such format. JSON stands for Javascript Object Notation. It’s a lightweight data interchange format. A JSON object is typically used to contain some data in key/value format. It looks like a string wrapped in curly braces with colons between the names and values, and commas between the values and names. See example below.

{ "id": "2", "name": "Abhishek", "message": "Read a book", "time": "2200" }

Here id, name, message and time are the keys with values 2, Abhishek, Read a book and 2200 respectively. It is analogous to hash(map). The order in which these keys appear don’t matter.

A JSON Array is a collection of JSON objects wrapped in [ ] separated by a comma. It is analogous to array. The order matters here. See example below.

[
  { "id": "2", "name": "Abhishek", "message": "Read a book", "time": "2200" },
  { "id": "3", "name": "Uneet", "message": "Order Lunch", "time": "1400" }
]

The JSON object and JSON arrays can be nested in each other as well. See example below.

{
  "room": "G75",
  "data": [
    { "id": "2", "name": "Abhishek" },
    { "id": "3", "name": "Uneet" }
  ]
}

JSON supports few basic data types.

Numbers
String
Boolean
Array(Ordered list)
Object (Unordered list)

Now we know how we can represent our data in JSON. We can encode our data in our program in JSON format. Some other program may ask for this data and decode it. Mostly all programming languages now-a-days have methods in their standard libraries which perform encoding and decoding of the data with just one line of code.

Protocol Buffer

Protocol Buffer is another such data-interchange format invented by Google. It is smaller, faster, and simpler than JSON and other data formats out there. JSON is a simple method to represent data. Protocol buffer introduces a schema in the data. For this we first define a (.proto) file. It contains how we want to structure our data. The first JSON object example that I stated earlier, can be given a structure in protocol buffer as given below.

message Todo {
optional int32 id = 1;
 required string name = 2;
 required string message = 3;
 required google.protobuf.Timestamp time = 4;
}

Note that how we can specify data types (int32, int64, string, timestamp etc), the data labels (optional, required) in our .proto file. Each field has a unique tag. The 1, 2, 3 and 4 in the RHS are the tags to fields. The numbered tags are used to match fields when serializing and deserializing the data.

This gives much more flexibility to extract meaning out of data in an easy manner as compared to JSON. This is a pretty simple example of a proto file. Protocol Buffer have support for namespaces, enumerations as well. After defining the structure, you run the protocol buffer compiler for your application’s language on your (.proto) file to generate data access classes. They will give you simple accessor function for each field.

JSON specifies the key(field) in every object which increases the data size. Protocol buffer uses the tags to identify fields. So it’s memory efficient than JSON.

A schema is very useful in maintaining the structure of data. It counters many inconsistencies that may get introduced if structure was not defined explicitly.

Protocol Buffers have several more advantages over JSON. You can add new fields without breaking anything. Old binaries simply ignore the new field when parsing. This is called backward compatibility.

I highly recommend you to see the code samples on the official documentation of Protocol Buffers. This makes concepts really clear via code samples and working examples in many languages.

Abhishek Sah — Blog

Executable Onboarding: From Docs to Agents

The Cost of a Real Local Test

The Plugin

Docker Gives State, the Plugin Gives Memory

What This Changed

The Broader Lesson

Future Improvements

Managing Infrastructure at Scale With Terraform

Why Not count

How for_each fixes this

Maps

Maps of Objects

Referencing for_each Resources

“Known Values” Constraint

Flattening Nested Data

Dynamic Blocks Inside for_each

Putting It Together: Multi-Environment EKS Node Groups

Summary

Go Contexts: What I Learned by Getting Things Wrong

1. Cancelling Work from the Outside

2. Cancellation Is Cooperative, Not Preemptive

The fix: pass the context down

3. Sequential Operations and Context in Kubernetes Operators

4. Parent-Child Context Chains and Cause Propagation

5. context.Background() Is the Root

6. Fire-and-Forget Goroutines and Context Lifetimes

7. Graceful Shutdown: Structuring the Context Hierarchy

8. Fan-Out: Cancelling the Stragglers

Concurrency Controls in Golang

Level 1: The Basic Pipeline for Stream Processing

Level 2: Scaling Up with Fan-Out and Fan-In

Level 3: Controlling Flow with a Rate Limiter

Level 4: Token Buckets and Handling Bursts

Final Thoughts and Best Practices

Terraform on CI - Part 2

Terraform Version

Cloud provider access

CI Workflow Design

Terraform Plan on Pull Requests

Terraform Apply

Best Practices and Learnings

Module structure

Handle Terraform state locking

Separate environments with workspaces or directories

Use variables for cross-account resource references

Security considerations

Multi Cloud Infra

Conclusion

Terraform on CI - Part 1

Typical Terraform Workflow

Going one step beyond

Deploying OTEL Collector as Daemonset

Setup

Service Discovery

Metric storage with Prometheus

Collector config

Node Exporter workload

Querying data

Debugging

cAdvisor high cardinality

Scraping cAdvisor

Metrics Cardinality

References:

Handling shutdowns, gracefully

Signals

Importance of handling signals

Golang example

Architecting with AWS VPC

VPC

Subnet

Route Tables

Architecting VPCs

AWS Networking in action

Siren - Alert management at scale

Introduction

Siren Alert Configuration

Siren Alert Routing Configuration

Google IAP auth automation for CLI apps

Automating Google IAP Auth for Golang applications

Why Not `count`

How `for_each` fixes this

Referencing `for_each` Resources

Dynamic Blocks Inside `for_each`

5. `context.Background()` Is the Root