RSS Feed | Mayank Raj

Idempotency Keys: Why Retrying Without Them Is Financial Suicide

Thu, 12 Feb 2026 18:30:00 GMT

You're standing at an ATM at 11:47 PM and you press "Withdraw $100." The screen freezes where the loading spinner spins... and spins... and spins. A minute later, the machine is silent. Did the transaction go through? Do you press the button again? Will you ever see your $100!? This is not a hypothetical. This is the Two Generals Problem, dressed in a bank lobby. And if you're building a payment API without understanding this then you're not building a financial system but you're building a slot machine where every retry pulls the lever on your customers' accounts.

Let me show you why idempotency keys are the only thing standing between "reliable payments" and "oops... we accidentally charged you three times."

The Network Is Your Enemy (And It Always Lies)

Here's the problem: when you make an HTTP request to charge a credit card, one of the three things can happen.

The request never reaches the server. Network jitter, DNS failure or a router hiccupped in Frankfurt. The payment never happened.
The server processes the request, but it's thumps-up aka 200 back never reaches you. The money changed db-entries, the charge did go through. But your client thinks it failed.
The server crashes mid-processing. The debit happened but the credit didn't. (Or vice versa, pick your nightmare.)

From your client's perspective, all three scenarios end up the same i.e. with timeout. The loading spinner never gets a signal to stop and you're left with a choice to either retry the request and risk a double-charge, or abandon it and risk the money vanishing into the ether.

This is the Two Generals Problem, a classic distributed computing paradox. Two generals need to coordinate an attack between them but they can only communicate via messengers who might get captured. General A sends a messenger: "Attack at noon." Did it reach? General B sends back: "Acknowledged." Did that arrive? No amount of back-and-forth acknowledgments can provide 100% certainty. We see this everywhere from HTTP protocol to DB commits. In our payment system, the client and server are the Generals. The network doesn't owe them reliability of any sort.

At-Least-Once Delivery

Modern systems don't just give up when a request times out. They retry, and if built rightly with some sort of backoff. Everything from Apache Kafka, the HTTP client library you use and even your mobile app retries. This is called at-least-once delivery, and it's the industry standard for one simple reason: losing a message is worse than duplicating it. But in some cases, especially in financial operations, duplication is not better than loss. It's categorically worse.

Let's say a customer clicks "Pay $100" on your checkout page. The request hits your server, charges their card, and updates the ledger, but before the server can respond with "200: Success," a network glitch eats up the acknowledgment. The client thinks the request failed. So it retries, maybe multiple times. Without idempotency, your backend sees three separate requests and rightfully so it charges the card three times. The customer's bank account: $300 lighter and all of a sudden the discount coupon makes no sense.

The Holy Grail of Exactly-Once Processing

You might be thinking: "Can't we just build exactly once delivery and avoid this whole mess?" No. It's mathematically impossible. The Two Generals Problem proves that you cannot guarantee delivery over an unreliable network. What is achievable, and what you must build is exactly-once processing. Here's the distinction:

Exactly-once delivery would mean the message arrives at the recipient exactly one time. In theory the network guarantees this, in reality it can't.
Exactly-once processing means the side effects of a message occur exactly once, even if the message itself is delivered multiple times.

This is the entire game. You accept the fact that the network will deliver duplicates at-least-once, but you make your server smart enough to recognize: "Looks like I've already processed this request. Here's the result I cached last time."

That intelligence is called idempotency. And the key to unlocking it is... well.. the idempotency key.

The Idempotency Key: A Deterministic Anchor in a Chaotic Sea

An idempotency key is a unique identifier the client generates before making a request. It's typically a UUID v4, attached as an HTTP header:

POST /api/payments
Idempotency-Key: 7f3e5c1a-9b2d-4f6e-8ae4-3f4e5f6a7b8c
Content-Type: application/json

{
  "amount": 10000,
  "currency": "USD",
  "user_id": "cust_12345"
}

When the server receives this request, it follows a specific protocol:

Check for existence. Query the idempotency store (database or cache): "Have I seen this key before?"
If yes, return the cached result. We don't re-execute the charge or touch the database.
If no, process the request. Execute the business logic, save the response in the idempotency store, and then return it to the client.

Now, when the client retries because it didn't receive the response (scenario #2 from earlier), the server recognizes the key. It skips the charge logic entirely. It returns the cached success response. The customer is charged once. The ledger is consistent. The idempotency key converts an unreliable at-least-once delivery into deterministic exactly-once processing.

Why the Client Generates the Key (Not the Server)

You might wonder: "Why doesn't the server just generate the key and return it to the client?"

Because that would reintroduce the problem at the key-exchange layer. If the server generates the key and sends it to the client, and that response gets lost, the client is blind to the key. It's as if it never existed. We've moved the problem around and not solved it. By having the client generate the key before the first attempt, the key becomes a deterministic anchor that survives network failures, app crashes, and even client-side restarts. The client can persist the key to local storage or a cookie. So if the app or webpage crashes mid-request, the next session can reload the key and retry safely.

Stripe does this. Airbnb does this. Every competent payment API does this. (And if your API doesn't, please stop reading and go implement it. I'll wait.)

Request Fingerprinting: Preventing Key Reuse Exploits

Here's a nasty edge case: what if a malicious (or buggy, but mostly malicious) client reuses an idempotency key from a $10 payment for a $1,000 payment? If your server just blindly returns the cached $10 success response for the $1,000 request, you just handed a "$990 off" coupon, and without any coupon code. The fix is to store a cryptographic hash of the request payload alongside the idempotency key.

import hashlib
import json

def compute_request_hash(method, uri, body):
    # Combine everything that makes this request unique
    payload = f"{method}:{uri}:{json.dumps(body, sort_keys=True)}"
    return hashlib.sha256(payload.encode()).hexdigest()  # Deterministic fingerprint

When a request with a known idempotency key arrives, compare the stored hash to the incoming request's hash:

If they match: Return the cached response.
If they don't match: Return HTTP 409 Conflict and force the client to use a new key.

This ensures that an idempotency key is uniquely and irrevocably bound to a specific set of parameters. You can retry the same charge safely but you cannot reuse the key for a different charge. This also means anything that can change with time, say current_timestamp cannot be part of this hash.

HTTP Status Codes: When to Cache, When to Retry

Not all errors should be cached under an idempotency key. Here's a common policy:

Status Code	Should Cache?	Client Action
2xx Success	✅ Yes	Done. Transaction succeeded.
400 Bad Request	✅ Yes	Invalid parameters. Fix data, use new key.
409 Conflict	❌ No	Idempotency key reused incorrectly. Use new key.
429 Rate Limit	❌ No	Retry with same key after backoff.
5xx Server Error	Usually ❌ No	Transient failure. Retry with same key unless your API explicitly persists failures.

Why cache 400s? Because if the client sent invalid data and retrying with the same key won't magically fix it. Caching the error prevents pointless retries and clarifies: "This request is fundamentally broken." So why not cache 5xx by default? Because the server might have crashed mid-transaction or a microservice was unavailable. A retry might succeed once the database recovers. That said, some APIs intentionally persist the first 5xx result to preserve strict idempotency semantics, so this part is a policy choice, not a law of physics.

Key Expiration

Storing idempotency keys forever is impractical and by design not required in the majority of cases. Your database would grow unbounded, and query performance would degrade. Most systems implement a time-to-live (TTL) for such keys. Say the key expires after 24 hours. This window is sufficient for any reasonable retry scenario. If a client retries after 24 hours then the key is gone, and the request is treated as new. Which is probably fine and if your retry loop is still running a day later, you have bigger problems to solve.

For high-stakes operations (like inter-bank transfers), you might extend the TTL to 7 days or store keys indefinitely in cold storage. The trade-off is between storage cost and the risk of re-executing a very old request.

Case Study: Airbnb's Orpheus Framework

Airbnb's transition to microservices required a centralized idempotency solution. They built Orpheus, a library that models every API request in three phases:

Pre-RPC Phase: Record the intent to perform an action. This happens in a database transaction alongside local state changes.
RPC Phase: Make the external call (e.g., to Stripe or Braintree). This is inherently non-atomic because the external service is a separate system.
Post-RPC Phase: Record the result of the external call. Update the idempotency record with the final response.

If a crash occurs during the Pre-RPC phase then the transaction rolls back and no side effects occurred. The client can retry safely. If a crash occurs during or after the RPC phase, the idempotency key allows recovery. On retry, the library checks the database and if it finds a recorded RPC result, it skips the external call and proceeds directly to the Post-RPC phase. This three-phase model ensures that guests are never double charged and hosts are always paid correctly, even when the network is actively conspiring against such holidays.

The Secret: Idempotency Is a Pinky Promise, Not a Feature

Idempotency keys aren't just a technical safeguard but they're a promise to your users that you understand the network is unreliable, and you've designed your system to be resilient anyway. When a customer clicks "Pay," they're trusting you with their money. They're trusting that your system is deterministic, even when the network is chaotic. They're trusting that you've done your homework. That trust is the foundation of every financial platform - Stripe, PayPal, Airbnb, Shopify, Amazon... you name it. These all use idempotency keys because they understand that in a distributed system, retrying without idempotency is not a technical risk but it's a betrayal of user trust.

That's not just good engineering. That's the only engineering that matters.

Why Your Cloud Bill Keeps Growing (And Why That Might Be Fine): The Jevons Paradox in Cloud Computing

Wed, 21 Jan 2026 18:30:00 GMT

You did it! You finally did it!! Months of planning, countless nights, POCs but your team migrated to serverless. You implemented auto-scaling, and optimized every last Lambda function. The unit costs deserve chef's kiss with each transaction now costing 42% less. You present the metrics to leadership, expecting applause and a hefty bonus.

Then the monthly bill arrives. It's 30%... higher than before, not lower - but higher. Let's look at why that's a even more beautiful metric.

Here's the thing about highways: adding more lanes doesn't reduce traffic congestion. You instead end up with even more traffic. Civil engineers call this phenomenon induced demand. The moment you make driving more convenient, more people drive. Those who were escaping rush hour traffic with train, now switch to cars. Suburban sprawl accelerates because the commute is now manageable. The new lanes fill up, and you're right back where you started, except now you're maintaining twice as much asphalt.

Highways are the cloud abstractions you built. Cars are the services running on them. Making it easy to run services increases the throughput, and with it the cost. Your cloud infrastructure works exactly the same way. Maybe less public commute is a bad thing for the city, but for you ? A higher bill could be a healthy indicator.

The Coal Question (160 Years Later)

In 1865, William Stanley Jevons published The Coal Question, a book that should be required reading for every engineering leader with a straightforward view on optimizing cloud spend. Jevons observed something counterintuitive: when James Watt's steam engine made coal power more efficient than Thomas's design, England's total coal consumption didn't decrease. It exploded. The efficiency didn't lead to conservation but it made coal so cost-effective and versatile that industries found thousands of new applications for it. Everything from blast furnaces to shipping to textile mills. The resource became cheaper per unit of work, so the economy consumed vastly more work.

This is now called the Jevons Paradox: improvements in efficiency lead to increases in total consumption. Today you're not burning coal but you're burning compute cycles. But the paradox? Still alive. Still expensive. Still smiling at your face.

The Efficiency Trap

Let me show you how this plays out in practice. A financial services company (let's call them "FinCorp") migrated from EC2 instances to a fully serverless architecture on AWS Lambda. The results were impressive:

Per-transaction cost: Down 42%
Infrastructure team time: Reduced by 60%
Response latency: Improved to P99 < 100ms

The CFO was thrilled. "If we're processing the same transactions at 42% lower unit cost, at least our compute bill should drop by nearly half, right?" Wrong. Here's what actually happened:

The Direct Rebound: Now that the transactions are cheaper, the product team launched a real time fraud detection service that was previously "too expensive" to run. Each transaction now triggers three additional Lambda invocations for risk scoring.
The Experimental Sprawl: Developers are no longer constrained by "the fixed pie" of EC2 capacity. They spun up 47 new microservices to test personalization features, A/B experiments, and ML model variations.
The Architectural Expansion: With auto-scaling removing the fear of capacity planning, the team added real-time dashboards. WebSocket connections for live updates, and a new mobile API that polls every 30 seconds are now in production.

While FinCorp's cloud bill instead increased by 100%, they're serving 400% more customers. On top of it three new revenue streams were launched. The CFO, once skeptical, now sees cloud spend as a growth indicator, not a cost center.

The Economics: Why Efficiency Breeds Consumption

To understand why this keeps happening time and again, we need to talk about price elasticity of demand. This is the relationship between the price of something and how much of it people consume. The formula is straightforward:

Elasticity = |Δ Quantity / Quantity| / |Δ Price / Price|

When demand is elastic (elasticity > 1), a drop in price causes a more than proportional increase in consumption. If your serverless migration cuts costs by 50%, but your usage increases by 150%, you've just experienced the Jevons Paradox.

Here's where it gets interesting: traditional IT infrastructure has had low elasticity. With physical servers, a price drop doesn't immediately allow you to buy more. You're constrained by procurement lead times, data center space, and the administrative overhead of racking hardware.

Infrastructure Model	Lead Time	Financial Model	Elasticity
Physical Servers	Weeks to Months	CapEx	Highly Inelastic
Colo/Rental	Days	OpEx	Modestly Inelastic
Public Cloud	Seconds	Pay-as-you-go	Elastic

The public cloud removed nearly all barrier to consuming more compute. Procurement, capacity planning is gone. There's no upfront capital. Just spin up another Lambda function and the AWS bill adjusts itself. This is why 83% of organizations report spending more on cloud services than anticipated, with an average overspend of 30%. The infrastructure isn't failing but it's working exactly as designed. You just unlocked infinite demand and means to fulfil it.

Architectural Sprawl: When Efficiency Enables Chaos

The Jevons Paradox doesn't just affect infrastructure but it also infects your architecture. In the old world of monolithic applications running on a somewhat fixed EC2 capacity, developers were constrained. You had 64GB of RAM and 16 vCPUs. If your code was inefficient, it impacted everyone else on the box. Scarcity bred discipline. And discipline called for sane choices upfront.

Serverless removed the constraint. Now, adding a new feature doesn't require a capacity planning meeting. It's just another Lambda. The marginal cost of experimentation approaches zero, so the organization experiments... a lot. This creates what we can call "architectural sprawl" - a silent but impactful explosion of microservices. Now there's one additional deployment pipeline, monitoring dashboard, and dependency graph.

Let's say you decompose your Django monolith into 72 microservices. Each service makes sense in isolation with clean boundaries, independent deployment, small team ownership. But the system as a whole became a distributed monolith with extra steps and with extra Lambda invocations.

A single user login now triggers:

1 auth service call
3 profile enrichment calls
2 analytics events
4 feature flag checks
1 A/B experiment assignment

That's ~11 Lambda invocations for what used to be a single database query. The cost per login, latency, overall load, they all went up. The blast radius is now bigger. But here's the uncomfortable truth: you're now shipping features 10x faster than before. The architectural sprawl is a feature, not a bug. Your business traded operational simplicity for velocity, and velocity won.

The Counter-Argument: Maybe the Paradox Has Limits

Before you despair, let's acknowledge the skeptics. Not everyone believes the Jevons Paradox applies unboundedly, especially now with the AI hype and cloud computing.

Here are the arguments for why consumption might eventually plateau at some point:

1. Market Saturation

LED lighting initially exhibited a rebound effect wherein people left lights on longer because they were cheap to run. But the market eventually saturated. You can only leave the lights on 24 hours a day. Is there a ceiling to how much AI generated slop content or real-time analytics a business can actually use?

2. Diminishing Returns

At some point, adding more microservices or running more ML experiments stops generating proportional business value. The ROI curve flattens and the capital allocation shifts to other investments.

3. Physical Constraints

Unlike 19th-century coal, AI compute requires specialized silicon (GPUs, TPUs, and whatever Groq has) and massive CapEx for data center construction. Supply chains have limits and so does Corporate budgets.

4. Regulatory Ceilings

Governments are waking up to the environmental cost of AI. We cannot keep milking the cow. If carbon taxes or energy caps become common, the paradox could hit a hard regulatory wall.

These are real constraints. The Jevons Paradox isn't a law of physics but it's an economic tendency. Whether it holds in the long term depends on how markets, technologies, and policies evolve. But for the next decade? I'll still bet on the paradox.

The Takeaway: Optimize for Value, Not Cost

Here's what all of us as leaders need to internalize:

Making your systems more efficient will not reduce your total cloud spending. It will make compute cheaper, which in turn will unlock new use cases, which will drive higher total consumption. This is not a failure of cost optimization but it's the natural outcome of lowering barriers to innovation.

Your job should not be just to minimize the AWS bill. Your job should be to ensure that every dollar spent generates maximum business value. That's a metric worth spending time on. This requires a shift in mindset:

Stop measuring absolute spend and start measuring cost per unit of value delivered (cost per transaction, cost per customer, cost per inference).
Embrace FinOps as a culture, not a project. Give engineering teams real-time visibility into their unit economics and empower them to make trade-offs.
Govern architectural sprawl. Efficiency enables innovation, but unchecked sprawl eventually leads to diminishing returns. Enforce principles around service granularity and dependency management.

The enterprises that thrive in the cloud era won't be the ones that spend the least. They'll be the ones that generate the most value from every Joule of energy and every millisecond of compute they consume.

The Jevons Paradox teaches us that efficiency isn't about using less. It's about doing more with what you have. The highway lanes will always fill up. The real question is: where are all those new drivers going?

The Stateless Lie: How AWS Lambda Finally Learned to Remember

Fri, 26 Dec 2025 18:30:00 GMT

Remember making long-distance phone calls in the 1990s? You'd have to plan the conversation like it's a military operation, rehearse your key points, and watch the clock with existential dread. Every minute costed real money. Being disconnected midway meant that you'd have to call back, re-explain the context, and hope you remembered where you left off. The anxiety wasn't about the technology failing but the architectural constraint that made natural conversations feel like a series of frantic sprints against time.

I remember discussing this exact constraint with a Principal Engineer from AWS during one of my interviews. We were talking about Lambda's 15-minute limit, and they were very clear: "We're smart enough to remove that limit, but there has to be a good reason behind it." Then they paused and added, "I'm sure needs will evolve ...and so will Lambda." At the time, I nodded like I understood the depth of what they meant. (well I didn't.)

For well over a decade, AWS Lambda has lived in that same 1990s phone call universe. Every function had a hard limit of 15 minutes to say what it needed to say, and if your business logic took longer ...well, you'd split it. You built external orchestrators, wrote "Lambda calling Lambda" chains. We had the whole of AWS Step Functions come in. We learned to express the application logic in Amazon States Language. Because nothing says "developer productivity" like maintaining separate infrastructure-as-code files for what should've been a simple while loop. We termed this "stateless computing" and treated it like some sort of immutable law of physics rather. In reality it was an architectural guardrail we'd outgrown.

At AWS re:Invent 2025, that guardrail came down.

The 15-Minute Wall Wasn't Technical ...It Was Philosophical

When Lambda launched in 2014, the 15-minute execution limit was presented as a feature, not a bug. We all brought it. Short-lived, atomic, idempotent functions were the doctrine. The reasoning made sense at the time: prevent runaway processes all while recycling the compute resources efficiently, and force developers to think in stateless, event-driven terms.

This was the era of microservices ascendant, where decoupling and independence were the answers to every architectural question. I caved in all well. The term "Serverless" was in my LinkedIn profile for years.

But real-world business processes are inconveniently continuous. Rarely does order fulfillment doesn't fit in 15 minutes, or that CSV processing. Multi-stage approval workflows laugh at arbitrary time limits. Chaining multiple Large Language Model calls together where each model might take 2-3 minutes and fail intermittently due to rate limiting. You'd checkpoint progress in DynamoDB, overengineer SQS queues for continuation signals, and write enough error-handling logic to make a Haskell programmer nod approvingly. (Okay, I take that back.)

The cognitive overhead was real. We called it the "context switching tax". While your business logic may have lived in code, but the flow of that logic lived in Step Functions configuration. Want to add a simple retry with exponential backoff? That's not a try/catch block anymore; that's a state machine definition update. The code and the coordination were architecturally divorced, and developers paid alimony in YAML ...or JSON.

AWS Lambda Durable Functions changes the game from ground up. A single function can now run for up to one year!!! All of this while preserving state across interruptions. The "pause" and "resume" is handled directly in the runtime. This isn't just a longer timeout but instead it's a fundamentally different programming model.

How Do You Pause a Function for Six Months Without Burning Money?

AWS came through with an elegant solution here. The checkpoint and replay mechanism. Unlike a traditional Lambda that executes linearly and discards its memory state when done, a durable function represents a complete lifecycle that can survive multiple physical invocations.

Here's the trick: When your function hits a "durable operation", say, a context.wait(Duration.days(180)) then the Lambda doesn't keep a warm compute instance idling. Instead, it:

Captures a checkpoint of the execution state (which variables hold what values, where in the code you are)
Terminates the compute environment (you stop paying for idle time)
Schedules a wake up event for 180 days later
Re-invokes the function when the wait expires

Now here's where it gets super interesting. When the function wakes back up, Lambda doesn't magically restore memory. Instead it replays the entire function from the beginning!! The SDK intercepts calls to durable operations like step() or wait() and lies to them. Instead of re-executing, it returns the cached results from the checkpoint store. The code speeds through its own history like a student cramming before an exam, catching up to where it was suspended, then continues with fresh compute.

Let's see what this looks like in practice:

import { DurableClient } from "@aws/durable-execution-sdk";

export const handler = DurableClient.handle(async (context) => {
  // Gets replayed every time, so better be quick about it
  const orderId = context.input.orderId;

  // Step 1: Call payment service (the SDK remembers this if we crash)
  const payment = await context.step("process-payment", async () => {
    // Only THIS step retries on failure, not the whole saga
    return await chargeCustomer(orderId);
  });

  // Step 2: Now we hibernate. No compute running. Just... waiting.
  await context.waitForCallback("warehouse-check");

  // Step 3: We wake up here when the warehouse finally responds
  await context.step("ship-order", async () => {
    return await scheduleShipment(orderId);
  });

  return { status: "completed", orderId };
});

That waitForCallback() is doing something magical. It generates a unique callback token, hibernates the function, and waits potentially for hours or days. This wait ends when an external system invokes a specific API with that token. When the warehouse system finally confirms availability, Lambda wakes the function up, replays it past the first two operations, and continues. No polling. No SQS queue management. No separate state machine tracking which step you're on.

The Determinism Tax: Your Code Runs Multiple Times, So It Better Get The Same Answer

This replay architecture introduces a constraint that will make some developers uncomfortable: your code must be deterministic. The function is re-executed multiple times for the same logical workflow, any logic outside of a durable step() must produce exactly identical results on every replay.

Consider the failure mode: You call Math.random() outside a step. On the first execution it generates "abc123". The function completes the step and checkpoints. Later when it replays, Math.random() generates "xyz789". The SDK is now panicking trying to reconcile checkpointed history with different data. The execution fails with a "replay divergence" error, and you're debugging distributed systems behavior at 2 AM. (Your on-call rotation thanks you.)

The fix: move non-deterministic operations inside durable steps. Wrap Math.random(), Date.now(), external API calls or anything that might change in context.step(). The result is checkpointed once, then cached for all subsequent replays.

Deterministic Hazard	Why It Breaks	Solution
`Math.random()`	Different value on every replay	Wrap in `context.step()`
`Date.now()`	Time keeps moving forward	Use `context.timestamp` or wrap in a step
Global variables	Might change between replays	Pass state through function arguments
External API calls	Network is a lie	Always wrap in `context.step()`
Iterating over `Map` or `Set`	Iteration order can vary by runtime	Use arrays or ensure stable ordering

This determinism requirement is the new tax of admission. You're getting a function that can pause for months and in exchange you're writing code with re-entrancy discipline. For teams used to stateless Lambdas where every invocation is a fresh slate, this is a mindset shift.

The Economic Pivot: When Durable Functions Are Cheaper Than Step Functions

Let's talk money, because architecture decisions that ignore cost are just expensive blog posts.

AWS Step Functions bill on state transitions at $25 per million transitions. Lambda Durable Functions use a different model: $8 per million durable operations plus standard Lambda compute cost. While you're waiting, you pay nothing for compute.

For a concrete example: Processing 10,000 items in a loop costs ~$0.75 with Step Functions (3 state transitions per item) versus ~$0.10 with Durable Functions (compute + durable operations). For high-frequency loops with minimal CPU overhead, Durable Functions can be ~8x cheaper. Step Functions charges you for every state transition while Durable Functions charges you for the durable checkpoints and actual compute consumed.

(Of course, if your workflow is 3 steps with a 6-month wait between each, both cost almost nothing. At that "scale", you're optimizing for developer sanity and not pennies.)

The AI Catalyst: Why This Matters Now

Timing of Durable Functions isn't just coincidental. The primary driver for this "stateless-to-stateful" shift is Generative AI and agentic workflows. AI-powered applications fundamentally break the 15-minute model in ways that traditional business logic never did.

LLM Chaining and Resilience

Chaining multiple Large Language Model calls together often takes several minutes per call and is subject to intermittent API failures or rate limiting. In a stateless Lambda, a failure at the third step of a five-step chain means restarting the entire process. This means wasting both time and token costs (which for frontier models are not trivial).

With Durable Functions, we checkpoint after each model call:

async def my_billion_dollar_funded_ai_research_agent(context: DurableContext):
    # Step 1: Generate questions (2-3 min, $0.15 in tokens we'd rather not waste)
    questions = await context.step("generate-questions",
        lambda: call_claude_opus(prompt="Generate 5 research questions"))

    # Step 2: Fan out to search APIs (they WILL rate limit us)
    results = await context.parallel([
        context.step(f"search-{i}", lambda q=q: search_api(q))
        for i, q in enumerate(questions)
    ])

    # Step 3: Synthesize (another 3-5 min, another $0.20 we don't want to repeat)
    synthesis = await context.step("synthesize",
        lambda: call_claude_opus(prompt=f"Synthesize: {results}"))

    return synthesis

If the synthesis step fails due to rate limiting, only that step retries. We don't re-run the expensive question generation or the search API calls. The previous results are cached in the checkpoint. For AI workflows where individual LLM calls cost dollars in API fees, this resilience model is a direct cost saver.

Human-in-the-Loop AI

Agentic AI often requires a "human-in-the-loop" to verify a high-stakes decision or provide domain expertise. Historically, this required complex external state machines to pause the workflow, send a notification, and resume hours or days later when the human responded.

With Durable Functions, the AI agent simply waits:

export const handler = DurableClient.handle(async (context) => {
  const analysis = await context.step("analyze-contract", async () => {
    // AI analyzes a legal contract, takes 5 minutes
    return await llm.analyze(context.input.contractText);
  });

  if (analysis.confidence < 0.85) {
    // Send notification to lawyer, wait for their review
    const reviewUrl = context.getCallbackUrl("legal-review");
    await sendEmail(lawyer, `Review needed: ${reviewUrl}`);

    // Function hibernates here (could be hours or days)
    const humanReview = await context.waitForCallback("legal-review");

    // Resume with human feedback
    return await context.step("final-decision", async () => {
      return await llm.finalizeDecision(analysis, humanReview);
    });
  }

  return analysis;
});

The lawyer clicks the link, provides feedback via a web form and the function resumes exactly where it left off. No Step Functions state machine tracking "which branch are we in." No DynamoDB table storing intermediate state. Just - plain - simple - code - that - works.

The Coin Flip: When Durable Functions Are The Wrong Choice

This wouldn't be a balanced architectural analysis without acknowledging where Durable Functions absolutely shouldn't be used.

Please don't over-engineer them for simple infrastructure orchestration. If your workflow is "write to DynamoDB, then invoke Lambda, then send SNS notification," Step Function's direct AWS service integrations are cleaner. Step Functions can do this entirely in a state machine definition and are built for this - now more so than ever.

Don't use them when non-technical stakeholders need to debug failures. The graphical Step Functions execution graph is irreplaceable for on-call engineers or support teams who didn't write the code. Durable Functions require logging, reading logs and understanding code structure. (Good luck explaining CloudWatch Insights queries to your VP of Operations.) If your workflows span multiple business units and require "management-friendly" observability, Step Function with their visual language justifies the cost premium.

Don't use them if your team isn't ready for the determinism discipline. The replay mechanism is elegant but can be unforgiving. If your developers aren't comfortable with re-entrancy, checkpointing semantics, and the constraints of deterministic code, the onboarding curve is steep. Organizations should pilot Durable Functions with a small tasks before migrating critical workflows.

Finally - Don't use them for workflows that are already well-served by Step Functions. If you have mature, battle-tested state machines that work - then let it work! The migration cost (rewriting ASL in code, testing replay behavior, training teams) might not justify the benefits. Durable Functions shine for new workflows or workflows where Step Functions was always an awkward fit.

Well ...Serverless Has Grown Up

The "stateless serverless" paradigm was never about technical necessity. It was an architectural training wheel. For a decade, we've been forced to fragment naturally continuous logic across multiple services all while accepting the context-switching tax. AWS Lambda Durable Functions represent the maturity of serverless computing. The programming model is now closer to how we think about business processes: as continuous flows with pauses, retries, and branches, not as disconnected atomic steps.

The 15-minute was never a constraint - it was a philosophy. And philosophies change.

The path forward is tactical:

Consolidate micro-orchestration. Migrate code heavy workflows from Step Functions to Durable Functions. If your state machine is 80% ASL and 20% Lambda then flip it.
Embrace AI agents as first-class workloads. Durable execution is the substrate for long-running AI workflows. Checkpoint after expensive LLM calls and use waitForCallback() for human-in-the-loop scenarios.
Mandate determinism audits in code review. Teams moving to Durable Functions need new habits. Are non-deterministic operations wrapped in steps? Is global state avoided? This intuition needs to be built.
Keep Step Functions for macro-orchestration. Don't throw out the visual state machine baby with the ASL bathwater. Use Step Functions for high-level infrastructure coordination and workflows that require non-technical observability. Use Durable Functions for application logic.

The future of serverless is durable. Not because AWS declared it, but because we finally have the tools to write code the way we actually think - continuously, resiliently, and without watching the clock. Let's use them....

Lambda Durable Functions vs Step Functions: When to Use Which Workflow Tool

Sat, 13 Dec 2025 18:30:00 GMT

We've all been there - assembling IKEA furniture and can't decide whether to follow the wordless pictorial instructions or just read the text description at the bottom? The pictures are great when you need to show someone else how you built it, or even to visually trace back when you forget a step. But sometimes you just want to say "attach bracket A to panel B with screws".

That's the exact tension between AWS Step Functions and Lambda Durable Functions. At surface they both orchestrate workflows. They both handle long-running processes. And yet, choosing between them isn't about "which one is better" but instead about matching the tool to how you think and how your team operates.

AWS didn't announce Durable Functions to deprecate Step Functions. They announced it because orchestration has two fundamentally different modes today: Macro-Orchestration (connecting separate systems, requiring visual oversight) and Micro-Orchestration (application logic that needs to live in code). Let's talk about when to use which.

The False Dilemma: "Will Durable Functions Replace Step Functions?"

No. And asking this question misses the point entirely. Here's the mental model: Step Functions are for infrastructure orchestration. So workflows that tie together disparate AWS services, need to be visible to non-technical stakeholders, and benefit from a graphical representation. Durable Functions instead are for application orchestration. That's the business logic that's intrinsic to your codebase, needs to be tested locally, and doesn't want to live in separate YAML files.

In my experience, the teams that struggle with this decision are the ones trying to force one tool to do the other's job. The arrival of Durable Functions doesn't obsolete Step Functions but instead it clarifies their roles.

When to Reach for Step Functions: The "Management-Friendly" Scenarios

1. Visual Observability for Non-Technical Stakeholders

IMO this is a big plus point. We may say we love logs, but GUI beats wall of text everytime. Step Functions' execution graph is a godsend at 3 AM for the on-call engineer who didn't write the code needs to see which state failed. The graphical Workflow Studio and execution timeline provide immediate insight. You literally can point at the red box and say, "This is where it broke."

For workflows that span multiple business units or require audit trails that executives actually look at, Step Functions' visual language is absolutely worth the cost. Explaining CloudWatch Logs mean dozens of Show & Tells, Runbooks for queries and what not. You don't want that. Trust me.

2. Low-Code AWS Service Integrations

Step Functions offers direct, SDK-less integrations with well over 200 AWS services. This is where the "managed service" shines. Need to write an item to DynamoDB, start an ECS task, or invoke another Lambda? Step Functions can do it without any code in a Lambda function. (No npm install, no dependency hell, no "works on my machine" debugging.)

No cold starts. No deployment packages. Just a state machine definition pointing at AWS APIs.

This is especially very powerful for infrastructure automation. Provisioning an RDS instance, waiting for it to be available, brewing coffee, running a migration script, updating Route 53, sipping that coffee - these are infrastructure tasks. They're not "business logic." They're system choreography and Step Functions being in the best moves possible. They provides a clear architectural boundary: the state machine is infrastructure, and it's managed as infrastructure.

3. Long-Running Workflows with Minimal Compute

If your workflow is "wait 7 days, send email, wait 14 days, send another email," you're not doing heavy computation between waits. Step Functions charges for state transitions, and those long waits are free. The $25 per million transitions makes sense here because you're not transitioning much and you're mostly just waiting.

When to Reach for Durable Functions: The "Developer-First" Scenarios

1. The Workflow Is Intrinsic to Your Application Code

Look here if your orchestration logic involves complex data manipulation, nested loops, conditional branching based on API responses, or reliance on third-party libraries. Expressing these in native Python or Node.js is exponentially more productive than writing hundreds of lines of Amazon States Language.

Let work with an example: Processing a CSV file where each row requires calling an external API and retrying on failure. Then aggregating results, and finally writing to multiple destinations based on the aggregated data. In Step Functions, that's a Choice state, a Map state, error handling with Retry and Catch, and probably a few Lambda functions anyway. In Durable Functions, it's a for loop with a try/catch and some context.step() calls. Which one do you want to maintain?

2. Local Debugging and Unit Testing

This is what makes me excited about Durable Functions. It's the "holy grail" of serverless development that Step Functions never delivered. With Durable Functions, we can:

Mock the DurableContext
Write standard Jest or Pytest tests
Step through the orchestration logic in our IDE WITH BREAKPOINTS !
Run the entire workflow locally without deploying to AWS - I repeat - Run and not Emulate

No more "deploy and pray" debugging where you push changes to AWS and watch the CloudWatch Logs to see if it worked. The feedback loop goes from minutes to seconds.

3. High-Frequency Loops and Data Processing

Remember the cost model: Step Functions charge $25 per million state transitions. Durable Functions charge $8 per million durable operations plus Lambda compute.

For a workflow that processes 10,000 items in a loop:

Step Functions: 3 state transitions per item (Task, Choice, Loop) = 30,000 transitions = $0.75
Durable Functions: 10,000 durable operations + minimal compute = ~$0.10

4. Your Team Lives in the IDE, Not the AWS Console

For teams that treat infrastructure-as-code as a necessary evil rather than a core workflow, Durable Functions fundamentally end up eliminating the context-switching tax. The orchestration logic and the application logic are in the same file, versioned together, reviewed together ...and break together. You don't have separate Terraform files for the state machine definition. You don't have to mentally map ASL states to Lambda function names.

The Hybrid Pattern: Using Both (Yes, Really)

Here's the thing nobody tells you: you can use both in the same architecture. In fact, you probably should and if you are not - you need to have strong reasons for it.

Use Step Functions for the macro-orchestration: "Provision infrastructure, deploy application, run smoke tests, update DNS." These are infrastructure tasks that benefit from the visual execution graph, native AWS integration. Less things can go wrong when you don't write another line of logic.

Use Durable Functions for the micro-orchestration within those steps: "Process customer data, chain multiple AI model calls, handle human-in-the-loop approval." These are application tasks that benefit from code flexibility. They are specific to your business logic, and you know the best way to handle them.

A Step Functions state machine can invoke a Durable Function. The Durable Function does its complex logic, checkpoints its progress, and returns a result. Step Functions continues with the next infrastructure task. You get the best of both worlds: visual infrastructure orchestration and flexible application logic.

The Takeaway: Two Tools, One Architecture

AWS didn't cave into the Durable Functions to create confusion. It well timed, and I think well thought of. We got it because orchestration has evolved into two very distinct modes today and we've been forcing one tool to do both jobs for a decade.

Step Functions excel at infrastructure orchestration that needs visual debugging and cross-team visibility. Keep using them for that. They're purpose-built for it.

Durable Functions excel at application logic that needs code flexibility and local testability. Start using them for that. They're purpose-built for it.

The maturity of serverless isn't about replacing old tools with new ones. It's about having the right tool for each job and also knowing when to reach for which. Let's build architectures that match how we think, not how we're forced to think by our tools.

The Library vs. The Warehouse: Why Your Database Can't Have It All

Mon, 17 Nov 2025 18:30:00 GMT

Your application just hit 10,000 writes per second, and your database is quite literally screaming. Looking at the write latency chart is like watching a heart attack ...while it's in progress! Your on-call engineer is panic texting, ALL CAPS, in the team chat. Someone just spoke out the obvious - "maybe we need to throw more RAM at it." You've been here before, many many times. Every engineering team eventually faces the same brutal question: do we optimize for reading data, or writing it?

We can't have both. Not really.

The Physics Problem (Or: Why Libraries and Warehouses Don't Work the Same Way)

Let's talk about two very different buildings. The first is a traditional library, the kind with the musty smell and the Dewey Decimal System. Every book has exactly one location, known neighbors and predictable rack. When a new book comes in, a librarian walks to a very precise shelf, shifts everything over (yes, physically moves books), and slots it into its ordained position. Want to get that book later? Instant. You look it up in the catalog, walk directly to QA76.76, and voila - there it is. The second building is an Amazon fulfillment center. Packages arrive by the truckload. Workers don't, or rather can't carefully organize them by the category which means that they throw them into the nearest available bin as fast as humanly possible. Need to find something? You end up having to scan barcodes, check multiple bins, and hope your "smart system" knows which bins might contain your item. But the receive rate? Absolutely insane.

B-Trees are the library while LSM Trees are the warehouse.

Every modern database is exactly one of these two buildings. Even worse, few act as the clever hybrid trying to be both, usually succeeding at neither. The fundamental constraint isn't in the software but the physics. Specifically, the physics of moving bits between RAM and disk.

The B-Tree: Pay Now, Read Fast Forever

B-Tree have dominated relational databases since the 1970s and for a good reason: they minimizes the number of times you have to ask the disk for data. Back then hard drives were physical entities with actual spinning platters and moving read/write heads. A random disk seek could take 10 milliseconds. That's couple of centuries in human timescale. In the time it takes to move a mechanical arm across a platter, a modern CPU could execute about 30 million instructions. We're looking at ~100x performance gap between sequential and random I/O on HDDs.

The B-Tree was designed for this mechanical sloth as per today's standards. It organizes data into fixed-size pages (~4-16 KB) and maintains a sorted tree structure where the internal nodes act as the navigation layer. The tree's fan-out is high wherein each internal node can point to hundreds of children. This enables databases with millions of records to remains only 3 or 4 levels deep. What you get in return is a point lookup requiring just a few disk reads, most of which are probably cached in memory anyway. Range scans are even better. Given that the leaf nodes are linked together, once you find your starting point, you just walk forward reading contiguous pages. The database is always organized, always sorted, always ready for your read query.

The Write Tax

Writes are where the librarian starts sweating. When you update a single 100 byte record in the B-Tree, you don't just write 100 bytes to disk. You instead write the whole of 16 KB page containing that record. That's 160x more data than you actually changed. It gets even worse. To ensure durability, every modification is written twice: once to a Write Ahead Log (WAL), and once to the B-Tree page itself. So now we're at 320x amplification for that poor 100-byte update.

And then there's fragmentation as B-Tree nodes are rarely 100% full. If they were then every insertion would trigger an expensive page split! In practice, a B-Tree under an active workload might have a 50-70% fill factor. That means 30-50% of your disk space is just... well... empty. Reserved for future insertions which may never come.

The B-Tree is the database equivalent of paying your taxes upfront. You may suffer during the write, but your reads never have to worry.

The LSM Tree: The Art of Organized Procrastination

In 1996 Patrick O'Neil (aka DB legend) and colleagues asked a dangerous question: What if we just... didn't organize data immediately? The Log-Structured Merge Tree, or LSM Tree was born from this very simple question. Sequential writes are 100x faster than random writes on HDDs. If we stop trying to update data in place and instead treat every write as an append operation, we can match the theoretical maximum throughput of the storage device, at least for the writing part.

Here's how it works. When your application writes a record:

Append to the Write-Ahead Log: The record is appended to a sequential log file on disk. This is extremely fast because we're just writing to the end of a file. There's no seeking, no shuffling, just raw sequential bandwidth.
Insert into the MemTable: The record is added to an in-memory sorted structure. This happens in RAM in nanoseconds.
Flush to Disk: When the MemTable fills up (say 64 MB), it's marked immutable and flushed to disk as a new Sorted String Table (SSTable). This flush is purely sequential and so at maximum speed.
Immutable writes: SSTable is strictly immutable. So If you update a record then you don't modify the old one but you write a new version. The old version sits there until a background process cleans it up later.

The throughput difference is staggering. LSM Trees like RocksDB can sustain 2-5x higher write rates than B-Trees. In Facebook's benchmarks, RocksDB regularly exceeded 100 MB/s ingestion while WiredTiger (a B-Tree engine) bottlenecked at 20 MB/s.

The Read Debt

You cannot escape tradeoffs. The warehouse model has a problem and it's in finding the stuff you stored. To read a single key in an LSM Tree, you have to check:

The active MemTable (in RAM)
Any immutable MemTables waiting to be flushed
All the Level 0 SSTables on disk with overlapping key ranges
One SSTable per level in the hierarchy

As you can imagine, this is slow. A single key lookup could require checking a dozen files across RAM and disk. So LSM Trees cheat in two ways:

Bloom Filters: Before reading an SSTable the DB engine checks a probabilistic data structure to see if the key is definitely not in this file? If the Bloom filter says no, then it skips the entire file or else it'll have to check. There are chances of false positive, but majority cases proceeds.

Sparse Indexes: Inside each SSTable, data is stored in sorted blocks. With this the system maintains a sparse index with pointers indicating the start and end keys for each block. This allows binary search within the file, narrowing down to a single 4 KB block instead of scanning the entire thing.

Even with these tricks, LSM's read latency is higher than B-Trees. The 95th percentile latency in production LSM systems can be 2-3x worse than B-Trees for point lookups.

The NVMe Twist

For decades the LSM Tree's dominance in write-heavy workloads was justified by the extreme cost of random writes on spinning disks. But NVMe SSDs have fundamentally changed the equations now. Modern NVMe drives don't have any mechanical heads. They use parallel flash channels which can handle thousands of random 4 KB writes per second at microsecond latency. The performance gap between random and sequential I/O has collapsed to as little as 2-3x. Now as the random writes are no longer prohibitively expensive, the value add to the LSM Tree's weakens.

Recent research on Bf-Trees shows write throughput 6x higher than traditional B-Trees and point lookups 2x faster than both B-Trees and LSM Trees! It's a next-generation B-Tree variant designed for SSDs. The future of database storage might not be LSM Trees and we might get back to the roots of B-Trees.

The Framework: Organize Now, or Organize Later?

You'll learn within years of working with databases - there is no perfect database. The space is as specialized as it gets. Each has its tradeoffs and is applicable for a given use case.

B-Trees organize at ingestion. Write pays the cost of finding the correct sorted position then updating pages in place, and finally maintaining the tree invariant. The benefit? Reads are orders of magnitude faster.

LSM Trees organize at compaction. Every write feels fast because you're just appending. The cost is deferred to background processes that merge, sort, and garbage-collect. The benefit? Extreme write throughput.

This isn't just a software problem but it's hardware and physics. The RUM conjecture (Read, Update, Memory overhead) states that you can optimize for two of these three dimensions, but not all three. B-Trees choose Read and Memory efficiency at the cost of Update performance. LSM Trees choose Update efficiency at the cost of Read complexity and Space overhead. The library and the warehouse both work. It's just that they are optimized for different realities. Understand the trade-offs well enough that when your database inevitably melts down at 3 AM, you'll know why and more importantly what lever to pull.

DynamoDB Single Table Design: A Brilliant Hack That's Probably Torturing Your Team

Tue, 21 Oct 2025 18:30:00 GMT

I'll be the brave one and say it out loud - Single Table Design in DynamoDB is premature optimization for 90% of applications.

Now before you close this tab in righteous indignation, just hear me out. I've seen this pattern implemented beautifully at hyper scale ...and I've also seen it turn a $200/month database into a $50,000/year developer productivity black hole. The difference isn't about the pattern itself. It's about whether your application actually needs it. And in most of the cases - it doesn't.

The Filing Cabinet Thought Experiment

Imagine you've just been hired at a company. Your first task: find Invoice #4521 from March.

In Office A, there are three filing cabinets: "Invoices," "Customer Records," and "Orders." You walk to the Invoices cabinet, flip to March, pull the file. Done. Your predecessor labeled everything clearly. Why? Because they weren't a psychopath.

In Office B, there's one gigantic filing cabinet. Everything is in there - labelled with postits - but all of it in there. Invoices, customer records, orders, important notes from 2019, someone's lunch receipt. Each file is labeled with a code: INVOICE#4521, CUSTOMER#789, ORDER#2024-03-15#001. The robot filing clerk who designed this system can retrieve any document in 3 milliseconds. But not you - you don't know the invoice number for the March recept, and so you have to look through each-and-every-file-one-by-one.

This is Single Table Design versus Multi-Table Design. And that 3-millisecond advantage? In my experience, it's rarely worth the tears.

A Brief History of a Brilliant Hack

This was a shocker for me - single Table Design wasn't born from best practices but was born from desperation.

In the early 2010s, DynamoDB was a far more constrained beast. It was and always been a specialised tool, for a specific job. Back then you got five Global Secondary Indexes (GSIs) per table. A hard limit of five. If you had six different query patterns, too bad. Spin up another table, provision separate capacity units, and watch your AWS bill as well as the operational overhead multiply.

A small group of architects at Amazon (led by Rick Houlihan) found a clever workaround. Instead of fighting the limitations, they embraced them. They created "Index Overloading": generic partition keys (PK) and sort keys (SK) that could store any entity type. A single GSI could now support users, orders, invoices, and product catalogs. The robot filing clerk was happy.

A bug was officially now a feature. This was genuinely brilliant. In 2012.

Here's what changed:

GSI limits increased from 5 to 20 to 25
On-Demand pricing eliminated the "table-per-entity" cost penalty
DynamoDB got faster, and networks got faster too

The original constraints that justified Single Table Design? Largely gone. There's no reason for the pattern to exist but it still does. It does not because it is still optimal, but because it has become dogma. Conference talks, blog posts, and certification exams continued to preach it as "the way." I'm literary adding to the debt here with this post!

The Trade-Off Nobody Talks About

Let's get mechanical for a moment. Single Table Design offers one undeniable advantage: pre-joined data retrieval.

By cramming a user details, their recent orders, their shipping addresses and what not into the same partition, you can fetch everything in a single query. One network round-trip instead of potential dozens. In theory, this is fantastic. The database is doing the work.

In practice, you've just shifted that work (and then some) to your application code.

Aspect	Single Table	Multi-Table
Query Speed	Marginally faster (single round-trip)	Slightly slower (parallel queries)
Code Complexity	High (maintain composite keys, prefixes)	Low (standard CRUD operations)
Debugging	Nightmare (generic `PK`, `SK` everywhere)	Straightforward (descriptive names)
Schema Changes	Super Expensive ETL migrations	Add a table, add an index
Analytics Readiness	Painful (everything's mixed)	Natural (one table = one concept)

The database is slightly more happier. Your developers - significantly less happy. And here's the thing about developer happiness: it directly correlates with how fast your team ships features and how often they introduce bugs.

The Cognitive Load Crisis

Forrest Brazeal once described a well-optimized Single Table Design as looking "more like machine code than a simple spreadsheet." He wasn't being hyperbolic.

Here's what debugging looks like in a Single Table:

PK                    SK                          Data
USER#123              METADATA#                   { "name": "Alice", "email": "..." }
USER#123              ORDER#2024-03-15#001        { "total": 59.99, ... }
USER#123              ORDER#2024-03-14#002        { "total": 124.50, ... }
USER#123              ADDRESS#HOME                { "street": "123 Main St", ... }
USER#123              ACTIVITYLOG#2024-03-15T10:  { "action": "login", ... }

Quick: find all the orders ...and filter by date. Now explain to the new hire why the sort key starts with ORDER# but the address one starts with ADDRESS#. Now explain the tilde hack (appending ~ to sort keys so parent items sort correctly)...

This is what I call ASCII sorcery: clever tricks that work beautifully for the database but create tribal knowledge dependencies. This is rarely documented, or even if it is - the documentation rarely keeps up with how the table grows. New team members can't be productive until they've absorbed months of context. Simple debugging sessions become archaeology expeditions. You cannot do a Distinct query to get all variation. Above all - every exporation is very very costly. Single tables are not self documented.

"Don't torture your developers for $5 a month."

That's not my quote. It's the emerging consensus from engineers who've lived through the Single Table experience.

The $5 vs. $5,000 Math

Let me make this concrete with some napkin math.

The Single Table pitch: By pooling all entities into one table, you share provisioned capacity. Economics of scale - less waste, better utilization, lower bill.

The reality: On-Demand pricing charges per request, not per table. Whether your reads hit one table or twenty, the per-RRU cost is identical.

But let's say you're on Provisioned Capacity, and Single Table genuinely saves you $100/month in capacity pooling efficiency. Sounds great, right?

Now consider:

40 hours of architect time designing the Single Table schema
20 hours of documentation (because nobody can understand it otherwise)
8 hours per bug debugging through cryptic partition keys
16 hours for every schema change requiring an ETL migration

At even $75/hour (conservative for senior engineers), you've spent $6,000 before you've saved your first dollar. Break-even is five years away. And that assumes no bugs, no churn, and no feature changes. (Spoiler: these assumptions have never been true for any project, ever.) You may have few blog posts, talks to speak of the design but the original goal remains a gaol for the future.

The Analytics Wall

Every application eventually needs to answer adhoc questions like:

"What was our year-over-year growth by product category?"
"Which users are most likely to churn?"
"What's the average order value by region?"
"Which user caused the spike in April?"

DynamoDB is extremely exceptional at transactional workloads ...but also equally poor for analytics. Asking DynamoDB to do analytics is like asking a sprinter to run a marathon: technically possible, deeply uncomfortable for everyone involved. When you inevitably need to stream your data into Redshift or a BI tool, Single Table Design becomes your worst enemy.

Remember that beautiful mixed table of users, orders, and addresses? Your analytics team now sees:

pk	sk	value (SUPER type)
USER#123	METADATA#	{"name": "Alice", ...}
USER#123	ORDER#2024-03-15#001	{"total": 59.99, ...}

They'll need to write PartiQL queries to "shred" this semi-structured data back into relational format. The performance gains you realized in your transactional database? You have to give it back, with interest, in the analytical layer.

Multi-Table Design, by contrast, maps naturally: Users table → users in Redshift. Orders table → orders in Redshift. Your analysts can write normal SQL on day one.

The Microservices Trap

Single Table Design and microservices are philosophical opposites pretending to be friends.

Microservices is built on each service having its own data. Independent deployment, independent scaling, clear separation boundaries. Single Table instead forces multiple services to share one table. A schema change for Service A can break Service B. A "noisy neighbor" service can end up throttling everyone. IAM permissions become a nightmare (good luck granting access to specific entity types in a shared table).

This is the Shared Database Anti-Pattern wearing a NoSQL costume. The whole point of separate services is reducing blast radius. A single shared table instead makes it a ticking clock.

When Single Table Actually Makes Sense (The 10%)

I'm not arguing Single Table Design is always wrong. In specific contexts, it's the right tool:

1. Hyper-scale core services: If you're processing millions of queries per second and operate at sub-millisecond latency which directly impacts revenue, the eliminated network round-trip matters. Think identity services, high-frequency trading systems, or core infrastructure at Netflix-scale.

2. Stable, immutable access patterns: URL shorteners, logging pipelines, or event stores where access patterns that are genuinely set in stone. If your queries haven't changed in three years and won't change in the next three, Single Table's rigidity is a feature to embrace.

3. Global Database: DynamoDB was one of the first truely globally available database I worked with. Regional database, across the globe, all synced with each other - it was magical. Understanding the consistency model was the key to making it work.

4. Expert teams with DynamoDB depth: If your team has shipped multiple Single Table implementations and genuinely understands the trade-offs, you're equipped to make an informed choice.

For everyone else (startups iterating on product-market fit, teams with normal DynamoDB experience, applications that will need analytics, systems where requirements change quarterly), Multi-Table Design is almost certainly the better bet.

The Pragmatic Path Forward

Whenever I hear DynamoDB, or see a Single Table design doc - the following goes through my mind:

Can we start with Multi-Table Design? One table per entity type, with descriptive attribute names. Standard GSIs for your query patterns. This gives us readable data in the AWS Console and straightforward debugging. At early stage, things would change, access patterns would be refined and data driven decissions need analytics layer with flexible querying.

Can we use parallel queries? DynamoDB is fast. But then two parallel queries at 15ms each often complete faster than developers expect — and the scaffolding is dramatically simpler. When you don't need to shave off milliseconds - don't try to.

I'd consider Single Table only when:

We've profiled the application and network latency is definately the bottleneck
The access patterns are stable and well-understood
The team has the expertise to maintain it, or we've experimented with non-critical use case first
The math actually works (savings > complexity cost)

Make use of tooling if you must. There are now libraries like ElectroDB, DynamoDB-Toolbox, and OneTable which can reduce Single Table complexity. They're good tools, genuinely. But they're also treating a symptom rather than questioning whether you need the disease.

The Bottom Line

Single Table Design was a brilliant architectural hack for a database service which had real limitations. Today those limitations have largely been removed. The pattern remains useful in specific large scale scenarios, but for the vast majority of applications - it sign of premature optimization of the most expensive kind: trading marginal database efficiency for massive developer inefficiency.

The most valuable resource in your organization isn't the RCU or the WCU but instead it's the focused, productive time of your engineering team. Go build for agility first. Optimize for database efficiency only when the profiler tells you to and not when a conference talk from 2018 implies you should.

Cell-Based Architecture: When Isolation Beats Integration

Wed, 10 Sep 2025 18:30:00 GMT

On June 30, 2021, Slack went dark. Not completely - that would have been easier to explain. Instead the platform entered a state we engineers dread more than a clean outage: the partial failure. Turns out that a single AWS Availability Zone was dropping packets. Just a few percentage points of loss. But in a distributed system where one user action fans out into hundreds of internal API calls, effect of these dropped packets propagated like poison through the bloodstream. Within minutes, users across all three availability zones couldn't send messages. The failure of one data center had somehow killed the entire platform.

The modern distributed systems are not like traditional applications wherein they crash cleanly when something breaks. They're more like submarines wherein the hull is divided into watertight compartments called bulkheads. So if one section floods, you seal it off and the rest of the vessel to survive. Without these compartments, a single breach sinks the ship. (famously Titanic had compartments, but not enough of them, and not sealed at the top. One breach became five, then fifteen...)

Traditional microservices architectures lack these bulkheads. I'll give you that you've split your monolith into dozens of services, each with its own codebase, it's own deployment pipeline etc. But scratch the surface and you'll find they all share the same database cluster, or maybe the same Redis cache, the same message queue, are in the same AZ ...you get the point. They're technically separate services living in a shared fate environment. When one service misbehaves, it sends a poison pill request, triggers a cascading retry storm, or simply experiences elevated latency. The blast radius is 100%.

Cell-based architecture is the antidote to shared fate.

The Blast Radius Problem

Let's get on the same page first. Blast radius is the scope of impact when something fails. In traditional thinking, we measured this spatially: how many servers went down, how many users were affected etc. Modern availability engineering adds a temporal dimension: how long did the damage last. Reducing an incident from one hour to one minute is the new and imo, the more accurate blast radius reduction, even if the same number of users were momentarily impacted.

Most distributed systems are designed to prevent failure. That's an unattainable goal. At any decent scale when you're serving millions of requests per second across dozens of microservices, failure is mathematically inevitable. Software bugs will slip through code review - no matter how many extra LLMS babysit PRs. Infrastructure will eventually experience "gray failures" where hardware degrades but doesn't cleanly die. A single tenant will send a request pattern that crashes your carefully load tested service.

The correct goal is containment. Not preventing the breach, but limiting how far the water spreads.

Cell-based architecture achieves this by partitioning your entire system and not just the code, but also the compute, storage, and all dependencies. This is broken into independent, self-sufficient units called "cells" (or "pods" or "shards" or "my-mini-app-1"). Each cell is a complete miniature replica of your entire stack. If you have 100 cells and one fails, only 1% of your users are affected. The failure is isolated and contained. It's now Mathematically predictable.

If your blast radius is 1/N where N is your cell count, reliability becomes a scaling problem, not an engineering breakthrough event.

When Shared Infrastructure Becomes Shared Fate

Here's what Slack learned the hard way. In a horizontally scaled microservices architecture, your service instances are broken down and distributed across multiple availability zones primarily for redundancy. The frontend services in Zone A can still talk to backend services in Zone B or Zone C. Databases are replicated and synced across zones. From a resource utilization perspective, this is optimal - every server can serve any request. You've maximized your capacity. Redundancy is on point as well.

But you've also created a perfectly connected graph. The failure spreads like wildfire When even one node in that graph starts zoning out just enough to cause timeouts but not enough to trigger circuit breakers cleanly. Your frontend in Zone A tries to call a backend in the degraded Zone B, times out, retries, and now that frontend is slow. Your load balancer keeps routing new requests to the slow frontend because it's technically still responding (just late). Users see errors. Your monitoring dashboards start lighting up like a Christmas tree. And the root cause? It's happening in a completely different availability zone than where the symptoms appear.

Contrast this with the cellular model. In Slack's post incident redesign, we implemented "AZ siloing." Traffic enters a specific availability zone and just stays there. Services in a Zone only talk to other services within the same Zone. The backend databases are all partitioned with each zone has its own dedicated slice of data. If Zone B starts dropping packets, the router simply stops sending traffic there. Users assigned to cells in Zone A and Zone C continue working normally. The blast radius is contained to one zone. In case case a zone failure can only ever affect 33% of the users.

The system is designed to lose a third of itself and keep running. That's not resilience through redundancy but it's resilience through isolation.

The Anatomy of a Cell

A cell is more than just a logical grouping as it acts like a hard isolation boundary. Let's break down the components to get a better picture fo it:

1. The Cell Router (The Thin Front Door)

Every request enters through a cell router which is a lightweight, stateless service whose only job is to answer one question: "Which cell should handle this request?" The router uses a partition key (tenant ID, user ID, resource ID, geographic region etc) to deterministically map the request to a specific cell. This mapping is cached locally at the router, so doesn't add to the latency in any significant manner.

The router must be dumb. If you start adding stuff like authentication logic, rate limiting, or complex business rules into the router, you've just created a new single point of failure. The router fails, every cell is unreachable. Game over.

2. The Cell Itself (The Bulkhead)

Inside the cell is a complete self contained deployment of your application. Remember that Duplication is the key here. All the microservices, all the databases, all the caches ...even the manually added "test IAM" roles. This rightfully feels wasteful at first especially from a pure cost perspective. But a database connection pool leak in Cell 7 cannot affect Cell 23. A cache stampede in Cell 42 doesn't cascade to Cell 38.

Cells should be designed to be unaware of each other. This means no cross cell API calls. No shared databases. No "just this one global Redis instance for session state." If two tenants in different cells need to communicate, that happens asynchronously via event buses or message queues but never ever synchronously on the hot path of a user request.

3. The Control Plane (The Orchestrator)

Cells don't deploy themselves. You need a control plane that manages the lifecycle of cells. It's very very important that every cell is identical to each other. This control plane includes:

Provisioning: Spinning up new cells as you grow
Health monitoring: Detecting when a cell is degraded
Deployment orchestration: Rolling out updates in waves (canary, blue-green etc)
Graceful evacuation: Draining traffic from a sick cell without dropping the in-flight requests

At Salesforce we have the Hyperforce platform, which uses a control plane that manages deployments across hundreds of cells globally for thousands of services. It's equipped with automated rollback, deployment strategies and what not. The control plane itself must be highly available, but it operates out-of-band from the data plane - it's not in the critical path of user requests.

Shuffle Sharding: Isolation on Steroids

Now that we have isolated cells - we can make better use of them. Standard cell based architecture gives you 1/N blast radius i.e. If you have 100 cells and one fails, 1% of your users are affected. But if you're operating a multi-tenant SaaS platform every tenant on the cell has equal power to poison pill every other tenant on that cell. Say they send a query that triggers a bug and crashes the services in their cell. With standard sharding, every tenant in that cell goes down together.

Shuffle sharding aims to solves this with combinatorics. Instead of assigning each tenant to a single cell, you assign them to a virtual shard composed of multiple randomly selected workers from the global fleet. Think of it like dealing a hand of cards from a shuffled deck—each tenant gets their own unique hand. The math is beautiful for this. In a cluster of 100 workers, if you assign each tenant a virtual shard of 5 workers, the number of unique combinations is:

$$ \binom{100}{5} = \frac{100!}{5!(100-5)!} = 75,287,520 $$

If one tenant crashes their 5 assigned workers, the probability that another tenant shares those exact same 5 workers is ...well ...effectively zero. Even if two tenants share one or two workers (which is statistically likely), each tenant retains 60-80% of their capacity albeit degraded, but still operational.

You want proof that this works? AWS uses this technique for Route 53. When you create a hosted zone, you're not assigned to a single cell. But instead you're assigned to a shuffle shard of 4 name servers out of a fleet of thousands. A targeted DDoS attack against your specific hosted zone can only affect the small subset of customers who also happen to share servers with you (and even then, only partially). It's isolation via statistical independence.

Real-World Implementations

Slack: AZ Siloing

After the June 2021 incident (blog article), at Slack we redesigned our architecture around availability zone boundaries. Each AZ became a cell and traffic is routed into an AZ only by an Envoy based layer. Services communicate strictly within their AZ. If an AZ becomes unhealthy, operators can "drain" it in under 5 minutes, then redirecting traffic at the edge while allowing the in-flight requests to complete gracefully.

The key innovation here is the whole zero external dependencies during evacuation. The draining mechanism cannot rely on any infrastructure in the AZ being drained as that AZ might be completely unreachable. The evacuation thus has to be driven from outside the failure boundary.

Salesforce: Hyperforce Cells

Salesforce's Hyperforce is perhaps the most ambitious cellular deployment in the industry. At first it seemed unecessary to me. But over the years as I saw one incidents after other and how they were all contained - the setup grew on me. Each cell is a software defined construct managing hundreds of thousands of customers. Cells are deployed across at least 3 availability zones, with immutable infrastructure and zero-trust security (every internal API call is authenticated, no implicit access).

Hyperforce processes over 100 billion requests per day. When we hit AWS's hard limit of 250,000 IP addresses per VPC (a constraint of Network Address Usage quotas), we ended up collaborating with AWS to implement prefix delegation. This ended up extending their capacity to 1 million IPs. The lesson: even with perfect cellular isolation, you can still hit scaling cliffs at the infrastructure layer.

AWS Physalia: Millions of Tiny Databases

The most extreme example of cell-based design is AWS Physalia, the configuration master for Elastic Block Store (EBS). Each EBS volume in an availability zone gets its own dedicated cell which is a 7-node Paxos cluster spread across different racks and power supplies. Physalia is not a one big database managing all EBS metadata; it's millions of tiny databases, each responsible for a single volume. The blast radius of a Physalia node failure is just one server rack. Not one AZ or one region, but just one rack. The grain of isolation is the individual resource making this cellular design taken to its logical extreme.

Building Your First Cell

If you're convinced this is the right pattern for your scale and risk profile, here's the high-level playbook:

1. Choose Your Partition Key

This is the most important decision you can make. The key must align with the grain of your service. The goal is to not have small number of very large partitions, and if you end up having one - there should be a good reason for it.

Tenant ID: For B2B SaaS (Slack, Salesforce) where customers expect isolation
User ID: For B2C apps where you want noisy neighbor mitigation
Resource ID: For storage services (S3, EBS) where data locality matters
Geography: For regulatory compliance (GDPR, data residency)

2. Design for Static Stability

Your cell router must work even if your control plane is completely down let alone degraded. Cache the routing map locally. Use DNS, not database lookups, for cell discovery if possible.

3. Deploy in Waves

Never roll out changes to all cells simultaneously. Use a canary cell with low-risk, low-traffic first. Monitor for say 30-60 minutes, and then deploy to 10% of cells, then 25%, then 100%. If anything looks off via error rates, latency, memory usage - HALT!. Pause the rollout and investigate. Automated rollback is non-negotiable.

4. Embrace Asynchronous Everything

Cross cell communication must be asynchronous. This is non-negotiable for isolation. Use event buses (like Kafka, Kinesis, EventBridge) for tenant-to-tenant interactions. Stream data to a global analytics warehouse for cross-cell reporting. Keep the hot path isolated across cells.

5. Test the Blast Radius

Someone, at some time, for some reason will break the rule. You just need to catch it in time and stop the practice from propagating. Run chaos experiments. Deliberately kill a cell in production (with approvals, obviously). Verify that only 1/N of users are affected and that evacuation happens automatically. If the failure spreads beyond the cell boundary, your isolation isn't real, and your next few weeks are blocked.

The Bottom Line

Cell-based architecture is not a best practice. It's a scaling strategy for when the cost of downtime exceeds the cost of operational complexity. If you're serving millions of users and a mere 5 minute outage ends up costing you six figures in revenue and customer trust, cells are your insurance policy. If you're a small team building an MVP, they're probably overkill (but keep the pattern in your back pocket for when you graduate). The trade-off is stark: you're duplicating infrastructure, increasing your monthly cloud bill, and adding layers of orchestration complexity. In return, you get mathematical predictability. When something fails - and it eventually for sure will - you know exactly how many users are affected: 1/N. That's my friend is not an estimate but it's a bare minimum guarantee.

Resilience at scale isn't about preventing failure. It's about making failure boring.

If you're aspiring to five-nines availability (99.999% uptime, which translates to just 5 minutes of downtime per year), cells aren't optional but foundation. Not because they prevent failure, but because they make failure survivable. And in a distributed system, that's the only guarantee worth having.

The Serverless Tax: Why Your LLM Wrapper is Bleeding You Dry

Mon, 07 Jul 2025 18:30:00 GMT

You shipped the shiny new "AI Powered" feature. Your users are happy. ChatGPT is answering questions, Claude is summarizing documents, and your product feels like magic - you can finally adda a new subscription tier. Then the AWS bill arrives and it's triple what you expected. You check the Lambda metrics and nothing looks unusual. The functions are fast, the memory usage is reasonable. But the charges keep climbing, and you can't figure out why.

Now imagine this - You hired a plumber to fix your clogged sink. They show up, do their checks and diagnose the issue in 90 seconds. They then sit in your kitchen for the next 45 minutes waiting for a specialty part to arrive from the warehouse. This whole time, their $200/hour meter is running. You're paying them to make small talks, sip water and scroll. This is what happens when you wrap an LLM in a Lambda function. The function does maybe 5 milliseconds of actual work of parsing the request, calling the API, formatting the response. Then it sits there, idle, while GPT-12.9-mini-high thinks for 20 seconds and streams back tokens at a leisurely 100 per second. AWS doesn't care that your function is waiting. The meter is running.

This for you - is the Serverless Tax.

The GB-Second Reckoning

Let's talk about how Lambda actually bills you as it reveals the entire problem. AWS charges you for two things: the number of requests (a flat $0.20 per million) and the duration of those requests, measured in gigabyte-seconds.

A gigabyte-second (GB-s) is a product of allocated memory of the function (in gigabytes) and how long it runs (in seconds). A function with 1 GB of memory running for 10 seconds consumes 10 GB-s. Simple enough. The current rate is about $0.0000166667 per GB-second for x86 instances (Arm Graviton2 is about 20% cheaper, but let's keep the math simple).

Nowhere in here did we talk about utilization. That's left on you. So if you use 1% or 90% of CPU - you're charged the same. CPU power is not a separate dimension. When you allocate memory to a Lambda function, you signal AWS to proportionally allocates vCPU. Want a full vCPU? Allocate 1,769 MB of memory. Want more compute? Crank the memory higher. This design made perfect sense in the era of compute heavy workloads like image processing, data transformation, video transcoding. More memory meant more CPU, and you paid for what you used.

But LLMs broke the model.

The 53x Problem

Let me show you a typical Retrieval-Augmented Generation (RAG) request. Your user asks a question. Your Lambda function:

Receives the HTTP request (2ms of CPU time)
Queries your vector database for relevant context (50ms)
Constructs the prompt with retrieved chunks (3ms)
Calls the OpenAI API with stream: true (0.5ms to fire the request)
Waits 4,800ms while GPT-4 generates 500 tokens and streams them back
Forwards the stream to the user (negligible CPU, it's just proxying bytes)

Total actual compute: ~55 milliseconds. Total billed duration: 4,855 milliseconds. You just paid for 88 times more time than you actually "used" the CPU.

(And this assumes a fast model. If you're using a slower model for complex reasoning, or Claude in extended "thinking" mode, you might be waiting 30 seconds or more. The ratio gets worse.)

Here's the math for a single request with a 1 GB function:

Billed duration: 4.855 seconds
Memory allocation: 1 GB
Cost: 4.855 × 0.0000166667 = $0.0000809 per request

That doesn't sound like much until you're handling 10 million requests per month. Then it's $809 in Lambda costs alone. The funny part is that the actual LLM tokens might cost you $300. Your wrapper function is more expensive than the AI.

When your infrastructure bill exceeds your API bill, the architecture is upside-down.

Response Streaming: A Beautiful Lie

AWS Lambda's streamifyResponse decorator was positioned to solve this problem. In theory instead of buffering the entire LLM response in memory (which could hit the 6 MB payload limit for traditional Lambda responses), you can stream the response back directly to the client. The user sees tokens appear in near real time.

Turns out that streaming doesn't reduce billed duration. The function is still well and alive, connected for the entire duration of the stream. If anything, streaming makes the problem worse, because now you're optimizing for user experience at the expense of infrastructure efficiency. The function can't terminate early. It has to sit there, forwarding bytes, until the last token arrives. The friction was removed which gave developers a cheat code to push the changes out. And we all know what happens with those "TODO: Replace with optimized...." comments.

Plus if your response exceeds 6 MB (maybe you're returning a long-form reasoning trace or a large document), you then start paying egress fees at the rate of $0.008 per GB beyond the first 6 MB. For most chat applications, this is irrelevant but for others dealing with images or files - it's a new cost.

(To be clear: streaming is the right architectural choice for UX in my opinion. But it doesn't fix the economic problem. It just makes it more palatable to the end user while making your CFO nervous.)

The Solutions - Good Old Web Sockets

Alright, enough doom. Let's talk about how to fix this. If your application is conversational (a chatbot, a coding assistant, a real-time agent), WebSockets flip the model entirely. Instead of the Lambda staying alive for the duration of the LLM call, you keep a persistent connection between the client and API Gateway. Lambda functions are invoked only when messages are sent or received.

The economics:

WebSocket connection time: $0.25 per million minutes
Lambda invocation: Only when there's actual work to do

A user might stay connected for 20 minutes but only send 5 messages. You're paying connection fees (pennies) and 5 Lambda invocations (also pennies). You're not paying for the 19 minutes of idle time between messages.

# Lambda is invoked ONLY on message receipt
def websocket_handler(event, context):
    connection_id = event['requestContext']['connectionId']
    user_message = json.loads(event['body'])['message']

    # Call LLM (async, with callback to API Gateway)
    llm_client.generate_streaming(
        prompt=user_message,
        callback=lambda token: send_to_websocket(connection_id, token)
    )

    return {"statusCode": 200}  # Lambda terminates; connection stays open w/t API Gateway

The streaming tokens get pushed to the client via the WebSocket, but the Lambda isn't sitting there playing the relay. It fires off the request and initiates a friendship. The connection is maintained by API Gateway, which costs almost nothing.

The Async-Native Future

The fundamental mismatch is that serverless was designed for a compute-heavy workloads whereas LLMs are I/O-heavy. It's the wrong tool for the job and that shows up in observability gaps, pricing, scaling etc.

We'll need serious retooling for these use cases. We've not had a demand for such I/O heavy jobs, at the scale we're currently seeing. Cloud platforms will need to bill for active CPU cycles, not wall-clock time. We're seeing early experiments and implementations on this already..

The secret: Serverless works beautifully when you're computing. It bleeds you dry when you're waiting. LLMs force you to wait.

The Matryoshka Architecture: Why Envelope Encryption Isn't About Security (It's About Economics)

Fri, 06 Jun 2025 18:30:00 GMT

Imagine a locked dropbox at the supermarket next to you. You've receive a letter but you don't get it right away. Instead of handing it over to you immediately, the staff hands you this funky looking key card. That key card doesn't open the dropbox directly but instead it grants you access to a vault at the far end of the counter. Here lies the real key to the dropbox. Once you retrieve that key, you walk back to the dropbox, unlock it, read your letter, and then immediately destroy the key. The key card remains safe in the vault, ready to issue new keys for the next letter.

This is envelope encryption in play ...plus with literal envelopes. And if your first reaction is "that sounds unnecessarily complicated," you're absolutely right. But here's the detail: when you're protecting at scale of petabytes, processing 1M transactions per second, and your cloud bill is approaching a few million dollars per month, unnecessary complexity becomes economic necessity. The goal of "encrypt everything" isn't theoretical anymore but instead it's a regulatory mandate enforced by PCI DSS, HIPAA, GDPR and your CEO's grandma. The question isn't whether to encrypt. It's how to encrypt. Doing that without bankrupting the company or grinding your database to a halt.

The Problem: When "Encrypt Everything" Meets Reality

Let's say you're an engineer at a SaaS company. Your CISO walks into a meeting and announces the new policy: all data at rest must be encrypted, effective immediately. You respond with - "EOD, Done". Sounds simple enough. You've got a master key already in AWS Key Management Service. You ask claude to create aPR that would encrypt every database record before it hits disk.

The PR goes to review, and you get your first review - "This is in critical path, let's test for regression and benchmark this". You run the first benchmark. Low and behold your transaction throughput drops by 80%. Your P99 latency balloons from 50ms to 2,000ms. Your AWS bill now includes a line item labeled "KMS API Calls" with a projected cost of $1,000,000 per month. What the hell just happened?

You've hit three fundamental failure modes of direct encryption at scale:

The Network Bottleneck: Your master key lives in a remote KMS as storing it on the application server itself would defeat the purpose of using a dedicated KMS. Every encryption operation requires a costly network round-trip, even if it's within region, or availability zone. At a speedy 300ms per call, encrypting 1,000 database records sequentially takes 300 seconds!!
The Cryptographic Wear-out: AES-GCM, the industry-standard encryption algorithm has a hard limit in it's maths: you can't use the same key for more than 2³² messages without risking IV collisions. In a high-throughput system, 2³² messages might occur in hours if not minutes.
The Blast Radius: A single master key protecting your entire database is a single point of failure. If that key is compromised through a memory exploit, an insider threat, or a simple misconfigured IAM policy then every byte of data you've ever encrypted is now exposed.

Direct encryption doesn't scale. Not because the cryptography is weak, but because of everything surrounding it. Theres the physics of networks and the mathematics of ciphers imposed constraints that no amount of hardware can overcome.

The Envelope Pattern: Keys Protecting Keys

Envelope encryption takes a stab at this problems by introducing a two-tiered hierarchy:

Data Encryption Key (DEK): A short-lived symmetric AES-256 key that is actually the one encrypting your data. This key is unique per object i.e. one key per database record, one key per file, one key per user session. It lives in plaintext only in memory, only for the duration of the encryption operation, and is immediately destroyed afterward.
Key Encryption Key (KEK): A long-lived master key that lives in the KMS and never leaves it in plaintext, with the sole job of encrypting (wrap) and decrypting (unwrap) DEKs. The KEK is stored in a Hardware Security Module (HSM) which are tamper-resistant cryptographic processor that will self-destruct its keys if someone tries to physically breach it.

This dance of encryption looks something like this...

Encryption (Wrapping the Envelope)

Generate: Your application asks the KMS for a new data key. The KMS generates a 256-bit AES key and returns two versions: a plaintext DEK (for immediate use) and a wrapped DEK (encrypted with the KEK).
Encrypt Locally: Your application uses the plaintext DEK to encrypt the data. This happens in-memory, at microsecond speeds, often using hardware-accelerated AES instructions (Intel AES-NI).
Bundle: The application stores the encrypted data (ciphertext) alongside the wrapped DEK. Think of it as a sealed envelope with the key taped to the outside but the key itself is locked in a safe.
Destroy: The application overwrites the plaintext DEK in memory. It's gone. If an attacker scrapes your application's memory five seconds later, they'll find nothing in that memory space.

Decryption (Opening the Envelope)

Retrieve: The application reads the ciphertext and the wrapped DEK alongside it from storage. Remember this DEK is specific to this ciphertext only.
Unwrap: The application sends the wrapped DEK to the KMS. The KMS verifies the requester's identity (say via IAM roles), decrypts the wrapped DEK using the KEK inside the HSM and then returns the plaintext DEK.
Decrypt Locally: The application uses the plaintext DEK to decrypt the data locally. This again happens locally at microsecond speeds.
Destroy: The plaintext DEK is purged from memory.

It's important to note the pattern that you only call the KMS once per session or per object and not once per byte. After unwrapping the DEK for that object, all the subsequent cryptographic operations happen locally. The 300ms network penalty is paid once, not a million times. You've also reduced your blast radius in the process, with DEK only responsible for a small number of objects.

The Three Wins: Performance, Security, and Economics

Win #1: Performance (The Network Bottleneck Solved)

By encrypting data locally with the DEK, envelope encryption eliminates any need of sending bulk data over the network for encryption. A 1 GB file encrypted with AES-256 on a modern CPU takes less than a second. The same file sent to a remote KMS for encryption would take minutes ...that is if it's not first blocked by KMS API's 4 KB payload limit. Benchmarks from PostgreSQL show that envelope encryption introduces ~5–15% overhead on transactional throughput. For Redis this overhead is less than ~3%.

Win #2: Security (The Blast Radius Contained)

Because each object has its own unique DEK, any compromise is localized. Let's say an attacker exploits a vulnerability in your application and manages to extract a DEK from memory. They've gained access to exactly one database record which is the one currently being processed. The KEK remains safely in the HSM, and the millions of other DEKs remain wrapped and unreadable. This is the "defense in depth" strategy applied to cryptography where a single breach doesn't cascade into a total system compromise.

Win #3: Economics (The Atlassian Story)

Managed KMS services charge by API call's. AWS KMS for example charges $0.03 per 10,000 requests. If you're calling the KMS for every database write in a high-traffic application then those pennies add up fast. Engineers at Atlassian calculated that their direct-encryption model would have cost approximately $1,000,000 per month in KMS fees alone. By implementing envelope encryption with a local cryptographic materials cache, they reduced this to $2,500 per month. That's a 400x cost reduction!

Let me repeat that: it's the same security posture, the same compliance checkboxes, but $997,500 less per month. Envelope encryption isn't just a technical pattern at this point but it's a business survival strategy.

The Operational Superpowers

Beyond the core performance and security wins, envelope encryption unlocks two operational capabilities that are impossible with direct encryption:

Superpower #1: Shallow Key Rotation

Regulatory frameworks like PCI DSS mandate that encryption keys be rotated periodically to protect from any breach's impact. In a direct encryption world, rotating the master key means decrypting and re-encrypting every byte of data. For a petabyte-scale database this is a multi-week high-risk operation.

With envelope encryption the key rotation is trivial. You rotate the KEK in the KMS and then you re-wrap the DEKs with the underlying data ciphertext never changing. Because DEKs are only often 32 bytes, re-wrapping millions of them can happen as a background process or lazily on the next access. No downtime. No bulk data migration. Just a metadata update for DEK mappings.

Superpower #2: Cryptographic Shredding

The GDPR's "Right to Erasure" mandate that organizations be able to prove data has been permanently deleted. In a distributed cloud environment it nearly impossible to ensure data is wiped from all backups, replicas, and logs is notoriously difficult.

Envelope encryption provides an elegant solution: delete the key and not the data. If you destroy the DEK (or the KEK that wraps it), the underlying ciphertext get demoted to garbage bytes. The data still exists on disk somewhere but it's indistinguishable from random noise. This for us is "cryptographic shredding," and it provides a tamper-proof audit trail for regulators.

The Hardware Root of Trust: Why HSMs Matter

The integrity of the entire envelope encryption system hinges on one assumption which is that the KEK is secure. If the KEK itself is compromised, the entire hierarchy collapses. This is why we generally use Hardware Security Module (HSM) for KEK's. An HSM is a dedicated cryptographic processor with actual physical tamper resistance. The KEK takes birth, lives and dies within the wall of HSMs i.e. it never exits the device in plaintext. All wrapping and unwrapping operations occur inside the HSM's hardened circuitry. If an attacker attempts to physically breach the device, the HSM detects the intrusion and automatically "zeroizes" its memory. This erases the keys before they can be extracted. For enterprise-grade systems or mission critical data, the HSM must be validated to FIPS 140-3 Level 3. It guarantees strong tamper resistance and identity-based authentication. This is the "root of trust" that makes the entire envelope pattern viable.

HSMs are fascinating and while at Salesforce I had the opportunity to work on the HSM layer that powered the whole of Salesforce ecosystem. I hope to write more about HSMs, and how critical they are to the whole "trust us with our data" claims someday.

When NOT to Use Envelope Encryption

Okay let's talk about the anti-pattern. Envelope encryption is not a one size fit's all, and introducing it prematurely can create unnecessary complexity.

Don't use envelope encryption if...

Your dataset is small (under 1 GB) and accessed infrequently
You're a startup with 10 users and no compliance requirements (you have bigger problems - users!)
Your application is CPU-bound, not I/O-bound (the overhead might matter more than the benefit)
You're using full-disk encryption (which already provides data-at-rest protection)

Envelope encryption shines at scale. If you're processing millions of transactions every day, storing sensitive data across the globe, or operating in a regulated industry, it's the only pattern that reconciles the competing demands of security, performance, and cost.

The Post-Quantum Future

In this series I've hyped up every cryptographic approach only to say - the cryptographic landscape is threatened by Quantum progress. A "cryptographically relevant quantum computer" (CRQC) would break the RSA and ECC algorithms which today protect many KEKs. This is the "harvest now, decrypt later" threat wherein the attackers storing encrypted data today with the intent to decrypt it once quantum computers are viable.

Envelope encryption provides some level of cryptographic agility. Because the KEK is separated from the DEK, we can upgrade KEKs to post-quantum cryptography (PQC) algorithms (eg lattice-based cryptography) without needing to re-encrypt their massive underlying data sets. As long as the AES-256 DEKs remain secure (symmetric encryption is largely quantum resistant with sufficient key lengths), the data remains protected.

This is the biggest draw of envelope encryption: it's not just about solving today's problems. It's about building a system that can evolve as the threat landscape changes.

The Reveal: It's Not About Security

Int he end, envelope encryption doesn't make your cryptography stronger. AES-256 is already unbreakable with todays tech. The cipher itself doesn't care whether you're using a single key or a million keys. What envelope encryption does is make encryption scalable, operable, and economically viable at the petabyte scale. It transforms encryption from a theoretical best practice into a production ready system that can sustain 100,000 transactions per second, all without melting your infrastructure or your budget.

The Matryoshka doll isn't just a clever metaphor (thank you fi you thought so). It's a fundamental architectural pattern that separates the concerns of data transformation (fast, local, ephemeral) from key governance (secure, centralized, long-lived). Envelope encryption is the reconciliation of cryptographic theory with operational reality. It's the difference between "encrypt everything" as a mandate and "encrypt everything" as a sustainable practice.

And to answer the question we started with - if you're an engineer at a SaaS company staring at a million dollar KMS bill, it's also the difference between a successful security program and a resume generating event.

The Math AWS Doesn't Want You to Do: Why S3 Intelligent Tiering Might Be Costing You More

Wed, 21 May 2025 18:30:00 GMT

You read the blog, validated the theory via ChatGPT and now you've enabled S3 Intelligent Tiering on your production bucket. The AWS marketing promised "automatic cost optimization" and "no operational overhead." Dream. Set it and forget it, they said. A month later, your bill arrives. You are excited to boast the % reduction in your next 1:1.

And ...It's higher than before.

Not by a little but by thousands of dollars. You check CloudWatch and verified that the data migrated correctly, you even open a support ticket. Everything is "working as designed." That's when you realize: S3 Intelligent Tiering isn't broken. You just didn't do the math right.

The Storage Unit That Charges Per Box

Imagine renting a storage unit for your stuff. The storage company offers you a deal: "We'll automatically move your boxes to cheaper warehouses when you don't access them! You'll save money!" Sounds perfect. Here's the catch: They charge you $5 per month for every box, to monitor which boxes you're accessing and then move them around. If you have 10 large boxes with expensive furniture, that's $50/month in monitoring fees to save $200/month in warehouse rent. Great deal.

But what if you have 1,000 small boxes? Tiny boxes. Say shoe boxes filled with old receipts, envelopes with birthday cards. Now you're paying $5,000/month in monitoring fees to save maybe $1,000/month in warehouse rent. The automation service costs magnitudes more than the storage it's supposed to optimize. You'd have been better off leaving everything in the expensive warehouse and canceling the "smart" service.

This is S3 Intelligent Tiering for small objects. And most people don't realize it until the bill arrives - very much guilty of this myself.

How S3 Intelligent Tiering Actually Works

Let's dismantle the mechanism so that it's easy to spot the patterns where it cracks...

S3 Intelligent Tiering is a storage class that monitors access patterns on individual objects and automatically moves them between tiers:

Frequent Access Tier (default): Same price as S3 Standard (~$0.023/GB). Millisecond latency.
Infrequent Access Tier (30 days idle): 40% cheaper (~$0.0125/GB). Still millisecond latency.
Archive Instant Access Tier (90 days idle): 68% cheaper. Still instant access.
Archive Access (optional, 90 days idle): 71% cheaper, but 3-5 hour retrieval.
Deep Archive Access (optional, 180 days idle): 95% cheaper, but 12 hour retrieval.

The idea by itself is elegant: Hot data stays fast and accessible and cold data gets cheap. If you suddenly try to access archived data, S3 will automatically promote it back to the Frequent Access tier (for free i.e no retrieval fee). You never have to think about tiers again.

The automation fairy handles everything while you sleep. ...Or does it?

The $0.0025 Tax You Didn't Notice

Here's the line item AWS doesn't bold in the pricing page:

Monitoring and automation charge: $0.0025 per 1,000 objects per month (for objects ≥128 KB).

Two and a half thousandths of a cent per object. That's... nothing, right? Who cares about fractions of a penny? Let's do the math...

Scenario 1: Large Number of Large Objects

You have 1 TB of data stored as 10 MB video files. That's roughly 100,000 objects.

Monitoring fee: 100,000 / 1,000 × $0.0025 = $0.25/month
Storage savings (moving to Infrequent Access): 1,000 GB × ($0.023 - $0.0125) = $10.50/month
Net savings: $10.50 - $0.25 = $10.25/month

Beautiful. This is what Intelligent Tiering was designed for and this is the use case AWS shows you in the case studies.

Scenario 2: Small number of Small Objects

You have 1 TB of data stored as 256 KB JSON logs. That's roughly 4,000,000 objects.

Monitoring fee: 4,000,000 / 1,000 × $0.0025 = $10.00/month
Storage savings (moving to Infrequent Access): 1,000 GB × ($0.023 - $0.0125) = $10.50/month
Net savings: $10.50 - $10.00 = $0.50/month

You're paying $10/month to save $10.50/month. Your ROI just evaporated. And this assumes 100% of your data reaches the Infrequent Access tier, which it won't (more on that later).

Scenario 3: Large number of Small Objects

You now have 19 TB of data stored as 300 KB files. That's roughly 66,000,000 objects. For logs, per-minute data dumps in any large scale, or high throughput scenarios, or even services wherein there's user submitted artefacts (eg images on Instagram) - this is very common.

Monitoring fee: 66,000,000 / 1,000 × $0.0025 = $165/month
Storage savings: 19,000 GB × ($0.025 - $0.0125) = $237.50/month
Net savings: $237.50 - $165.00 = $72.50/month

You just burned $165/month on monitoring fees just to save $237.50/month in storage. That's a 70% reduction in your expected savings. And we still haven't accounted for:

The transition fee to move the data into Intelligent Tiering ($660 one-time for 66M objects)
The request fees for your application's API calls
The KMS encryption overhead (if you're not using S3 Bucket Keys)
The metadata storage tax in archive tiers

In the real case, the bill was $2,000/month instead of the expected $400. The monitoring fee was just the beginning of the cost cascade.

The 128 KB Trap: Where Small Objects Go to Die

In 2021, AWS updated S3 Intelligent Tiering with a "cost-saving" feature: Objects smaller than 128 KB are exempt from the monitoring fee.

Sounds generous ...which is very unlike AWS! AWS is giving small objects a free pass, right?

Wrong. Here's what they don't tell you: If an object is under 128 KB, it's exempt from the monitoring fee because it's also exempt from tiering!!. These objects never move to cheaper tiers. They stay in the Frequent Access tier forever, billed at S3 Standard rates.

Let's say you have a dataset with:

50 million objects under 128 KB (microservice logs, metadata files)
10 million objects over 128 KB (some larger files)

You enable Intelligent Tiering expecting the whole dataset to benefit. What actually happens:

The 50M small objects stay in Frequent Access ($0.023/GB), paying S3 Standard prices forever
The 10M larger objects get monitored (costing you $25/month) and maybe move to cheaper tiers
You now have the complexity of a "managed storage class" with almost none of the benefits

The worst part is that you're under a false sense of security that your bucket needs no further work!! For high cardinality datasets with lots of tiny files, S3 Intelligent Tiering is strictly worse than S3 Standard. At best you get the same cost but with a more complicated storage architecture and even less predictability.

The Break-Even Formula (Do This Before You Enable)

Let's derive the actual math. When is S3 Intelligent Tiering profitable?

For sake of simplicity, let's say that the monitoring fee ($0.0025 per 1,000 objects) must be less than the storage savings (difference between S3 Standard and Infrequent Access tier):

Monitoring Cost < Storage Savings
(N / 1000) × $0.0025 < G × ($0.023 - $0.0125)

Where:
  N = Number of objects ≥128 KB
  G = Total storage in GB

Simplifying (and assuming 100% of data reaches Infrequent Access):

N × $0.0000025 < G × $0.0125
N / G < $0.0125 / $0.0000025
N / G < 5,000 objects per GB

Since N / G is the inverse of average object size in GB:

Average Object Size > 1 GB / 5,000
Average Object Size > 0.0002 GB
Average Object Size > ~204.8 KB

Translation: If your average object size is less than ~205 KB then the S3 Intelligent Tiering will cost more than S3 Standard, even if 100% of your data is cold. That's the best case scenario. In practice, not all data reaches the Infrequent tier (some objects get accessed before the 30-day threshold). So the real break-even point is closer to 300-500 KB average object size.

If you don't know your average object size, you're gambling with your cloud bill.

The Hidden Costs They Don't Mention

The monitoring fee is just the most visible tax. Here are the other costs that make S3 Intelligent Tiering a minefield.

1. The Transition Fee: The $3 Million Entry Toll

AWS charges $0.01 per 1,000 objects to transition data from S3 Standard to Intelligent Tiering via lifecycle policies.

For the Canva engineering team (which reportedly has 300 billion objects) transitioning to Intelligent Tiering would cost $3,000,000 upfront. Even for a modest bucket with 100 million objects of any size, that's a $1,000 entry fee. If your monthly savings is only $100, you'll need 10 months just to break even on the transition cost.

You can bypass this fee but only for future objects. You can upload new data directly to Intelligent Tiering (using x-amz-storage-class header). But this requires code changes, which contradicts the "zero operational overhead" promise. And it means all new data starts in the Frequent Access tier, accruing monitoring fees immediately.

2. The 40 KB Metadata Tax

When objects move to the Archive Access or Deep Archive Access tiers, AWS charges you for 40 KB of metadata overhead per object:

8 KB billed at S3 Standard rates (for object name and attributes)
32 KB billed at Glacier rates (for indexing)

For a 1 MB file, 40 KB is negligible (4% overhead). But for a 128 KB file, it's a 31% increase in billable storage. And if you're archiving billions of tiny objects? The metadata can cost more than the actual data.

Example: Archiving 1 billion objects of 1 KB each to Glacier Deep Archive.

Data storage: 1,024 GB × $0.00099/GB = $1.01/month
Metadata overhead: 40 KB × 1B objects = 38.1 TB × $0.00099/GB = $39.60/month

Well the metadata now costs 40× more than the data. This is why the "Average Object Size" metric isn't just important but it's existential.

3. The Promotion Reset Loop

When you access an object in the Infrequent or Archive tiers, it signals the Intelligent Tiering to automatically promotes it back to the Frequent Access tier. Free promotion, no retrieval fee! Sounds great ...but here's the problem: The 30-day idle clock resets.

If you have a compliance scanner, a backup verification tool, or an audit system that touches every object once every 29 days, your data will never make the move to a cheaper tier. You'll pay the monitoring fee every month, but 100% of your data will stay in Frequent Access (billed at S3 Standard rates). You're paying for automation that can't activate because your access pattern is just frequent enough to keep resetting the timer.

4. The "One-Way Door" Problem

Once you transition data into S3 Intelligent Tiering via a lifecycle policy, there's no automated way to move it back to S3 Standard. That is - If you discover six months later that the monitoring fees are eating your savings then you're stuck. Moving data out of Intelligent Tiering requires:

A manual operation (e.g., S3 Batch Operations)
Paying PUT request fees ($0.005 per 1,000 objects)
Dealing with potential data transfer costs
Rewriting lifecycle policies

This is the "set it and forget it" trap: You set it, you forget it, and then you're financially locked in.

When S3 Intelligent Tiering Actually Works

Let me be very clear about this - S3 Intelligent Tiering isn't a scam but it's just wildly misunderstood. There are workloads where it's genuinely the best choice and I almost always default to it. Just any other tool - you have to use the right one for the right job.

The Sweet Spot: Large Objects, Chaotic Access

I would consider Intelligent Tiering when:

Average object size > 500 KB (preferably > 1 MB)
Unpredictable access patterns (you genuinely don't know which files will be accessed when)
Long retention periods (data lives for months and years, not days or weeks)
High-frequency spot access
Can use SSE-KMS (Without S3 Bucket Keys enabled, AWS KMS fees will destroy you)

If my use case doesn't fit this - I would need some good convincing to look at Intelligent Tiering. Lifecycle rules may be the best option for transitions in such cases - ones that are tuned for the specific needs.

The Real "Set It and Forget It" Lie

The central issue with S3 Intelligent Tiering isn't technical but instead it's epistemological. AWS marketing materials presents it as a revolutionary "hands-off" solution, which trains engineers to treat it as a configuration checkbox rather than a proper decision requiring ongoing validation.

In reality, S3 cost optimization is not an event but it's an ongoing process:

Audit your object size distribution (~quarterly)
Model the costs (monitoring fee + metadata + transitions vs. storage savings)
Test on a single representative prefix before enabling globally
Monitor the actual costs vs. projections (~monthly)
Refine by disabling on prefixes where ROI is negative (~quarterly)

This is the opposite of "set it and forget it." It's "set it, measure it, course-correct, and continuously optimize." Build automations, add it to ceremonies and most importantly - document the reasoning so that the next person can follow through.

Takeaways

If you're walking away from this article thinking "S3 Intelligent Tiering is bad," you missed the point. Let me reframe:

S3 Intelligent Tiering is excellent for:

Large objects (≥1 MB average)
Unpredictable, bursty access patterns
Long-lived datasets with low object cardinality

S3 Intelligent Tiering is catastrophic for:

Small objects (<200 KB average)
Predictable "hot then cold" lifecycles
High-cardinality datasets (billions of files)

The tool isn't the problem. The "set it and forget it" framing is the problem. AWS gave you a sophisticated tool but they marketed it as a magic wand. The difference matters. And that could either make your next review cycle a breeze or a difficult conversation.

Insights in Plaintext: Elliptic Curve Cryptography (ECC) — The Efficient Enigma

Tue, 11 Mar 2025 18:30:00 GMT

Why modern privacy relies on playing billiards on a mathematical doughnut.

If you have worked in software long enough - you have interacted with an RSA certificate. You generate a 2048-bit key, paste it somewhere, and move on with your life. The math is complex, the keys are long. If you've been paying attention, you might have noticed GitHub defaulting away from RSA to smaller, faster Ed25519 keys for SSH authentication. Then you hear about ECC and wonder why anyone bothered with anything else.

Think of cryptography like transporting a secret across enemy territory. RSA's approach is to hide a needle in an enormous haystack—make the haystack big enough, and nobody can find your needle. ECC takes a completely different path: imagine a game of billiards where you hit a ball, it bounces around the table millions of times, and even if I tell you the start and end positions, you cannot figure out how many times it bounced. That one-way hardness is what now secures your WhatsApp messages, your Bitcoin wallet, and increasingly, every HTTPS connection you make.

In this deep dive, we strip away the algebraic geometry to reveal the simple mechanics underneath. We'll explore why "adding" points on a curve is the secret to smaller keys, how the "Pac-Man" effect of finite fields turns smooth curves into secure scatter plots, and why the NSA might have backdoored the very standards we use today (yes, really). No conspiracy theories here - I promise.

The Heavyweight vs. The Gymnast

In a previous installment, we dissected RSA, which in every sense the granddaddy of public-key cryptography. RSA's security rests on a very simple bet: multiplying two massive prime numbers is trivial, but factoring the result back into those primes is computationally brutal.

The problem? RSA is struggling to keep up.

To match the security of a 256-bit AES key (the gold standard for symmetric encryption), RSA would need a key that's over 15,360 bits long. That's like needing a dump truck to carry a single envelope. Your mobile device is shuffling around these monstrous numbers, burning CPU cycles and battery life - all to "securely" view the next reel. The math works. The efficiency doesn't.

Enter ECC, the shiny new gymnast to RSA's heavyweight.

ECC offers military-grade security with keys that fit on a sticky note. A 256-bit ECC key provides roughly the same security as that 15,360-bit RSA behemoth. All by abandoning the "needle in a haystack" approach entirely and exploiting a different mathematical labyrinth.

ECC doesn't make the haystack bigger but instead it makes the search algorithm itself useless.

Playing Billiards on a Curve

At the heart of ECC lies an elegant piece of mathematics: the elliptic curve. The name may fool you but these curves have nothing to do with ellipses. The name is a leftover from their use in calculating the arc length of ellipses centuries ago ...and mathematicians not stressing about naming conventions like you and me do.

The Equation

An elliptic curve is defined by the Weierstrass equation:

$$y^2 = x^3 + ax + b$$

Where $a$ and $b$ are constants that determine the curve's shape. There's one constraint: the curve must be "smooth" (no cusps or self-intersections), which requires $4a^3 + 27b^2 \neq 0$. Plot this equation, and you get a distinctive shape—imagine a sideways bell curve with a smooth hump. It's symmetric about the x-axis, which becomes crucial for our "game."

[DIAGRAM SUGGESTION: Show a smooth elliptic curve with labeled axes and the characteristic "hump" shape]

The Game: Point Addition

Let's look under the hood now. We'll start by defining a strange operation called "point addition". It lets us combine two points on the curve to get a third point.

The Rule: Draw a straight line through any two points ($P$ and $Q$) on an elliptic curve. That line will always intersect the curve at exactly one more point.

The Move:

Draw a line through points $P$ and $Q$
Find where the line hits the curve again (call this $R'$)
Reflect $R'$ across the x-axis to get $R$

We define: $P + Q = R$

That's it!! That's all that there is to elliptic curve "addition." It doesn't look like normal addition, and it shouldn't, as the curve is playing by its own rules here. The curve doesn't care about your intuition, It has its own physics.

[DIAGRAM SUGGESTION: Animated or step-by-step visual showing P + Q = R with the reflection]

Point Doubling: When P Meets P

What happens when $P$ and $Q$ are the same point? You can't draw a unique line through a single point... or can you?

When doubling a point, we use the tangent line at that point. The tangent touches the curve exactly at $P$. By the same rules it will intersect the curve at exactly one other point. Reflect that point, and you have $2P$. The curve is stubborn: even when you give it the same point twice, it still produces a meaningful answer.

This operation, point doubling is the key to making ECC computationally efficient.

The Trapdoor: Where One-Way Becomes No-Way

Now we can define the actual hard problem that secures ECC.

The Setup

Every ECC system starts with a Generator Point $G$. It is a specific point on the curve that everyone agrees on. Think of it as the starting position of our billiard ball.

The Easy Way: Scalar Multiplication

We can "multiply" a point by a number through repeated addition:

$2G = G + G$ (point doubling)
$3G = 2G + G$
$kG = G + G + G + ...$ ($k$ times)

Here's the beautiful part: we actually don't need to add $G$ to itself $k$ times. Using "Double and Add", we can compute $kG$ in roughly $\log_2(k)$ operations. This algorithm is greedy and it takes shortcuts wherever it can.

For example, to compute $100G$:

$100 = 64 + 32 + 4$ (in binary: 1100100)
Calculate $2G, 4G, 8G, 16G, 32G, 64G$ through successive doubling
Add the relevant results: $64G + 32G + 4G = 100G$

This, my friends was 100 additions in about 7 operations. Scale this up: we can compute $k \cdot G$ for astronomically large $k$ (think 256-bit numbers) in mere milliseconds. The CPU is happy doing this.

The Hard Way: The Discrete Logarithm Problem

Now flip the problem:

Given the starting point $G$ and the ending point $Q = kG$, find $k$.

This is the Elliptic Curve Discrete Logarithm Problem (ECDLP).

We know where the ball started and also where it landed. But how many times did it bounce?

With properly chosen curves and sufficiently large numbers, the best known algorithms to solve this problem would take longer than the age of the universe. The math is elegant. The computational wall is absolute.

Your Keys

This asymmetry gives us our key pair:

Private Key: The scalar $k$ (a random number you keep secret)
Public Key: The point $Q = kG$ (you share this with the world)

Computing $Q$ from $k$ is trivial—your phone does it in milliseconds. Computing $k$ from $Q$? The sun will burn out first.

Finite Fields: When Smooth Curves Become Chaos

There's a problem with everything we've discussed: computers hate infinity.

The smooth, continuous curves we've been visualizing have infinitely many points with infinite-precision decimal coordinates. Your CPU can't work with that (and neither can your RAM ...or your patience). We need integers.

The Pac-Man Effect

To get around this, we restrict our curve to be a finite field. Instead of working with all real numbers, we work with integers modulo a large prime number $p$.

Our curve equation then becomes: $$y^2 \equiv x^3 + ax + b \pmod{p}$$

Every calculation simply wraps around when it exceeds $p$.

Next up: imagine our billiard table has Pac-Man physics. When the ball reaches the right edge, it teleports to the left. When it goes off the top, it reappears at the bottom. Now imagine playing billiards on this table. After millions of bounces, could you trace the path backward just by knowing the start and end positions?

Nope.

The modular arithmetic creates this "teleportation" effect. Addition and multiplication still work, but the results wrap around unpredictably. The mathematical relationships that make the curve secure are preserved, but the visual patterns are obliterated.

The Transformation

Something strange happens when we apply this constraint:

The smooth, elegant curve vanishes.

In its place: a seemingly random cloud of scattered points. No pattern. No shape. Just dots sprayed across a grid. The curve hasn't changed mathematically.

[DIAGRAM SUGGESTION: Side-by-side comparison of smooth curve vs. finite field scatter plot]

This is the genius of finite field ECC: all the algebraic properties we need for cryptography survive, while the "structure" an attacker might exploit is annihilated. The math works perfectly but the pattern recognition fails completely.

The Efficiency Showdown: Why ECC Won

Let's talk numbers. Here's why the entire internet is migrating to ECC:

Security Level (bits)	RSA Key Size	ECC Key Size	Ratio
80	1,024	160	6.4x
112	2,048	224	9.1x
128	3,072	256	12x
192	7,680	384	20x
256	15,360	512	30x

The ratio gets worse for RSA as security requirements increase. The more security you need, the more ECC outperforms RSA.

The Real-World Impact

A 256-bit ECC key vs. a 3,072-bit RSA key (both providing 128-bit security):

Key Generation: ECC keys are generated ~10x faster
Signatures: ECDSA is ~10x faster than RSA signatures (equivalent security)
Bandwidth: ECC public keys are 12x smaller
Battery: Mobile devices consume significantly less power (your phone thanks you)

This is why TLS 1.3 defaults to ECDHE (Elliptic Curve Diffie-Hellman Ephemeral) for key exchange. It's enabled our phone to establish hundreds of secure connections without draining the battery. It's why Bitcoin can process thousands of transaction signatures every second.

The efficiency isn't marginal. It's transformational.

Implementation: Code You Can Touch

Theory is beautiful, but code is truth. Here's a simplified implementation of point addition on an elliptic curve:

class Point:
    """A point on an elliptic curve y² = x³ + ax + b (mod p)"""

    def __init__(self, x, y, curve):
        self.x = x
        self.y = y
        self.curve = curve  # Contains a, b, and p

    def __add__(self, other):
        # The identity element: point at infinity
        # (Yes, elliptic curves have their own version of zero)
        if self.x is None:
            return other
        if other.x is None:
            return self

        p = self.curve.p

        # When points are vertical opposites, they cancel out
        # P + (-P) = infinity (the curve shrugs and returns nothing)
        if self.x == other.x and self.y != other.y:
            return Point(None, None, self.curve)

        # Point doubling: P + P
        # The tangent line takes over when we're adding a point to itself
        if self.x == other.x and self.y == other.y:
            # slope = (3x² + a) / (2y) mod p
            # The curve leans into the tangent here
            numerator = (3 * self.x**2 + self.curve.a) % p
            denominator = (2 * self.y) % p
            slope = (numerator * mod_inverse(denominator, p)) % p
        else:
            # Standard point addition: P + Q
            # Just draw the line and find where it hits
            numerator = (other.y - self.y) % p
            denominator = (other.x - self.x) % p
            slope = (numerator * mod_inverse(denominator, p)) % p

        # Calculate the new point
        # This is where the reflection magic happens
        x3 = (slope**2 - self.x - other.x) % p
        y3 = (slope * (self.x - x3) - self.y) % p

        return Point(x3, y3, self.curve)


def mod_inverse(a, p):
    """Fermat's little theorem saves us from division headaches"""
    return pow(a, p - 2, p)


def scalar_multiply(k, point):
    """
    Double-and-add: the algorithm that makes ECC practical.
    We're computing k * Point without actually adding k times.
    """
    result = Point(None, None, point.curve)  # Start at infinity
    addend = point

    while k:
        if k & 1:  # If lowest bit is set, add current point
            result = result + addend
        addend = addend + addend  # Double (always)
        k >>= 1  # Shift right, on to the next bit

    return result

This code demonstrates the core mechanics:

Point addition with slope calculation (the geometry becomes algebra)
The modulo operator wrapping all values (Pac-Man physics)
Double-and-add for efficient scalar multiplication (the shortcut that makes it all practical)

Production Code

For anything real, use a library. Here's what that looks like:

from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.backends import default_backend

# Generate a key pair on P-256
# (The library handles all the scary math)
private_key = ec.generate_private_key(ec.SECP256R1(), default_backend())
public_key = private_key.public_key()

# For Curve25519 key exchange (the cool kids' choice)
from cryptography.hazmat.primitives.asymmetric.x25519 import X25519PrivateKey

private_key = X25519PrivateKey.generate()
public_key = private_key.public_key()
# Done. That's it. The library did the heavy lifting.

The Quantum Horizon: ECC's Expiration Date

Here's the uncomfortable truth that keeps cryptographers up at night: ECC has an expiration date on the box.

The Threat

Shor's Algorithm, discovered in 1994, can solve both the integer factorization problem (breaking RSA) and the discrete logarithm problem (breaking ECC). All in polynomial time on a quantum computer.

And here's the cruel irony: ECC's efficiency today that make it compelling becomes its downfall. Because ECC uses smaller keys, a quantum computer needs fewer qubits to crack it.

Algorithm	Key Size	Qubits Needed (estimate)
RSA-2048	2,048	~4,000
ECC-256	256	~2,330

The Timeline

Current quantum computers have roughly 1,000-1,500 noisy qubits. We'll need millions of error corrected qubits to run Shor's Algorithm. By estimates we're 10-20 years away from cryptographically relevant quantum computers but only time will tell. We're also a bit distracted with all the LLM stuff today.

But here's the catch: "harvest now, decrypt later."

Adversaries can record encrypted communications today and decrypt them once quantum computers arrive. If you're protecting secrets that need to stay secret for decades, say state secrets, medical records, intellectual property then the quantum threat is already here. Your secrets are sitting in someone's storage, waiting for the future.

The Future: Post-Quantum Cryptography

The cryptographic community is already building the next generation of defenses: Post-Quantum Cryptography (PQC).

NIST finalized its first post-quantum standards in 2024:

ML-KEM (Kyber): For key encapsulation
ML-DSA (Dilithium): For digital signatures
SLH-DSA (SPHINCS+): Hash-based signatures

These algorithms are based on mathematical problems that even quantum computers can't solve efficiently like lattice problems, hash functions etc. The trade-off? Larger keys and signatures. We're going back to the dump truck era, at least partially.

When NOT to Use ECC

Long-term secrets requiring 30+ year confidentiality: Consider hybrid schemes (ECC + post-quantum)
Systems that can't be updated post-deployment: Embedded devices, satellites, IoT with 20+ year lifespans
Environments where timing attacks are feasible and you can't use Curve25519: Some older curves require constant-time implementations that are hard to get right

The Takeaway

Elliptic Curve Cryptography represents the pinnacle of classical cryptography. It's the most elegant balance of security and efficiency we've ever achieved. It transformed mobile security, enabled cryptocurrency, and quietly protects billions of daily transactions.

ECC's core insight is in the geometric game of bouncing points on a curve, constrained by modular arithmetic, creates a one-way function so powerful that all the computers on Earth couldn't reverse it.

Having said that, nothing lasts forever. As the quantum era approaches, and with it, new mathematical foundations will fundamentally replace the curves we've come to trust. But for the next decade or two, ECC remains the workhorse of internet security.

Aurora DSQL vs DynamoDB MRSC: The Global Database Duel

Tue, 04 Feb 2025 18:30:00 GMT

Scene: You're sitting in an architecture review. The CTO leans forward and asks the question that'll decide the next quarter's roadmap: "Can we guarantee that a user in Tokyo sees the exact same inventory count as a user in Frankfurt, instantly, without either of them waiting 200 milliseconds for the other region to phone home?"

The room goes quiet. Because for two decades, the honest answer was: "Pick two: global, consistent, or fast."

Well ! Not anymore!!

The City Hall Problem

Let me paint you a picture. Imagine that you're running a global operation with two very different coordination challenges:

Challenge One: The City Hall Records Office. You have birth certificates, property deeds, marriage licenses, business permits and what not. Every document references other documents. A marriage license references two birth certificates. A business permit references a property deed, which references an owner, who has a marriage license, which... you get the idea, right?

Now zoom out. When someone in the London office updates a property deed, the clerk in New York needs to see that change immediately. Why? Because they're about to issue a business permit that depends on it. The system is structured, cross-referenced, and deeply relational. One wrong reference and the entire legal framework collapses.

Challenge Two: The FedEx Tracking Network. You have millions of packages flying around the planet. Each package has a tracking number, a destination, a weight, and maybe some custom attributes (fragile, perishable, "definitely not a birthday present"). Packages don't really reference each other.

When you look up tracking number 111222333444555, you're only interested in that one package's status. You don't care about complex joins across manifests and route tables. You care about speed, scale, and the flexibility to slap new attributes on packages without redesigning the entire tracking database schema.

This is the duel between Amazon Aurora DSQL and DynamoDB Multi-Region Strong Consistency (MRSC).

City Hall vs. FedEx. Relational rigor vs. NoSQL agility. Both now with a superpower we were told was impossible - Global Strong Consistency.

What Changed? (Spoiler: Physics Didn't)

For years, the CAP theorem was gospel. For many it still is. In any distributed system during a network Partition, you can have Consistency or Availability, but not both. Most architects chose their side early:

Team Consistency: PostgreSQL, MySQL with synchronous replication. Slow, regional, but correct.
Team Availability: Cassandra, original DynamoDB. Fast, global, but eventually consistent.

Then Google Spanner came along in 2012 and said, "What if we just... synchronized atomic clocks across data centers?" TrueTime felt like cheating but It was brilliant. Fast forward to 2024, and AWS dropped two services that bring Spanner level strong consistency to the masses...

Aurora DSQL — A distributed SQL database that uses hardware-assisted time synchronization and optimistic concurrency control.
DynamoDB MRSC — Multi-region strong consistency for DynamoDB Global Tables using a global journal sequencer.

Both systems achieve PC/EC in the PACELC theorem. Translation: When there's a partition (P), they choose consistency (C). When everything's running normally (E), they still choose consistency (C) over latency (L). This is the "hard mode" of distributed databases, and AWS is now offering it with the fan favorite flavour of serverless.

Aurora DSQL: City Hall Goes Serverless

The Architecture (Or: How to Disaggregate Everything)

With traditional databases a single monolithic process makes up compute, storage, and transaction management. Aurora DSQL saw that, and took a different path...

Here's the flow when you run a BEGIN; UPDATE users SET balance = balance - 100 WHERE id = 42; COMMIT; transaction:

The Relay gets your connection via TLS SNI and routes you to a Query Processor (QP).
The QP is a stateless PostgreSQL compatible compute node running in a Firecracker microVM. No noisy neighbors. No state. It starts by parsing your SQL, planing your query, and then during the transaction it just remembers your changes in memory (like a waiter taking your order but not sending it to the kitchen yet).
When you say COMMIT, the QP screams the transaction to the Adjudicator — a lease-based coordinator that owns specific key ranges. The Adjudicator uses Optimistic Concurrency Control (OCC). It checks: "Did anyone else touch these rows since you started?" If not, it proceeds. If yes, the Adjudicator throws up its hands and gives you a PostgreSQL serialization error (40001), and you must retry.
If validation passes, the transaction gets committed to the Journal — a distributed log service (think S3-grade durability). The Journal is replicated across two active regions and a witness region. Once the Journal says "committed," your data is durable.
Asynchronously, the Storage Layer (independant service) learns about committed transactions from the Journal and updates its local view. When you read, DSQL uses a "pushdown" model: filters and aggregations happen at the storage layer, so only the relevant rows come back to the QP. No shuffling entire table scans across the network.

This is how AWS can claim DSQL being 4x faster than traditional distributed SQL. Because all of the chatty PostgreSQL protocol round trips (SELECT, UPDATE, UPDATE, UPDATE...) happen locally in the QP's memory. The QP buffers everything like a patient scribe, taking notes. The global coordination tax is paid exactly once, at COMMIT time.

The Trade-Off: Optimism Has a Price

OCC is fantastic when contention is low (different users updating different rows). It's painful when contention is high (everyone trying to decrement the same inventory counter simultaneously). In high-contention scenarios, one transaction wins. Everyone else gets a serialization error and retries.

Your application must be idempotent. Because retrying "charge this credit card" without safeguards is how you get angry customers and chargebacks.

DynamoDB MRSC: FedEx Gets a Global Sequencer

The Architecture (Or: How to Make Key-Value Coordination Not Terrible)

In standard DynamoDB Global Tables each region has its own partition. Writes are local and fast. Replicators then ship changes to other regions asynchronously. If two regions update the same item simultaneously, DynamoDB uses a simple last-writer-wins (LWW) based on timestamps. This is eventual consistency. It works for 90% of use cases (session state, cache invalidation, analytics).

But what if you're running a global leaderboard, and two players in different regions both claim to be #1 at the exact same millisecond? LWW picks one based on timestamps, and the other player's update silently vanishes into the void.

That's... not great for player retention (or your app store rating).

Enter Multi-Region Strong Consistency (MRSC). Here's how it works:

When you write to an MRSC enabled table, the write goes to a Multi-Region Journal (MRJ) first. It assigns a globally unique sequence number to the write.
The write must be acknowledged by a quorum of regions (say 2 out of 3) before the client gets a success response. This is synchronous replication. Your RPO (Recovery Point Objective) is zero — if Region A explodes, Region B already has your data.
When you try a strongly consistent read (ConsistentRead=True), the regional replica sends a heartbeat to other regions or to MRJ to ensure it's caught up with all globally committed writes. The replica is basically asking, "Hey, did I miss anything?" Only after getting confirmation does it return data.

This ensures you never read stale data. But it also means that a strongly consistent read in MRSC is no longer a local operation. You're paying cross-region latency for correctness.

The Trade-Off: Consistency Isn't Free (Literally)

DynamoDB pricing got a massive overhaul in late 2024: 50% cut in on-demand throughput pricing and 67% cut in global tables replication costs.

But here's the thing: at millions of requests per second, DynamoDB's per-item pricing model can get expensive. A single RDBMS or DSQL cluster might handle the same query load for less money because you're paying for compute time, not individual request units. Also, if two regions write to the same item simultaneously, you get a replicated write conflict exception. Just like DSQL's serialization errors, your application must catch this and retry. Idempotency is non-negotiable.

The Performance Showdown

Let's talk numbers...

Operation	Aurora DSQL	DynamoDB MRSC
Local Read	~1-5ms	~1-5ms
Strong Read (Global)	~1-5ms (snapshot)	~20-200ms+ (heartbeat)
Write (Interactive)	~1-5ms (buffered)	~20-200ms+ (synchronous)
Write (Commit/Ack)	2 RTTs (sync)	1-2 RTTs (sync)
Max Throughput	Virtually unlimited	20M+ requests/sec

The absolute killer feature of DSQL is that strongly consistent reads are still local. Because DSQL uses snapshot isolation tied to physical time, a QP can serve reads from its local storage without consulting other regions.

DynamoDB MRSC, by contrast, has to send a heartbeat to verify consistency. Every. Single. Strong. Read.

This is the speed-of-light problem. Tokyo to Frankfurt is roughly 150-200ms round-trip. Just to be on the same page... you can't cheat physics.

The Decision Framework (Or: How to Not Screw This Up)

Here's a mental math I generally use...

Q: "Do I need complex joins, foreign keys, or multi-table transactions?"

If yes: Aurora DSQL. DynamoDB doesn't do joins. You can work around them with Global Secondary Indexes and multiple queries. But that classic case of unnecessary overengineering.

If no: Keep reading.

Q: "Is my schema stable, or am I still figuring out what attributes my items need?"

If stable: Aurora DSQL is fine. Schema migrations in DSQL are just ALTER TABLE statements.

If volatile: DynamoDB. You can add new attributes to items on the fly without ALTER TABLE hell. Hate to say this in 2025, but this is the superpower of NoSQL.

Q: "What's my read/write ratio, and what's the access pattern?"

If mostly reads, complex queries, or ad-hoc analytics: Aurora DSQL. SQL is expressive, beautiful and self-documented.

If mostly writes, or simple key lookups at massive scale: DynamoDB. It's built for throughput.

Q: "What's my tolerance for operational complexity?"

If you want maximum abstraction: Both are serverless. But DSQL is newer. It's in preview. DynamoDB is battle-tested and powers AWS's own internal systems.

If you're risk-averse: DynamoDB MRSC. It's a feature flag on an existing, mature service.

The Gotchas...

For Aurora DSQL:

No foreign keys (yet). You're left to handle it with application enforced referential integrity.
3,000 row / 10MB limit per transaction. If you're doing bulk imports, break them.
Preview status means features are missing. No PostGIS, no triggers, no full-text search (yet).
Sequential primary keys are death. Auto-incrementing IDs will create a hot spot where every write hammers the same Adjudicator. Instead use UUIDs or random keys to distribute the load.

For DynamoDB MRSC:

Strongly consistent reads are slow. Plan your access patterns really really really really carefully.
Item size limit is 400KB. If you're storing blobs, use S3 and reference them.
Idempotency by design. Conflict exceptions require retry logic.
Pricing at scale can surprise you. Run the cost calculator and compare against DSQL for your specific query profile.

The Punchline

Just two decades ago if you wanted global strong consistency, you'd built a monolithic Oracle RAC cluster ...prayed to the database gods ..and also accepted that "disaster recovery" meant "restore from backup in 4-6 hours."

Today you spin up a serverless DSQL cluster or flip a feature flag on DynamoDB. You get active-active multi-region replication with zero RPO and near-zero RTO! The commit latency is still capped by the speed of light itself, but at least you're not also fighting decades-old database architectures designed for a pre-cloud world.

So just to recap - when your CTO asks, "Can we guarantee consistency globally?" the answer is finally, confidently: "Yes. Now let's talk about your access patterns."

Insights in Plaintext: RSA - The Asymmetric Anchor

Sat, 02 Nov 2024 18:30:00 GMT

Fresh from our exploration of Diffie-Hellman's elegant key exchange protocol, we're diving into RSA. It's the algorithm that transformed the theoretical possibility of public-key cryptography into a practical reality. While DH solved the key exchange puzzle, RSA took it a step ahead and opened up entirely new possibilities with its asymmetric approach to encryption and digital signatures. Today, it stands as a cornerstone of modern cryptography, but faces a formidable challenge. Yes, you guessed it - quantum computing.

This article builds on the mathematical foundations we've explored throughout this series. We'll revisit everything from the prime numbers that fascinated us in our DES discussion to the modular arithmetic that powered our AES implementations. Now the pieces of the puzzle will start to fit in as we'll see how these concepts combine in RSA's ingenious design. While not yet obsolete but RSA's vulnerability to Shor's algorithm demands a thorough understanding of its strengths and weaknesses. Grab your favorite beverage as we dissect RSA's inner workings.

The RSA Story - More Than Just Initials

The 70s was truely a wild era for cryptography. Computationlly fast symmetric encryption suffered from a killer flaw of key distribution. How do you securely share a secret key with someone across the internet without any guarentee that someone is already intercepting the traffic? You might think - didn't we discuss Diffie-Hellman's 1976 masterpiece in the last article? Well yes, and while DH introduced the concept of public-key cryptography but that theory needed a real-world algorithm. And this voice created space for Rivest, Shamir, and Adleman to rise up. 1997 saw the paper that introduced RSA algorithm. This wasn't just a paper but instead it was the birth of practical public-key cryptography. Today we can say that it is a foundation for much of our modern digital security infrastructure.

The genius of RSA? It leverages the computational asymmetry of prime factorization i.e. while multiplying two large primes is easy but factoring their product is incredibly difficult. Even today powerful supercomputers struggle with this task. This asymmetry allows for a public key (used for encryption and verification) and a private key (used for decryption and signing) - the core of public-key cryptography.

A Deeper Dive into RSA's Number Theory Magic

RSA's security isn't magic but rather elegant mathematics. Let's dissect the core components and step a bit beyond the surface level:

Prime Numbers: The Foundation of Foundations

Two incredibly large prime numbers, say p and q form the bedrock of RSA. These aren't just any primes but they're very large primes, often thousands of bits long. Finding these primes isn't trivial. We have few tried and tested means to get these two numbers like the Miller-Rabin test. It offer a high probability (but not absolute certainty) that a number is prime. However the probability of error is so incredibly small that for all practical purposes we can consider the numbers we find to be prime. The product of these primes, n = pq*, is the modulus. Modulus is the public parameter used in both encryption and decryption. The size of n directly influences the security of the system. A larger *n* implies a much more difficult factoring problem for attackers.
Euler's Totient Function: Counting the Coprimes

Euler's totient function, φ(n), is then used to get co-primes for n. Co-primes are the positive integers up to n that are relatively prime to n i.e. they share no common factors other than 1. For RSA, where n = pq*, this simplifies beautifully to φ(n) = (p - 1)(*q* - 1). While it seems insignificant it really is not. It dictates the structure of the mathematical group within which the encryption and decryption operations take place.
Modular Arithmetic: The Clockwork of Cryptography

Modular arithmetic is the arithmetic of remainders. We perform operations (addition, multiplication, exponentiation) and then take the remainder after dividing by a modulus (n in our case). This dance is what's depicted by the "clock arithmetic" analogy. If you add 5 hours to 10 o'clock on a 12-hour clock, you get 3 o'clock (because 15 mod 12 = 3). This is fundamental to RSA's efficiency as it helps to keep the numbers within manageable bounds. Just to reiterate we're dealing with incredibly large numbers to begin with.
Public Exponent (e): The Public Face

The public exponent, e, is a relatively small number (often 65537, which is a Fermat prime) that's coprime to φ(n) (their greatest common divisor is 1). This is done to ensures that it has a multiplicative inverse modulo φ(n). The public key (n, e) is openly advertised and freely available to anyone. Somone who wants to send you an encrypted message can do so with this public key. The choice of e thus involves a tradeoff. A small e leads to slightly faster encryption but could potentially lead to vulnerabilities. For that reason 65537 is a good balance.
Private Exponent (d): The Secret Keeper

We're now getting into the heart of RSA's security. The private exponent d is the multiplicative inverse of e modulo φ(n). This means e * d ≡ 1 (mod φ(n)). In much simpler terms, if you multiply e and d and then take the remainder after dividing by φ(n), you get 1. Finding d without knowing p and q (and thus φ(n)) is computationally infeasible using classical algorithms. This is because of the fact that finding this inverse requires knowing the factorization of n. This is exactly the problem we're leveraging for security. For all of this to work the private key, (n, *d*), must remain absolutely secret.
Chinese Remainder Theorem (CRT): The Speed Demon

The CRT is a mathematical theorem which involves breaking down the problem with large modulus into smaller more manageable subproblems. In the context of RSA decryption it permits performing the decryption operation modulo p and modulo q separately. From there we combine the results efficiently to obtain the final result modulo n. This significantly accelerates decryption often by a factor of four or more. It helps make RSA practical for real-world applications.

Key Generation: The Art of Crafting Cryptographic Secrets

The food can only be as good as it's ingrediants by themself. While RSA relies on strong mathematics but it will all be for nothing of the key pair generation is sloppy. Here's a detailed breakdown:

Prime Number Generation: This is the most computationally expensive part and also the most important. We use some sophisticated probabilistic primality testing algorithms (like the Miller-Rabin test) to generate two large prime numbers. The larger these primes, the more secure the resulting RSA key pair. The process of prime generation deserves some careful setup to ensure that the selected primes do not have any special properties which makes them vulnerable to advanced factoring techniques.
Modulus Calculation: The modulus, n, is calculated as the product of the two primes i.e. n = pq*. As you can imagine, n forms the foundation of the public and private keys. The size of *n* (typically expressed in bits) dictates the security level of the RSA system.
Totient Calculation: Next, we compute Euler's totient function, φ(n) = (p - 1)(q - 1). This value is crucial for determining the private exponent. This value, φ(n), must be kept secret to maintain security of RSA.
Public Exponent Selection: We choose the public exponent, e, such that 1 < e < φ(n) and gcd(e, φ(n)) = 1 (i.e., e and φ(n) are coprime). A common and generally safe choice is e = 65537.
Private Exponent Calculation: This is where the Extended Euclidean Algorithm joins the party. It's used to calculate the modular multiplicative inverse of e modulo φ(n). This inverse d is the private exponent. The equation e * d ≡ 1 (mod φ(n)) must hold true. The Extended Euclidean Algorithm efficiently finds d. The calculation of d requires knowledge of φ(n), which in turn requires knowledge of p and q. It is the secrecy of d that protects the RSA system at all times. If an attacker could determine *d*, they could decrypt messages intended for the private key holder.
Key Pair Formation: Finally, the public key is formed as the pair (n, e), while the private key is (n, d). The public key can be freely distributed, while the private key must be kept strictly secret and protected.

This process requires careful attention to detail to ensure the generation of strong and secure RSA key pairs. Any flaws in this generation process can severely compromise the security of the entire system.

Implementation Details

The theoretical beauty of RSA needs practical implementation considerations:

Key Size: Larger keys (2048 bits or more are recommended now) offer stronger security but slower performance. The choice involves a trade-off between security and efficiency.
Padding Schemes: Raw RSA is vulnerable. Padding schemes like Optimal Asymmetric Encryption Padding (OAEP) add randomness and structure to the data before encryption. This in turn significantly enhancing security against various attacks (including chosen-ciphertext attacks). PKCS#1 v1.5, while historically prevalent, is now considered insecure.
Hardware Acceleration: For high-performance applications, hardware-based acceleration via specialized cryptographic processors or Hardware Security Modules (HSMs) is often necessary. This significantly improves performance and security.

import random
import hashlib
from Crypto.Util.number import getPrime, inverse

def generate_rsa_keys(key_size):
    p = getPrime(key_size//2)
    q = getPrime(key_size//2)
    n = p*q
    phi = (p-1)*(q-1)
    e = 65537                           #Common value
    d = inverse(e, phi)
    return ((n, e), (n, d))

def oaep_padding(message, key_size):
    #Simplified OAEP -  In a real system, use a robust library like pycryptodome
    k = key_size // 8
    mLen = len(message)
    mgf = hashlib.SHA256()
    mgf.update(message)
    maskedDB = mgf.digest()[:k - mLen - 2] + b'\x00' + message
    seed = os.urandom(k//2)
    dbMask = mgf(seed, k - k//2)
    maskedSeed = mgf(maskedDB, k//2) ^ seed
    return maskedSeed + maskedDB

Security Analysis: RSA's Vulnerabilities

RSA's security while strong for appropriately sized keys and careful implementation is by no means absolute. Key vulnerabilities include:

Factoring Attacks: RSA's core relies on the difficulty of factoring large numbers. While currently infeasible for sufficiently large keys with classical computers, quantum computing (via Shor's algorithm) poses a significant long-term threat.
Side-Channel Attacks: These exploit information leaked during computation, such as timing variations (timing attacks) or power consumption patterns (power analysis). Robust countermeasures, including constant-time implementations as well as blinding are essential for mitigating these attacks.
Implementation Errors: It's surprising how many times a poorly written code, inadequate testing, or insecure libraries end up creating vulnerabilities. This is regardless of key size or padding. Using well-vetted libraries, secure coding practices, and rigorous testing are essential for robust security.

Addressing these vulnerabilities requires a multi-pronged approach which involves strong key management, secure implementation, and regular security audits. It's not just about the math but it's about secure implementation and proactive risk mitigation.

The Looming Threat of Quantum Computing

We've referenced Shor's algorithm many times in this series already. It can efficiently factor large numbers. This poses a significant threat to RSA, potentially rendering it insecure. The development of large-scale, fault-tolerant quantum computers is a matter of "when", not "if." This necessitates the urgent exploration and adoption of post-quantum cryptographic algorithms.

The NIST Post-Quantum Cryptography standardization process has identified several quantum-resistant algorithms. Transitioning to these algorithms is a crucial task. It requiring careful planning, testing, and implementation. This is not a short-term project; it's a generational shift in how we secure our digital world.

The Bottom Line: RSA - Still Relevant, But Prepare for the Future!

RSA stands as a testament to the power of mathematical elegance in cryptography. We've journeyed from its prime number foundations through practical implementations, exploring how this revolutionary algorithm transformed digital security. We however cannot ignore how quantum computing poses an existential threat via Shor's algorithm. RSA's design principles - particularly the creative use of computational asymmetry, till date continue to inspire modern cryptographic solutions. This includes many post-quantum candidates.

As we look ahead to our next deep dive into Elliptic Curve Cryptography (ECC), it's worth reflecting on RSA's enduring legacy. Its elegant mathematical foundation didn't just solve the public-key cryptography challenge, but it showed us how creative thinking about computational complexity could revolutionize security. Do share your experiences with RSA and possibly it's implementations! Have you encountered interesting challenges with key sizes or padding schemes?

Insights in Plaintext: Hiding in Plain Sight with Diffie-Hellman

Sun, 20 Oct 2024 18:30:00 GMT

After exploring the elegant world of SHA-3's sponge construction in Part 5, we're shifting gears to tackle one of cryptography's most revolutionary breakthroughs - the Diffie-Hellman key exchange. Unlike the patent-free approach we saw with Bruce Schneier's Blowfish, DH sparked intense patent disputes that highlighted its groundbreaking importance. This fundamentally innovative protocol has been the silent guardian of our digital secrets for decades. It's been powering everything from HTTPS connections to secure messaging apps.

You've likely crossed paths with DH countless times today alone - every time you've seen that little padlock in your browser or sent an encrypted message. But today we're ditching the surface-level chatter to dive deep into the elegant mathematics and clever design that make this protocol tick. From its mathematical foundations to its quantum-resistant future, we'll explore how this cryptographic legend continues to evolve and remain relevant in today's wild west of cybersecurity.

The Genesis of Diffie-Hellman

At one point cryptography was a heavily guarded realm that was primarily controlled by governments and military organizations. This was before Diffie-Hellman broke into the cryptography scene. Secure communication relied on prior shared secret keys. As you can imagine this posed a significant logistical challenge: how do you securely share these keys without them falling into the wrong hands? It was like trying to deliver a top-secret message via carrier pigeon through a flock of hungry hawks.

Then came Whitfield Diffie and Martin Hellman, without any doubt two cryptographic revolutionaries who dared to dream of a different world – a world where secure communication was accessible to everyone. In their groundbreaking 1976 paper, "New Directions in Cryptography," (IEEE, 1976) they introduced the concept of public-key cryptography. This changed the face of secure communication from the ground up. They also brought in Ralph Merkle to the project. Ralph's work on public key distribution was fundamental but often forgotten, so sometimes this protocol is also called Diffie-Hellman-Merkle key exchange. At the time during its research and publishing there was heavy interference from the NSA who believed cryptography should be under government control. Diffie and Hellman fought to publish their revolutionary ideas and because of that we now have widely accessible secure communication.

Diffie-Hellman was the first viable implementation of public-key cryptography. It enabled two parties to establish a shared secret key over an insecure channel. This was all done without ever exchanging the secret itself. This allowed for the two sides to be on the other end of the world and establish a secure form of communication over an insecure line.

The Math Behind the Magic

Let's get to the heart and soul of Diffie-Hellman: the intricate dance of numbers that makes secure communication possible. It does indeed involve some seriously clever math but don't worry we'll get through it together.

At the center of this DH-dance-floor is a discrete logarithm problem. The basis of DH starts from a computational challenge - it's relatively easy to multiply two numbers, but given a number it's difficult to get to its exact roots. With that now imagine a one-way street in the world of numbers. You can easily multiply a number by itself multiple times (aka the easy street). But try to figure out how many times it was multiplied to get to this result? That's like navigating a dance floor blindfolded (aka the hard street).

How about we bring in the classical dancers of DH-World - Alice, Bob, and Eve. Picture this: Alice, wants to share a secret with Bob, but Eve is eavesdropping on every word. Difficult to deal with, I know. How do you whisper secrets in a crowded room? Enter the gurus - Diffie-Hellman.

Here's the choreography:

Setting the Stage: Alice and Bob agree on two public numbers: a base 'g' and a large prime 'p'. These are the rules of the cryptographic dance, visible to everyone, including Eve.
The Secret Spin: Alice now chooses a secret number 'a' (private key), and then raises 'g' to the power of 'a'. From there she takes the result modulo 'p'. This gives you her public key 'A'. Bob does the same with his secret number 'b', calculating his public key 'B'.
The Exchange: Alice and Bob now exchange their public keys – 'A' and 'B' – loud and clear for everyone to hear. Eve now knows 'g', 'p', 'A', and 'B', but crucially, she doesn't know 'a' or 'b'.
The Magical Merge: Alice takes Bob's public key 'B' and raises it to the power of her secret 'a'. She then takes the result modulo 'p'. This gives her the shared secret 's'. Bob does the same with Alice's public key 'A' and his secret 'b'. And here's the kicker: due to the magic of modular arithmetic, both of them arrive at the exact same secret 's'! Amazing right!?

A simple diagram illustrating the key exchange process, showing the flow of public keys and the calculation of the shared secret. Source: Wikipedia

Let's break down the math, step by step:

Alice calculates: A = g^a mod p
Bob calculates: B = g^b mod p
Alice receives B and calculates: s = B^a mod p = (g^b mod p)^a mod p = g^ab mod p
Bob receives A and calculates: s = A^b mod p = (g^a mod p)^b mod p = g^ab mod p

Spot the magic, Pen-and-Teller? Both Alice and Bob arrive at g^ab mod p, the shared secret, without ever directly exchanging 'a' or 'b'. Eve, despite knowing 'g', 'p', 'A', and 'B', is left scratching her head, unable to efficiently calculate 's' without knowing either 'a' or 'b'. This is the power of the discrete logarithm problem in action.

# Simplified DH example (don't use this in production, seriously!)
import random

def diffie_hellman(p, g):
    a = random.randint(2, p - 1)    # Alice's private key – shhh!
    A = pow(g, a, p)                # Alice's public key – shout it from the rooftops!

    b = random.randint(2, p - 1)    # Bob's private key – top secret!
    B = pow(g, b, p)                # Bob's public key – for everyone to see!

    s_a = pow(B, a, p)              # Alice computes the shared secret
    s_b = pow(A, b, p)              # Bob computes the shared secret

    assert s_a == s_b               # They should be the same! – Magic!
    return s_a

# Example usage
p = 23                              # A small prime (in reality, use much larger primes)
g = 5                               # A generator
shared_secret = diffie_hellman(p, g)
print(f"Shared secret: {shared_secret}") # Shared secret: 1

Needless to say, like with samples all throughout this series - this code is just a simplified illustration. Real-world implementations use industrial-strength primes and handle all sorts of edge cases.

A Family of Secrets

Don't be fooled by Diffie-Hellman, it isn't just one move. It's a whole damn dance crew where each member brings in their own unique style to the cryptographic floor. Over time we've seen some seriously slick movers and shakers that have expanded on the original DH. We've got bright minds to pick up DH and create variations to address specific security threats and boost performance. It's the World Dance Faceoff: you've got the OG DH, the breakdancing duo ECDH, the security-conscious hip-hop crew Authenticated DH, and the flash mob that is Group DH. Let's meet the crew:

Elliptic Curve DH (ECDH): Picture DH but as a pair of insane breakdancers who are pulling off moves you never thought possible. ECDH leverages the mind-bending math of elliptic curves (think finite fields and geometric wizardry) to achieve the same level of security. The unique selling point is that they do this with way smaller key sizes. This makes it the MVP for resource-constrained environments like IoT devices.
Authenticated DH: The OG DH, bless its heart, is nothing short of a free spirit. It trusts everyone and doesn't inherently protect against man-in-the-middle (MITM) attacks. Authenticated DH comes in with the hip-hop swagger by adding an identity check to the routine. It checks against digital signature to make sure that you're really grooving with Bob and not some malicious Eve trying to steal your moves.
Group DH: Sometimes a bigger party is what the routing needs and in such cases you need to share the secret handshake with the whole crew. Imagine planning a surprise flash mob. It's a surprise only if you find a way to coordinate everyone without spilling the beans. That's where Group DH walks in. It takes the core DH moves and adapts them for group communication, letting multiple parties establish a shared secret.

Beyond these headliners, there are many more like the Station-to-Station (STS) protocol. It combines DH with signatures for mutual authentication (like a choreographed duet with a signed contract). There's also the MTI/A0 key agreement protocol, a frequent performer in GSM networks. The DH dance crew is truly a diverse bunch where each member bringing their unique skills and style to the cryptographic stage.

DH Dance Crew Face-Off: Comparing the Moves

Okay, so we've met the DH dance crew. Now let's see how they stack up against each other in a head-to-head comparison:

Feature	Original DH (The Founder)	ECDH (The Breakdancing Duo)	Authenticated DH (The Hip-Hop Crew)	Group DH (The Flash Mob)
Key Size	Larger	Smaller	Similar to Original DH	Varies depending on the group size
Performance	Slower	Faster	Slightly slower than Original DH	Can be complex and slower
Resource Usage	Higher	Lower	Slightly higher than Original DH	Higher, especially with large groups
Security	Good, but needs backup	Higher (for equivalent key size)	Significantly higher	Good, but coordination is key
Vulnerability	Susceptible to MITM	Susceptible to MITM	Resistant to MITM	Subgroup attacks, complexity issues

Quantum Quandaries (The Future of DH)

We cannot escape the looming threat of quantum computing. As we've seen many times in the past few articles of the series - these futuristic machines bring with them their mind-boggling power. This raw computational power poses a serious challenge to many cryptographic algorithms that including DH. Shor's algorithm which is a quantum algorithm for factoring large numbers can in theory crack the discrete logarithm problem. This means that it can render DH vulnerable.

But fear not! Cryptographers are already working on post-quantum cryptography – algorithms resistant to attacks from both classical and quantum computers. Promising contenders include lattice-based cryptography, code-based cryptography, and hash-based cryptography. These technologies are still evolving, but they represent the future of secure communication in a post-quantum world.

Keeping Secrets Safe (Security Considerations)

No system is impenetrable, and DH has its vulnerabilities:

Man-in-the-Middle (MITM) Attacks: A sneaky attacker can intercept the public key exchange and then establish separate shared secrets with each party. This allows this attacker to relay messages in plaintext effectively eavesdropping on the conversation. Authenticated DH is your shield against this threat.
Small Subgroup Attacks: If parameters aren't carefully chosen, attackers might exploit weaknesses in smaller subgroups to compromise the key exchange. This is why the seniors keep on repeating - choose your parameters wisely!
Safe Prime Selection: The choice of the prime number 'p' is crucial. Using a "safe prime" (a prime number where (p-1)/2 is also prime) significantly strengthens the protocol.

DH in the Wild (Modern Applications)

Diffie-Hellman is everywhere, quietly securing our digital lives:

TLS/SSL: That little padlock in your browser? DH often plays a key role in establishing a secure connection between your browser and the server.
Signal Protocol: This end-to-end encrypted messaging app relies on DH for its key exchange magic.
VPN: VPNs use DH to create secure tunnels, protecting your data as it travels across the internet.
IoT Device Pairing: When you pair your smart devices to your home network, DH is often working behind the scenes to secure the connection.

Bottom Line

Diffie-Hellman stands as a masterpiece of cryptographic innovation by elegantly solving the key distribution problem that plagued secure communication for centuries. From its mathematical foundations in discrete logarithms to modern elliptic curve variants - we've seen how DH's clever design enables secure key exchange over insecure channels. While quantum computing poses future challenges for all cryptographic operations, the protocol's adaptability has already spawned quantum-resistant variants. This goes a long way in ensuring its continued relevance in tomorrow's cryptographic landscape.

That's not all though. Stay tuned for our next deep dive into RSA. We'll explore another revolutionary public-key cryptography system that builds upon many concepts we've covered here. Till then remember that every time you see that padlock in your browser, you're witnessing Diffie and Hellman's elegant mathematics at work. It's working silently in protecting your digital communications. What are your thoughts on DH's evolution? Do share your experiences implementing or working with this cryptographic cornerstone!

Insights in Plaintext: SHA-3 – Absorbing the Complexity, Squeezing Out Security

Tue, 08 Oct 2024 18:30:00 GMT

Fresh from our exploration of HMAC's authentication prowess in Part 4, we're diving into SHA-3. For a change, It's the quantum-resistant powerhouse of modern cryptographic hashing. While its predecessors SHA-1 and SHA-2 laid the groundwork, SHA-3 represents a revolutionary approach with its unique sponge construction. It's a complete departure from traditional Merkle-Damgård designs.

In this fifth installment of our series, we'll dissect SHA-3's innovative architecture, from its permutation-based core to its robust security properties. Whether you're building blockchain applications, implementing digital signatures, or preparing systems for the quantum computing era, understanding SHA-3's capabilities is crucial. Get ready to absorb the complexity and squeeze out the security benefits as we unravel the algorithm that's reshaping the future of cryptographic hashing!

The Dawn of SHA-3: A Preemptive Strike Against Cryptographic Monoculture

It is somewhat surprising how SHA-3 started off. Most other algorithms in this class started to fix something with other algorithm. Well, not this one. SHA-3's origin story begins not with a crisis, but for a change with foresight. While SHA-2 remained a stalwart in the cryptographic world, the community wisely recognized the underlying risk of relying heavily on a single family of hash functions. A classic case of having all the eggs in one basket. Imagine a digital ecosystem where one vulnerability could be enough to bring down the entire security infrastructure. This very concern prompted NIST to launch the hash function competition in 2007, a preemptive strike against cryptographic monoculture.

The goal wasn't to replace SHA-2 but to instead promote a diverse and resilient cryptographic toolkit. From this competition, we got Keccak - the elegant sponge construction. Think of it like a sponge absorbing data: it soaks up the input and then squeezes out a unique cryptographic fingerprint (the hash). Unlike SHA-2's Merkle-Damgård construction, the sponge brings to the table adaptable output lengths and the innovative extendable-output functions (XOFs) like SHAKE128 and SHAKE256. This adaptability is what ended up standing strong in the face of emerging challenges like lightweight cryptography and the looming threat of quantum computing.

The Quantum Shadow: From Theoretical Threat to Looming Reality

Quantum computing, is no longer a theoretical concept. For crying out loud, you have offerings like Azure Quantum Cloud Computing. Quantum world now casts a long shadow over the cryptographic landscape. While large-scale, fault-tolerant quantum computers are still some time away, their potential to shatter widely used algorithms like RSA and ECC is not just a theoretical concern. Quantum algorithms like Shor's algorithm for efficient factorization and discrete logarithm computation, pose a direct threat to these foundational cryptographic primitives. Even hash functions are not entirely immune to the quantum threat. Grover's algorithm can speed up brute-force attacks, making larger hash outputs necessary for robust security.

This is where SHA-3's inherent quantum resistance and flexibility come into the picture. Unlike SHA-2, whose underlying mathematical structures are vulnerable to Shor's algorithm, SHA-3's security is rooted in the sponge construction. This construction is resistant to known quantum attacks. Combining this with its support for larger output sizes provides a significantly higher level of confidence in a post-quantum world. Furthermore, SHA-3's adaptability allows for adjustments to its internal parameters. Like increasing the state size which further strengthens its defenses as quantum computing technology advances. This positions SHA-3 as a vital component in the ongoing evolution of cryptography.

Dissecting the Sponge: A Deep Dive into SHA-3's Inner Workings

SHA-3 which is based on the Keccak sponge construction operates on a state of 1600 bits. These bits are visualized as a 5x5x64-bit three-dimensional array, often represented as a[x][y][z]. It processes data through a series of permutation rounds, each with five steps:

θ (Theta): Here the focus is on diffusion. Imagine each column C[x][z] as a vertical line through the state. θ calculates parity for each column and then mixes this parity into two other columns. This spreads changes across the entire state quickly.
```
State (simplified):
[ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ]
[ ][ ][C][ ][ ]  <- Calculate parity for this column
[ ][ ][ ][ ][ ]
[ ][ ][ ][ ][ ]

Mix parity into other columns:
[ ][X][ ][ ][X]
[ ][X][ ][ ][X]
[ ][X][C][ ][X]
[ ][X][ ][ ][X]
[ ][X][ ][ ][X]
```
Where 'X' represents the columns affected by the parity of column 'C'.
ρ (Rho): Next up we rotates each 64-bit lane a[x][y]. If you visualize each lane as a long horizontal sequence of bits and then rotate them around, and do this by a different amount for each lane - you would have to create asymmetry and further mixing up of the state.
```
Lane (simplified):
[0,1,2,3,4,5,6,7]  <- Original lane

Rotated lane (example offset of 3):
[3,4,5,6,7,0,1,2]
```
π (Pi): We then rearrange the lanes within each slice. Think of it as shuffling the lanes according to a specific pattern, further scattering the bits and increasing the complexity of the state.
```
Simplified Lane Arrangement:
[L1][L2][L3]
[L4][L5][L6]

After π:
[L4][L2][L1]
[L5][L3][L6]  <- Lanes rearranged
```
χ (Chi): This step is the only non-linear one. It operates on rows within each slice. It applies a non-linear mapping based on neighboring bits, introducing crucial non-linearity for security.
```
Row (simplified):
[0,1,0,1,1]

After χ (example):
[1,0,1,0,0] <- Non-linear transformation based on neighbors
```
ι (Iota): Finally in this step we add a round constant to a single lane, a[0][0]. This constant is different for each round, preventing repetitive patterns and ensuring each round is unique.

These five steps are repeated for 24 rounds for SHA3-224/256 or for 12 rounds in the case of SHAKE128/256. In the end, we have created a complex and highly diffusive transformation. After absorption, the next phase which is the squeezing phase extracts the variable-length hash output. SHA-3 also offers fixed-output variants: SHA3-224, SHA3-256, SHA3-384, and SHA3-512.

Show Me the Code!

Fortunately, you don't have to implement this all from scratch. Here's a simple example to get SHA3-256:

import hashlib

def sha3_256_hash(data):
    sha3_hash = hashlib.sha3_256(data.encode('utf-8')).hexdigest()
    return sha3_hash

my_string = "This is a test string"
hashed_string = sha3_256_hash(my_string)
print(f"The SHA3-256 hash of '{my_string}' is: {hashed_string}")

# The SHA3-256 hash of 'This is a test string' is: 22cda3a2d2053f0a81fbf89ac531f724989f2308f0a2f29d894b07b4fec0320e

The Security Fortress: SHA-3's Robust Defenses

A lot of SHA-3's security is rooted in its design and extensive cryptanalysis over the years. Formal security proofs and years of scrutiny support its robustness. It exhibits:

Collision Resistance: Finding two different inputs producing the same hash is computationally infeasible.
Preimage Resistance: Retrieving the original input from a given hash is practically impossible.
Second Preimage Resistance: Finding another input with the same hash, given an input and its hash, is extremely difficult.
Resistance to Length-Extension Attacks: SHA-3's sponge construction inherently avoids this SHA-2 vulnerability.

SHA-2 vs. SHA-3: A Tale of Two Hashes

Feature	SHA-2	SHA-3
Construction	Merkle-Damgård	Keccak Sponge Construction
Internal Structure	Based on block ciphers	Based on permutation functions
Quantum Resistance	Vulnerable	More resistant
Speed	Generally faster	Generally slower
Flexibility	Fixed output lengths	Variable and extendable output lengths

Performance and Efficiency: SHA-3 in a World of Varying Resources

We unfortunately aren't getting the best of both worlds here as SHA-3 isn't known for its blazing speed. It is generally slower when compared to SHA-2. This comes from the fact that it has more complex permutation operations within its sponge construction. SHA-2 over the years has also gone through extensive optimization efforts both in terms of software and hardware. Raw speed however isn't the sole determinant of an algorithm's suitability. For many applications, the enhanced security and quantum resistance offered by SHA-3 outweigh the performance trade-off. Take for example digital signatures or blockchain implementations - the overall time impact of hashing is often negligible compared to other operations like network latency or even the disk I/O.

However, in performance-critical systems processing high volumes of data, every milliseconds matter. SHA2-256 is typically 1.5-2x faster than SHA3-256. To put that in perspective - Google processes over 8.5B searches daily. Adding even a single milli-second to each search means adding over ~98.3 days or ~3.2months of computational load. All this to say that in the real world, careful benchmarking and performance testing are crucial to determining whether SHA-3's added security justifies the potential performance hit. Consider the specific requirements of your application and the available hardware resources before making any choice.

This performance consideration becomes even more critical decision makers in resource-constrained environments. Think of use cases around IoT devices, mobile phones, etc. These devices often operate with limited processing power, memory, and energy. Deploying SHA-3 on such devices may not make sense as it will lead to significant performance bottlenecks and more importantly - battery drains. This is where lightweight cryptography comes into play. Look at algorithms like Ascon and SPHINCS+ which are designed specifically for resource-constrained devices. They offer a trade-off between security and efficiency. Choosing between SHA-3 and a lightweight alternative depends on the specific security requirements as well as the resources at the disposal. If security is paramount and resources are plentiful, SHA-3 is a no-brainer. In other cases, a lightweight algorithm might be the more practical choice.

SHA-3 in Action

SHA-3's versatility extends beyond theoretical cryptographic discussions. It has already found practical use in diverse applications:

Digital Signatures: SHA-3 plays a crucial role in digital signatures, ensuring data integrity and authenticity. By encrypting the hash of a document with a private key, the sender creates a digital signature. The recipient can then verify the signature using the sender's public key. This helps to establish that the document hasn't been tampered with and that it originated from the claimed sender. SHA-3's collision resistance is the basis of the confidence here. Forging a valid signature would require finding another document with the same hash which as we discussed is a computationally infeasible task.

Blockchain and Cryptocurrency: In the world of blockchain and cryptocurrencies SHA-3 plays a significant role. It secures transactions and maintains the integrity of the distributed ledger. Cryptocurrencies like Ethereum utilize SHA-3 for various cryptographic operations. This includes transaction verification as well as mining. SHA-3's preimage resistance is vital here as it prevents attackers from reversing transactions or creating fraudulent ones.

Random Number Generation (RNG): SHA-3 can be used to build cryptographically secure random number generators (CSPRNGs). This is done by hashing a seed value and then repeatedly hashing the output. This gives us a sequence of unpredictable random numbers. This is essential for applications requiring high-quality randomness. Think of use cases like generating cryptographic keys, session IDs, and nonces.

Password Hashing: While traditionally bcrypt and scrypt are preferred for password hashing, the shift is coming in. SHA-3 offers a robust alternative. Generally, in such cases, we make use of a sufficiently large output size and incorporate a salt (a random value unique to each password). SHA-3 can effectively protect passwords against brute-force and rainbow table attacks.

The Bottom Line: SHA-3's Legacy and Future

SHA-3 stands as a testament to cryptographic innovation. It introduces the revolutionary sponge construction that fundamentally changed how we approach hash functions. Its quantum resistance and flexible output capabilities make it particularly valuable as we face evolving security challenges. While it may not always be the fastest option, its mathematical elegance and robust security properties have earned it a crucial place in modern cryptography.

The true power of SHA-3 lies not just in its technical capabilities but in how it demonstrates the importance of cryptographic diversity. We've seen how different use cases can all make use of it. Be it anything from blockchain implementations to resource-constrained IoT devices. They all demand different trade-offs between security, performance, and flexibility and SHA-3 stands strong. SHA-3's adaptable design principles continue to influence the development of new cryptographic primitives, including many post-quantum candidates.

Do share your experiences with SHA-3! Have you encountered interesting performance trade-offs? How are you balancing security requirements with resource constraints?

Insights in Plaintext: Hashing out HMAC

Fri, 27 Sep 2024 18:30:00 GMT

Continuing our journey through the cryptographic landscape in the "Insights in Plaintext" series, we're shifting gears from symmetric encryption. We'll now look into the critical world of message authentication. Part 4 brings us face-to-face with HMAC. It's a cryptographic heavyweight that's been silently securing our digital world for decades now.

While HMAC might not be the newest kid on the cryptographic block, its battle-tested reliability makes it the backbone of countless security protocols today. It plays vital role in securing API calls to protecting blockchain transactions. HMAC's elegant design continues to shine in our increasingly complex digital landscape. So grab your favorite debugging mug as we're about to dissect the inner workings of Hash-based Message Authentication Codes and understand why this seemingly simple construction remains a cornerstone of modern security architecture.

The HMAC Origin Story

In the digital Wild West of the mid-90s, one of the top challenges in front of engineering teams was to secure online communications. Existing Message Authentication Codes (MACs) were either computationally expensive, relying on slower block ciphers, or suffered from security vulnerabilities, either ones that were already exploitable or could be in the next few years. This cryptographic conundrum called for a hero, and then in 1996 we witnessed HMAC riding into the town with guns blazing (metaphorically, of course). Born from the minds of cryptographic pioneers Mihir Bellare, Ran Canetti, and Hugo Krawczyk, HMAC came in with a simple yet elegant solution: provide robust security all while leveraging the speed and ubiquity of hash functions. Their groundbreaking paper, "Keying Hash Functions for Message Authentication," laid the groundwork for a MAC function that was three - fast, secure, and easy to implement.

The nested hashing construction is at the core of HMACs, incorporating inner and outer padding to thwart length extension attacks and other vulnerabilities that plagued earlier MAC designs. This clever approach, formalized in RFC 2104 in 1997, quickly propelled HMAC to the forefront of web security. It became everyone's go-to solution for protecting everything under the sun - from financial transactions to sensitive data. HMAC’s enduring popularity after all these years is a testament to its ingenious design and the foresight of its creators. They were able to recognize the need for a practical and robust authentication mechanism in the burgeoning digital age.

HMAC Under the Microscope

Enough with the hand-waving. Let's get down to the brass tacks of how HMAC actually operates. As we discussed earlier, it's all about nested hashing modules. But there's more to it than just slapping some padding on and calling it a day. The devil, as they say, is in the details ...or in this case - the hashing modules.

Let's start our discussion with the Keys. HMAC expects a key of a certain length. If your key is shorter than the block size of the underlying hash function (e.g., 64 bytes for SHA-256), then the key is padded with zeros up to that block size. If it's longer, it's first hashed down to a smaller size such that it fits in the block size. From there if it's smaller than the block size then it's padded up to the size. All of this is to make sure that we are dealing with a consistent key size, irrespective of what we started with.

Next comes the padding. We have two options at our disposal here - inner padding (ipad) and outer padding (opad). Both are derived from the key using a bitwise XOR operation with fixed constants: 0x36 for ipad and 0x5c for opad. Why these specific constants? Well, they are carefully chosen so as to have good bitwise difference properties. This helps in ensuring the security of the construction even if the underlying hash function has some weaknesses. Think of it as adding a layer of obfuscation. All of this is for one goal - making it harder for attackers to reverse-engineer the process.

import hmac
import hashlib

def hmac_calculate(key, message, blocksize=64):
    """Calculate HMAC using SHA-256."""

    # Ensure key and message are bytes
    if isinstance(key, str):
        key = key.encode('utf-8')
    if isinstance(message, str):
        message = message.encode('utf-8')

    # Hash the key if it exceeds the block size
    if len(key) > blocksize:
        key = hashlib.sha256(key).digest()

    # Pad the key if it's shorter than the block size
    if len(key) < blocksize:
        key = key + b'\x00' * (blocksize - len(key))

    # Create the inner (ipad) and outer (opad) padding
    opad = bytes(x ^ 0x5c for x in key)
    ipad = bytes(x ^ 0x36 for x in key)

    # Perform inner hash
    inner_hash = hashlib.sha256(ipad + message).digest()
    # Perform outer hash
    outer_hash = hashlib.sha256(opad + inner_hash).digest()

    return outer_hash.hex()

# Example usage
key = "MySuperSecretKey"
message = "Top Secret Message"
hmac_output = hmac_calculate(key, message)
print(f"HMAC: {hmac_output}")

# Verification against Python's built-in hmac library
built_in_hmac = hmac.new(
    key.encode('utf-8'),
    message.encode('utf-8'),
    hashlib.sha256
).hexdigest()
print(f"Built-in HMAC: {built_in_hmac}")
assert hmac_output == built_in_hmac, "HMAC implementation doesn't match built-in function"

# $> Python my-hmac-py
#       HMAC: a8da02b39f6144341be7b70adda46893255c6de31cadc44b90f6c9d02fb9bbac
#       Built-in HMAC: a8da02b39f6144341be7b70adda46893255c6de31cadc44b90f6c9d02fb9bbac

Now, for the start of the show: the nested hashing. First, the key is XORed with the ipad, and then the message is concatenated to the result. This whole shebang is then hashed using the chosen hash function (SHA-256 in our example). The output of this first hash becomes the input of the second hash. For the second round, the key is XORed with the opad, and then the output of the first hash is concatenated to it. Finally, this is hashed again, producing the final HMAC output.

This two-step hashing process is what makes HMAC so secure. Even if an attacker finds a collision in the inner hash, they're still faced with the outer hash, which effectively acts as a one-way function protecting the secret key. Sounds simple right? It is. A simple-er yet elegant solution.

Padding: More Than Just Fluff

As you can make out by now - the ipad and opad aren't just random voodoo. These constants (0x36 and 0x5c) are XORed with the key. Why XOR and why these specific values? Well, they're chosen such that the chances of collision the minimized, and to make sure that the inner and outer keys are different. This is done even if the original key is short or contains repeated bytes. This is very important for preventing attacks like length extension. Imagine if an attacker could append some random data to your legitimate message and then somehow calculate the new valid HMAC without knowing your key. A complete disaster, right? The padding is what helps prevent this by effectively rendering it computationally infeasible to extend the message without knowing the original key.

HMAC Security: From Street Smarts to Mathematical Muscle

Let's now turn our attention to security. In the world of cryptography, paranoia is not just a virtue; it's a damn necessity. Over the years HMAC has proven itself to earn its reputation as a robust MAC. Let's look at both the practical security considerations and the theoretical underpinnings that make it so tough to crack.

From a practical standpoint, HMAC boasts some impressive defenses:

Length Extension Attacks: Thanks to the clever use of inner and outer padding, HMAC is immune to length extension attacks. This style of attack has plauged many of the more simpler hash-based MACs. Those simple paddings effectively prevent an attacker from appending data to a message and calculating a valid MAC without knowing the secret key.
Timing Attacks: Timing attacks exploit the fact that cryptographic operations can take slightly different amounts of time depending on the data being processed. At times it's mere CPU clock cycles that are enough to give away the secrets. To mitigate this, constant-time comparison functions are essential when verifying HMACs.
Collision Resistance: HMAC inherits the collision resistance of the underlying hash function. So, if you're using a strong hash function like SHA-256 or, even SHA-3, you're in good shape. The topic is further addressed with the nested hashing construction, making finding collisions even harder.
Key Size: The bigger, the better. A general guideline is to use a key at least as long as the output of the hash function. A longer key increases the computational effort required to brute-force it, making your HMAC more resistant to attacks.

HMAC just keeps on giving. For the true crypto nerds, HMAC's security isn't just based on empirical testing and real-world observations. We also have the mighty mathematicians backing it! As long as we can make sure that the underlying hash function behaves like a pseudorandom function (PRF), formal security proofs show that forging an HMAC is computationally infeasible. A PRF outputs values that are indistinguishable from random to an attacker, even if they know the input (except, of course, for the secret key). This PRF assumption allows us to mathematically prove the security of HMAC, giving us a high level of confidence in its overall robustness.

HMAC in the Real World: Practical Considerations and Applications

Okay, we've covered the theoretical muscle and security properties of HMAC. But how does this all play out in practice? That's where it should shine right? Well, let's explore some practical considerations for implementation and deployment, along with a look at HMAC's wide range of applications.

Choosing the Right Ingredients: Selecting the appropriate hash function is crucial for HMAC's security and performance. SHA-256 and SHA-3 are generally solid choices. Goes without saying but the key management is equally critical. Store your keys securely, using robust key management systems, and always rotate them regularly. And please, for the love of all that is holy, don't hardcode your keys directly into your code!

Key Derivation: Where do those secret keys come from? Are they really secret? Key Derivation Functions (KDFs) are the answer. KDFs take a high-entropy input (like a password or a random seed) and churn out a strong, uniformly distributed key suitable for cryptographic use. Popular KDFs include PBKDF2, bcrypt, scrypt, and Argon2. Remember - the best choice is the one you choose, and then the next moment it becomes the one you didn't.

Beyond HMAC: While HMAC is the undisputed king of the MAC hill, it's not the only name in town, far from it. Other contenders like CMAC (Cipher-based Message Authentication Code) and Poly1305 offer alternatives, each with its own strengths and weaknesses. However, HMAC's simplicity, widespread availability, and extensive vetting make it a compelling default choice for many applications.

Advanced Attacks and Staying Ahead of the Curve: No cryptographic system is ever impenetrable. While HMAC is generally very secure, advanced attacks like side-channel attacks, exploiting information leakage through timing, power consumption, or electromagnetic emissions, can pose a threat that you will find difficult to design for. Vulnerabilities in the underlying hash function can also weaken HMAC's security. Staying up-to-date on the latest security research and best practices is paramount.

HMAC in Action: HMAC is ubiquitous in today's interconnected world - you may not realize it. It safeguards API authentication, ensures data integrity, secures blockchain transactions, and plays a crucial role in protocols like SSL/TLS. It's the silent guardian, the watchful protector, working tirelessly behind the scenes to keep your data safe from those pesky eavesdroppers and malicious actors.

The Bottom Line: HMAC's Enduring Legacy in Modern Security

Our deep dive into HMAC has revealed why this elegant cryptographic construction continues to dominate the authentication landscape. Its clever combination of nested hashing and key management provides robust security. It does so while remaining remarkably simple to implement. From its mathematical foundations to practical Python implementations, we saw how HMAC effectively addresses critical security challenges. This becomes crucial in domains like API authentication, blockchain transactions, and secure communications.

Stay tuned for our next article, where we'll explore SHA-3 and its revolutionary sponge construction. We'll also see why it's a perfect complement to HMAC in building comprehensive cryptographic solutions. Until then, remember that HMAC's enduring success stems from its elegant simplicity: sometimes the most effective solutions are built with simple fundamentals rather than chasing complexity for complexity's sake.

JWT Deep Dive: Beyond the Basics - Unmasking Claim Types and Their Secrets

Wed, 26 Jun 2024 18:30:00 GMT

Decoding the JWT Enigma

We've looked a fair bit at JWT's in the last few posts. I want to close this round of discussion by talking about JWTs claims. if you have used JWTs, you've danced with them in authentication flows. The real question then is - are you truly unlocking their full potential? Beyond the familiar sub and exp claims, not so far away lies a rich world of customization just waiting to be explored. We're going to crack the code on JWT claims, revealing their power for fine-grained authorization, efficient data transmission, and much more.

The Usual Suspects: Standard Claims

Before we even attempt to dive into the deep end, let's revisit the standard claims you'll encounter in almost every JWT:

iss (Issuer): Trace to the token's origin. A critical block in multi-tenant systems and when integrating with external authentication providers. Think of it as the token's return home address.
sub (Subject): The star of the show – the user or entity the token was issued for. This is usually a unique identifier.
aud (Audience): The intended recipient. This prevents token misuse by unintended services by scoping it appropriately.
exp (Expiration Time): The token's expiry date. Everything has a limited time, and the expiry claim makes sure that JWT do so as well. They prevent tokens from living forever and becoming a security liability.
nbf (Not Before): The token's activation time. Useful for scheduling access or delayed functionality. Think of it as a post-dated check.
iat (Issued At): The token's birth timestamp. Helpful for tracking its age and detecting potential issues. Like a fine wine, some tokens get better with age (just kidding!).
jti (JWT ID): A unique identifier for the specific token. Helpful in preventing replay attacks, ensuring each token is a one-time-use ticket.

These are the foundation, but the real magic happens when the capability of JWT claims are used to drive application use cases.

Claim Customization: Unleashing the Power of JWTs

Here's where things start to get exciting. JWTs allow you to define custom claims allowing you to add rich contextual information directly into the token itself. This unlocks powerful capabilities for, but not limited to use cases around authorization, personalization, and streamlined data access.

Roles and Permissions: Fine-Grained Access Control

Imagine a user with multiple roles, like "admin" and "editor." You can embed these roles directly into the JWT:

{
  "sub": "12345",
  "roles": ["admin", "editor"],
  "permissions": ["create_article", "edit_article", "publish_article"]
}

This reduces the database round-trips for authorization as now your application can directly check these claims and grant or deny access based on the user's roles and permissions. This not only improves performance but also reduces latency, all this while also reducing the load on the overall system.

User Context: Personalization and Beyond

In a multi-tenant application, include the tenant_id directly in the JWT:

{
  "sub": "12345",
  "tenant_id": "acme_corp",
  "preferences": {
    "theme": "dark",
    "language": "en"
  }
}

This not only simplifies tenant-specific logic but also allows for fine grained user experience personalization. Imagine pre-loading user preferences, UI settings, or even A/B testing flags - all within the token. Think of the last time you saw a web page load in light mode only for it to automatically switch to dark mode because you had set a preference for the dark mode. Well now your users will never have to experience it.

Data Minimization: Privacy by Design

Never ever overshare! Instead of sending the entire user profile with every request, include only the essential data relevant to the current scenario in the JWT:

{
  "sub": "12345",
  "username": "johndoe",
  "email": "john.doe@example.com",
  "profile_picture": "https://example.com/avatars/johndoe.png"
}

This minimizes the amount of sensitive data in transit which in turn helps in reducing the impact of potential breaches. To make the deal even more sweet, it's more efficient than sending large payloads with every request. This requires careful planning ahead of time to make sure everything needed for the job is availble in the JWT, and nothing more.

Walking the Tightrope: Security Considerations and Best Practices

Custom claims are powerful, but like everything powerful they come with responsibilities. Remember - never treat your JWTs like a digital junk drawer. Keep them lean, mean, and secure:

Size Matters: Large JWTs impact overall performance. Include only the essential data. For everything else - consider using a dedicated caching mechanism.
Sensitive Data Alert: Needless to say - avoid putting highly sensitive data (e.g., credit card numbers, social security numbers) directly into the JWT unless you have robust encryption in place. Think end-to-end encryption or payload encryption, as JWT claims are by themselves visible (but not verifiable).
Validation is Key: Always validate every claim on the server-side. Don't ever trust the client-provided data. Validate data types, formats, and ranges to prevent man-in-the-middle styled attacks.
Expiration Dates Are Your Friends: Use appropriate, short-lived expiration times to limit the damage for the inevitable event of token being compromised. Implement refresh tokens for long-lived sessions.

Signing Ceremony: Choosing the Right Algorithm

The security of the JWT token relies heavily on signing algorithms, or I may go as far as to say - only on the signing algorithm. These algorithms ensure the integrity as well as the authenticity of the token. Here's a breakdown of the most common choices:

HS256 (HMAC with SHA-256): Fast and efficient, uses a shared secret key. Great for internal services where key management is relatively straightforward. However, if the secret is compromised, all tokens become vulnerable. Think of one master key for all locks in the nuclear facility.
RS256 (RSA with SHA-256): More secure than HS256, uses a public/private key pair. The private key signs the token, and the public key verifies it. Suitable for scenarios where you have the use case to share the verification key publicly, such as with external APIs or third-party services. It's slower than HS256 but offers better security and the option to share responsibility.
ES256 (ECDSA with SHA-256): Provides strong security with smaller key sizes compared to RSA - a real win-win. A great alternative for resource-constrained environments. Elliptic curve cryptography makes it performant while maintaining a high level of security.

Choosing the right algorithm depends on your specific needs - security requirements, performance needs, along with key management capabilities. For highly sensitive data or external APIs, RS256 or ES256 are generally preferred. For internal services, HS256 could be a viable option given that you keep the secret ..well ...a secret. In more sensitive use cases you could use a hardware security module (HSM) for enhanced key security.

Real-World Adventures: JWTs in Enterprise Applications

Let's also look at some real-world scenarios where JWTs take the spotlight:

E-commerce: JWTs often carry product entitlements, seamlessly controlling access to premium features as well as content. Imagine unlocking exclusive content based on a user's subscription level, all from the data encoded within the JWT itself.
Healthcare: Securely transmit patient identifiers and consent information within JWTs, while maintaining integrity. Thus streamlining data access for authorized personnel while adhering to HIPAA regulations.
Single Sign-On (SSO): Enable smooth access to multiple applications within an organization thus allowing users to log in once and access various services without re-authenticating.
Microservices Communication: Secure communication between microservices using JWTs for service-to-service authentication and authorization. This enhances security and decoupling within a microservices architecture.

JWT Toolkit: Libraries and Tools

JWT is not experimental by any stretch of imagination. You don't have to figure the details out by yourself. Numerous libraries and tools simplify JWT management:

Auth0: A comprehensive platform for authentication and authorization, with robust JWT support and extensive documentation.
Firebase Auth: Another great option with built-in JWT capabilities, especially for mobile and web applications. It integrates seamlessly with other Firebase services.
Language-Specific Libraries: From Python's PyJWT to Node.js's jsonwebtoken and Java's jjwt, you'll find a library for virtually every language. These libraries provide convenient functions for creating, signing, and verifying JWTs.

The Bottom Line: JWTs Are Your Secret Weapon

JWTs while often associated with just authentication tokens, they are more than that. They're versatile tools for secure data transmission, fine-grained authorization, and efficient data management. By understanding claims, signing algorithms, and best practices, you can build more efficient, secure, and personalized applications.

Insights in Plaintext: A Deep Dive in the Sea with Blowfish

Tue, 18 Jun 2024 18:30:00 GMT

Welcome to Part 3 of our "Insights in Plaintext" series! After exploring the rise and fall of DES/3DES in our previous deep dive, we're turning our attention to Blowfish. It's a cipher that revolutionized cryptography by championing open-source principles. Think of it as that trusty old university hoodie you still keep around - while you might not wear it daily anymore, but it represents a pivotal moment in your journey. Just as DES taught us about the dangers of limited key sizes, Blowfish has its own crucial lessons about open security and algorithmic evolution. So grab your coffee, put on your nostalgia goggles, and let's dive into this game-changing algorithm that bridged the gap between legacy ciphers and modern encryption.

Schneier's Brainchild: Open Source Before It Was Cool

Ready!? Picture this: it's the early 90s. Bell bottoms are a thing, the internet is still a baby and cryptography? Well, let's just say it was a different world. DES, the reigning champ, was starting to show cracks, and proprietary algorithms, shrouded in secrecy, were the norm. Imagine a world where you couldn't peek under the hood. This is when security relied on obscurity rather than robust design. It was a bit like buying a car and not being allowed to lift the hood to see the engine. Sketchy, right?

Into this scene strides Bruce Schneier, a security guru with a rebellious streak. He believed in open cryptography and in the power of community scrutiny. He saw the limitations of closed-source security and unlike many decided to do something about it. Blowfish, released in 1993, was his answer – a fast, free, and unapologetically open-source encryption algorithm. It was a bold move, a direct challenge to the established order.

You may ask why did Schneier create Blowfish? Several factors fueled its development:

Limitations of DES: DES, with its 56-bit key, was becoming increasingly vulnerable to brute-force attacks. The US government's involvement also raised concerns about potential backdoors (never gets old, does it?).
Cost and Availability of Cryptography: Existing alternatives were often expensive and encumbered by patents, limiting access for developers and researchers. Schneier wanted to democratize cryptography, making strong encryption available to everyone.
The Philosophy of Openness: Schneier firmly believed that open scrutiny leads to stronger security. By exposing the algorithm to public review, he invited the entire cryptographic community to find and fix potential weaknesses.

Open source was not the only selling point of Blowfish; it was designed for speed and simplicity. Schneier prioritized performance, especially in software implementations. This made it suitable for a wide range of applications, from securing email communications to protecting hard drives. Its compact size made it particularly attractive for resource-constrained environments, a crucial factor in the early days of computing.

It would be an understatement to say that the impact of Blowfish's open-source nature was profound. It encouraged collaboration and peer review, which in turn ended up accelerating the advancement of cryptographic research. It demonstrated that security through obscurity is a fallacy, and that open algorithms, subjected to rigorous analysis, can be far more secure. Blowfish became a textbook example of how to do cryptography right. This ended up influencing the design and development of countless ciphers that followed it.

Under the Hood: Gears and Gadgets

Alright, out from history, into the maths class. Let's roll up our sleeves and get into the nitty-gritty of how Blowfish actually works.

Variable Key Length (32-448 bits): Yes! This flexibility allows you to fine-tune the security level to match your needs and resources. A shorter key might be acceptable for low-risk scenarios or resource-constrained devices, while a longer key provides greater protection against brute-force attacks. Remember that with longer key length comes more sophisticated key-management responsibility.
64-bit Block Size: This block size, while efficient for its time, is now a major security vulnerability due to the Birthday Paradox. It's a trade-off between performance and security that hasn't aged well.
Feistel Network This is where the real magic happens. Imagine a cryptographic dance floor where data is constantly being paired, shuffled, and transformed. The Feistel network is a series of rounds, typically 16 in Blowfish, where the input data is split into two halves. One half undergoes a transformation with the key, and the result is then XORed with the other half. Then the two swap places, and the process repeats. This repeated shuffling and XOR operations end up creating a strong avalanche effect wherein relatively small changes in the input lead to significantly large and measurable changes in the output.
Key Scheduling Why limit yourself to a small key eh? Key scheduling takes your initial key which is generally in the range of 32-448 bits, and expands it into a much larger array of subkeys. This array could be a whopping 4168 bytes! These subkeys are then used in the rounds of the Feistel network, injecting key material throughout the encryption process. The key scheduling itself involves complex operations using the digits of Pi and iterative encryption of a zero block. This seemingly random process was put in place to ensure that the subkeys are thoroughly derived from the original key, making it difficult to reverse-engineer the key from the subkeys.
P-array and S-boxes The P-array is a set of 18 32-bit subkeys used in the key scheduling process and during encryption. The S-boxes are four substitution boxes, each containing 256 32-bit entries. These S-boxes play a fundamental role in introducing non-linearity into the encryption process. They act as lookup tables, mapping 8-bit input values to 32-bit output values. The contents of both the P-array and the S-boxes are initialized using the digits of Pi and then modified using the key material during the key scheduling process. Why Pi? Maybe because it's a number that never stops giving!

Let's Get Our Hands Dirty: Code Time!

Here's a slightly more detailed look at the F-function, incorporating the S-boxes. Again, this is for educational purposes only, and not to be used even in "dev-envs":

def f(x, s_boxes):
    # Split the 32-bit input into four 8-bit bytes
    x1 = (x >> 24) & 0xFF                               # Extract the bytes individually
    x2 = (x >> 16) & 0xFF
    x3 = (x >> 8) & 0xFF
    x4 = x & 0xFF

    # The core F-function calculation
    result = (s_boxes[0][x1] + s_boxes[1][x2]) % 2**32  # Modulo for 32-bit overflow
    result ^= s_boxes[2][x3]                            # XOR operation
    result = (result + s_boxes[3][x4]) % 2**32

    return result

This function starts with a 32-bit input x and the pre-initialized S-boxes. It splits x into four 8-bit bytes, which are then used as indices in the S-boxes. The resulting values from the S-boxes are combined using addition and XOR operations, with modulo operations ensuring the result stays within 32 bits. This seemingly simple function plays a crucial role in creating complex transformations within the Feistel network.

The Birthday Paradox: It's Not About Cake

Unfortunately, the Birthday Paradox isn't about some shared cake; it's about the surprising probability of shared birthdays. It deals with the probability that in a room of just 23 people, there's more than a 50% chance that two share a birthday! That's the real surprise right!? This counterintuitive result stems from the fact that we're looking for any two people to match, not one specific birthday. With more people, the number of possible pairs grows faster and so does the overall probability. This same principle applies to cryptography. Blowfish's 64-bit block size means there are 2^64 possible ciphertext blocks. While this seems like a HUGE number, the Birthday Paradox tells us that collisions become likely after encrypting roughly the square root of that, around 2^32 blocks. That's just 4GB of data or the images from your last vacation. These collisions can leak information about the plaintext, weakening the encryption.

Speed Demons: Blowfish vs. The New Kids

Now to the real question - how does Blowfish stack up against the competition?

Cipher	Key Setup	Encryption Speed	Block Size	Key Size	Security
Blowfish	Slow	Fast (software)	64 bits	32-448 bits	Good
Twofish	Moderate	Fast	128 bits	128-256 bits	Excellent
AES	Fast	Very Fast	128 bits	128, 192, 256	Excellent

Blowfish's key setup is painfully slow. AES takes the top spot for raw encryption speed, thanks to widespread hardware-level acceleration. Twofish sits comfortably in the middle. This comparison is here to paint the general idea, and as you can imagine - specialized hardware will always speed things up.

Beyond the Basics: Modes of Operation

Choosing the right mode of operation is very important. Just like you wouldn't use a hammer to screw in a lightbulb, the wrong implementation can make even the best cryptographic algorithm fail. Here's a quick rundown of the options we have:

ECB (Electronic Codebook): The simplest but (to no one's surprise) the least secure. Encrypting the same block always produces the same ciphertext, making patterns visible.
CBC (Cipher Block Chaining): Each block is XORed with the previous ciphertext block before encryption. This hides patterns but requires an Initialization Vector (IV) to get the first block rolling.
CTR (Counter): Turns a block cipher into a stream cipher. Fast and allows for parallel processing.
GCM (Galois/Counter Mode): Provides both confidentiality and authentication. A popular choice for modern applications.

Each mode has its trade-offs. Choose wisely!

The Quantum Menace: Is Crypto Doomed?

Quantum computing is the elephant in the room, only that it's 20x the size of a normal elephant. While still in its early stages, it has the potential to break many of our current state-of-the-art cryptographic algorithms. This includes Blowfish. Shor's algorithm, a quantum algorithm, could efficiently factor large numbers and compute discrete logarithms, rendering RSA and ECC useless. Symmetric-key algorithms like Blowfish are relatively less vulnerable, but they are prone to brute force attacks. And boy are quantum computers fast. The key sizes of these algorithms would need to increase significantly to resist brute-force attacks from quantum computers. It's like preparing for a hurricane – we don't know exactly when it will hit, but we need to start building our defenses now. Post-quantum cryptography is an active area of research, exploring new algorithms that can withstand the quantum onslaught.

Blowfish Today: A Legacy System's Best Friend?

Blowfish isn't the shiny new toy on the shelf anymore. AES and Twofish (Blowfish's beefier successor) have now become the go-to choices. But Blowfish still lurks in legacy systems, especially in embedded ones with limited resources - maybe one day our dishwashers will have hardware to make them smart. If you're dealing with one of these, migration is a very delicate dance. You have to balance security risks with the cost and disruption of an overhaul. It's like renovating an old house - you want to modernize it in place, but you also definitely don't want the whole thing to collapse.

The Legacy of Open Cryptography

Blowfish stands strong as a pivotal milestone in cryptographic history. It was successful in marking the transition from closed, proprietary algorithms to open, community-reviewed security solutions. It's the bases of trusted security today. While AES and Twofish have superseded it for most of the modern applications, Blowfish still holds it's place. It's revolutionary approach to open-source security and its elegant design principles continue to influence how we develop and evaluate encryption algorithms today. Its journey from cutting-edge innovation to legacy system staple offers crucial insights into the evolution of security standards and the importance of forward-thinking design.

Ready to explore more modern encryption algorithms? Join us in the next article where we'll examine HMAC and its crucial role in modern authentication systems.

Insights in Plaintext: Decrypting the Legacy of DES and 3DES

Mon, 06 May 2024 18:30:00 GMT

Welcome to Part 2 of our "Insights in Plaintext" series! After laying the groundwork in our introduction to cryptography, we're diving deep into our first case study: DES and 3DES. These algorithms once reigned supreme but have now been standing as fascinating monuments to the never-ending march of technical progress. Not so long ago, these ciphers were the guardians of sensitive data all over the world. Today they serve as crucial cautionary tales about complacency in cryptography. So, grab your virtual (or real) coffee mug and let's explore how these pioneering encryption standards shaped the security landscape we covered in Part 1.

From Bell Labs to Bit Rot: A History Lesson

Our story starts in the 1970s, a decade of groovy music, now questionable fashion choices, and the birth of modern cryptography. The powerhouse IBM, in its infinite wisdom (or perhaps with a nudge from the NSA), developed the Data Encryption Standard (DES). YES! It was the IBM. DES was a block cipher designed to protect sensitive government and commercial data. It was designed with a 56-bit key, a seemingly insurmountable barrier at the time, which gave a false sense of security. Remember this was the time when 800Kb floppy disks were state-of-the-art. Banks, eager to safeguard their customers' financial transactions, embraced DES with open arms. Little did they know that the seeds of its demise were already sown.

What followed was the rise of distributed computing and Moore's Law. The technological advancement gradually chipped away at DES's armor. What once seemed unbreakable began to show cracks. The final blow came in 1998, with the infamous EFF's Deep Crack project, a crowdsourced effort to break DES. This sent shockwaves through the security community. One thing became painfully clear: Fort Knox was made of cardboard, and the chihuahua was asleep on the job.

The scramble for a solution led to the creation of 3DES, or Triple DES. It was a band-aid attempt to bolster DES's security without completely reinventing the wheel. Why? Because we didn't have the time to do so back then (more on this later). 3DES, as the name suggests, applies the DES algorithm three times, effectively increasing the key size to 112 or 168 bits. It was always a temporary fix, never a cure. An intermediate solution to buy us more time to move to something else. While 3DES offered improved security compared to its predecessor, it also ended up inheriting DES's fundamental limitations. All of this while adding the burden of significantly increased computational overhead.

Under the Hood: Gears and Gizmos

DES, at its heart, is a Feistel network, a structure that involves breaking up the data block into two halves and then passing them through a series of intricate transformations. While it may look so this isn't just some random shuffling of bits; it's a carefully orchestrated ballet of cryptographic operations designed to create confusion and diffusion.

Let's break down a single round of DES (remember, there are 16 of these bad boys):

Initial Permutation (IP): The 64-bit input block is permuted according to a predefined table. This shuffles the bits around, creating the initial state for the upcoming round.
Key Transformation: A 56-bit key (derived from the original 64-bit key with parity bits removed) is used to generate 16 subkeys, one for each of the 16 rounds. This involves further permutations and shifts.
Splitting: The permuted block is split into two 32-bit halves, Left (L) and Right (R).
F-function (The Heart of DES): This is where the magic happens. The R half undergoes a series of operations:
- Expansion Permutation (E): The 32-bit R half is expanded to 48 bits, duplicating some bits according to a specific permutation table. This increases the diffusion effect.
- Key Mixing: The expanded R half is XORed with the round key. This introduces the key's influence on the encryption process.
- S-box Substitution (The Secret Sauce): The 48-bit result is then divided into eight 6-bit chunks. Each chunk is fed into a corresponding S-box (aka Substitution box). Each S-box is a 4x16 lookup table. The 6-bit input selects a row and column in the S-box, and the corresponding 4-bit value is the output. This non-linear substitution is crucial for DES's security.
- Permutation (P): The 32-bit output from the S-boxes is permuted again, further scrambling the bits.
XOR and Swap: The output of the F-function is XORed with the L half. Then, the L and R halves are swapped. This completes one round.
Repeat: Steps 3-5 are repeated for 16 rounds, using a different subkey for each round.
Final Permutation (IP^-1): After 16 rounds, the two halves are combined, and with the resulting 64-bit block a final permutation is carried out. This is an inverse of the initial permutation (IP). This finally produces the ciphertext.

3DES is simply three applications of DES (encrypt-decrypt-encrypt). It inherits this complex structure but applies it multiple times. Increasing the computational cost but also (theoretically) the security. To be honest, I tried to float 5DES, and 7DES as even more secure versions of 3DES, but the community just rejects them every time.

This intricate dance of permutations, substitutions, and XOR operations makes DES a seemingly complex yet simple algorithm. The non-linearity introduced by the S-boxes is critical for its security, but the relatively small key size and block size ultimately led to its downfall. Understanding this underlying structure is key (get it!? I just had to!) to acknowledge the limitations of DES which were also the motivations behind the development of more robust ciphers like AES.

Show Me The Code!

Using DES in production today is about as advisable as wearing a "kick me" post-it note through a high school hallway. But for educational purposes only, here's a snippet of Python code using the pycryptodome library. Remember that you shouldn't be doing this:

def simplified_f_function(right_half, subkey):
    """A simplified F-function (XOR with subkey)."""
    return right_half ^ subkey

def simplified_des_round(left_half, right_half, subkey):
    """A simplified DES round."""
    new_right = left_half ^ simplified_f_function(right_half, subkey)
    new_left = right_half
    return new_left, new_right

def simplified_des(plaintext, key, num_rounds=16):
    """A simplified DES implementation."""
    # Dummy subkey generation (in real DES, this is much more complex)
    subkeys = [key >> i for i in range(num_rounds)]  # Just shifting the key

    # Initial "permutation" (just splitting the plaintext for this example)
    left, right = plaintext[:len(plaintext) // 2], plaintext[len(plaintext) // 2:]

    # Rounds
    for i in range(num_rounds):
        left_int, right_int = int(left, 2), int(right, 2)
        left_int, right_int = simplified_des_round(left_int, right_int, subkeys[i])

        # Convert back to binary strings with padding
        left = bin(left_int)[2:].zfill(len(plaintext) // 2)
        right = bin(right_int)[2:].zfill(len(plaintext) // 2)

    # Final "permutation" (combining the halves)
    ciphertext = left + right
    return ciphertext

# Example usage
plaintext = "1010101001010101"  # Example 16-bit plaintext
key = 0b1111000011001010        # Example 16-bit key (highly insecure)
num_rounds = 4                  # Reduced number of rounds for illustration

ciphertext = simplified_des(plaintext, key, num_rounds)
print(f"Plaintext:  {plaintext}")   # 1010101001010101
print(f"Ciphertext: {ciphertext}")  # 1000100000000101101001000011110

# ... and decryption

Again, this is a museum piece, not even a local-testing-ready implementation.

Cracking the Code: From Brute Force to Sweet Treats

The real wake-up call came with the EFF's Deep Crack project. It demonstrated that brute-forcing a 56-bit key was no longer a theoretical exercise but a practical reality with the computational power available at the time. This, combined with advancements in cryptanalysis, like differential cryptanalysis, further weakened DES's defenses.

Wait a minute - what's this differential cryptanalysis all about? Imagine you're a codebreaker, and you don't know the key, but you can encrypt chosen plaintexts. Not so difficult to imagine right? Practically every system today. Differential cryptanalysis exploits the fact that changes in the input (plaintext) lead to predictable differences in the output (ciphertext), even without knowing the key. With this knowledge, by carefully analyzing these differences, or "differentials," across multiple chosen plaintexts and their corresponding ciphertexts, you can deduce information about the key itself. Neat!

In the context of DES, differential cryptanalysis targets the S-boxes, those lookup tables that are supposed to be the source of DES's cryptographic strength are after all a static lookup table. The attack works by identifying pairs of plaintexts with specific differences and observing the resulting differences in the intermediate values after passing through the S-boxes. By analyzing these differential patterns, cryptanalysts can deduce information about the key bits used in each round of DES.

While differential cryptanalysis doesn't instantly break DES, it significantly reduces the computational effort required to find the key. It gives you a cheat sheet for a really hard exam – you still need to put in some effort, but you have a significant advantage from having nothing. This type of attack, along with the ever-increasing power of computers, ultimately led to DES's demise. It also highlighted the need for ciphers with a much stronger resistance to cryptanalysis.

3DES, while offering a larger key space, wasn't entirely immune to these types of attacks, although the triple encryption made them significantly more difficult. The Sweet32 attack, exploiting a birthday paradox-style collision vulnerability, demonstrated that even triple encryption could be broken with enough effort (and a whole lot of birthday cakes, apparently). You can delve into the details of this attack here. For the official pronouncements, refer to NIST SP 800-67.

Tale of the Tape: DES vs. 3DES vs. AES

Feature	DES	3DES	AES
Key Size	56 bits	112/168 bits	128/192/256 bits
Block Size	64 bits	64 bits	128 bits
Security	Weak	Deprecated	Strong
Performance	Fast (but insecure)	Slow	Fast and Secure
Mode of Operation	Limited (ECB is a no-no)	More flexible, but still...	Wide range of secure modes

Legacy Lament: The Ghosts of Crypto Past (and Present, unfortunately)

DES and 3DES, despite their known vulnerabilities and exploits, continue to haunt the digital landscape even today. Migrating away from these cryptographic relics can be a herculean task, often fraught with challenges.

The Great Crypto Migration: Why It's Harder Than It Sounds

Imagine trying to replace the engine of a car ...that's moving ...and has a leaking gas line. That's essentially what migrating away from DES/3DES in a legacy system feels like. Here's a breakdown of the pain points:

Tight Coupling: These old ciphers are often deeply integrated with the application logic, woven into the very fabric of the system. It's not a simple matter of swapping libraries; it can require extensive code refactoring.
Dependency Hell: Legacy systems are notorious for their complex and often undocumented dependencies. Changing one thing can trigger a cascade of breakages, turning a simple migration into a descent into the abyss of compatibility issues.
Testing Nightmares: Answer this - how do you thoroughly test a cryptographic migration without bringing your entire system to a screeching halt? Comprehensive testing is crucial, but it can be incredibly time-consuming, resource-intensive, and a logistical nightmare, often involving late nights, copious amounts of caffeine, and the occasional existential crisis - at times all at once.
The "If It Ain't Broke..." Mentality: Who are we kidding !? Sometimes there's resistance to change, especially when the system appears to be functioning (ignoring the ticking time bomb). Convincing stakeholders to invest time, resources, and money in migration can be like pulling teeth.
Cost: All of the above translates to one thing: money. Migrations are expensive, and justifying the cost to management can be a challenge, especially when there's no immediate, tangible benefit (besides, you know, ...preventing a potentially catastrophic security breach).

The Legacy Lives On: Lessons for Modern Cryptography

If nothing else, our journey through DES and 3DES reveals a timeless lesson - one about cryptographic evolution. These algorithms demonstrate how even the most trusted security solutions become vulnerable over time. It does a great job at reinforcing the principles of key size importance and algorithm agility that we have discussed in Part 1. As we look ahead to our next deep dive into Blowfish, do remember that today's gold standard could be tomorrow's cautionary tale. If you're still running systems with DES or 3DES, consider this your wake-up call. You only have limited time to start planning that migration to modern algorithms. Your future self (and your security team) will thank you.

Ready to explore more encryption algorithms? Join me in the next article where we'll dissect Blowfish and see how it addressed some of DES's shortcomings while introducing innovations of its own. Do share your thoughts and DES/3DES migration war stories!

JWT Revocation: Taming the Stateless Beast

Wed, 01 May 2024 18:30:00 GMT

Let's just admit it, we've all been seduced by the siren song of stateless authentication with JSON Web Tokens (JWT). No more dealing with sticky sessions, no more multiple database lookups for every request… bliss,right? That's until you need to revoke a token. That's when it feels less like bliss and more like stepping on a LEGO brick ...barefoot ..at 3 AM. The promise of statelessness, while alluring, introduces a unique challenge no one asked for: how do you effectively revoke access when the very nature of your architecture eschews persistent state?

In a much more comfortable, traditional, stateful world, revocation is relatively straightforward. You simply invalidate a session on the server, and boom, the user is locked out. But in the ephemeral realm of JWTs, things get a just a little bit trickier. The token is self contained, i.e. it has all the information needed for authentication. This means the server doesn't inherently "remember" who has access. So, how do we wrestle with this beast of revocation in a stateless world? Let's dive into the mosh pit of strategies, evaluate them and see what comes out alive.

The Usual Suspects (Standard Approaches)

Blacklisting: This is the "if it ain't broke, don't fix it" approach, albeit with some duct tape and good old-trusty WD-40. You start by maintaining a list of revoked tokens, often in a database or distributed cache. Then you check against it on every single request. Simple in concept, but a potential scalability nightmare. Imagine the latency implications of checking against a list containing thousands of revoked tokens, if not millions. Furthermore, managing this ever-growing list introduces its own set of operational headaches. You also have to flush the expired token from this list from time to time. All in all, While suitable for smaller applications with a relatively low revocation rates, blacklisting quickly becomes unwieldy in enterprise environments. From a security standpoint, it's reactive rather than proactive which involves dealing with the aftermath of a potential breach instead of preventing it in real-time. Furthermore, a failure of system leads to token being valid, which means it's not failsafe in any sizable production environment.
Short-lived Tokens: This strategy embraces the ephemeral nature of JWTs and cranks it all the way up to eleven. By issuing tokens with very short expiry times (e.g., a few seconds to minutes), the need for explicit revocation diminishes. Compromised tokens become less of a threat because they expire quickly anyway. This time-to-live can be tuned according to the use case and what time seems acceptable for a stray token to be active. However, this introduces the overhead on the systems wherein frequent token refreshes increases the load on your authorization servers. It's a delicate balancing act, just a tad bit simpler than trying to find the perfect water temperature in a shared shower. Security-wise, it minimizes the window of opportunity for attackers, but requires careful consideration of the refresh token mechanism itself. If you did not guess it already - this mechanism itself becomes a new attack vector if not implemented securely.

Scaling Shenanigans

Token Introspection: Shifting gears from client-side checks to a more predictable server-side validation, token introspection involves querying an authorization server to determine a token's validity. Instead of maintaining a cumbersome blacklist, the app servers simply ask the authorization server, "Is this token still good?". This centralizes revocation management while also providing real-time validation. On the flip side, it places a significant burden on the authorization server. Think of it as hiring a highly efficient but also super expensive bouncer for your API party – they'll keep the riff-raff out, but you'll need deep pockets to keep them on the payroll. This approach is better suited for scenarios where real-time revocation and centralized control are paramount to everything else.
Refresh Token Rotation: This strategy adds a layer of indirection by introducing two sets of tokens - refresh and access tokens. Access tokens remain short-lived, while refresh tokens have a relatively longer lifespan. When an access token expires, the client uses the already issued refresh token to obtain a new access token. This can also be considered as refreshing a token pair, hence refresh token. More importantly, the old refresh token is invalidated during this process. This sets up mechanisms for indirect revocation i.e. if a refresh token is compromised, only future access is blocked thus limiting the blast radius. It's like changing the locks after losing your keys – a bit of a hassle, but seems reasonable than having your house ransacked. This approach strikes a good balance between security and usability, while limiting the impact of compromised tokens without constantly interrupting user sessions.
Database Lookup (with Caching): While seemingly contradictory to the stateless paradigm we started with, database lookups, when combined with very aggressive caching strategies, can provide a robust and performant revocation solution. Token's validity is stored in a database, but a distributed cache like Redis acts as a first line of defense, significantly reducing database hits. This approach offers a good balance between security and performance, providing real-time revocation without stressing out your database. Think of it as having a well-organized cheat sheet for your exam – much faster than flipping through the entire textbook and gives you the answers you need. However, this requires careful consideration of cache invalidation strategies to ensure consistency between the cache and the database. The value of still using JWT here lies in the fact that the tokens have other metadata like user details, preferences etc which reduce DB roundtrips.

Code (Example: Refresh Token Rotation in Go)

// ...Import packages

// Define token claims
type Claims struct {
	UserID uint `json:"user_id"`
	jwt.RegisteredClaims
}

func generateRefreshToken(userID uint, secretKey []byte) (refreshToken string, err error) {
	// Start by creating refresh token
	refreshClaims := &Claims{
		UserID: userID,
		RegisteredClaims: jwt.RegisteredClaims{
			ExpiresAt: jwt.NewNumericDate(time.Now().Add(7 * 24 * time.Hour)),
			Issuer:    "my_important_issuer",
			ID:        "unique_id_for_tracking",
            // ...other claims
		},
	}
	refreshToken, err = jwt.NewWithClaims(jwt.SigningMethodHS256, refreshClaims).SignedString(secretKey)
	return
}

func main() {
	secretKey := []byte("my_super_secret_key")
	userID := uint(123)

	refreshToken, err := generateRefreshToken(userID, secretKey)
	if err != nil {
		fmt.Println("Error generating token:", err)
		return
	}
	fmt.Println("Refresh Token:", refreshToken)

	// ... (Code to store and invalidate refresh tokens would go here)
}

Pushing the Boundaries: Emerging Approaches

Push-based Revocation: Cue with WebSockets. Imagine proactively pushing revocation notifications to your services instead of constantly querying from the clients. This minimizes the reliance on checking lists or managing querying endpoints, offering near real-time revocation. The approach is still relatively new and requiring more sophisticated infrastructure, it holds significant promises in terms of enhanced security and performance.
- WebSockets: Setup a persistent, bi-directional connection between your authorization server and your resource servers. When a token is revoked, the authorization server pushes a notification over the WebSocket in near-real time. This notification instruct the resource servers to invalidate the token in their local cache. This approach provides the most responsive revocation mechanism, significantly reducing the window of vulnerability. However, it requires managing persistent connections, a cache on the resource servers and also handling of potential disruptions.
- Server-Sent Events (SSE): We don't really need a bi-directional communication channel and this is where SSE comes in. It's a lighter-weight alternative to WebSockets. SSE allows the authorization server to push updates to resource servers. This is ideal for scenarios where real-time revocation is desired, but the complexity of WebSockets is not called for. SSE provides a unidirectional communication channel, making it simpler to implement, maintain and debug.
- Challenges: Implementing a push-based revocation introduces complexities related to scalability as well as reliability. Your messaging infrastructure should be able to handle the volume of revocation events, and resource servers need to process these events efficiently. Robust error handling and retry mechanisms are essential to account for the inevitable network interruptions. While not a trivial undertaking, the potential benefits in terms of responsiveness and reduced latency make this an area worth exploring for applications demanding high-security and high-performance applications. Imagine the possibilities!

The Good, the Bad, and the Ugly (Trade-offs) -

Strategy	Pros	Cons	Security Implications
Blacklisting	Simple to implement	Doesn't scale well, potential latency issues	Reactive, relies on identifying compromised tokens after potential misuse
Short-lived Tokens	Reduces impact of compromised tokens	Increased token refresh overhead	Minimizes attack window but doesn't eliminate risk
Token Introspection	Centralized revocation management	Increased load on authorization server	Real-time validation, reduces risk of stale data
Refresh Token Rotation	Limits damage from compromised refresh tokens	Requires careful management of refresh tokens	Good balance, limits damage without constant interruptions
Database Lookup (with caching)	Balance of security and performance	Relies on caching infrastructure	Effective with proper cache invalidation

Choosing the Right Strategy: A Balancing Act

Choosing the right JWT revocation strategy is not a decision for the faint of heart. It requires carefully balancing security requirements, performance goals, all while staying within the budget constraints. Consider the following factors:

Scale: How many users and tokens will your system handle?
Security Requirements: How sensitive is the data you are protecting?
Performance Goals: What are your acceptable latency targets?
Budget: How much can you invest in revocation infrastructure?
Development Resources: What expertise and resources do you have available?

For smaller applications with less stringent security requirements to deal with, short-lived tokens or blacklisting might be enough. For much larger, enterprise-level applications with high-security needs, token introspection, refresh token rotation, or database lookup with caching are more suitable. Push-based revocation is an emerging approach which is gaining popularity because it offers promising performance benefits but requires more specialized expertise and infrastructure.

Bottom Line

Sadly but to no one's surprise - there's no silver bullet for JWT revocation in stateless environments. The best approach depends on a nuanced understanding of your specific needs and scale of operations. Carefully consider the trade-offs, balancing security requirements, performance goals, and budget constraints. Make sure that you get creative and combine approaches – a hybrid strategy might be the perfect fit. And remember, a well-designed revocation strategy can be the difference between a secure system and a security nightmare.

What's your go-to strategy? Share your thoughts and experiences. Let's learn from each other. I'm especially interested in hearing from folks who've implemented push-based revocation in production!

JWTs: Beyond the Buzzwords - What You Really Need to Know

Tue, 16 Apr 2024 18:30:00 GMT

Let's talk Json Web Tokens aka JWT, shall we? Not just a surface level skim, but a much needed deep dive. You've probably already worked with them, and maybe even fought a JWT-fueled fire drill it the past. But are we really doing our bit in squeezing all the juice out of these little JSON nuggets? Let me tell you something ...let me tell you something - there's a whole damn universe inside a JWT just waiting to be explored and we are going to do just that!

Decoding the Damn Thing

Think of a JWT like a self-contained, digitally signed and verifyable envelope. It's all about secure, portable data for your applications. Three parts, each separated by dots: Header, Payload, and the Signature. Sounds simple enough, right? Well, to no one's surprise the devil's in the details. Let's peel back the layers and explore them.

Anatomy of a JWT

Here’s a JWT broken down, piece by piece. We'll get into what each part really means:

// Header (Base64Url encoded)
{
  "alg": "HS256",       // Specifies the algorithm
  "typ": "JWT"          // Just letting everyone know it's a JWT - standard practice
}

.

// Payload (Base64Url encoded) - This good stuff
{
  "sub": "user123",     // Subject - who this token is about
  "name": "John Doe",   // User's name - or whatever you want to store
  "exp": 1678886400,    // Expiration time - crucial for security
  "roles": ["admin", "editor"] // User roles - for authorization magic
}

.

// Signature (Base64Url encoded) - The cryptographic seal of approval
HMACSHA256(
  base64UrlEncode(header) + "." +
  base64UrlEncode(payload),
  your-256-bit-secret // Keep this REALLY secret - like your Netflix password
)

Decoding a Real JWT

Enough of the theory! Let's get practical. Here's an actual encoded JWT as it would appear in the wild (don't worry, the secret's fake):

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

You can decode this using quite a few standard library like jsonwebtoken in Node.js (version 16.x.x as of this writing):

const jwt = require("jsonwebtoken");

const token =
  "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c";

try {
  const decoded = jwt.verify(token, "your-256-bit-secret");
  console.log(decoded);
} catch (err) {
  console.error("Invalid token:", err);
}

Signing Ceremony: Choosing Your Cryptographic Weapon Wisely

The beauty of JWT is in the signing algorithm. It is very critical and not the area where you slack off. It’s one of assets in your digital fortress against token forgery. HS256 (HMAC with SHA-256) is relatively fast while also using a shared secret. This is great for internal systems where you control both the ends. But if that secret ever makes it's way out of the walled garden, you're toast. As an alternative, RS256 (with SHA-256) makes use of a public-private keyspair. It bring in much stronger security and thus making it ideal for external APIs as well as distributed systems. This however comes with a performance cost of working with RSA.

Choosing the right algorithm is always a balancing act – security versus speed. There's never a one-size-fits-all, especially when the topic at hand is security. You have to consider your applications context in the larger ecosystem, your exposure, the application's blast radius, the target audience and their willingness to invest into breaking your mechanisms. Ask yourself - are you dealing with highly sensitive data? What is your performance budget? Are you ready to spend the next 7 weekends looking at signatures?

JWT Superpowers: Beyond Just Simple Authentication

While JWTs are often just used as digital bouncers checking IDs at the door but they are much much more than just that. Fine-grained authorization? Let's go! You can simply embed roles and permissions directly in the payload. Skip all that redundant database trips. Have a need for single sign-on (SSO)? JWTs can make that happen without even a sweat. Letting users slide between services without much friction. Do I hear - Auditing? logging? Inspect theJWT claims to track user activity like a hawk.

Performance Anxiety: Taming JWTs at Scale

I hate to do this but there's bad news as well - JWTs do end up adding overhead and this overhead is not negligible. At scale, that overhead can become a real pain in the app. Keeping your payloads concise and to the bare minimum can make a really big difference. Think about how frugally you pack for a trip - only the essentials and not even a napkin more. Every extra byte eventually adds up. Signature verification is computationally expensive, and so caching them appropriately can be your friend. With that now you have an additional caching layer to deal with. And remember, while statelessness has it's own perks, it also makes revocation a lot more tricky. Short expiration times and a blacklist cache at the host of more computational power can help you sleep at night, that is if at all in the first place.

JWT vs. The World

JWTs, while fun aren’t the only name in town. Opaque tokens are much simpler to use and revocable by design but require server-side storage. We also have the good old session cookies which are easy to manage but can struggle with scalability in any size of distributed workloads. How can we forget the API keys which while great for service-to-service communication, lacks the granular control tha today's complex app need. In the end, choosing the right authentication mechanism depends on your application's specific needs and the enviornment that it'll operate in. It's always agood idea to not follow the hype. Take the day off, get a cup of cooffee and think it through.

JWT in the Wild: Tales from the Trenches

Case Study 1: The Exploding Payload: I once inherited a service that crammed everything under the sun into the JWT payloads - think user data, preferences, the kitchen sink. While it worked great initially, even giving a false sense of security that the developers have reduced the DB roundtrips and with that improved the users experience. But as the tale goes - the application scaled and these very massive tokens crippled performance. Network calls became sluggish, users complained, costs skyrocketed. Lesson learned: keep payloads lean and mean!

Pitfall 1: The "none" Algorithm Nightmare: Let me be very clear here - never, ever, ever ever, ever ever ever ever even think of using the "none" algorithm! Not even when "testing on the local", definately not on the "test env". It disables signature verification altogether, making your JWTs as secure as a screen door on a submarine. While a rookie mistake, I’ve seen it happen in real customer facing apps. Don't be that person for a change.

Wielding JWTs Effectively

JWTs are powerful tools - no questions on that. But they require some careful considerations. Understand their strengths and weaknesses, and only then choose the right signing algorithm for your specific security and performance needs. Always be mindful of performance, especially at scale. When used correctly, they can enhance your application's security and user experience. Used incorrectly, they can be a source of not only headaches but also the even more expensive - vulnerabilities. So choose wisely, my friend.

What are your experiences with JWTs, especially at scale? Do share them! Let's learn from each other.

Insights in Plaintext: Deep Dives into Cryptographic Algorithms

Mon, 01 Apr 2024 18:30:00 GMT

Gear up! We're about to embark on an exciting journey through the intricate maze that is cryptography! This is Part 1 of our "Insights in Plaintext" series. In here where we'll strip away the complexity and them dive deep into the algorithms that keep our digital world secure. My goal is to explore these algorithms with you. I'm just curious about how our data stays safe online. Hopefully this series will arm you with practical knowledge about encryption - from battle-tested classics to quantum-ready innovations. Let's kick things off by going over the fundamentals and setting the stage for our deep dives into specific algorithms in upcoming articles.

The Wild West of Data Security

Let's be honest for a second, the digital world is a wild west jungle. Data breaches, identity theft, online snooping – it's a freakin' free-for-all out there. In 2023 over 32% of digital attacks were intended for data theft and leak (Source: IBM Security X-Force Threat Intelligence Index 2023). That's why encryption is no longer a nice-to-have line item, but a damn necessity. Consider it your data's very own personal bodyguard, decked out in Kevlar and ready to rumble at a moment's notice. From your grandma's cat videos to your company's top-secret cola recipe, everything has value to someone, and encryption is the key to keeping it in safe hands ...or hard drives.

Encryption 101: Ciphers, Keys, and the Quantum Threat

At its core, encryption essentially scrambles your data using a secret code (a cipher) and a key, much like a lockbox. The cipher is the locking mechanism, and the key is what opens it. Only those with the correct key can unscramble the data. We have two main types:

Symmetric Encryption: Uses the same key for both encryption and decryption. While it's fast and efficient, key security is paramount. Examples include AES (Advanced Encryption Standard), the current gold standard, and older algorithms like DES and 3DES.
Asymmetric Encryption: Uses two keys: one for encryption (a public key) and one for decryption (a private key). It's like a mailbox with a public slot and a private key to control who can open it. While it's slower in practice it's incredibly secure for key exchange and digital signatures. RSA (Rivest–Shamir–Adleman) and ECC (Elliptic Curve Cryptography) are prime examples.

Then there's Hashing, a one-way function transforming data into a unique, fixed-size string (a hash). This is crucial for verifying data integrity. Any change in the data alters the hash in a significant manner. SHA-3 (Secure Hash Algorithm 3) is a leading hashing algorithm today. HMAC (Hash-Based Message Authentication Code) uses hashing for authentication.

Now, a word of caution. While these methods are strong today, quantum computing is just around the corner. Quantum computers, with their immense processing power, could potentially break widely used encryption algorithms at play today. The good news? The industry is developing quantum-resistant cryptography. We'll discuss this more later in the series.

Decrypting the Future: What's Coming Next

This series will arm you with the knowledge you need to navigate the complex landscape of encryption. Here's a sneak peek at what's coming up:

DES/3DES: The Golden Oldies (and Why They're Not So Golden Anymore): We'll explore these once-dominant algorithms, and examine their drawbacks in today's age.
Blowfish: A Fast Fish in a Big Pond: Discover the strengths and limitations of this speedy algorithm and its suitability for modern applications.
HMAC: Hashing for Authentication: Learn how HMAC leverages hashing to provide robust authentication while also providing data integrity checks.
SHA-3: The Hashing Heavyweight Champion: We'll delve into the low-level intricacies of SHA-3 and the significant role it plays in securing digital signatures as well as data integrity.
Diffie-Hellman: The Key Exchange Mastermind: Understand how Diffie-Hellman enables secure key exchange, even over insecure channels.
RSA: The Asymmetric Anchor: Explore the enduring power of RSA and its continued importance in secure communication.
AES: The Reigning King of Symmetric Encryption: We'll dissect AES, the current gold standard, and understand why it's so widely adopted.
ECC: The Elliptic Curve Enigma: Discover the elegance and efficiency of ECC, especially in resource-constrained environments like mobile devices.
OTP: The One-Time Wonder: Explore the theoretical un-breakability of OTP and the practical challenges that limit its widespread use.

Get ready to unlock the secrets of encryption!

The Road Ahead: Your Journey into Cryptography

We've covered enough ground for today even if it's just the fundamental principles of encryption and the looming challenges of quantum computing. I hope that this foundation will serve as our launching pad for deep dives into specific algorithms in upcoming articles. Whether it's about securing sensitive data, building secure applications, or just being fascinated by cryptography - understanding these concepts is crucial in our increasingly digital world.

Ready to continue this cryptographic journey? Do share your questions and thoughts. Let's decode these challenges together!

The Great Cloud Egress Escape: Why Your Cloud Bill Gives You Nightmares

Sat, 09 Mar 2024 06:30:00 GMT

So, you've migrated to the cloud - let's say AWS, and you're basking in the glow of scalability and cost savings... right? Well not so soon my friend. The next bill arrives, and it's higher than Taylor Swift's backstage pass. What gives? One sneaky culprit often overlooked is the gold old data egress fees. The cost of transferring data out of the cloud. It's a trap - you were warned. If not, I'll take this opportunity to discuss egress with you...

The Egress Elephant in the Room

Data transfer within a cloud region (say an AWS region) is usually free (or very very cheap), lulling you into a false sense of security. But the moment your data needs to get out of that cozy AWS region walled garden – whether to your on-premise servers, another cloud provider, or even just your users browsing the internet – BAM! Egress charges hit you like nothing has ever before.

For the remainder of this post, I'll use AWS as an example to drive a point - but be rest assured that it's not alone in the game.

Decoding the Data Egress Matrix

AWS's data egress pricing is nothing short of a choose-your-own-adventure novel, except all of the endings involve you paying. The costs vary based on many factors like...

Destination: Transferring data to the world-wide-internet is generally a bit cheaper than transferring to your own data center. Go figure that out...
Volume: The more data you move ...well ...the more you pay (duh). But on the other end of the spectrum volume discounts do exist, so bulk transfers can be slightly less painful.
Region: Egress charges differ between different AWS regions. This means choosing the right region for your workload can leave you with some savings in the end.
Service: To no one's surprise - different AWS services have their own egress pricing quirks. S3, for example, has its own set of rules. Fun times all around.

Show Me the Money! (Or Rather, How Much It's Going to Cost Me)

Let's shift gears and get down to brass tacks. Imagine that you need to transfer over 1TB of data. Here’s a hypothetical cost comparison (because AWS pricing changes faster than crypto prices):

Egress Destination	Hypothetical Cost
Internet	$90
Same Region	$0 (or very minimal)
Different Region	$20
Direct Connect (to your data center)	$50

Disclaimer: Please note that these figures are for illustrative purposes only. The actual costs will vary based on your specific setup, usage, region, timing, etc. You certainly don't want to be surprised by a bill in the end that looks nothing short of the national debt.

Static Assets: The CDN vs. The Rest

Now let's consider a fairly common scenario - a website serving ~1TB of static assets (images, JavaScript, CSS) per month. Let's compare three approaches available to us:

Serving assets directly from an EC2 instance: This option is nothing less than the egress cost equivalent of setting your money on fire. Every asset - image, script, stylesheet and more, downloaded incurs data transfer charges. Let's say, hypothetically, transferring 1TB out of us-east-1 costs around $90. To add to it, you will need a beefy instance running in the first place. We'll ignore that for this comparison.
Serving via API Gateway: Slightly better, but still far from ideal. API Gateway adds its own costs over and above the egress fees. You're paying for the data transfer and then some more for API Gateway to process the data. Let's assume this adds another $10-$20 to our bill, bringing the total to $100-$110.
Serving via CloudFront (aka CDN): This is the clear winner! CloudFront not only caches your static assets at edge locations around the world but also reduces the load on your origin. When a user requests for an asset, it's served from the closest edge location. This minimizes data transfer from your origin server (S3 or EC2) and dramatically reduces the egress costs. This also reduces the load on your servers thus allowing you to scale them down - further reducing the bill. Think of it as having a wide network of squirrels stashing copies of your website all over the garden. With CloudFront, you might pay significantly less for the same data transfer out to the internet. This could be in the range of ~$25 for that same 1TB, and even less for requests served from the cache.

In this scenario, using CloudFront helped you save $65-$85 per month compared to serving directly from EC2. Not to even mention that you save further by reducing the capacity needed to serve the same traffic. Finally as your traffic grows, those savings multiply. Cha-ching!

Supercharging Your CDN: Tips and Tricks

Okay CDN sounds convincing and now you're using a CDN like CloudFront. Amazing! But are you making sure to squeeze every last drop of available performance and cost savings out of it? Don't scratch your head, here are a few quick tips:

Caching Policies to Control Your Destiny: Caching policies end up deciding how long assets are stored at edge locations. Configure these policies wisely based on your requirements:
- Example 1: Aggressive Caching for Static Assets: Assets like your brand logo - rarely change. Hopefully. Cache it for a long long time, like a year. This means every request for the logo will be served from the CDN's cache itself. This means hitting your origin server (and in turn incurring egress fees) only once a year. Neat!
- Example 2: Moderate Caching for Frequently Updated Content: Let's say you have a popular blog online (maybe like this one!?). While new articles are posted regularly, they might not be posted every minute. Cache these articles for a relatively shorter duration, perhaps a few hours or a day. This balances freshness with cache efficiency. An update to the post will be visible only after the Cache expires. While you'll incur some egress costs when new articles are published, eventually the requests will be served from the cache itself.
- Example 3: No Caching for Highly Dynamic Content: For data that changes frequently, like I don't know ...Mr. Elon Musk's rant on a topic - caching might not be appropriate. In such cases, requests should always hit your origin server. But thankfully, your users will at least have up-to-the-second data!
HTTPS - Security with Savings: Two-for-one !? Serving content over HTTPS goes hand in hand for security, but it can also end up impacting your CDN costs. CloudFront for example, offers different pricing tiers for secure traffic. Make sure you calculate with the pricing structure in mind and only then choose the most cost-effective option. At times, using dedicated IPs or different SSL certificate configurations can leave you to discover some surprising cost differences.
Geo-Restrictions: Keep Your Content Where It Belongs: If all your users are in specific geographic regions then it makes sense to use geo-restrictions. This will prevent access from other areas and thus reduce unnecessary data transfer as well as egress costs. Not only this but it can also improve performance for your target audience.

The Development Delusion: "Egress? What Egress?"

We've all been there - bright-eyed and bushy-tailed in the early stages of development, focused on building cool features and shipping code. Cost optimization? Pfft, we'll worry about that later, right? Wrong!! Ignoring egress costs during the initial designing stage is like ignoring a leaky faucet – it seems small at first, but it can quickly turn into a massive flood of expenses. When you're prototyping and testing - sure, the data transfer will seem negligible. But as your application scales, the user base grows, the traffic increases, and with that those egress costs can snowball faster than a runaway snowball rolling down a really big, snowy hill.

Taming the Egress Beast: Strategies for Savings

So in the end - it does matter. How do you wrestle these costs into submission? Here are a few nifty tactics:

Minimize Data Transfer: Seems obvious, but often overlooked. Do you really need to move all that data? Can you not process it within AWS first and only then move the results out? Think lean, think mean, think data-transfer-reduction machine.
Content Delivery Networks (CDNs): For static content like images, videos etc, a CDN like CloudFront can be your best friend. CDNs cache your content closer to users thus reducing the amount of data that needs to travel from your origin server. This in turn reduces egress costs, all while reducing the load on your servers.
Data Compression: Do you want another no-brainer? How about compressing your data before transferring it? You'd be surprised by how many folks forget this basic step. We all pack our suitcases efficiently right? You can fit more in, it's easier to carry and cheaper to transport.
Right-Sizing Instances: Abundant choices but choosing the right instance type for your workload can minimize unnecessary data transfer.
Use Specialised Tools: Explore specialized tools like AWS DataSync for efficient data transfer and cost optimization. Someone has thought about it, why not use it?

Show Me the Code! (Because Every Good Blog Post Needs Code)

Here's a high-level sample Python snippet demonstrating a simple compression technique using gzip:

import gzip
import shutil

def compress_file(input_file, output_file):
    with open(input_file, 'rb') as f_in:
        with gzip.open(output_file, 'wb') as f_out:  # Gzipping like a boss
 shutil.copyfileobj(f_in, f_out)


# Example usage (because we're helpful like that)
compress_file('massive_data_file.txt', 'massive_data_file.txt.gz')  # Boom! Shrinkage!

Don't Let Egress Eat Your Lunch

Data egress costs can be a real pain-in-the-a**. But with a little ahead-of-time planning and some clever strategies, you can tame the beast. Don't let those sneaky egress fees turn your cloud dream into a financial nightmare. What are your favorite egress-busting strategies? Do share them!

Bits, Bytes, and Buttons: The Unexpected But Obvious Link Between Elevators and Hard Drives

Fri, 02 Feb 2024 18:30:00 GMT

Ever wondered about the connection between elevators and hard drives? Well I have, and surprisingly while on an elevator! If you give it a thought - both involve scheduling, resource management, and optimization. Both have many pending request to go from A-B, for one it's floors, and for the other the disk location.

Up and Down and All Around: Scheduling in Distributed Systems

Elevators and hard drives have to handle multiple requests all while minimizing wait times and seek times. Both make use of scheduling algorithms in order to optimize performance. Here are few such commonly used algorithm.

SCAN: The pointer moves in a single direction, servicing requests, then turns around and moves back. Like an elevator going to the top most floor, then back down to the ground floor. It'll wait on any floor in between which have been requested.
LOOK: Similar to SCAN, but the head only travels as far as the last request in each direction. The elevator will still strictly travel in one direction, but will reverse it's direction of there is no request for any other floor in that direction.
SSTF (Shortest Seek Time First): Prioritizes the nearest request and thus minimizing seek time but at the cost of starving distant requests. In this case, the elevator may end up only serving the first few floors of a 50 floor building.
C-SCAN (Circular SCAN): Moves in one direction, returning to the beginning after reaching the end. In this case, the elevator only goes up, and when it reaches the top it starts over again from the ground floor going up.

Here's a simplified Python implementation of these algorithms:

# algos.py
import matplotlib.pyplot as plt

def scan(requests, head, direction, disk_size=200):
    requests = sorted(set(requests))  # Remove duplicates and sort
    left = [r for r in requests if r < head]
    right = [r for r in requests if r >= head]

    if direction == "left":
        result = left[::-1] + [0] + right
    else:  # direction == "right"
        result = right + [disk_size - 1] + left[::-1]

    print(f"SCAN: {result}")
    return result

def look(requests, head, direction):
    requests = sorted(set(requests))  # Remove duplicates and sort
    left = [r for r in requests if r < head]
    right = [r for r in requests if r >= head]

    if direction == "left":
        result = left[::-1] + right
    else:  # direction == "right"
        result = right + left[::-1]

    print(f"LOOK: {result}")
    return result

def sstf(requests, head):
    requests = list(set(requests))  # Remove duplicates
    result = []
    current_head = head
    while requests:
        closest_request = min(requests, key=lambda x: abs(x - current_head))
        result.append(closest_request)
        current_head = closest_request
        requests.remove(closest_request)

    print(f"SSTF: {result}")
    return result

def cscan(requests, head, direction, disk_size=200):
    requests = sorted(set(requests))  # Remove duplicates and sort
    right = [r for r in requests if r >= head]
    left = [r for r in requests if r < head]

    result = right + [disk_size - 1] + [0] + left

    print(f"C-SCAN: {result}")
    return result

def calculate_total_seek_time(schedule, start_position):
    total_seek_time = 0
    current_position = start_position

    for request in schedule:
        seek_distance = abs(request - current_position)
        total_seek_time += seek_distance
        current_position = request

    return total_seek_time

def plot_seek_operations(algorithm_name, schedule, start_position):
    plt.figure(figsize=(10, 6))
    plt.title(f"{algorithm_name} Disk Scheduling")
    plt.xlabel("Request Sequence")
    plt.ylabel("Disk Position")

    x = range(len(schedule) + 1)
    y = [start_position] + schedule

    plt.plot(x, y, marker='o')
    plt.grid(True)
    plt.show()

# Example usage:
if __name__ == "__main__":
    requests = [176, 79, 34, 60, 92, 11, 41, 114]
    head = 50
    direction = "right"
    disk_size = 200

    print("Initial state:")
    print(f"Requests: {requests}")
    print(f"Initial head position: {head}")
    print(f"Direction: {direction}")
    print(f"Disk size: {disk_size}")
    print()

    for algorithm in [scan, look, sstf, cscan]:
        if algorithm == sstf:
            result = algorithm(requests.copy(), head)
        elif algorithm == look:
            result = algorithm(requests.copy(), head, direction)
        else:
            result = algorithm(requests.copy(), head, direction, disk_size)

        total_seek_time = calculate_total_seek_time(result, head)
        print(f"Total seek time: {total_seek_time}")
        plot_seek_operations(algorithm.__name__.upper(), result, head)
        print()

$> python3 algos.py
    Initial state:
    Requests: [176, 79, 34, 60, 92, 11, 41, 114]
    Initial head position: 50
    Direction: right
    Disk size: 200

    SCAN: [60, 79, 92, 114, 176, 199, 41, 34, 11]
    Total seek time: 337

    LOOK: [60, 79, 92, 114, 176, 41, 34, 11]
    Total seek time: 291

    SSTF: [41, 34, 11, 60, 79, 92, 114, 176]
    Total seek time: 204

    C-SCAN: [60, 79, 92, 114, 176, 199, 0, 11, 34, 41]
    Total seek time: 389

In a distributed system it becomes tricky to coordinate multiple "elevators" (or nodes). We end up introducing challenges like fault tolerance, consistency and conflict resolution. Algorithms like Paxos or Raft, which rely on distributed consensus often use a leader node to manage the scheduling.

The Enterprise Elevator: Scaling Up Our Metaphor

With the basics down, let's crank up this metaphor up to 11 to see how it applies to enterprise-scale systems. Because let's be real for a second, if you're writing code, you're not just managing one elevator in a small four-story building. You are instead running the entire transportation system of not one but five megacities.

Distributed Elevators: A Cluster of Confusion

Imagine that you are tasked with optimizing the elevator system for the gold standard - Burj Khalifa. Well all of a sudden our simple scheduling algorithms look about as effective as a kids paper airplane in a hurricane. We're talking over 163 floors, 57 elevators, and thousands of people trying to get to their destinations before the coffee cools down.

This is where our elevator-hard drive analogy really starts to shine, especially in the context of distributed systems. Let's break it down:

Load Balancing: Just like you would make sure to distribute requests across multiple hard drives in a RAID array, you need to efficiently distribute passengers across multiple elevators. The goal isn't just to reduce the wait time, that the by-product - the goal is to preventing system failures and ensuring consistent performance.
Predictive Analytics: Just like the coffee machine next to you, modern elevator systems use AI to predict the usage patterns. Sound familiar eh? It's the same principle behind predictive read-ahead available in modern SSDs.
Fault Tolerance: What happens when an elevator breaks down? After you have rescued the people - you need to also redistribute the remaining load. This is just like a distributed database handling node failures on the fly.

Here's a simplified example of how we might model this in code:

import random
from collections import deque

class Elevator:
    def __init__(self, id, max_floor):
        self.id = id
        self.current_floor = 1
        self.destination = 1
        self.passengers = []
        self.max_floor = max_floor
        self.direction = "up"
        self.status = "operational"

    def move(self):
        if self.current_floor < self.destination:
            self.current_floor += 1
            self.direction = "up"
        elif self.current_floor > self.destination:
            self.current_floor -= 1
            self.direction = "down"

    def add_passenger(self, passenger):
        self.passengers.append(passenger)
        self.update_destination()

    def update_destination(self):
        if self.passengers:
            self.destination = max(p.destination for p in self.passengers)
        else:
            self.destination = self.current_floor

class ElevatorSystem:
    def __init__(self, num_elevators, max_floor):
        self.elevators = [Elevator(i, max_floor) for i in range(num_elevators)]
        self.max_floor = max_floor
        self.requests = deque()

    def request_elevator(self, from_floor, to_floor):
        # In a real system, we'd use a more sophisticated algorithm here
        available_elevators = [e for e in self.elevators if e.status == "operational"]
        if available_elevators:
            elevator = min(available_elevators, key=lambda e: abs(e.current_floor - from_floor))
            elevator.add_passenger(Passenger(from_floor, to_floor))
        else:
            self.requests.append((from_floor, to_floor))

    def simulate_step(self):
        for elevator in self.elevators:
            if elevator.status == "operational":
                elevator.move()
                # Handle passenger drop-offs, etc.

        # Handle queued requests if elevators become available
        while self.requests and any(e.status == "operational" for e in self.elevators):
            from_floor, to_floor = self.requests.popleft()
            self.request_elevator(from_floor, to_floor)

class Passenger:
    def __init__(self, start_floor, destination):
        self.start_floor = start_floor
        self.destination = destination

# Usage
system = ElevatorSystem(num_elevators=57, max_floor=163)
for _ in range(1000):  # Simulate 1000 steps
    system.simulate_step()
    if random.random() < 0.1:  # 10% chance of a new request each step
        from_floor = random.randint(1, system.max_floor)
        to_floor = random.randint(1, system.max_floor)
        system.request_elevator(from_floor, to_floor)

Obviously this code is a gross oversimplification, but it illustrates the point we are after: managing a distributed system, whether it's elevators or databases, requires designing around resource allocation, fault tolerance, and optimizing. All of this for the global efficiency rather than local optimums.

Elevating Your Thinking

So, what's the point of this you ask? It's not just to make you think twice next time you're in an elevator (though that's a fun side effect). The real takeaway is about how we approach a seemingly artifical topic of system design at scale, with something that we have a day to day experience of:

Adapt and Predict: A well designed system, whether it's moving people or data - anticipate needs, adapt to challenge and falls back to the next optimal move.
Think Holistically: Just as an efficient elevator system considers the entire building, it's layout, where the elevators are places and the high traffic for the floor with the candy shop, we need to design our systems with the big picture in mind as well.
Balance Efficiency and Fairness: Pure efficiency (like prioritising serving the closest request) can lead to unfairness at large. Your design needs to balance everything with at sometimes competing objectives.
Prepare to Failure: In any sufficiently complex system, something will go wrong - and eventualy everything will go wrong. Design with redundancy and graceful degradation in mind from the get go.
Optimize, Observe, Optimize: The job is never done! Just as the smart elevator systems are constantly learning and adjusting themselfs, our data systems should be continuously optimized based on the real-world usage patterns.

Remember that whether you're moving bits or bodies, the principles of efficient, scalable system design remain the same. Next time you're architecting a distributed system - pause and take a moment to think about elevators. Why? Because it might just elevate your solution to new heights.

Now, if you'll excuse me, I still have several floors of data to go through...

Base64 Encoding Demystified: Padding, Security, and Practical Applications

Thu, 11 Jan 2024 18:30:00 GMT

Let's look at Base64, shall we? There's a good change that you have already seen it atleast twice today. From URLs to email attachments, they are everywhere. But what exactly is this funny looking encoding scheme? Why does it sometimes end with those equal signs (==)? Why isn’t it encryption? Why was my application to Standford rejected? Well answer to some lies in my introspection, and for others let's dive into this essential piece of the web's infrastructure.

Base64 in the World-Wild-Web

Before we get into the technical nitty gritty, let's look at where exactly does Base64 earns its spot in the enterprise:

Image Data URIs: Used to encode and embedding small images directly in HTML or CSS using. This can improve page load times. Think icons or small logos, with no aditional requests.
Email Attachments: Encoding attachments like PDFs, images, etc. as Base64 strings allows us club them together within the email email body. This stramlines email handling.
Data Storage and Retrieval: Base64 makes storing binary data in text-based databases or configuration files more possible. This means the data lies closely together with each other.
Security Tokens and Credentials: While Base64 isn't encryption, you’ll often see it used within the security contexts. For example in encoding parts of JSON Web Tokens (JWTs). But remember - encoding is not encryption!

Decoding the Encoding: How Base64 Works

Base64 at their core encodes binary data using only printable ASCII characters. This is very important for interoperability because some systems struggle with raw binary data. They can potentially misinterpret byte sequences as control characters and cause unexpected behavior. Base64 provides a safe, text-based encoding to avoid these issues in the first place.

Here’s the process at a very high level - binary data is chopped into 6-bit groups, each mapped to a specific printable ASCII character using a lookup table (A-Z, a-z, 0-9, +, /).

Visualizing the Transformation: "MayankRaj.com" Goes Base64

Let's see this in action with "MayankRaj.com":

# Step 1: String to Bytes (ASCII)
MayankRaj.com  ->  77 97 121 97 110 107 82 97 106 46 99 111 109

# Step 2: Bytes to Binary (8-bit)
77  -> 01001101
97  -> 01100001
121 -> 01111001
97  -> 01100001
110 -> 01101110
107 -> 01101011
82  -> 01010010
97  -> 01100001
106 -> 01101010
46  -> 00101110
99  -> 01100011
111 -> 01101111
109 -> 01101101

# Step 3: Regrouping into 6-bit Chunks (with annotations)
010011 010110 000101 111001 011000 010110  <- First six 6-bit groups
111001 101011 010100 100110 000101 101010  <- Next six 6-bit groups
001011 100110 001101 101111 011011 010000  <- Last six 6-bit groups (padded)


# Step 4: Mapping to Base64 Characters (Tabular Format)
| 6-bit Chunk | Base64 Character |
|---|---|
| 010011 | M |
| 010110 | W |
| 000101 | F |
| 111001 | 5 |
| 011000 | Y |
| 010110 | W |
| 111001 | 5 |
| 101011 | r |
| 010100 | R |
| 100110 | a |
| 000101 | F |
| 101010 | q |
| 001011 | L |
| 100110 | a |
| 001101 | M |
| 101111 | v |
| 011011 | b |
| 010000 | Q |


# Final Base64 Encoded String:
TWF5YW5rUmFqLmNvbQ== (The == indicates padding)

Show Me the Action!

import base64

binary_data = b'Hello, world!'
encoded_data = base64.b64encode(binary_data)
print(encoded_data)  # Output: b'SGVsbG8sIHdvcmxkIQ=='

decoded_data = base64.b64decode(encoded_data)
print(decoded_data)  # Output: b'Hello, world!'

The Security angle - Encoding vs. Encryption

Let's be very sure of one key aspect of Base64 - It is encoding, not encryption. Encoding just changes the data's representation, but not its underlying meaning. Anyone can decode it, let alone a computer. On the flip side, Encryption scrambles the data in such a way that you can only make sense of it with a key. This key is usually called encryption key and is handled with cared. Make sure to not confuse the two! Use proper encryption for security, and encoding for reliable data transfer.

The Port Party: A Crash Course on Ephemeral Ports and Their Security Implications

Sun, 03 Dec 2023 18:30:00 GMT

Let's continue our deep dive into a critical, yet often overlooked aspect of distributed systems — ephemeral ports. We looked at what they are with Journey Through the Silicon and also looked at proxies with From Forward to Reverse Proxies. Let's look at how these two play with each other. Ephemeral ports are always in the backstage control room, but trust me, you don't want to ignore them. They might be small, but they can cause big problems if you don't understand how they work.

Think of ephemeral ports as the temporary phone numbers used by your client applications to communicate with the servers. Each time any client initiates a connection, the operating system assigns it a port number randomly picked from a reserved range. These ports are tightly coupled with this connection and only active during the connection lifetime — hence the name "ephemeral". When the job is done, the port is put back in the bowl to be picked up for another connection.

So, why should you care? After all these are temporary, ever-changing ports often short lived for the lifecycle of the connection. Well - Because they play a crucial role in making sure a smooth communication in distributed systems can take place. They are the reason for the connection to happen in the first place. You wouldn't want your shiny new fancy distributed application to become a tangled mess, right?

Load Balancers: Who Let the Port-Numbers Out !?

Let's say you have a bunch of servers, which host your backend. These are the army powering your distributed application. How do you direct traffic to them efficiently? Cue the load balancer, the overworked traffic cop that takes all the incoming connections and splits them evenly across your servers.

Load balancers are typically deployed in front of your backend services as the first line of defence. Today load balancers can also operate at Layer 7, which means they not only distribute the incoming traffic, but have the ability to peek inside the request. This also means they have to terminate the incoming request from the client, and establish a new one with your servers.

When a client request comes in, the load balancer assigns it a unique ephemeral port at its level. It then directs it to an appropriate backend server. It's like the load balancer gives each client a temporary phone number, and then connects them to the right server based on its internal routing logic.

Load balancers, a type of proxies was discussed in much more details over at From Forward to Reverse Proxies.

Sticky Sessions: The Double-Edged Sword of Performance

Applications are not always built "ideal". Some end up having session state stored locally on the server. To tackle this one common strategy used by load balancers is "sticky sessions." It makes sure to keep a client connected to the same server throughout a series of requests. This can improve performance by utilizing cached data which is tied to a given server. Imagine a shopping cart: just because developers choose to use a load balancer, you don't want to start adding items on one server and then end up on a different one at checkout, right? That would be a recipe for a frustrated customer (and potentially a lost sale)!

To flip the coin around, sticky sessions can cause issues too. If the server that a given client's request is attached to goes down, the load balancer needs to find a new server and new port. This could potentially disrupt the client's ongoing session. To add to it, if you have a slow server, all the clients stuck on it will experience performance problems as sticky sessions will not allow them to go to another server. It's like having the party guest stuck in a slow moving elevator - everyone is late to the party, and with that the fun is getting disrupted.

NAT Gateways: Translating Traffic Like a Pro (With Some Ephemeral Magic)

Ever heard of NAT? No !? It stands for Network Address Translation. It's a common means for hiding internal IP addresses from the wild outside world. NAT gateways behave like translation services - they are constantly converting the client's IP address and port to different ones, so that your internal servers can communicate safely. Above all, all of this without exposing themselves directly to the clients.

Usually NAT gateways utilize SNAT (Source Network Address Translation) to rewrite the source IP address and port. Think of it as the gateway taking your client's phone number, switching it up, and then giving it to the server. Generally only the NAT has these mapping with it. This helps protect your internal network while maintaining communication flow. The ports are allocated at NAT level, for either end of the communication.

The Ephemeral Port Pool: A Shared Resource

But just like life - there's a catch. NAT gateways often use ephemeral ports for this translation process. This ends up creating an ephemeral port pool which is then shared across all translated connections. What this means is that when your gateway is handling a ton of traffic, it can run into issues with port exhaustion. This in turn could lead to dropped connections or performance degradation. It's like everyone wants Sheldon's spot because it's ..well ..the perfect spot - it's a recipe for chaos!

Tuning NAT Gateways for Optimal Performance

Just like with Linux iptables, NAT gateways can be tweaked as well which can help improve their performance. The key here is to configure the appropriate values for ephemeral ports, connection timeout, etc which eventually can lead to optimizing the translation process. All of this will significantly impact your NAT's ability to scale and respond. NAT should be a part of the stack that are not noticed by anyone. You should be able to set it and forget it. So time spent in optimisations here are well worth it.

Proxies: Middleman or Bottleneck? How Ephemeral Ports Behave in Proxy Scenarios

Proxies act as intermediaries between the clients and servers. When set as forward proxies they are forwarding requests from clients to servers. When as reverse proxies they end up as a gateway to your backend services. It's like having a friendly bartender who takes orders from guests and delivers them to the kitchen. It's forwarding requests from one point-of-view, and proxying request from the others.

When a client makes a request via a proxy, the proxy will almost certainly assign an ephemeral port to manage this connection. What this means is that the proxy becomes a temporary relay station, handling the traffic flowing between the client and the server. On the other hand, if the proxy isn't configured appropriately, it can create a bottleneck, especially when dealing with a high volume of requests. This is where tuning knobs like HTTP keep-alive, connection pooling etc come into play. They can not only help reduce the burden on the proxy but also improve efficiency. You always choose to have your Amazon packages be delivered in "fewer boxes" right ? Why not do the same with your network packages?

Proxy Technologies: A Choice of Tools

Choices are everywhere, and proxies are not behind. Popular technologies like Squid, Varnish Cache, and HAProxy are often used in large scale distributed systems. These offer different features and optimizations like connection pooling, caching, load balancing and more. All of these can be critical for managing ephemeral ports effectively. And as we have seen time and again - managing ephemeral ports are key to seamless operations.

Scaling Shenanigans: How Ephemeral Ports Affect Distributed System Performance

In the world of distributed systems, scaling is the buzzword. And ephemeral ports play a crucial role in achieving high performance and scalability. But they can also end up being the source of unexpected performance issues, particularly in a high-throughput environments.

Real-World Examples: Port Exhaustion in Action

Consider a large online website handling a massive influx of traffic during a concert ticket sale. You have been there, haven't you? The requests come in all-at-once, quite literally. The system might experience port exhaustion due to the overwhelming number of connections being established via the load balancers, NAT gateways and proxies. This can lead to connections being dropped, throttling and overall customer frustration.

Performance Tuning Tips: Avoiding Port Starvation

In my experience the best way to avoid port exhaustion, is to consider the following...

Increase Ephemeral Port Range: Expand the range of ports available for ephemeral use. Every system has a certain number of ports, and when you know that the server would need a high number of port - it's best to allocate it ahead of time.
Use Connection Pooling: Tackle the problem at the source - reduce the number of connections established in the first place. Reusing existing connections wherever possible.
Setup Load Balancing: Optimize the load balancing algorithms. If the load balancer can make sure all the available resources are used optimally, you won't have to scale any one individually or overallocate.
Monitor Port Usage: Track ephemeral port usage, so that you can identify potential bottlenecks ahead of time and address them. Remember - logs, logs and even more logs.

Security Considerations: Guarding Your Port Party

Ephemeral ports facilitate communication but they also pose a major security challenges. If a bad actor knows that ephermal ports will be used for the requests, it could be worthwhile to check "all" the ports - maybe once in a while the bad actor will hit gold! Ephemeral ports can be vulnerable to various attacks like port scanning, port exhaustion attacks, and brute force attacks.

Firewall Rules: Firewall rules can be used to limit access to specific ports. Thus restricting connections from unauthorized sources.

Network Segmentation: Breaking up the network into smaller, isolated segments help prevent malicious attacks from spreading. This also reduces the blast radius.

At last - Avoiding Ephemeral Port Hell

Ephemeral ports are crucial components of the modern distributed system stack. Better understanding how they work within components like load balancers, NAT gateways and proxies is the key to building reliable and scalable applications. Avoiding common pitfalls like port exhaustion and properly configuring your infrastructure, can ensure that your system operates as it is designed to.

Remember, ephemeral ports might be small behind the scenes players, but they have a big impact on your overall performance. So, be aware of them when designing your applications. Don't let those pesky little ephemeral ports turn your system into a party gone wrong!

From Forward to Reverse Proxies: Enhancing Network Performance and Security

Fri, 10 Nov 2023 18:30:00 GMT

From Forward to Reverse Proxies: Enhancing Network Performance and Security

Let's be honest, folks. Any engineer who has worked for more than a year has already seen it all – from spaghetti code that would make a seasoned Italian chef weep to production outages that would make even a grown engineer cry. And let's not even talk about those "super-critical-path-breaking-urgent" requests from marketing that magically appear at 4:59 PM on a Friday.

But here's the thing – we don't have to navigate this crazy world alone. Proxies, my friends, are the unsung heroes of scalable and a more stable architecture, always there to lend a helping hand (or at least a helpful network hop).

Forward vs. Reverse Proxies: Two Sides of the Same Coin (Sort of)

Forget the technical definitions just for a second. A proxy, in its simplest form, is just a middle-person. It's that friend who always intercepts your calls when your ex is blowing up your phone, except instead of drama, it's dealing with network traffic.

Now, there are two main flavors of proxies that you will come across – the forward and reverse.

1. Forward Proxies: The Client's Best Friend (and Firewall Foe)

For a second picture this: you're at a tech conference, desperately trying to update your LinkedIn profile before that crucial networking event. As with every conference that you have been to, the conference Wi-Fi is not reliable. With that now you're stuck behind a firewall.

That's where a forward proxy comes in handy. It's like having a secret tunnel right under the firewall, allowing you to bypass those pesky restrictions and access the websites you need (even if it's just to stalk your new "acquaintance" on LinkedIn... I mean, research potential employers).

Forward proxies are masters of disguise, masking your IP address and making it look like all your requests are coming from the proxy server itself. It's like browsing the web incognito, but with a slightly more sophisticated fedora and with that trusted hacker-sunglasses combo.

But wait, there's more! Forward proxies can also cache content, just like your browser does, but on a much larger scale. This means faster load times for everyone and less strain on the origin server (which is probably already sweating under the pressure of a thousand engineers trying to update their LinkedIn profiles simultaneously).

Popular forward proxies: Squid, Apache HTTP Server (with mod_proxy), HAProxy

2. Reverse Proxies: Shielding Your Servers from the Wild, Wild Web

If forward proxies are all about protecting the client, reverse proxies are the guardians of your servers. Think of them as the bouncers at the club – all requests go through them first, and only the legit ones get through to the party inside (your precious servers).

Reverse proxies are the multi-tasking superheroes of the server world. It is designed to distribute traffic like a seasoned air traffic controller, preventing any single server from getting overwhelmed (and crashing and burning in a fiery blaze of server errors). They can handle the SSL encryption, offloading the heavy lifting from your servers and making sure your users' data is safe and sound. And yes, they can even cache content, because who doesn't love a good performance boost?

That's not all, here's where things get really interesting – reverse proxies can also be used for some pretty amazing stuff. Stuff like A/B testing and blue-green deployments. We can instruct this air traffic controller to direct traffic to different set of servers, all while your users continue to watch the add that they should be watching, and definitely not blocking off with an ad-blocker (...right?).

Popular Reverse Proxies: Nginx, HAProxy, Varnish Cache

Let's talk business: Show Me The Code!

Alright, enough with the metaphors. Let's see some real-world action with one of the popular options out there - Nginx, the Swiss Army Knife of web servers and proxies.

Scenario 1: Load Balancing

Imagine you just launched a killer new feature that is bound to change the course of humanity, and your website traffic is exploding faster than a bag of microwave popcorn. Nginx can swoop in and distribute those requests across multiple servers like a boss, preventing any single server from becoming a smoldering pile of silicon.

http {
    upstream myapp {
        server web1.example.com;   # Server 1: Probably sipping margaritas somewhere
        server web2.example.com;   # Server 2: Trying to keep up with the requests
        server web3.example.com;   # Server 3: Wishing it had invested in more RAM
    }

    server {
        listen 80;
        server_name www.example.com;

        location / {
            proxy_pass http://myapp;  # Nginx doing its magic, redirecting traffic
        }
    }
}

Scenario 2: SSL Termination – Because Security Shouldn't Give Your Servers a Nervous Breakdown

SSL encryption is like the secret handshake of the internet, ensuring that only authorized parties can access sensitive data. But handling all that encryption and decryption can be a real drag on your servers, especially when they're already juggling a million other tasks. These involve relatively complex cryptographic operations, which are not easy to pull off.

Nginx to the rescue! Again !? YES ! It can handle SSL termination at the proxy level, freeing up your servers to focus on what they do best – serving up those beautiful web pages with even more adds.

server {
    listen 443 ssl;
    server_name www.example.com;

    ssl_certificate /path/to/certificate.crt;   # Your shiny SSL certificate
    ssl_certificate_key /path/to/private.key;   # Don't lose this!

    location / {
        proxy_pass http://localhost:8080;  # Assuming your web server is running locally
    }
}

Proxies in the Wild

Here's the thing about modern software development – just like the real world, it's messy. We're talking microservices, containers, serverless functions, microservices built with serverless functions, microservices, microservices built with serverless functions hosted on containers... you get the point. But in the heart of this chaotic jungle, proxies are our trusted guides, helping us navigate the complexities and emerge victorious (...or maybe at least with just our sanity intact).

Load Balancing: Not Just for Your Overworked Web Servers

Remember when load balancing was as simple as distributing traffic across a few web servers? Wait what !? You do !!?? Those were the good old days, my friend. Now we're dealing with sprawling microservices architectures, where dozens, hundreds, or even hundreds of thousands of services are fighting for resources.

But fear not, for proxies are here to save the day (again!). They've evolved and leveled up their load balancing game, acting as intelligent traffic cops at different layers of our architecture. They route requests, optimize resource utilization, and prevent those dreaded cascading failures that can bring down your entire system faster than you can say "blame it on the network."

Service Mesh

Ah, the service mesh – the latest and arguably the greatest buzzword in the world of microservices. It's like the glue that holds your entire distributed system together, ensuring that all those tiny services can communicate and collaborate without descending into an inevitable anarchy.

And guess what plays a starring role in this intricate dance of microservices? You guessed it – proxies!

In a service mesh, proxies come out to be the heroes, quietly working behind the scenes to handle service discovery, routing, security, and even observability - all while maintaining their pazzaz.

Istio: Built on the ever-popular Envoy proxy, Istio is like the Swiss Army Knife of service meshes. It's packed with features, but be warned – it comes with a very very very very steep learning curve.
Linkerd: If Istio is the overachieving older sibling, Linkerd is the arguably cool, minimalist younger one. It's known for its simplicity, performance, and ease of use – perfect for dipping your toes into the service mesh pool without getting overwhelmed.
Consul Connect: HashiCorp knows their stuff when it comes to infrastructure management. Consul Connect is no exception in that list. It integrates seamlessly with their service discovery tool, making it a breeze to set up, scale and manage.

Global Traffic Routing: Because Distance Shouldn't Matter

As your user base expands beyond your city, state, or even continent, you need to ensure that everyone gets a fast and reliable experience, regardless of their geographical location. And guess what can help you achieve that? You got it once again – proxies! This is getting too predictable now isn't it ?

Proxies are the masters of global traffic routing, using a variety of techniques to bring your application closer to your users:

CDN (Content Delivery Network): CDNs are like having a network of caches strategically placed around the world which themself act as an extention to your servers. They not only store copies of your static content (images, videos, etc.) closer to your users, so they don't have to wait an eternity for things to load but also reduce the load on your origin servers by not asking it for content time and again.
Geo-DNS: Remember that time you accidentally booked a flight to the wrong London? (Been there - done that) Geo-DNS is like the GPS of the internet, routing users to the nearest data center - all of that based on just their IP address.
Anycast Routing: Anycast routing is like having a team of identical twins working for you – it sends traffic to the closest available server, ensuring high availability and fault tolerance.

The Challenges of Managing Proxies at Scale: Because With Great Power Comes Great Responsibility (...and Headaches)

Let's take a step back, and reflect on a developers life shall we - Life can be like a puzzle—at first, it's fun. Then, it’s like trying to finish it with quite a few missing pieces. Scaling proxies? That’s doing the puzzle while the pieces change shape in your hands.

Here are a few challenges you might encounter on your proxy-powered journey:

Configuration Management: With dozens or even hundreds of proxies scattered across your infrastructure, keeping track of all those configurations can make your head spin. You may now need a proxy layer to communicate the configurations to all your proxies, which themself are in the proxy layer. Centralized configuration management tools and automation are your best friends here (trust me, you don't want to be manually updating configurations at 3 AM ...on saturday night ...after things are on fire).
Observability: How do you know if your proxies are actually doing their job? You need eyes everywhere! We all want more logs, and there's not enough of it. Robust monitoring, logging, and tracing are essential to better understand traffic flow, identifying bottlenecks, all while troubleshooting issues in a proxy-heavy environment.
Performance Tuning: Remember that extra hop we talked about? Well, it can come at a cost if you're not careful about it. Proxies can introduce latency if not configured correctly, and sometimes even if configured properly. You need to optimize those caching mechanisms, connection pooling settings, and all those other knobs and dials that make proxies sing.

So, You Think You Want to Be a Proxy Master?

This is where you come in.. your time to rise and shine. Have you wrestled with proxies in your own projects? What war stories can you share about those late-night debugging sessions with a once trusted proxy that now a rogue culprit? Do share your experiences, tips, and questions.

Journey Through the Silicon : Network communications via Ephemeral Ports

Sun, 03 Sep 2023 12:13:00 GMT

Do you ever stop to think about the tiny marvels of technology that make our digital world go 'round? We're diving into the building blocks of the tech universe, unraveling the mysteries behind what enables you to read this very article. We're talking DNS, ephemeral ports, proxies, tunnels, and more—these are the unsung heroes ensuring data flows seamlessly from one device to another on your network.

Today, our spotlight is on ephemeral ports. These unsung heroes are the reason you can have multiple online conversations at once. Imagine only being able to chat with one friend at a time—how dull would that be?

The Basics: Can You Juggle Two Conversations?

Let's put you in the spotlight. Have you ever chatted with more than one friend simultaneously? Of course, you have! Picture a scenario where you're conversing with each of them on different topics. Now, think about how you manage that. Let's start with a situation where these conversations are happening over text. It's as simple as opening separate chat windows for each conversation. You effortlessly switch between these windows, seamlessly adding to each discussion. With minimal effort, you maintain the context of each conversation and respond appropriately. The topics can vary widely, but you've got it all under control.

Now, shift gears and imagine you're doing this in person. Not so challenging, is it?

Replace those chat windows with the faces of your friends, and you can still smoothly engage in two or more parallel conversations at the same time.

Let's dissect what's happening here: You initiate each conversation, creating a context (even if it's subconscious) for that specific chat. Without breaking a sweat, you receive responses and place them neatly within the corresponding context. When it's time to respond, you effortlessly switch back to the relevant context and continue the conversation.

Ephemeral Ports: Where Your Tech Devices Get Playful

Have you ever wondered how your tech gadgets, be it your mobile, laptop, or even your trusty smartwatch, manage to mimic human-like communication? Well, behind the scenes, they rely on something called "ephemeral ports" to keep their digital conversations flowing smoothly. But before we dive into this world of tech wizardry, let's take a quick detour to refresh our memory on what ports are all about.

Ports: The Gateways to Digital Conversations

Think of ports as the doors to a bustling digital building, the faces in a crowd, or even the chat windows in our digital lives. They are the identifiers that machines use to keep track of different ongoing conversations. Every message that goes out or comes in gets neatly sorted into these virtual buckets, and it's the operating system's job to manage and process the data within them like a maestro conducting a symphony.

Numbers, Numbers Everywhere

Now, here's where it gets interesting. When you type in a web address like "https://mayankraj.com/" (a little shameless plug never hurts, right?), you're actually making a request to "https://mayankraj.com:443/". That ":443" at the end? That's the port number! These numbers range from 0 to 65,353. The first 1023 are reserved for super common TCP/IP applications, aptly named "well-known ports." There are a few global favorites, like 22 for Secure Socket Shell (SSH), 80 for Hypertext Transfer Protocol (HTTP), 443 for the super-secure Hypertext Transfer Protocol Secure (HTTPS), and many more. Then come the "registered ports," ranging from 1,024 to 49,151, where applications on your operating system can request to listen in. Ports like 8000 or 8080 are often used for local development. Finally, the last block, from 49,152 to 65,535, houses the dynamic ports, reserved for short-lived, on-the-fly communications.

The Ephemeral Dance

In the grand scheme of things, when your favorite app wants to talk to anything else, be it on your local network or across the vast internet, it starts by asking the operating system for a free port. Usually, it goes for one from the dynamic port range, but it can also venture into the reserved territory. The OS kindly grants it a port, let's say 57000, which means any data received on this port will be sent straight to that application.

Now, when our app wants to chat with another digital entity (say, a web server), it makes a request on the well-known port of the destination (like 443 for HTTPS). But here's the fun part – it also includes the port number it received from the OS as its return address, which is 57000 in our case. The destination, our trusty web server, gets the message on port 443, processes it gracefully, constructs a reply, and sends it off to the port number 57000. When the OS on the other end receives this data packet on 57000, it's like the final act in a fantastic play; it forwards it to the waiting application. This special 57000 port? That, my friends, is what we call an "ephemeral port."

So, the next time you see your tech devices communicating seamlessly, remember the ephemeral ports doing a lively dance behind the scenes, ensuring that your digital world stays beautifully connected. It's all part of the delightful play that is modern technology! 🎉🌐

Data archival on cloud and how to not do it

Fri, 02 Jun 2023 18:30:00 GMT

Making data-driven decisions is the new norm. It won't come as a surprise that the rate at which we are generating data has gone up. As the volume of data continues to grow exponentially, businesses are increasingly turning to cloud storage for efficient and scalable archiving solutions. The cloud offers numerous advantages, such as cost-effectiveness, flexibility, and easy accessibility. However, archiving data in the cloud requires careful consideration and planning to ensure data integrity, security, and long-term accessibility. In this blog post, we will explore some common mistakes to avoid when archiving data in the cloud.

In this blog, we will look at Amazon S3 Glacier which is a popular cloud storage service provided by Amazon Web Services (AWS) that offers durable, secure, and cost-effective storage for long-term data archiving and backup. While S3 Glacier provides numerous benefits, it is important to understand the scenarios in which it may not be the ideal choice. In this blog post, we will delve into the strengths of S3 Glacier and highlight situations where alternative storage solutions might be more appropriate.

Key Areas to Look Out for

The need for Data Classification is one of the key mistakes organizations make when archiving data in the cloud. It may seem logical but try to think about the last time your team put in conscious effort in classifying the data properly. Not all data is created equal, and without proper classification, it becomes difficult to prioritize data for archiving, apply appropriate retention policies, and allocate storage resources effectively. Invest time in understanding your data, identifying its value, and categorizing it based on its sensitivity, compliance requirements, and business importance.

Overlooking the backup strategy often leads to bad experiences down the line. Archiving data does not mean it's exempt from the risk of loss or corruption. Failing to implement a robust backup and recovery strategy is a grave mistake. Many organizations assume that cloud service providers automatically handle backups, but this is not always the case. Cloud providers may offer infrastructure-level redundancy, but it's essential to have your data backup strategy in place to protect against accidental deletion, data corruption, or service provider failures. Regularly test your backup and recovery processes to ensure they are effective and reliable.

When in cloud compliance becomes even more important. Compliance regulations and legal requirements dictate how long certain data must be retained. Ignoring or overlooking these policies when archiving data can lead to serious consequences, such as legal liabilities or financial penalties. Ensure you understand the specific data retention requirements for your industry and region. Implement proper retention policies and procedures, and regularly review and update them to stay compliant with evolving regulations.

You might also be overlooking data validation and integrity checks. Data integrity is crucial for successful long-term archiving. Neglecting to perform regular data validation and integrity checks can result in silent data corruption that goes undetected until it's too late. Implement checksums, hash functions, or other integrity validation mechanisms to ensure data remains intact and unaltered during the archiving process. Regularly validate archived data to identify and rectify any integrity issues promptly.

From Day-0, keep an eye on the Long-Term Storage Costs. While cloud storage offers scalability and flexibility, it's important to consider the long-term costs associated with archiving data. Cloud storage costs can accumulate over time, especially for large-scale archiving projects. Evaluate different storage options, including lower-cost tiers specifically designed for archiving, and consider a mix of storage solutions to optimize cost-effectiveness. Additionally, periodically review and analyze your archiving needs to identify and remove obsolete or unnecessary data.

S3 Glacier

S3 Glacier is a highly advantageous option for archiving data in the cloud due to its cost-effectiveness, durability, availability, security, and integration capabilities. It offers a low-cost storage solution for long-term data retention, making it suitable for infrequently accessed data without immediate retrieval requirements. With a remarkable durability of 99.999999999% (11 nines), data stored in S3 Glacier is spread across multiple facilities to ensure high availability. Security is ensured through server-side encryption, access controls, and seamless integration with AWS Identity and Access Management (IAM). Moreover, S3 Glacier seamlessly integrates with S3 lifecycle policies, enabling automated transitions from hot storage tiers to Glacier based on user-defined rules.

Limitations and When Not to Use S3 Glacier:

Despite its advantages, there are certain scenarios where S3 Glacier might not be the optimal storage solution. If you have data that requires frequent or real-time access, S3 Glacier's retrieval times (ranging from minutes to hours) may not meet your requirements. In such cases, consider using other storage classes like S3 Standard or S3 Intelligent Tiering. S3 Glacier is optimized for large file sizes. If you predominantly work with small files, the overhead associated with Glacier's minimum storage duration and retrieval costs might outweigh the benefits. Consider other storage options like S3 One Zone-IA or S3 Standard-IA for smaller files.

If you need short-term storage for data that will be frequently accessed or modified, S3 Glacier is not the appropriate choice. Instead, opt for storage classes like S3 Standard or S3 Intelligent-Tiering, which offer low-latency and high-performance characteristics. While S3 Glacier provides expedited retrieval options, the associated costs can be higher. If you have strict recovery time objectives (RTOs) and need rapid access to your data, alternatives like S3 Standard orIntelligent Tieringring will better suit your needs.

In conclusion, if you are looking for a cost-effective and scalable solution to manage the increasing volume of data then archiving data in the cloud could be a great option. But to ensure successful archiving, you should make sure to look into a few aspects. Proper data classification is essential for prioritizing data, applying retention policies, and so is effectively allocating storage resources. Implementing a robust strategy to back up the data, including regular testing, protects against data loss and corruption. Compliance with data retention requirements and regular validation checks ensure data integrity and regulatory adherence cannot be ignored. Considering long-term storage costs and periodically reviewing archiving needs optimizes cost-effectiveness.

Amazon S3 Glacier is a great option but remember that it is just that - an option. I would use Glacier with my eyes closed for certain use cases but certainly not for all of them. Once you know of the limitations of the tool, you can make better and more informed decisions. For some cases, alternative storage classes like S3 Standard or S3 Intelligent Tiering are more appropriate. Short-term storage needs with frequent access or modification are better served by S3 Standard or S3 Intelligent-Tiering, which offer low-latency and high-performance characteristics. Considering recovery time objectives and the associated costs, organizations can make informed decisions about the most suitable storage option for their specific requirements.

Zero Day Vulnerabilities and You : Introduction

Wed, 15 Feb 2023 16:53:24 GMT

You may not have realised it but there are a handful if not more zero day vurlnerabilities running on the very device you are holding. Weather that is a mobile device, laptop, tablet or even your smart fridge. you can get a grip on their seriousness by knowing that Zero day vurlnerabilties are traded for hundreeds if not millions of dollars among Cyber Criminals and nation state actors.

To name a few, attacks like Wanacry which affected hundreds of thousands of computers in over 150 countries in May 2017, Stuxne which used a sophisticated piece of malware that targeted Iran's nuclear program and was discovered in 2010 were done with help of one or more Zero Day Vurnerability at it's core.

In this series, I will attempt to tear apart a few such attacks. We will look at how the attack was caries out, what the exploit was, it's impact and how it was eventually fixed. The language of the articles in the series will ensure that any and everyone will be able to appreciate the execution behind these attacks, in makign them happen and fixing them. This is the most facinating part for me and is what I want to share.

Unique Quirks of DynamoDB: Things you will notice when building at scale

Sat, 28 Jan 2023 18:30:00 GMT

DynamoDB is part of the Database family of AWS. It sits beside the heavyweights like RDS, Redshift, etc. It is a fully managed NoSQL database. The core value of DynamoDB is that it provides a high-performance storage solution for applications that require low-latency access to large amounts of data. Above all, it is scalable and reliable, even at large data volumes in tunes of a couple of hundred gigabytes.

Above all, it's a managed solution. While that is great news to start with but also means that you don't get to finetune it when the need arises. So you have to make sure that from Day 0, right from the time when you are designing the database patterns, you also design for the unique behavior of DynamoDB.

Flexible Schema is the headlining feature of the database, like any other NoSQL database. Flexible schemas sound like great news but when you go from a few records to a billion, it can come back to bite you. The flexible schema allows you to store items with varying attributes within the same table. This provides great flexibility when dealing with evolving data structures. For example, let's say you have an e-commerce application where different product categories have different attributes. With DynamoDB, you can store products of various categories in a single table, without needing to define a fixed schema upfront. DynamoDB indexes play a big role in how you make use of this schema. Without an index, it's as good as you are not using a database but reading the whole data for every query from a large json file.

Secondary Indexes will quickly become your friends when you scale. They provide flexible querying capabilities. These indexes can be global, spanning the entire table, or local, limited to a specific partition. Secondary indexes allow you to query your data using different attributes, enhancing the flexibility and efficiency of your application.

Let's take an example wherein you are building a social media application and need to retrieve posts by both creation date and user ID. By creating a global secondary index on the user ID attribute, you can efficiently query the DynamoDB table to retrieve all posts made by a particular user. Integrating DynamoDB with AWS AppSync and AWS Amplify can further simplify the development process by providing managed GraphQL APIs for your frontend applications, seamlessly integrating with DynamoDB's secondary indexes.

Transparent Scaling is a big draw of DynamoDB. It dynamically adjusts its capacity based on the workload. This eliminates the need for manual provisioning and ensures consistent performance as your application's demand fluctuates. Scaling can be done both vertically (throughput per partition) and horizontally (number of partitions).

Suppose you have a real-time analytics application that experiences varying traffic patterns throughout the day. By integrating DynamoDB with Amazon CloudWatch and AWS Application Auto Scaling, you can monitor the workload and automatically adjust the provisioned capacity of your DynamoDB tables. This enables your application to handle high-traffic periods without compromising performance or incurring unnecessary costs during low-traffic periods.

Item Time to Live is a unique but underrated feature, which automatically deletes expired items from a table. This can be useful for managing temporary data or purging stale records, reducing storage costs and query overhead. This allows you to think of DynamoDB for a lot more use cases than just durable long-term data storage. Consider a mobile gaming application that tracks user session data. By setting a TTL attribute on the session records in DynamoDB, you can ensure that expired sessions are automatically deleted from the table. Furthermore, you can use DynamoDB Streams in conjunction with AWS Lambda to trigger additional actions whenever an item is deleted, such as updating analytics or sending notifications.

DynamoDB, in general, seems like a one size fits all, but it is far from it. While it offers several unique quirks that make it a powerful choice for scalable, low-latency data storage but it comes at. cost. Its flexible schema, automatic scaling, secondary indexes, and Time to Live feature provide developers with powerful tools to build efficient and dynamic applications. By combining DynamoDB with other AWS services, you can unleash its full potential and create robust, scalable solutions to meet your application's specific requirements.

Serverless Architecture 101: Key areas to look into from Day 0

Mon, 05 Dec 2022 18:30:00 GMT

You might have heard about Serverless recently. It has become increasingly popular, with more and more businesses adopting it as a way to build and deploy applications. Moreover, developers are finding it easy to build applications with it. Serverless computing allows developers to focus on writing code, without worrying about the underlying infrastructure.

Over the years, I've built many applications with serverless components embedded into them. Today I prefer to use my serverless templates for even my hobby projects let alone API endpoints that process millions of requests. There are a few key areas that I always find myself coming back to. In this post, I've collected 10 such areas that you should pay attention to when building serverless applications, especially using AWS services.

Let's start with the three basics - API, Compute, and Storage.

1. API Endpoints

API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. It acts as a “front door” for your serverless application, allowing you to define RESTful APIs and route requests to backend services. However, if GraphQL is more of your thing, look no further than AWS AppSync.

For Example: If you want to build a serverless application that requires an API, you can use AWS API Gateway to define your RESTful APIs and route requests to your Lambda functions.

2. Compute

No brownie points for guessing it. Compute is the foundation of serverless architecture. AWS provides two main computing services: AWS Lambda and AWS Fargate. AWS Lambda is a computing service that lets you run code without provisioning or managing servers. Fargate is a serverless compute engine for containers that allows you to run containers without managing servers or clusters. For Example: If you want to build a serverless application that requires a lot of computing power, you can use AWS Lambda to run your code. If you have your containers with you fargate may be better suited for you.

3. Storage

Serverless applications require storage for application data, logs, and other assets. AWS provides several storage services, including Amazon S3, Amazon DynamoDB, and Amazon Aurora serverless. For Example: If you want to build a serverless application that requires a database, you can use Amazon DynamoDB to store your data.

Now with basics out of the way, you have your POC in place. We now get into the interesting aspects of making this application ready for the wide and interesting audience of the public internet.

5. Breaking down the core logic:

Break down your application into smaller, focused functions to take full advantage of serverless scalability. Avoid creating monolithic functions that handle multiple tasks. For example, in an e-commerce application, a lambda function for processing orders and another for sending email notifications would be ideal. You can club them with SQS and make the two logical flows async.

6. Events and more Events:

Always think of breaking into smaller modules and syncing them together with events. Make use of the event-driven architecture to trigger serverless functions. Utilize AWS services such as Amazon S3, Amazon DynamoDB, or Amazon Simple Notification Service (SNS) to trigger functions based on specific events. For instance, when a new image is uploaded to an S3 bucket, you can automatically resize and optimize it using AWS Lambda.

7. Data and State Durability:

Serverless functions are inherently stateless, which means they do not maintain a session state. Use managed services like Amazon DynamoDB, Amazon Aurora Serverless, or Amazon Simple Queue Service (SQS) to persist and manage application state across invocations.

8. Balance Scalability with Concurrency :

Design your serverless application to handle concurrent invocations effectively. Configure the maximum concurrency limits for your functions to avoid resource exhaustion. AWS provides services like AWS Auto Scaling and Amazon API Gateway to automatically scale your serverless application based on demand.

9. Design for Cold Starts:

Serverless functions may experience latency due to cold starts when invoked infrequently. Employ strategies such as function warmers, provisioned concurrency, or keeping functions warm with periodic invocations. AWS Lambda provides provisioned concurrency to keep functions ready for instant response.

10. Distributed Tracing and Monitoring:

Ensure visibility into your serverless application by implementing distributed tracing and monitoring. AWS X-Ray allows you to trace requests as they flow across different serverless functions, helping you identify performance bottlenecks and optimize your application.

11. Security and Access Control:

Implement proper security measures to protect your serverless applications. Leverage AWS Identity and Access Management (IAM) for fine-grained access control. Apply security best practices, such as using secure API gateways, encrypting data at rest and in transit, and following least privilege principles.

12. Error Handling and Retry Mechanisms:

Design your serverless application to handle errors gracefully. Utilize features like AWS Step Functions for building resilient workflows, or implement retries with exponential backoff to handle transient failures. AWS Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS) can be used for reliable event processing.

13. Cost Optimization:

Optimize the cost of running your serverless application. Configure auto-scaling policies based on demand to avoid over-provisioning. Use AWS Cost Explorer and AWS Budgets to monitor and analyze your serverless costs. Additionally, consider using AWS Lambda Layers to share code across functions and reduce duplication.

14. Integration with Existing Systems:

Leverage AWS services like AWS API Gateway and AWS EventBridge to seamlessly integrate your serverless application with existing systems. Use AWS Lambda as an integration layer to connect disparate components of your application architecture.

Serverless is a great tool. But remember that at the end of the day, it's just another tool. You should not be looking at forcing the tools to do the job. Designing for serverless architecture requires careful consideration of various aspects to ensure scalability, reliability, and cost-effectiveness. By focusing on function granularity, event triggering, state management, scalability, and other key areas, you can harness the full potential of serverless computing. AWS provides a rich set of services that can be used to address these considerations and build highly efficient and scalable serverless applications.

An Associate Director talks opportunities, flexibility, and work culture at CACTUS Tech

Mon, 16 Aug 2021 19:40:46 GMT

Reference

by Cactus Tech

Mayank Raj is the Associate Director, Engineering, at Cactus Labs. He has extensive experience in building high-performance products powered by AI/ML & Big Data. He has designed cloud native architecture that can scale and yet remain cost-effective. He has also worked on more experimental applications like AR/VR.

You joined CACTUS when you were still in university. How did that work out?

That’s right. I joined CACTUS right after my second year of engineering. I had a brief stint with an established name in tech in my second-year summer vacations; I was on contract with their HR team and working on a few apps. I started looking for other contractual opportunities.

I figured out a hack on LinkedIn and applied to some 400 organizations, including CACTUS, within a few minutes. I wasn’t particularly looking at CACTUS. At that point, I hadn’t even heard about it. But I got a call from the recruitment team and an interview was scheduled. Till date, it remains the best 15 lines of JavaScript I have written.

During my third and fourth year of university, I started working with CACTUS. I joined the two-member UI team where I formed the other half. I used to come into the office right after college every day; I’d spend a couple of hours at the office, some from home, and some from the last bench of my university class. You can say that this was my introduction to “work from anywhere.”

You mentioned that you had not even heard of CACTUS. So what drew you to the organization?

CACTUS gave me a lot of flexibility with work hours, and I needed this since I was still in university. They were ok with me visiting the office for 12–14 hours a week and then doing the rest of the work from anywhere.

Also, the projects that were discussed during the interview seemed interesting; these were things I would have wanted to work on.

And most importantly, I was not being treated as an intern.

This was one of the most off-putting things I had to face at other places. I was able to work on fairly complex things, but because I was in university, I was being offered a measly stipend and the promise of “exposure.” I didn’t get the concept. CACTUS didn’t treat me like an intern. I was made a good contractual offer that had good projects and strong ownership from the get-go.

Finally, the fact that I would have a real job even before graduating seemed like a big bragging right. In retrospect, I don’t think I bragged enough about it. :)

Can you describe your role at CACTUS? What is your typical day like?

As Associate Director, Engineering, I serve as a conduit between the business and technology sides of the organization. I take the business requirements and translate them for the tech team. I interface with heads of departments, product managers, and vendors.

I also make sure that our tech roadmap aligns with our business needs and I manage expectations on both sides. My hands-on programming has reduced over the last year, but I do a lot of Proof-Of-Concepts (POCs) and product bootstrapping.

I am involved at the very early stages of all products. I work on the first couple of POCs as well as the initial MVP. After that, I help set up a team and move on to the next project.

I also work on the more experimental stuff. I like to work with new technologies and tools and I get to experiment with something completely new. Almost a year ago, I worked on a VR application. Today I work with a team of BigData Engineers.

What is the most exciting aspect of your role?

This is something that we joke about internally — everyone within the team has equal access to almost everything.

What that means is that I have the flexibility of working in the capacity that I would want to. I don’t have to wait for approvals. I don’t have to wait for someone to give me a green light to start working on something.

If I pick up a problem to work on, I have the freedom to innovate. This has been the case for all the projects I have worked on. This applies to everyone on the team. This essentially means that all of us are constantly looking for ways to improve and at times geek out on the new offerings.

You make CACTUS sound like a startup.

It’s the best of both worlds. The ownership on a project is like what you’d see in a startup. This is especially true in the tech team. But I don’t have the restrictions that a startup would generally have. It’s not that we are directionless or that we don’t have the in-house capabilities to do most of the things.

One of the things that startups struggle with is that they don’t know whom to reach out to. In our case, if I have, say, a problem with the language automation solutions we are working on, I can leverage the expertise of the massive team of editors and the close to 20 years of experience we have. All of that is one chat away. I can simply drop in a message and get the conversation going.

The same goes for budgets. If I can justify the usage, I can have a server farm running to meet my needs.

When UNSILO joined us, they were surprised with the fact that they no longer had to worry about access to subject-matter experts; we have access to Ph.D. holders that we can collaborate with anytime.

What has been the one coolest, biggest, most interesting, or most challenging problem that you’ve worked on?

It’s difficult to choose. The first major feature launch I was involved in end-to-end was a big success within the tech team and the entire organization. It was very well received by the stakeholders.

There was also the time when we were migrating to a new workflow management system, and I was put in charge of revamping the client-facing forms that were the revenue funnels. When I was gearing up for my final year exam in university, I was put in charge of revamping some of the most business critical modules. During that whole process, I took a two-week break for the final university exam and came back to work once the exam was over. So while my batchmates were getting ready for 2–3 months of vacation time, I had kickstarted my professional life!

Then there’s the time we were working on data generation for a tool. We had to process data, which would have taken close to 6 years if done sequentially, and whittle it down to a couple of hours. Around this time, I along with a team member and the CTO were preparing for a trip to the US for that external collaboration. If we didn’t have the data in time, we would essentially not have anything to do while we were in the US. A waste of everyone’s time, money, and effort! That was the most exciting couple of weeks where I was introduced to the problem statement just a month before the plan date. We were actually able to complete the task just a day before our flight.

What would you say to people with your background who probably find working with a startup or the Big Tech companies more exciting? What’s the big draw about CACTUS’s work culture?

For me, the most important consideration is the independence that the organization offers. At CACTUS, I am given a problem and the room to ideate and figure things out on my own. I am given access to leaders with experience whom we can consult, but I am responsible for execution. That has led to a lot of learning opportunities and I can proudly claim to have built something.

CACTUS is at that sweet spot where it’s not too small and facing a funding crunch or too big and saddled with bureaucratic processes. If I think that I can spend some funds on running a pipeline, I don’t have to wait for approvals as long as I can justify it.

In my free time, I consult with some startups on technical aspects and the conversations are always around saving a few hundred dollars here and there. You are putting in more effort into saving something that doesn’t really make sense. It’s not the kind of environment I would like to be in.

Many of my friends working with some big names talk about how restrictive the environment is. Too many hoops to jump through before anything can get started. They are told that they are just starting out and hence cannot call the shots. But it’s the opposite for me and my friends at CACTUS.

A couple of months ago — this is after some 5 years at CACTUS — I had the opportunity of joining a major cloud computing platform. I spoke to a few people who work there to understand the work culture and it didn’t appeal to me.

Some of my friends have stuck around with some big names even though they don’t enjoy the work. I’d rather work at a place where I can do interesting work.

A university batchmate who joined CACTUS recently and who has about two years of work experience is the lead engineer of one of our key tech offerings — a product that generates a couple of hundred thousand dollars in revenue. Those are the kinds of opportunities that people can look forward to here.

We’re hiring!

CACTUS Tech is looking for passionate innovators, ideators, and problem solvers to join its team and build exciting products and solutions. Learn more: https://tech.cactusglobal.io/careers/

Python Building Blocks - Generators

Thu, 15 Jul 2021 19:40:46 GMT

Reference

by Mayank Raj

Every programming language has few aspects that always remain in play and go unnoticed. They take the front seat only in two scenarios - when you read about it and it clicks, “Hey ! I’ve been using this all along. Looks like there’s a term for it” or have some niche use case that is satisfied by that exact phenomena. There are few such aspects common across all languages like Scopes, Closures etc and few that are language specific. In the Python ecosystem, generators, iterators, comprehensions etc fall in this category. There’s a good chance you have already used them if you have written more than 100 lines in python.

In this article, a part of a series touching on such details of the languages we will be looking at one such aspect of Python - Generators. Generators are really powerful if used well. Just like any other tool it works the other way around as well, it can be really bad if not used well or used incorrectly. To a great extent, one can also say that generators are a specialised iterators. Iterators is another beast that deserves a spot of its’s own. For this post, we will focus on just generators.

Before Generators…

Let’s start by looking at a few cases wherein you would want to iterate over a set of objects. You would write something like this…

for item in items:  # items is a list of _something_
     process(item)  # We process each of them one by one

This looks good. You get the job done and move on. Nothing too fancy here. …or is there? Can it really be that simple?

The bottlenecks

Notice how when you are looping over items you have to have the items available to the python interpreter. This means the whole object has to be in memory.

**What if you had to loop over something that is HUGE ! **What if you had to process lines in a 2TB file. You certainly cannot hold the whole of the 2TB in memory to loop over it.
**What happens when the items are dynamically generated ? **In such cases external factors might affect the items that have to be processed. eg, You took a fresh dump of 100K subscribers of your newsletter and started sending out email to them in a loop. While sending out, you were 2/5th in and someone unsubscribed from the mailing list. Ideally you would not want to take a fresh dump of subscribers after every email sent out nor would you want to send email to this user after they have unsubscribed because of a race condition.
**What if you had to abort the loop midway ? **If the items of the loop were processed beforehand to be made available for looping through them, then you have essentially wasted the processing units.
How do you structure your code such that it’s intuitive to read and is not sprinkled with list aggregations and processes all over the place ?

Enter - Generators…

Generator functions allow you to declare a function that behaves like an iterator, i.e. it can be used in a for loop. (source: Python docs)

Essentially Generators are simple python functions that make it possible to loop through each item of a list in a truly sequential manner_,_ without the need of having all the items of the list available beforehand. If it took some computation to generate these items then these computations are done during the loop execution rather than before the loop execution.

Let’s look at a simple example

Generators are simple case wherein instead of returning the value, we are instead giving a token for it. This token can be exchanged for the actual value and this exchange happens at the run time. Things become more interesting when you go about using them…

Do you notice how the response is not the value itself but rather a memory reference ?

At this point, all that the interpreter has done is take the instructions that you have given it and kept it in memory along with every detail that it needs to execute them and give you the correct answer. It is however holding back on actually executing on this bit of information until you give the green light. Loosely speaking, this can be called lazy execution. Figuratively and literally. It has the same attitude that you had for your university submissions - procrastinate everything to the last moment until it is absolutely necessary to execute on the plan.

When you finally need the data…

Usage Patterns for Generators

If you had to read a large file and only process a part of it at once, then generators can help you achieve the goal in an easy manner.

If your operation on the other hand is expensive, takes time or consumes a lot of resources and you have a case wherein if certain conditions are met you can skip performing the operation on the rest of unchecked items, generators can save you from building your own logic to handle the use case.

Finally, you may have also noticed that generators also help in making the code easier to read. Instead of building the logic as a part of your application, it’s a built in functionality at your disposal.

Antipatterns for Generators

Just like any tool, generators do have a flip side as well. You should not throw around generators for every loop statement in your application but rather assess the situation and then use it wisely.

Remember that _generator_s are executed when you actually ask for the value. This makes it a good candidate when you need to factor in the most accurate environmental conditions in your execution (eg, the email newsletter that we discussed earlier)

However when you need the environments to be locked in and do not do so explicitly, generators will pick these dynamically. In some cases, this might not be desirable

For example, in a stock exchange wherein every second the condition changes, the bets placed have to be evaluated with the parameters (or the environment) present when the orders were placed. If you have a backlog of bets to process, you cannot process a bet with new prices. Let’s see this in action

Can you guess what the output of the snippet above would be ? Notice how we change theGlobalMultiplier midway through our processing. Can you do a dryrun to come up with how the multipliedNumber would turn out ?

_Followup question: Can you fix the issue ? We expect all the numbers to be multiplied by the _theGlobalMultiplier at the time of trigger and not execution.

Conclusion

Generators should be treated as a tool. Any good tool has a way to use it optimally, at the right time and under the correct situation. If done so, generators can not only simplify your logic, but also improve the flow of your code and increase its readability. However you should know its working well enough because if used under the wrong situation, it can go undetected. It becomes very tricky and hard to spot the source of the problem in such cases.

Next time you are looping over something, ask yourself a few questions… Is there any space or memory constraints that I should be taking into consideration ? Do I need to process a chunk of data at a time or the whole data ? Do I know of cases which I can use to exit for the loop early ? How do I make the best use of those to optimise for computation time ?

Do share the situations in which you have seen a clever use of this gem of python or better yet, used it.

Of Aspirations and Achievements- Front-end Intern to Solutions Architect in 3.5 Years

Tue, 14 Jan 2020 19:40:46 GMT

Reference

by Cactus Tech

Look around and the world has a lot of inspiration and motivation to offer. But nothing quite compares to a story of humble beginnings and great success.

Mayank’s is one such story- about faith and risk, about the courage to dive into the unknown and do what’s never been done before, about the right energy focused at the right places, about progress as an outcome of being ever-learning, always-exploring, and growing out-of-the-mould you’ve been set in.

The story behind the story

Mayank became a part of CACTUS while he was entering the third year of his engineering undergraduate study. He started freelancing a little before that. Initially, it was all about seniors and college committees. He was “the go-to guy” when it came to developing websites for anything, at a time when everyone wanted websites for their next startup or to simply get an online presence. He did an internship with Tata Consultancy Services in the summer break of his first year of college and completed in a week and half, what his manager had planned for him for two months. In the break of his second year, he did a project with Directi. By this time, he had completed multiple projects and built up his portfolio. But working at CACTUS would get him involved in a long-term project and that was a big leap from the unstructured, erratic in-flow of freelancing work. CACTUS seemed like the perfect opportunity to hone his skills and start a career — all at the same time.

First impressions

“I was a second-year college student with only some freelance projects and workshop talks to brag about. I didn’t have an under-100 CodeChef rank, some national-level competition medal to show. When I was called for an interview at CACTUS, I did not expect it to be so big. When I got to the office for interview, I was taken aback. The office spanned multiple floors, Cactus had offices in 7 countries, and I saw lot of expats working in the office, and I was interviewed by two people, then team lead and the VP of engineering.” exclaims Mayank, who was surprised at the scale of work done at CACTUS, just like we were amazed at the amount of work he could handle and the pace at which he could deliver results.

“When I finally joined, I was fortunate enough to have a manager who knew what he was doing and was very good at it. On top of that he was open to my suggestions. Also, I had only heard of terms like ‘no hierarchy at work’, ‘open structure’. I suggested a few things to the then Technical Architect of the team, and in a weeks’ time was working on implementing them.” he recounts.

So far, so great

Mayank joined CACTUS at a stage when the company started using technology not as a support system but to drive transformation and to scale up. This shift, coupled with the fact that even at that time CACTUS was not a small company by any means meant that there was a big list of ‘To-Dos’. This being the perfect environment for him to grow, meant that Mayank would occasionally pick up something exploratory to work on, create a POC, demo it and then start working on it as a full-blown project. Most of the initial big tickets have been this way.

Given the scale of business, whatever built here is shipped to a huge global audience. Mayank has been able to grow in understanding regarding potential problems that may arise when a product is used by someone in Japan, China, Brazil, USA, etc. and how to handle them, thus, opening up many new avenues for him as a developer.

The developer’s ladder

The best thing about CACTUS is that your title doesn’t define your work but rather, it’s the other way around. Mayank recounts, “Even while in college, I was sleeping during the lectures in the morning and working on improving the then Angular2 Core to accommodate multilingual builds from the same codebase. I had the opportunity to design the architecture and develop the pipeline for a module that processes a great load of events every hour. The project introduced me to AWS, BigData, Analytics and many more things.”

With more responsibility after each project, its quite evident that a company that focuses on merit gained through action, doesn’t make years of experience, the team you’re a part of, or your professional title, a mandate for where you can go.

“Every other day for me at Cactus is different. Today I get to work on multiple exciting projects in the domain of Artificial Intelligence, NLP (Natural Language Processing), Big Data, cloud architecture and also cutting-edge technologies like Web based Augmented/Virtual Reality. I also have the privilege of managing a small but extremely talented and motivated team, and together we not only bootstrap a product but take it from an idea to a production-ready, scalable and fault-tolerant system. In the process we have learned what it takes to build, scale, and manage large scale applications. I also manage recruitment, take interviews for the team and conduct campus placements. I visited my own university campus in just under two years of graduation, only this time, on the other side of the table to recruit students for my team.

I’ve also visited NYC, USA, multiple conferences in India and much more. I’ve interacted with core members of the engineering team from AWS, GCP, etc. for various products. In the process I’ve met some brilliant people — both within CACTUS and outside, who are amazing at what they do.”, he remarks.

Some people learn and deliver more in two years than others do in a decade. It’s not the time that matters, it’s the individual, and CACTUS gives you the right opportunities and freedom to turn your aspirations into your achievements.

Demystifying JavaScript Closures

Sat, 30 Nov 2019 19:40:46 GMT

Reference

by Mayank Raj

JavaScript as a language is easy to get started with but difficult to master. This is partly owing to the fact that you can get up and running with a fairly complex application built with JS and not know in detail the workings of it and somewhat because once you have a functional system there is not much motivation to go back and break it down. Hoisting, Lexical Scopes, Closures, this and even IIFFE are not well understood by developers. Understanding concepts like these not only make the language interesting but also explain a lot of weird - “Oh this works!! I don't know why or how, but it does!”. Let's break down Closures today.

A write up on Closure is not complete without the line, ‘you have already used them before’, so its only fitting that I start with it as well.

Understanding Lexical Scopes

A good point to start with would be understanding what scopes actually are. Scope, at the most basic level is exactly what it sounds like — scope of the variables. It’s how the compiler answers the following question -

I have two variable _foo_ and _bar_, if someone (function, assignment, etc.) asks for it should I give it to the requester? Just to break it to you, yes JavaScript is a compiled language. It all happens at run time and not ahead of time like in the case of C++, Java, etc. so it's not very evident. Hoisting is one of the proof of this. Anyways, getting back to the topic, a compiler should know where a variable was declared and who has access to that variable. Apart from this being the basic requirement of a program, it is also useful for garbage collecting (once I know no one can access a variable, I can safely delete it). Everything in JS is defined under a scope. That scope can either be of global execution context or that of a function. Every new scope is a bubble inside of its parent scope. Now the crucial part is that each scope has access to everything declared in itself and also in the scope of its parents, ancestors etc. The first place to look for is the local scope, then one step above and so on. Let's see it in action.

We’ll break down the execution of the above example when we call greet. There are three scopes in play here - Global execution, that of function greet and of function printGreet. The flow of how variables are discovered is as follows. When the execution comes to line 6, there are three variables that are called for here.

heyString: There is no declaration found in the local scope i.e. that of the function printGreet. We move to an upper bubble and into the scope of the function greet. Voila, we got the reference of the variable here.
name: Local scope of printGreet doesn't have any references of the variable, we go a step above and we get the reference in the function greet again. This time it was an argument that was passed to the function.
greetingEmoji: The story repeats here as well but we don't find any reference in the scope of the function greet as well. We know what we have to do and move up in the scope bubble. We are now in the Global Execution scope and here we find a reference of the variable we were looking for. Thus declaring scope is in the hands of the author. It depends on where you declare something i.e. the scope of the variable, the bubble that it is kept in is dependent on the position of its declaration in the program. The nesting of scopes is a byproduct of this functionality. This is what is termed as lexical scope.

Closure

Now on to the main business. Closure can be summed up as:

The ability of a function to hold a reference to it’s Lexical scope even when the execution is happening outside it. With a new profound knowledge of Lexical Scopes and the above line, have a look at the below code and try to guess the final output and reason it.

Let’s make our observations loud and clear:

A function customGreet accepts an argument which is assigned to a local variable customGreetingPrefix. It also has a declaration of a function which is eventually returned.
This declaration is of a function printGreet which accepts an argument name and prints a greeting message.
We execute the function customGreet twice at line 8 and 9. With that we store the response in the respective variables.
The response itself is a function, thus these two variables now hold a reference to a function (i.e. printGreet)
These two functions are then executed. If you notice, the function printGreet was declared in the lexical scope of customGreet. But when we finally execute it at line 10 and 11, the lexical scopes have now changed, it is executed in the parent of customGreet. So going by the theory that access to a variable is only successful if it is present in either the current scope or any of the parent scope, we can say that the function printGreet would not be able to access customGreetingPrefix. But to our surprise, it can access it! What you see my friend, is closure in pay here. By definition, a function holds the reference to it's scope. So in our case, the function printGreet holds the reference to the lexical scope of customGreet, which, (you guessed it) brings the reference of customGreetingPrefix with it (technically also of greetingPrefix). Thus when we create the scope of customGreet at line 8 and 9, the two bubbles are preserved by the JS engine as it knows that printGreet may ask for anything in that scope at any point later. Thus we get the following output.

...
greetWithHey('Tom');     // Hey Tom.
greetWithHello('Jerry'); // Hello Jerry.

Now that you know the signature of Closure, they are not very hard to find. Every JS module that you use adopts closure in some form or the other. Consider the following example.

function theBestModule(config) {
    var propertyA = config['propertyA'];
    var propertyB = config['propertyB'];
    var propertyC = config['propertyC'];
    def combinePropertyAandB() {
        return (propertyA + propertyB);
    }
    def combinePropertyAandC() {
        return (propertyA + propertyC);
    }
    return {
        'combineAandB': combinePropertyAandB,
        'combineAandC': combinePropertyAandC
    }
}
myModulesConfig = { /* ... */ }
myModule = theBestModule(myModulesConfig);
console.log(myModule.combineAandB) // TADA, we found closure
console.log(myModule.combineAandC) // ...yet again. Not that difficult huh.

Virtually all modules in JS accept a set of configuration in some form or the another. Internally, they set these configurations to default and then override it if the user passed something custom. If you notice, these configurations are accessed by methods that the module (a function to be precise) exposes. You don’t have access to these configurations because the lexical scope is not available to you but the functions declared inside the module hold the reference.

With all the knowledge that you have collected, I’ll leave you with the following program. Try to guess what the output would be at each log statement and reason it.

for(var i = 0; i<5; i++){
    setTimeout(() => console.log(i), 0); // ?
}

Conclusion

Lexical scope is not unique to above examples or to some special cases. It comes into play every time you define a function or create a code block. Even a plain and simple function call has closure over its parent scope. It just becomes more evident in some cases and can quickly become confusing to developers.

You don’t have to know it to use it but the knowledge of the pattern enables you to effectively use it to your benefit. In many cases, it becomes easy to reason why something is happening the way it is. Closure can help you write clean code by best utilizing resources.

Managing Continuous Integration Pipelines with Jenkins

Tue, 11 Jun 2019 19:40:46 GMT

Introduction

Jenkins is acclaimed as the "leading open-source automation server," a versatile tool for automating diverse aspects of a development workflow. It enables the streamlining of processes such as running test cases, deploying the latest builds, or even automating entire Continuous Integration (CI) pipelines. Jenkins can be configured to execute these tasks in various environments, including dedicated containers. Its distributed nature allows for scalable workloads and simultaneous task execution across different environments, such as running test cases for an Angular project in multiple browsers and versions concurrently. With an extensive library of plugins, Jenkins seamlessly integrates and communicates with external services like Git, Slack, and more.

In this tutorial, you will learn to set up a Jenkins instance. You will also explore how to leverage the modern Blue Ocean plugin interface and integrate with GitHub to automate tests. You can use this GitHub repository as a reference if needed.

“Learn how to manage Continuous Integration pipelines with Jenkins.” Tweet This

1. Installing Jenkins

To begin, you'll need a server to host Jenkins, making it accessible globally to you and any services you integrate with it. At a minimum, this requires a static IP address or a valid DNS name pointing to the server. This domain (or IP) will be used by external services like GitHub to notify Jenkins of new events.

If you already have a server, you can skip to section 1.2.

1.1. Setting up a DigitalOcean Droplet

DigitalOcean is a cloud provider simplifying the setup of virtual servers, known as droplets. If you don't have a DigitalOcean account, you can use this link to get $100 in credit over 60 days.

Navigate to the droplets management console and click on "Create Droplet".
Choose an image: Select "Ubuntu 18.04" (or a current LTS version).
Choose a plan: Start with a basic plan (e.g., $5/month).
Choose a datacenter region: Select one closest to you (e.g., Bangalore).
Authentication: Add SSH keys if you have them, or you can use password authentication (explained later).
Hostname: Assign a descriptive hostname (e.g., jenkins-test).
Verify details and click "Create".

Once the droplet is created, you'll receive its IP address, username, and password via email (if you didn't use SSH keys). Log into the server using a terminal:

ssh <username>@<public_ip_address>

Note: Replace <username> and <public_ip_address> with your droplet's credentials.

You might be prompted to add the IP address to your list of known hosts; accept it. Enter your password when prompted. You may be asked to reset the password.

You now have a virtual server ready for Jenkins installation.

1.2. Installing Jenkins

After accessing your server, update its packages:

sudo apt update
sudo apt upgrade

Install the Java 8 runtime environment, which is required by Jenkins:

# Install Java 8
sudo apt install openjdk-8-jdk

# Verify installation
java -version

If the output is similar to openjdk version "1.8.0_212", proceed to install Jenkins.

Add the Jenkins repository key to the system:

wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -

Add the Debian package repository to your system's sources:

sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'

Update apt to use the new source:

sudo apt-get update

Finally, install Jenkins:

sudo apt-get install jenkins

Congratulations! Jenkins is installed. Access your Jenkins setup at http://<public_ip_address>:8080. Jenkins listens on port 8080 by default.

2. Setting Up Root User

When you first access your Jenkins setup (http://<public_ip_address>:8080), you'll be asked to "Unlock Jenkins" with a password. This password ensures that the person setting up Jenkins has server access.

Fetch this password from your server:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Copy the output and paste it into the Jenkins setup page.

On the next screen, select "Install suggested plugins." You'll see the installation progress.

After plugin installation, create your first admin user by filling in the required details.

Image: Filling in the details about the first Jenkins user.

Next, provide the domain name Jenkins will use (e.g., jenkins.example.com). For this tutorial, you can use your server's public IP address.

Click "Save and Finish" to complete the setup. You'll be redirected to your Jenkins dashboard.

You have successfully installed Jenkins and are ready to connect it with GitHub.

3. Installing Jenkins Blue Ocean

Jenkins Blue Ocean offers a modern interface for interacting with Jenkins.

From your Jenkins dashboard, select "Manage Jenkins" from the left navigation bar.
Select "Manage Plugins."
Click on the "Available" tab.
Search for "Blue Ocean."
Check the box next to it and click "Install without restart."

Image: Installing Jenkins Blue Ocean plugin.

Once installation completes, access the new interface at http://<public_ip_address>:8080/blue.

4. Setting Up a Pipeline and Connecting to GitHub

When you open the Blue Ocean URL for the first time, you'll be prompted to create a new pipeline.

Image: Create your first Blue Ocean pipeline.

A Pipeline is a defined set of procedures with tasks like running test cases, verifying code, creating deployment packages, and deploying to servers. Triggers initiate these tasks, and each run is called a Job.

The goal is to use GitHub actions (like commits) as triggers.

Click "Create a new Pipeline."
Select "GitHub."
You'll need to create a GitHub access token. Click the "Create access token here" link.
- Note: "Jenkins Integration" (or similar).
- Select scopes: Leave with default values.
Click "Generate token." Copy this token (GitHub shows it only once).
Paste the token into the "Your GitHub access token" field in Blue Ocean and click "Connect."

Jenkins will now ask you to select a GitHub organization and repository.

Open GitHub in a new tab and create a new repository named jenkins-test.
Back in Jenkins:
- Select the organization where the repo resides.
- Select the jenkins-test repository.
Click "Create Pipeline."

Image: Using Jenkins to create a new pipeline on a GitHub repository.

Your pipeline will be ready in a few seconds, and you'll be redirected to the pipeline editor.

5. Adding Stages to the Pipeline

A Stage is a logical block in a pipeline housing various Steps (individual actions).

For this tutorial, the pipeline will check for the presence of two files: file-1.txt and file-2.txt.

First, define the Agent (the environment where the pipeline runs). Jenkins supports distributed architectures with master and agent nodes, or Docker containers. For this tutorial, Jenkins will run the pipeline on the host server itself.

In the pipeline editor, select "any" from the "Agent" dropdown on the right.
Leave environment variables blank for now.

To add a stage:

Click the plus (+) button next to "Start."
Name the stage (e.g., "Check file 1").
Click "Add Step."
Choose step type: "Shell Script."
Enter the command: cat file-1.txt (this checks if the file exists; if not, the step fails).

Similarly, add a second stage named "Check file 2" with a "Shell Script" step using the command cat file-2.txt.

Image: Adding multiple stages to a Jenkins job.

Save this pipeline to your repository. Pipelines are configurations stored in a Jenkinsfile at the root of your repository. You can also write this file directly (learn more here).

Click "Save" on the top right.
Add a commit message (e.g., "configuring Jenkins").
Select the master branch.
Click "Save and Run." This commits the Jenkinsfile to your branch and triggers the build.

You've defined pipeline stages and triggered your first build.

6. Setting Up GitHub Webhook

To automate builds upon new commits:

Go to your GitHub repository (jenkins-test).
Click the "Settings" tab.
In the left navigation bar, select "Webhooks."
Click "Add Webhook."
Payload URL: http://<public_ip_address>:8080/github-webhook/ (replace <public_ip_address>).
Leave other settings as default and click "Add Webhook."

Image: Creating a GitHub webhook to notify Jenkins about new commits.

GitHub will now notify Jenkins of any new commits.

7. Triggering the First Automated Build

Test the integration by making a commit. You can either:

Clone your repository locally, add files, commit, and push.
Use GitHub's website to create the files.

Add two files to the project root: file-1.txt and file-2.txt (content doesn't matter for this test).

If working locally, commit and push the new files. GitHub will notify Jenkins, triggering an automatic build (which should succeed if both files are present).

If using GitHub's website, adding the first file will trigger a build (which will likely fail if the second file check is active). Adding the second file will trigger another build, which should then succeed.

Image: Jenkins showing the list of builds that were automatically triggered after GitHub's webhooks notifications.

New commits to this repository will now automatically trigger builds, and results will be updated on GitHub.

8. Recording Test Results and Artifacts

Each Jenkins build typically starts with a shallow clone of the branch in a new, isolated folder. After the build, Jenkins performs a cleanup, deleting this folder. To preserve test results or build outputs, use build artifacts.

Note: Build artifacts are stored in /var/lib/jenkins/workspace/ on your Jenkins server if you're curious.

This section transforms your repository into a simple npm project to demonstrate recording test results. (npm/Node.js knowledge isn't strictly required to follow, but you need them installed locally for these steps.)

Note: Jenkins understands JUnit XML format for test results, a standard supported by most test runners. This tutorial uses MochaJS.

Ensure npm and Node.js are installed locally.
Clone your jenkins-test repository.

Initialize an npm project:

# from the repository project root
npm init -y

Install dependencies:

# install the test case runner
npm install --save-dev mocha

# install the JUnit XML reporter
npm install --save-dev mocha-jenkins-reporter

Create test.js with the following content:

const assert = require('assert');

describe('Addition', function() {
  it('2 + 2 should be 4', function() {
    assert.equal((2 + 2), 4);
  });

  it('1 + 5 should be 6', function() {
    assert.equal((1 + 5), 6);
  });
});

describe('Type Comparison', function() {
  it('\'5\' == 5 should be true', function() {
    assert.equal(('5' == 5), true);
  });

  it('\'5\' === 5 should be false', function() {
    assert.equal(('5' === 5), false);
  });
});

Run tests locally:
```
npx mocha
```

Generate reports in JUnit XML format:

# define where you want the test results
export JUNIT_REPORT_PATH=./test-results.xml

# run mocha and tell it to use the JUnit reporter
npx mocha --reporter mocha-jenkins-reporter

This saves results to test-results.xml.

Update your GitHub repository:

# make git ignore the node_modules directory
echo "node_modules/" > .gitignore

# stage your new code
git add .

# commit it to git
git commit -m "Added Test Cases"

# push it to GitHub
git push origin master

Now, configure Jenkins to run these tests and find the XML report.

Go to the Blue Ocean pipeline editor (e.g., http://<public_ip_address>:8080/blue/organizations/jenkins/pipeline-editor/jenkins-ci-cd/master/).
Add a new stage after "Check file 2" called "Install dependencies".
- Add a "Shell Script" step: npm install -d

Add another stage after "Install dependencies" called "Run test cases".

Add a "Shell Script" step:

# define where you want the test results
export JUNIT_REPORT_PATH=./test-results.xml

# run mocha and tell it to use the JUnit reporter
npx mocha --reporter mocha-jenkins-reporter

Click "Save" on the top right.

Install npm and Node.js on your Jenkins server (as global dependencies):

ssh <username>@<public_ip_address>
# Then, on the server:
sudo apt install nodejs npm

To capture JUnit-formatted test results, you need to update the Jenkinsfile directly, as the visual editor might not support the post stage for this. Open Jenkinsfile (at the project root) and update it:

pipeline {
  agent any
  stages {
    // ... all existing stages remain untouched ...
    stage('Install dependencies') {
        steps {
            sh 'npm install -d'
        }
    }
    stage('Run test cases') {
        steps {
            sh '''
            export JUNIT_REPORT_PATH=./test-results.xml
            npx mocha --reporter mocha-jenkins-reporter
            '''
        }
    }
  }
  post {
    always {
        junit 'test-results.xml'
    }
  }
}

Commit and push this change. You should now find test results on the "Tests" tab of your Jenkins pipeline.

Image: Checking tests results on a Jenkins pipeline.

You've learned to record test results and track their status across builds.

9. Exploring Plugins

Plugins are the backbone of Jenkins' power and flexibility. Explore available plugins here. Examples:

Slack Notification: Notify your Slack channel about build statuses.
Kubernetes Plugin: Integrate with Kubernetes.
Cloud container service plugins like Amazon Elastic Container Service and Azure Container Service.

“I just built a Continuous Integration pipeline with Jenkins. So cool!” Tweet This

Aside: Syncing Authentication in CI using the Auth0 Deploy CLI Tool

Managing authentication configuration across different environments (e.g., testing, production) can be tricky. Auth0 can help manage this part of your Jenkins pipeline using the Auth0 Deploy CLI tool.

The Deploy CLI tool allows you to:

Import and export Auth0 tenant configuration objects (tenant settings, rules, connections).
Export data to a predefined directory structure or a YAML configuration file.
Call the tool programmatically.
Replace environment variables.

To try this:

Sign up for a free Auth0 account.
Check the documentation on how to install the CLI tool and incorporate it into your build environment.

Note: This tool can be destructive to your Auth0 tenant. Please read the documentation and test on a development tenant before using it in production.

Conclusion

Congratulations! You now have a Jenkins setup capable of automatically testing your codebase with each push. You can track build history, identify when test cases fail, and perform integration testing after pull request merges.

What are your thoughts? Will you use Jenkins in production? Let us know in the comment box below.

How To Build a Search Bar with RxJS

Wed, 17 Apr 2019 19:40:46 GMT

Introduction

Reactive Programming is a paradigm concerned with asynchronous data streams, in which the programming model considers everything to be a stream of data spread over time. This includes keystrokes, HTTP requests, files to be printed, and even elements of an array, which can be considered to be timed over very small intervals. This makes it a perfect fit for JavaScript as asynchronous data is common in the language.

RxJS is a popular library for reactive programming in JavaScript. ReactiveX, the umbrella under which RxJS lies, has its extensions in many other languages like Java, Python, C++, Swift, and Dart. RxJS is also widely used by libraries like Angular and React.

RxJS’s implementation is based on chained functions that are aware and capable of handling data over a range of time. This means that one could implement virtually every aspect of RxJS with nothing more than functions that receive a list of arguments and callbacks, and then execute them when signaled to do so. The community around RxJS has done this heavy lifting, and the result is an API that you can directly use in any application to write clean and maintainable code.

In this tutorial, you will use RxJS to build a feature-rich search bar that returns real-time results to users. You will also use HTML and CSS to format the search bar.

(Image from the original tutorial: Demonstration of Search Bar)

Something as common and seemingly simple as a search bar needs to have various checks in place. This tutorial will show you how RxJS can turn a fairly complex set of requirements into code that is manageable and easy to understand.

Prerequisites

Before you begin this tutorial you’ll need the following:

A text editor that supports JavaScript syntax highlighting, such as Atom, Visual Studio Code, or Sublime Text. These editors are available on Windows, macOS, and Linux.
Familiarity with using HTML and JavaScript together. Learn more in How To Add JavaScript to HTML.
Familiarity with the JSON data format, which you can learn more about in How to Work with JSON in JavaScript.

The full code for the tutorial is available on Github.

Step 1 — Creating and Styling Your Search Bar

In this step, you will create and style the search bar with HTML and CSS. The code will use a few common elements from Bootstrap to speed up the process of structuring and styling the page so you can focus on adding custom elements. Bootstrap is a CSS framework that contains templates for common elements like typography, forms, buttons, navigation, grids, and other interface components. Your application will also use Animate.css to add animation to the search bar.

First, create a file named search-bar.html.

Next, create the basic structure for your application. Add the following HTML to the new file:

<!DOCTYPE html>
<html>
  <head>
    <title>RxJS Tutorial</title>
    <!-- Load CSS -->

    <!-- Load Rubik font -->

    <!-- Add Custom inline CSS -->

  </head>
  <body>
      <!-- Content -->

      <!-- Page Header and Search Bar -->

      <!-- Results -->

      <!-- Load External RxJS -->

      <!-- Add custom inline JavaScript -->
      <script>

      </script>
  </body>
</html>

Load the CSS for Bootstrap and Animate.css. Add the following code under the  comment:

<!-- Load CSS -->
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css" integrity="sha384-GJzZqFGwb1QTTN6wy59ffF1BuGJpLSa9DkKMp0DgiMDm4iYMj70gZWKYbI706tWS" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/animate.css/3.7.0/animate.min.css" />

This tutorial will use a custom font called Rubik from the Google Fonts library. Load the font by adding the following code under the  comment:

<!-- Load Rubik font -->
<link href="https://fonts.googleapis.com/css?family=Rubik" rel="stylesheet">

Next, add the custom CSS to the page under the  comment. This will style the headings, search bar, and results.

<!-- Add Custom inline CSS -->
<style>
  body {
    background-color: #f5f5f5;
    font-family: "Rubik", sans-serif;
  }
  
  .search-container {
    margin-top: 50px;
  }
  .search-container .search-heading {
    display: block;
    margin-bottom: 50px;
  }
  .search-container input,
  .search-container input:focus {
    padding: 16px 16px 16px;
    border: none;
    background: rgb(255, 255, 255);
    box-shadow: 0 2px 4px 0 rgba(0, 0, 0, 0.2), 0 25px 50px 0 rgba(0, 0, 0, 0.1) !important;
  }

  .results-container {
    margin-top: 50px;
  }
  .results-container .list-group .list-group-item {
    background-color: transparent;
    border-top: none !important;
    border-bottom: 1px solid rgba(236, 229, 229, 0.64);
  }

  .float-bottom-right {
    position: fixed;
    bottom: 20px;
    left: 20px;
    font-size: 20px;
    font-weight: 700;
    z-index: 1000;
  }
  .float-bottom-right .info-container .card {
    display: none;
  }
  .float-bottom-right .info-container:hover .card,
  .float-bottom-right .info-container .card:hover {
    display: block;
  }
</style>

Now that the styles are in place, add the HTML for the header and input bar under the  comment:

<!-- Content -->
<!-- Page Header and Search Bar -->
<div class="container search-container">
    <div class="row justify-content-center">
        <div class="col-md-auto">
        <div class="search-heading">
            <h2>Search for Materials Published by Author Name</h2>
            <p class="text-right">powered by <a href="https://www.crossref.org/">Crossref</a></p>
        </div>
        </div>
    </div>
    <div class="row justify-content-center">
        <div class="col-sm-8">
        <div class="input-group input-group-md">
            <input id="search-input" type="text" class="form-control" placeholder="eg. Richard" aria-label="eg. Richard" autofocus>
        </div>
        </div>
    </div>
</div>

This uses Bootstrap's grid system. The search bar has an id of search-input, which will be used to bind a listener.

Next, create a location to display search results. Under the  comment, add the following:

<!-- Results -->
<div class="container results-container">
    <div class="row justify-content-center">
        <div class="col-sm-8">
        <ul id="response-list" class="list-group list-group-flush"></ul>
        </div>
    </div>
</div>

The ul element with the id response-list will hold the search results.

At this point, the search-bar.html file has its basic structure and styling. In the next step, you will write the JavaScript function to handle search terms and results.

Step 2 — Writing the JavaScript

With the HTML structure formatted, you can write the JavaScript code. This code will serve as the foundation for the RxJS implementation.

Load the RxJS library by adding the following under the  comment in search-bar.html:

<!-- Load RxJS -->
<script src="https://unpkg.com/@reactivex/rxjs@5.0.3/dist/global/Rx.js"></script>

Inside the <script> tag (under the  comment), store a reference to the HTML element where results will be displayed:

// Add custom inline JavaScript
const output = document.getElementById("response-list");

Next, add a function to convert the JSON response from the API into HTML elements. This function will clear previous results and set a delay for search result animation. Add the showResults function:

const output = document.getElementById("response-list");

function showResults(resp) {
    var items = resp['message']['items'];
    output.innerHTML = "";
    let animationDelay = 0; // Corrected: Use let or var for declaration
    if (items.length == 0) {
        output.innerHTML = "<li>Could not find any :(</li>"; // Corrected: Wrap message in <li> for consistency
    } else {
        items.forEach(item => {
        let resultItem = `
        <div class="list-group-item animated fadeInUp" style="animation-delay: ${animationDelay}s;">
            <div class="d-flex w-100 justify-content-between">
            <h5 class="mb-1">${(item['title'] && item['title']) || "&lt;Title not available&gt;"}</h5>
            </div>
            <p class="mb-1">${(item['container-title'] && item['container-title']) || ""}</p>
            <small class="text-muted"><a href="${item['URL']}" target="_blank">${item['URL']}</a></small>
            <div> 
            <p class="badge badge-primary badge-pill">${item['publisher'] || ''}</p>
            <p class="badge badge-primary badge-pill">${item['type'] || ''}</p> 
            </div>
        </div>
        `;
        output.insertAdjacentHTML("beforeend", resultItem);
        animationDelay += 0.1;                        
        });
    }
}

The if block checks for search results. If results are found, the forEach loop displays them with an animation.

You've now laid the groundwork for RxJS by creating a function that accepts results and renders them on the page.

Step 3 — Setting Up a Listener

RxJS deals with data streams. In this project, the stream is the series of characters a user types into the search bar. You will add a listener to the input element.

Recall the search-input identifier used for the input field:

<input id="search-input" type="text" class="form-control" placeholder="eg. Richard" aria-label="eg. Richard" autofocus>

Create a variable to hold a reference to the search-input element. This will be the Observable for input events. Add this line within your <script> tags, after the showResults function:

// ... (output variable and showResults function)

let searchInput = document.getElementById("search-input");

Use the fromEvent operator from RxJS to listen for input events on this DOM element. Add the following line:

// ... (output variable and showResults function)

let searchInput = document.getElementById("search-input");
Rx.Observable.fromEvent(searchInput, 'input')
// ... (more operators will be chained here)

The listener is now set up to notify your code of updates to the input element.

Step 4 — Adding Operators

Operators are pure functions that perform operations on data streams. You'll use operators for tasks like buffering input, making HTTP requests, and filtering results.

Pluck Target Value: The DOM input event contains various details. We are interested in the value typed into the target element. Use the pluck operator:
```
let searchInput = document.getElementById("search-input");
Rx.Observable.fromEvent(searchInput, 'input')
    .pluck('target', 'value')
// ...
```
Filter by Search Term Length: Set a minimum search term length (e.g., three characters) because shorter terms might not yield relevant results or the user might still be typing. Use the filter operator:
```
// ...
Rx.Observable.fromEvent(searchInput, 'input')
    .pluck('target', 'value')
    .filter(searchTerm => searchTerm.length > 2)
// ...
```

Debounce Input: To ease the load on the API server, ensure requests are sent only at intervals (e.g., 500ms). Use the debounceTime operator:

// ...
Rx.Observable.fromEvent(searchInput, 'input')
    .pluck('target', 'value')
    .filter(searchTerm => searchTerm.length > 2)
    .debounceTime(500)
// ...

Distinct Until Changed (Input): Ignore the search term if it hasn't changed since the last API call. This further optimizes API calls. Use the distinctUntilChanged operator:

// ...
Rx.Observable.fromEvent(searchInput, 'input')
    .pluck('target', 'value')
    .filter(searchTerm => searchTerm.length > 2)
    .debounceTime(500)
    .distinctUntilChanged()
// ...

SwitchMap for API Calls: Query the API with the search term using RxJS's AJAX implementation. switchMap is used to chain AJAX calls. It cancels previous pending requests if a new search term comes in. The map operator within switchMap structures the API response. The example uses the Crossref API.

// ...
Rx.Observable.fromEvent(searchInput, 'input')
    .pluck('target', 'value')
    .filter(searchTerm => searchTerm.length > 2)
    .debounceTime(500)
    .distinctUntilChanged()
    .switchMap(searchKey => Rx.Observable.ajax(`https://api.crossref.org/works?rows=50&query.author=${searchKey}`)
        .map(resp => ({
            "status" : resp["status"] == 200,
            "details" : resp["status"] == 200 ? resp["response"] : [],
            "result_hash": Date.now() // A simple way to check if response content changed
        }))
    )
// ...

The response is broken into:

status: HTTP status (true if 200).
details: The actual response data.
result_hash: A timestamp to help detect if results have changed.

Filter Unsuccessful API Responses: Use the filter operator to only accept successful (status 200) API responses:

// ... (previous operators)
    .switchMap(searchKey => Rx.Observable.ajax(`https://api.crossref.org/works?rows=50&query.author=${searchKey}`)
        .map(resp => ({
            "status" : resp["status"] == 200,
            "details" : resp["status"] == 200 ? resp["response"] : [],
            "result_hash": Date.now()
        }))
    )
    .filter(resp => resp.status !== false) // or resp.status === true
// ...

Distinct Until Changed (Response): Only update the DOM if the API response has actually changed. This reduces resource-heavy DOM updates. Use distinctUntilChanged with a custom comparator function checking the result_hash.
```
// ... (previous operators)
    .filter(resp => resp.status !== false)
    .distinctUntilChanged((a, b) => a.result_hash === b.result_hash)
// ...
```
This compares the result_hash of the current and previous emissions. If they are the same, the data is filtered out.

You've now created a pipeline that processes user input, performs checks, makes an API call, and formats the response, optimizing for resource usage.

Step 5 — Activating Everything with a Subscription

The subscribe operator is the final link that connects an Observer to the Observable, enabling the flow of data. It typically implements three methods:

onNext: Specifies what to do when an event is received.
onError: Handles errors. No further onNext or onCompleted calls are made after an error.
onCompleted: Called when onNext has been called for the final time (no more data).

This enables lazy execution: the Observable pipeline is defined but only starts emitting data upon subscription.

Subscribe to the Observable and route the data to the showResults function:

// Full RxJS chain:
let searchInput = document.getElementById("search-input");

Rx.Observable.fromEvent(searchInput, 'input')
    .pluck('target', 'value')
    .filter(searchTerm => searchTerm.length > 2)
    .debounceTime(500)
    .distinctUntilChanged()
    .switchMap(searchKey => Rx.Observable.ajax(`https://api.crossref.org/works?rows=50&query.author=${searchKey}`)
        .map(resp => ({
            "status" : resp["status"] == 200,
            "details" : resp["status"] == 200 ? resp["response"] : [],
            "result_hash": Date.now()
        }))
    )
    .filter(resp => resp.status !== false)
    .distinctUntilChanged((a, b) => a.result_hash === b.result_hash)
    .subscribe(resp => showResults(resp.details)); // Pass only the details to showResults

Save the search-bar.html file. Open it in your web browser to test the search bar.

(Image from the original tutorial: The completed search bar) (GIF from the original tutorial: Content being entered into the search bar, showing behavior with few characters)

You have now subscribed to the Observable, activating your search bar.

Conclusion

In this tutorial, you created a feature-rich search bar using RxJS, CSS, and HTML that provides real-time results. The search bar requires a minimum of three characters, updates automatically, and is optimized for both client and API server load.

Complex requirements were addressed with concise RxJS code, leading to a solution that is reader-friendly and maintainable.

For further reading, refer to the official RxJS API documentation.

Generating Synthetic Music with JavaScript - Introduction to Neural Networks

Wed, 17 Apr 2019 19:40:46 GMT

Reference

by Mayank Raj

Introduction

Recurrent Neural Networks (RNN) are a way to consider the dimension of time when training or inferring from the Neural Network. While talking about Neural Networks in general it is assumed that inputs map to the same output irrespective of the order or sequence of inputs. This is true in cases when you have to identify objects in an image or predict a candidate’s eligibility for loan based on various circumstances. This is because each entity is independent of the previous entity. But Neural Networks built on this assumption do not perform well in cases where the input sequence is one of the most crucial features like machine translation where a sequence of words matter, or in case of system log analysis where the order of events matter among other things.

In this tutorial we will briefly understand how LSTM considers a certain sequence of events. We will then explore Magenta a, an ML library that can be used to generate music and art. We will also build a small app that will play a different sequence of drums for us each time by using DrumRNN in the browser using Magenta.js.

Step 1: Understanding Neural Networks and Recurrent Neural Networks

Neural Networks are universal function approximators, i.e.given an adequate size of neural networks you can use it to define any function. In it’s simplest form neural network can be understood as a Perceptron. It has the following components:

Inputs: The inputs to the perceptron
Weights: These are the weights assigned to each input
Bias: This is a constant that is added to the inputs
Activation function: It maps input to output

Weights help to give importance to certain inputs more than others. For instance, in order to decide if John should be granted a loan or not, the input related to his income should have a higher weight compared to the one that represents the initials of his name. Bias helps to make sure that given the same input, different perceptrons get different summed input by adding or subtracting a constant value from the input. This summed input is then passed through an activation function. This function has one job - to map the input to the output. There are many variations of activation functions, each suited for a different job. You would use a sigmoid activation function when you want to scale values in the range of 0 to 1, which is generally the case in a multi-class classification. In cases of binary classification where you want to know just one of the two classes- 0 or 1, you would use something like a binary step.

In essence, this perceptron is also a single-layer neural network. When you stack number of such perceptrons in a layer and increase the number of layers you get a deep neural network. The training data is used to adjust the parameters of each such perceptron in this network such as weights, biases etc. As a convention, the layer that receives the input is also known as the input layer and the one that provides output is called the output layer. All the layers in between, if present are known as hidden layers.

As you might have observed before, the output of a perceptron is only dependent on its input at a certain time. As they are the building blocks of a neural network, the output of the neural network is also dependent on only the input at a certain time. This is where recurrent neural networks come into the picture.

The output from the current state is fed again during the next state viz. the hidden layer. This makes the network capable of knowing what happened in the previous state or when the last input was passed through the network. As the previous state also has information about the states before it, this input represents the complete history to some extent. The term - “some extend” is used as there is a concept of a vanishing gradient which makes the memory of a state shrink as and when new states are seen.

In this section you leaned about what makes up a neural network and how they work. You also saw how recurrent neural networks are different from normal neural network and also how they work with time series data.

Step 2: Introduction to TensorFlow.js to Magenta.js

In this section you will learn about the Magenta project and look at Magenta.js. Although training the models is out of scope of this tutorial you will see how to use pretrained models to generate synthetic music using only JavaScript.

Magenta is a project lead by Google that focuses on using neural network in the domain of music and art. Although the core concepts can be implemented with any machine leaning library, the team behind Magenta has used TensorFlow for the task. TensorFlow is a library for machine learning developed and used by Google. It is an end-to-end system which means it is easy to not only develop with TensorFlow by using GPU’s and TPU’s for training but also deploy with it to multiple machines and even IOT devices. It also has TensorFlow.js which makes it easy to develop models and infer from them all within the JavaScript ecosystem. This means you can train and use the model in the browser itself.

In the next section you will use Magenta.js to create synthetic music.

Step 3: Creating the base

In this section we will design the base of the app. We will load all the libraries we need using Content Delivery Network or CDN. Start by creating a file named index.html and write the following snippet to it:

[index.html]
<html>
<head>
    <title>Synthetic Music</title>
    <style type="text/css">
        body {
            margin: 0;
            padding: 0;
            background-color: #f6f6f6;
            font-family: sans-serif;
        }        .play {
            width: fit-content;
            margin: 0 auto;
            padding: 12px;
            border: 2px solid #323232;
            border-radius: 50%;
        }        .play div {
            border-top: 10px solid transparent;
            border-bottom: 10px solid transparent;
            border-left: 20px solid #323232;
            height: 0px;
            width: fit-content;
            margin: 0 auto;
        }        .heading {
            text-align: center;
            margin-top: 5rem;
        }        #pattern-container {
            margin-top: 2rem;
            width: fit-content;
            border-radius: 5px;
            margin: 0 auto;
        }        .pattern-group {
            display: inline-block;
            background-color: #e4f9f5;
            padding: 5px 6px;
        }        .pattern-group.seed {
            background-color: #a8e6cf;
        }        .pattern.active {
            background-color: #11999e;
        }        .pattern {
            height: 10px;
            width: 5px;
            display: block;
            margin: 5px 0;
            padding: 3px;
            border-radius: 4px;
            color: transparent;
            background-color: #cbf1f5;
        }
    </style>
</head><body>
<h2 class="heading">Synthetic Music with Neural Networks </h2>
<div class="play" onclick="createAndPlayPattern(this)">
    <div></div>
</div>
<div>
    <div id='pattern-container'></div>
</div>
<script src='https://code.jquery.com/jquery-3.3.1.slim.min.js'></script>
<script type='text/javascript' src='https://cdn.jsdelivr.net/npm/lodash@4.17.5/lodash.min.js'></script>
<script type='text/javascript' src='https://gogul09.github.io/js/tone.js'></script>
<script type='text/javascript'
        src='https://cdn.jsdelivr.net/npm/@magenta/music@0.0.8/dist/magentamusic.min.js'></script>
<script>
    // Continue here...
</script>
</body>
</html>

We have defined a few custom CSS styles in the head section. Additional DOM elements were also defined to which we will later add the music patterns generated by out neural network. We have also loaded a few libraries like jQuery and loadash to clean up our code by using the API provided by them, tone.js to play tones and finally magenta.js which we will use to generate music patterns.

Step 4: Setting up Tone.js

In this section we will setup Tone.js so that it knows what notes are available to play. We will be playing drums, so we need sounds for different pieces of drums. You can get these sound files from the assets folder in this repository. You will find files named 808-hihat-open-vh.mp3, 808-hihat-vh.mp3, 808-kick-vh.mp3, 909-clap-vh.mp3, 909-rim-vh.wav, flares-snare-vh.mp3, slamdam-tom-high-vh.mp3, slamdam-tom-low-vh.mp3, slamdam-tom-mid-vh.mp3 and small-drum-room.wav. Among these small-drum-room.wav will be used to create reverb.

Start by creating a convolver which you will later use to create reverb. Also make sure that the wet value is set to 0.3 which means 30% of this effect will be applied to the tone on which it is used. Write the following snippet to do so.

let reverb = new Tone.Convolver(`assets/small-drum-room.wav`).toMaster();
reverb.wet.value = 0.3;

Next we have to define the individual components of the drum kit. These are Kick, Snare, Hi-hat closed, Hi-hat open, Tom low, Tom mid, Tom high, Clap and Rim. Write the following snippet that creates a drumkit with these components in order.

let drumKit = [    new Tone.Player(`assets/808-kick-vh.mp3`).toMaster(),
    new Tone.Player(`assets/flares-snare-vh.mp3`).toMaster(),
    new Tone.Player(`assets/808-hihat-vh.mp3`).connect(new Tone.Panner(-0.5).connect(reverb)),
    new Tone.Player(`assets/808-hihat-open-vh.mp3`).connect(new Tone.Panner(-0.5).connect(reverb)),
    new Tone.Player(`assets/slamdam-tom-low-vh.mp3`).connect(new Tone.Panner(-0.4).connect(reverb)),
    new Tone.Player(`assets/slamdam-tom-mid-vh.mp3`).connect(reverb),
    new Tone.Player(`assets/slamdam-tom-high-vh.mp3`).connect(new Tone.Panner(0.4).connect(reverb)),
    new Tone.Player(`assets/909-clap-vh.mp3`).connect(new Tone.Panner(0.5).connect(reverb)),
    new Tone.Player(`assets/909-rim-vh.wav`).connect(new Tone.Panner(0.5).connect(reverb))
];

Notice how each component is defined as an instance of Tone Player. Components that need reverb effect applied to them, a tone panner was connected to the original tone player. Panning is a process of placing the instrument in the 3D space by deciding the shape of the signal that is given to each channel of audio. Suppose, in a 2 channel the audio, you pass a tone signal equally through both the channels, that creates the illusion of the tone originating from the centre, you can then move this perceived source of the tone by controlling the signal that goes to each of the channel.

In this section you created the components that make up your drumkit. You also loaded the sound files for each tone and added reverbs to the tones that require it.

Setting up Magenta.js

In this section you will use neural network to continue a provided seed pattern and create synthetic music. Before we actually start using the neural networks these are a few methods that need to be setup that will convert the music notes to sequences that the model understands. Similarly, we will have to convert back the sequence that the model gave back to a tone that can be played.

Start by creating a mapping of MIDI values for each tone. MIDI or Musical Instrument Digital Interface is the protocol by which various devices that deal with music communicate with each other. If you connect some synthesizer with your computer, this is the interface used for communication. As you may have guessed this is also how we will communicate with our model. Magenta has been designed in this way as it makes it a easy plug in existing devices into the model directly. You may have guessed it, there are already devices available that do just that.

A machine leaning model performs best when the range of input is defined. Thus we will create a mapping of values from MIDI range into the range that the model can work with. Use the following variables to create this mapping:

const midiDrums = [36, 38, 42, 46, 41, 43, 45, 49, 51];
const reverseMidiMapping = new Map([    [36, 0], [35, 0], [38, 1], [27, 1], [28, 1], [31, 1], [32, 1], [33, 1],
    [34, 1], [37, 1], [39, 1], [40, 1], [56, 1], [65, 1], [66, 1], [75, 1],
    [85, 1], [42, 2], [44, 2], [54, 2], [68, 2], [69, 2], [70, 2], [71, 2],
    [73, 2], [78, 2], [80, 2], [46, 3], [67, 3], [72, 3], [74, 3], [79, 3],
    [81, 3], [45, 4], [29, 4], [41, 4], [61, 4], [64, 4], [84, 4], [48, 5],
    [47, 5], [60, 5], [63, 5], [77, 5], [86, 5], [87, 5], [50, 6], [30, 6],
    [43, 6], [62, 6], [76, 6], [83, 6], [49, 7], [55, 7], [57, 7], [58, 7],
    [51, 8], [52, 8], [53, 8], [59, 8], [82, 8]]);

Next, also set the temperature of the music you desire and also the length of the pattern as follows:

const temperature = 1.0;
const patternLength = 32;

To convert the notes into a sequence, write down the following function:

function fromNoteSequence(seq, patternLength) {
    let res = _.times(patternLength, () => []);
    for (let { pitch, quantizedStartStep } of seq.notes) {
        res[quantizedStartStep].push(reverseMidiMapping.get(pitch));
    }
    return res;
}

It takes two inputs, the sequence of notes and the pattern length. Loadash was used to first creat a list of length defined by patternLength consisting of empty lists that then filled up next. Individual notes are then iterated over to unpack the start time of the note and then the note is placed at that interval by using the reverseMidiMapping defined earlier.

The output from model can now be converted into a sequence that can be used to play the drum kit. The reverse has to happen as well i.e. pattern sequences should be converted into note sequences that the model can understand. Write down the following function that performs this job:

function toNoteSequence(pattern) {
        return mm.sequences.quantizeNoteSequence({
                ticksPerQuarter: 220,
                totalTime: pattern.length / 2,
                timeSignatures: [{
                    time: 0,
                    numerator: 4,
                    denominator: 4
                }],
                tempos: [{
                    time: 0,
                    qpm: 120
                }],
                notes: _.flatMap(pattern, (step, index) =>
                    step.map(d => ({
                        pitch: midiDrums[d],
                        startTime: index * 0.5,
                        endTime: (index + 1) * 0.5
                    }))
                )
            },
            1
        );
    };

Here are few details on the attributes that have been defined when creating this note sequence:

ticksPerQuarter: ticks is the unit of time in the MIDI standard. totalTime: The length of the sequence that has been provided in the notes attribute. This length is measured in terms of quantized steps. timeSignatures: This defines the time signature used in musical notation. tempos: Define the temps used in the tone sequence provided. qpm here refers to quarter notes per minute. notes: This represents the notes with the pitch and duration of each note in the sequence.

Once we have the pattern, there should be a way to play it as well. To play the pattern we will use the drumkit we created in the previous section. Write the following method that takes a pattern and plays it using Tone.js:

function playPattern(pattern) {
    sequence = new Tone.Sequence(
        (time, {drums, index}) => {
            drums.forEach(d => {
                drumKit[d].start(time)
            });
        },
        pattern.map((drums, index) => ({ drums, index })),
        '16n'
    );    Tone.context.resume();
    Tone.Transport.start();
    sequence.start();
}

Tone.Sequence has been used to accomplish this. It expects three inputs: callback: This method would be called for each event event: The individual events of this sequence. Here we map each individual pattern as an object that has the drum components to play as drums and the index of the pattern in the sequence as stepId subdivision: The subdivision between the events Tone.context.resume then starts the audio context which is required to connect to the audio interface provided by the browser. Tone.Transport makes sure the timing of the music stays perfect by not directly relying on browsers timing. sequence.start finally sets everything in motion.

With this you are now ready to finally use the model to create music. Before we actually start using the model let us look at how note pattern look. A specific pattern in a pattern sequence is an array of indexes from 0 to 8 of the 9 components in our drumkit. So a pattern [0,2,4] would play the Kick, Hi-hat closed and Tom low.

We now use a seed pattern that our model will then improvise upon. Use the following seed pattern for it.

var seedPattern = [    [0, 2],
    [0],
    [2, 5, 8],
    [],
    [2, 5, 8],
    [],
    [0, 2, 5, 8],
    [4, 5, 8],
    [],
    [0, 5, 8]
];

Use the following snippet to define a function that creates this seed pattern and then creates a pattern of length patternLength which was previously set to 32:

let drumRnn = new mm.MusicRNN('https://storage.googleapis.com/download.magenta.tensorflow.org/tfjs_checkpoints/music_rnn/drum_kit_rnn');
drumRnn.initialize();function createAndPlayPattern() {
    drumRnn
        .continueSequence(seedSeq, patternLength, temperature)
        .then(r => seedPattern.concat(fromNoteSequence(r, patternLength)))
        .then(displayPattern)
        .then(playPattern)
}

We start by first loading the Drum RNN model and then initializing it. We have then created a function that uses this model to take a seed pattern and then continue it to create a new pattern using the continueSequence method. We then pipe this note sequence to convert it to sequence that we can play our drumkits with. We have also chained the displayPattern to visualize the pattern and then finally play the pattern with playPattern method. We will create the visualization method in the next section.

Adding visualization

In this section we will toe up everything to create visualization for our pattern. Modern browsers like Chrome require the user to make some interaction on the page before it allows the music to be played. This is actually a usability feature. You would not like it if a pop-up that opened in the background suddenly starts playing something. To make sure that we start our process only after the user interacts with the page, we had created a play button when writing the HTML and added an onclick attribute to it. The method referenced in this attribute was createAndPlayPattern which we created in the previous step that takes a seed pattern, generates music, visualizes it and plays it. We do not want this play button to be visible after the first click. Modify the createAndPlayPattern pattern to accept the click event and delete the element from the page as follows:

function createAndPlayPattern(e) {
    $(e).remove()
    let seedSeq = toNoteSequence(seedPattern);
    drumRnn
        .continueSequence(seedSeq, patternLength, temperature)
        .then(r => seedPattern.concat(fromNoteSequence(r, patternLength)))
        .then(displayPattern)
        .then(playPattern)
}

You are now ready to add the last and final part to the puzzle - displayPattern method that visualizes the pattern. Write down the following method to achieve this:

function displayPattern(patterns) {
    for (let patternIndex = 0; patternIndex < patterns.length; patternIndex++) {
        let pattern = patterns[patternIndex];
        patternBtnGroup = $('<div></div>').addClass('pattern-group');
        if (patternIndex <= seedPattern.length)
            patternBtnGroup.addClass('seed');        for (let i = 0; i <= 8; i++) {
            if (pattern.includes(i))
                patternBtnGroup.append($(`<span></span>`).addClass('pattern active'));
            else
                patternBtnGroup.append($(`<span></span>`).addClass('pattern'));
        }
        $('#pattern-container').append(patternBtnGroup)
    }
    return patterns;
}

It takes the pattern and creates multiple spans on the page. Each span is sorted out in a vertical stack of 9 blocks representing each component of the drum kit. It is highlighted if it is on or played when the respective sequence is played. Also the seed pattern is highlighted to make it easy to see what the model has generated.

Conclusion

Recurrent neural networks provide a mechanism to retain state over input. This makes RNN a great fit to predict music as at it’s core, music is a sequence of tones which are all dependent on each other. Magenta is a great starting point for any such projects.

Check out the source code at github and a live demo here.

Automating windows with COM object

Sat, 01 Dec 2018 16:54:41 GMT

A few weeks ago I got a project wherein I had to get various statistical data about a document. One of the important parameters was to get a precise count of grammatical errors in the document. I tried to go the open source way and tested out LanguageTool, PorseLink among others. None of these were up to the mark of the good old MS-Word. So I had to now work on a way to problematically access Word, load a document in it and get the statistics.

Python being versatile and great at memory management, we would be using it to control COM object and do what we need to do.

This is the Part 1 of the two series post.

Part 1 : Introduction to COMobject, initialization, examples etc

Part 2 : Setting up a server environment. Making use of threads and opening using thread for each process. This is how to communicate with the COMobject in a multi-thread environment as it cannot be passed to a thread directly.

Before we dig deep, what is COMobject ?

The Microsoft Component Object Model (COM) is a platform-independent, distributed, object-oriented system for creating binary software components that can interact. COM is the foundation technology for Microsoft's OLE (compound documents), ActiveX (Internet-enabled components), as well as others. read more...

In a nutshell it can be used to control various aspects of Windows. We would be focusing on using the COMobject to control MS-Office applications like Word and Excel. Although it is not limited to just MS-Office applications, it can be used to communicate with other softwares like IE (if you're still into it...)

You can do things like :

Create triggers to perform various tasks
- Add new rows to an excel sheet in real time with the current scores of a game. This excel sheet can then have some complex algorithms to predict which team would win by performing calculations on all the rows. Although it can be done with implementation in even NodeJS but we don't want to take the pain to port the functions/macros from Excel to JS
Log the current changes made by the user XYZ in the database

The possibilities are limitless. I guess you already have a use case in mind and landing to this page to know the process, if not it's always good to know if there is possibility to get a certain functionality done.

Python and COM

We will need the following tools to communicate with MS-Word :

MS Office or Specific applications already installed
PyWin32 library

PyWin32 is a great library that gives us the same set of methods/properties that are exposed by COM which are natively in Visual Basic for Applications (VBA).

NOTE: I'm using Python3 in the code samples in this guide.

Let's start

To get hold of a COM object

import win32com.client

# for MS-Word
word = win32.gencache.EnsureDispatch('Word.Application')
# OR
word = win32com.client.DispatchEx('Word.Application')

# for MS-Excel
excel = win32.gencache.EnsureDispatch('Excel.Application')
# OR
excel = win32com.client.DispatchEx('Excel.Application')

There are some key differences to be noted here

EnsureDispatch : In dead simple terms, it makes sure that a object which references the specified application is returned. If the application is already open, it will return the same instance, if not it will start a instance and return it. It is useful in cases where you would want to make use of single application that holds certain meta-data, over the life of the process.

DispatchEx : For each call, you would get a new object returned. This would mean that if you call it 7 times, you would see 7 instances of MS-Word in task manager. It makes sense to get independent instances when you would want to handle each process independently and close the application when one process completes.

Understand that if you run a web-server that takes in data, puts it in an Excel, runs a macro and return a result. You would want to have a dedicated instance of Excel for each process so that you can close it as soon as the response is ready. If you used EnsureDispatch or Dispatch, that would mean that at the end of each process, you would have to check if any other process is in progress, if not shut down the application.

It's requires a bit extra memory but helps to keep memory leak in check as using same application instance to load up multiple files that do not require each others data, means that with each load, some meta-data is loaded which is not cleared when closing the process and it builds up over time. We will cover this aspect in detail in the Part-2 of the post.

Coming back to the point, let's go ahead.

Now that we have the COMobject we can use it a open a Word Document.

doc = word.Documents.Open( os.path.join( os.getcwd(), 'path/to/files/', filename), ReadOnly=True)

NOTE : As you might have had guessed, you can use the same word object to open multiple doc objects. This is what I was referring to using same application to open multiple documents in the last paragraph

References

At this point, we can refer to Microsoft's documentation to get gist of various properties & functions available.

Word 2007 Developers Reference
Word Object Model reference
Application Object : List of methods and properties available

Let's have look at some usage examples:

Make the application window invisible :
```
word.Visible = False
```
To add a new Document
```
doc = word.Documents.Add()
```

To get all the grammatical or spelling errors

grammaticalErrors = doc.GrammaticalErrors
spellingErrors = doc.doc.SpellingErrors

#or to simple get the count
grammaticalErrorsCount = doc.GrammaticalErrors.Count
spellingErrorsCount = doc.SpellingErrors.Count

If you have a look at the Word BuiltInProperty, you'll see you can do the following

wordCount = int(doc.BuiltInDocumentProperties(15))
characterCount = int(doc.BuiltInDocumentProperties(16))
paragraphCount = int(doc.BuiltInDocumentProperties(24))

or do something much more complex as

characterPerWord = doc.ReadabilityStatistics("Characters per Word").Value

or something as simple as

shapeCount = doc.Shapes.Count
tableCount = doc.Tables.Count
sentenceCount = doc.Sentences.Count
tableOfContentCount = doc.TablesOfContents.Count

#loop through each paragraphs
for paras in doc.Paragraphs:
    # paras will now be each individual paragraph

Conclusion

As you can see the possibilities are endless. MS-Office tools are pretty powerful in what they do. We did not even dig into the excel territory. Majority of the questions on stackoverflow are based on excel as automating tasks there makes can be of great use.

In the next post we will focus on how to wrap all this up into dedicated thread for each process and word with COMobject in threads.