Blog

  • How to Choose Chunk Sizes and Rank Retrieved Chunks

    In the last post I said: retrieve first, then generate. A reader asked the obvious next question — how do you choose the chunk size, and how do you decide which retrieved chunks actually reach the model? This post is the answer I promised.

    There is no single right chunk size. There is a default that works, a way to tell when it is wrong, and a ranking step that matters more than most people think.

    Why chunk size matters

    Chunking is the step where your documents get cut into pieces, each piece gets embedded, and those embeddings become what the retriever searches. Get the cut wrong and the best embedding model in the world cannot save you — the right passage is either split across two chunks, or buried inside a chunk about something else.

    Two failure modes, and they pull in opposite directions:

    • Too small. A 64-token chunk is precise, but it has no context. “The company’s revenue grew by 3% over the previous quarter” is useless as a chunk — which company, which quarter? The retriever finds it, the model cannot use it.
    • Too large. A 2,000-token chunk has all the context, but its embedding is an average of several ideas. It matches many queries weakly and none strongly, and it eats context window that could have held two better chunks.

    Chunk size is the most consequential setting in a RAG pipeline. It shapes retrieval precision, answer quality, embedding cost, and index size, all at once.

    The default that works

    Start here, and only move when you have measured a reason to:

    • 512 tokens, 10–20% overlap, recursive splitting.

    Recursive splitting means: try to split on paragraph breaks first, then line breaks, then sentences, then spaces. It is the default in LangChain’s RecursiveCharacterTextSplitter and the approach most 2026 guides converge on. A 512-token chunk is roughly 350–400 words, or about 2,000 characters in English.

    Overlap exists for one reason: an idea that straddles a split point should not fall through the cracks. At 20% overlap on 512-token chunks you generate roughly 25% more chunks than with zero overlap, so overlap is not free. Keep it under 25% unless you can show it helps.

    Match the size to the question, not the document

    The same corpus serves different questions. A useful rule, reflected across several 2026 guides:

    Question type Chunk size
    Factoid — “what is the ticket ID format?” 128–256 tokens
    Technical docs, code + explanation 512–768 tokens
    Analytical, multi-hop questions 512–1024 tokens
    Long-form reports, legal, contracts 768–1024 tokens
    Whole-section comprehension 1024–2048 tokens

    The principle is simple: a chunk should be big enough to contain the answer plus its local context, and small enough that the top results are about the answer, not about everything around it.

    One more constraint that surprises people: the best size depends on the embedding model. A 2025 multi-dataset study (arXiv:2505.21700) found that some embedding models do better with larger chunks and others with smaller ones. There is no universal optimum — which is why the default above is a starting point, not a law.

    When to break the default

    Three techniques that beat “just tweak the size” when the default is not enough:

    Semantic chunking. Embed each sentence, split where the meaning changes. It groups sentences by topic instead of by length. It is slower, and it can produce tiny 40-token fragments that hurt retrieval — always enforce a minimum chunk size if you use it. One 2026 practitioner benchmark reports semantic chunking giving large accuracy lifts; another reports recursive 512-token chunking beating it (Vectara 2026 study, as reported second-hand — I have not verified the original study, so treat that result as contested, not settled).

    Small-to-big (parent–child) retrieval. Retrieve small, precise child chunks, then hand the model the larger parent chunk they came from. You get tight retrieval and full context. This is the pattern to reach for with legal and contract documents: a 200-token child for retrieval, a 1,000-token parent for generation.

    Contextual retrieval. Before embedding a chunk, prepend a short line saying where it fits — which document, which section, which product. Anthropic published this in September 2024 and reported that it cut failed retrievals by 49%, and by 67% when combined with a reranker. Their measured failure rate went from 5.7% to 2.9%, then to 1.9% (top-20 retrieval). The cost they reported was about $1.02 per million document tokens, paid once at index time. Those are Anthropic’s own figures on their own test sets — but the direction is consistent with everything else in this post: chunks without context are the problem.

    Late chunking is worth knowing about, though it is newer: embed the whole document first, then pool the token embeddings into chunk embeddings. It keeps each chunk aware of the document around it. Published by Günther et al. in 2024, presented at ICLR 2025. Not yet a production default.

    Ranking: retrieve wide, rerank narrow

    Getting the right chunk into the candidate set is only half the job. The other half is ordering — because the model only ever sees the top few.

    A practical pipeline, and the one most production RAG systems use in 2026:

    1. Retrieve wide. Run dense vector search and BM25 keyword search in parallel, each returning its top 50. Dense search misses exact terms like ticket IDs and error codes; BM25 misses paraphrases. You need both. This is the hybrid search I mentioned in the last post.
    2. Fuse. Merge the two ranked lists with Reciprocal Rank Fusion (RRF). The formula scores each document by 1 / (60 + rank) in each list and adds them up. It works on ranks, not scores, so you never have to normalise two different scoring systems. The constant 60 comes from the original 2009 paper (Cormack, Clarke, and Buettcher) and has held up.
    3. Rerank. Pass the fused top 50 to a cross-encoder reranker — a model that reads the query and each passage together and scores true relevance. Cohere Rerank, Voyage rerank, and the open BGE reranker are the common choices in 2026. Keep the top 3–5.
    4. Generate. Only those top chunks go to the model.

    Practitioner benchmarks report hybrid retrieval reaching roughly 91% recall@10 against about 78% for dense-only (aiworkflowlab.dev — a practitioner blog, not a peer-reviewed result). Adding a cross-encoder reranker is commonly reported as a +5 to +15 NDCG@10 lift over first-stage retrieval alone. Treat both numbers as indicative, not gospel — but the shape of the finding is consistent across sources: hybrid search fixes recall, reranking fixes precision.

    The cost is real but small: the two parallel searches add a few milliseconds; reranking 50 candidates adds roughly 50–200 ms depending on the model. That is cheap compared to a wrong answer.

    How to know if any of this helped

    Do not tune chunk size by feel. Build a small evaluation set first — 30 to 50 real questions from your users, with the correct answer or the correct source passage noted for each. Then measure two things before and after every change:

    • Did the right chunk get retrieved at all? (Recall — the hybrid search step owns this.)
    • Did the right chunk rank near the top? (Precision — the reranker owns this.)

    Change one thing at a time: chunk size, or overlap, or the reranker — never all three at once. Keep the change only if the numbers move. This is the same discipline as any other performance work, and RAG is no exception.

    The short version

    • Start at 512 tokens, 10–20% overlap, recursive splitting.
    • Small chunks for facts, larger chunks for reasoning. Match the chunk to the question.
    • Retrieve small, generate from the parent chunk, when documents are dense.
    • Add context to each chunk before embedding — it is the cheapest large win available.
    • Retrieve wide with hybrid search, fuse with RRF, rerank to the top 3–5.
    • Measure on your own questions. Keep only what the numbers justify.

    Next up: GraphRAG — building the graph lazily, only over what gets retrieved.

    References

  • Hybrid Search + Reranking: the Production Default for RAG

    In the first post, I introduced retrieval-augmented generation: retrieve first, then generate. That works well until retrieval itself becomes the bottleneck. In production, the most common failure is not the language model. It is the retriever returning the wrong passages, or missing the right one entirely. This post is about the retrieval pipeline most production RAG systems actually run: keyword search and vector search side by side, fused with Reciprocal Rank Fusion, then reranked with a cross-encoder.

    The idea is straightforward. No single retrieval method is good at everything. Keyword search is precise but brittle. Vector search is broad but misses exact identifiers. Running both, and letting a reranker order the survivors, covers each method’s weakness. The rest of this post walks through each stage in order, what it costs, and when it does not help.

    The problem: two kinds of queries

    Every RAG corpus gets two kinds of questions, and each kind breaks a different retriever.

    The first kind is conceptual. “How do I reduce customer churn?” The answer lives in documents about retention strategies and preventing subscriber loss. Those documents share almost no words with the question. Vector search handles this well, because it matches meaning rather than words.

    The second kind is exact. “What does error code E-4217 mean?” The answer is the one document that contains that exact string. Vector search handles this badly. An embedding places “E-4217” in a general “error handling” region of the vector space, and returns documents about error handling in general. Rare tokens — ticket IDs, product SKUs, version strings, RFC numbers, clause references like “Section 4(b)(iii)” — collapse into near-identical embeddings. The answer is sitting in the corpus, and pure vector search walks past it.

    Keyword search has the opposite profile. It finds the exact token you typed, every time. But it cannot connect “contract termination procedures” to “end of agreement protocols”. Zero shared terms means zero relevance. It cannot handle typos, abbreviations, or paraphrases.

    So the two methods fail in opposite directions. That is the whole argument for running both: their failure modes are complementary, and each one catches what the other misses.

    Anthropic measured this directly in their contextual retrieval work. Embeddings, they wrote, excel at semantic relationships but can miss crucial exact matches, and BM25 is the older technique that catches them. Their example was “Error code TS-999” in a support database: the embedding finds error-code content in general, while BM25 finds the exact string. On their own test sets, adding contextual BM25 cut the top-20 retrieval failure rate from 5.7% to 2.9% — a 49% reduction. Adding a reranker on top took it to 1.9%, a 67% reduction. Those are Anthropic’s figures on their own data, so treat the exact percentages as theirs, not as universal. The pattern, though, is consistent across every source I checked: keyword search first, reranker second, each stage fixing a different failure.

    BM25: the boring half

    BM25 is a 30-year-old ranking function. It scores documents by term frequency, inverse document frequency, and document length normalisation. There is no model to host, no GPU to provision, and no drift to monitor. On millions of documents it answers in single-digit milliseconds on a single core.

    That simplicity is the point. BM25 does one job — exact matching — and it does it cheaply and predictably. It is not a replacement for vector search. It is the half of the pipeline that catches the queries vector search drops.

    Dense search: the broad half

    Dense search does the job BM25 cannot. It encodes the query and every document into vectors, and ranks by similarity. It bridges vocabulary gaps, handles paraphrases, and finds conceptually related material that shares no words with the question.

    Its weakness is the one described above: exactness. It is also the more expensive half to run and to keep current, since every document change means re-embedding. But for conceptual queries it is the only method that works at all, which is why it became the default — and why, on its own, it is not enough.

    Putting them together: RRF fusion

    Running two retrievers gives you two ranked lists. The problem is that their scores live on different scales. BM25 scores are unbounded and depend on corpus statistics. Cosine similarity lives between -1 and 1, with most relevant documents clustering above 0.7. A weighted sum of the two scores looks natural, but it is fragile in practice: min-max normalisation is sensitive to outliers, and z-score normalisation changes shape with every query.

    Reciprocal Rank Fusion avoids the problem by ignoring scores entirely and working on ranks only. The formula, from Cormack, Clarke, and Buettcher’s 2009 SIGIR paper, is:

    RRF_score(d) = sum over each list of 1 / (k + rank(d))

    A document ranked 1st in one list and 3rd in another scores 1/61 + 1/63. A document ranked 40th in both scores 1/100 + 1/100. Documents near the top of multiple lists win. The constant k = 60 comes from the original paper, and it has held up well enough that most implementations never touch it. Lower k makes top ranks dominate more; higher k flattens the distribution.

    The one thing worth tuning is not k, but the weight given to each retriever. Most vector databases let you weight each branch. A reasonable starting point:

    • Identifier-heavy queries (catalogs, error codes, ticket IDs): BM25 0.7, dense 0.3.
    • Conceptual or paraphrase-heavy queries: BM25 0.3, dense 0.7.
    • Mixed corpora: 0.5 / 0.5, and let the reranker sort it out.

    Do not tune k until you have a labelled eval set of at least 200 queries. Anything less and you are chasing noise. And do not tune anything at all until you have measured the 50/50 default on your own questions first.

    How much does fusion actually gain? Adaptive Recall, in a July 2026 guide to production RAG, puts the hybrid gain at 5% to 15% higher recall@10 and NDCG@10 than the stronger of the two methods alone, with the biggest wins on queries where one method alone would fail. A practitioner guide (AI Workflow Lab) reports a starker split on enterprise corpora — recall@10 around 91% for hybrid, 78% for dense-only, 65% for BM25-only — but that is a practitioner blog, not a peer-reviewed result, so treat it as indicative rather than exact. The direction is the same in both: hybrid beats either method alone, and the gap is largest exactly where one method fails outright.

    Reranking: precision is a separate job

    Fusion fixes recall. It gets the right document into the candidate set. Reranking fixes precision. It puts the right document at the top of the list. These are complementary stages, not alternatives, and confusing them is the most common mistake I see.

    A cross-encoder reranker reads the query and each candidate passage together, through every layer of the model, and scores true relevance. A bi-encoder — your embedding model — encodes the query and the document separately and compares the two vectors. The cross-encoder is far more accurate and far slower per pair. That is why you never run it over the whole corpus. You run it over the top 20 to 50 fused candidates and keep the top 3 to 5.

    The pipeline, in order:

    1. Retrieve wide. Dense search and BM25 in parallel, top 50 from each. Retrieve at least 20 from each — because the two lists overlap, the merged set is smaller than the sum.
    2. Fuse. RRF with k = 60.
    3. Rerank. Pass the fused top 20–50 to a cross-encoder. Keep the top 3–5.
    4. Generate. Only those chunks go to the model.

    The reported lift from reranking is roughly +5 to +15 NDCG@10 over first-stage retrieval alone. That figure comes from a practitioner synthesis of BEIR and MTEB-style benchmarks (The Agent Ecosystem), not a single peer-reviewed study, so read it as a range rather than a promise. Anthropic’s numbers are the cleanest measured case: as noted above, adding a reranker to their hybrid setup cut the failure rate from 2.9% to 1.9%.

    One more point from the same sources, and it matters: a reranker cannot recover a document that was never retrieved. It can only re-order what retrieval surfaced. If your labelled set shows the correct chunk missing from the top 50, the problem is recall, not ranking. Fix chunking, metadata, or the retrieval stage first. And the reverse is worth saying too: if latency rules out a reranker, hybrid search without reranking is still better than dense-only with reranking in most evaluations. Recall is the more fundamental fix.

    What it costs

    The cost is real but small, and it is mostly latency, not money.

    Running two retrievers in parallel adds roughly 6 ms to p50 latency over dense-only search (AI Workflow Lab, practitioner measurement). BM25 itself is single-digit milliseconds. Reranking 50 candidates adds roughly 50–200 ms, depending on the model — self-hosted on a GPU is often 50–100 ms, a hosted API 100–400 ms for the same batch. Total retrieval lands around 60–220 ms. The LLM generation step takes 500–3,000 ms, so the whole retrieval cascade is essentially free relative to generation — unless your latency budget is a typeahead search box or a voice pipeline, where 150–300 ms is unacceptable.

    Money is a rounding error next to the LLM bill, and reranking can even cut total cost: sending three clean chunks instead of ten noisy ones lowers the completion tokens you pay for.

    Which reranker? The practical choice in 2026:

    • Cohere Rerank 3.5 / Rerank 4 — managed API, the fastest path, no GPU to operate. Best under roughly 50,000 queries a day.
    • Voyage Rerank 2.5 — managed API, cheaper, the natural pick if you already use Voyage embeddings.
    • BGE reranker v2-m3 — open weights, Apache 2.0, multilingual, the default for self-hosted enterprise deployments. A single GPU handles it; at high volume the unit economics beat the API.
    • FlashRank (MiniLM-L-12) — quantized cross-encoder, sub-100 ms on CPU, the cheap default when you cannot send data to a third-party API.
    • LLM-as-reranker — prompt a large model to score candidates. Highest accuracy on subtle relevance judgements, but 1–3 seconds per batch. Use it only for async pipelines or high-value queries.

    Pricing quoted in 2026 guides — Cohere around $2 per 1,000 searches, Voyage around $0.05 per million tokens — should be checked against the vendor pricing pages before you rely on them. Cohere moved self-serve pricing during 2026.

    The spread between the top rerankers is small. One practitioner benchmark table (AI Workflow Lab) puts them close together on quality, with nDCG@10 from 0.66 (FlashRank, CPU) to 0.74 (Cohere). The bigger truth from the same source: the retriever sets the ceiling. No reranker pushes the hit rate above roughly 88% if the retrieval stage missed the document. Fix recall before you shop for rerankers.

    When hybrid search does not help

    Hybrid is the default, not a law. Three cases where it adds little or nothing:

    Tiny corpora. Under roughly 2,000 chunks, vector search alone usually retrieves the right material, and a reranker adds latency without moving accuracy. If the whole relevant document set fits in the prompt, retrieval quality barely matters — send more chunks instead.

    Narrow, consistent vocabulary. In highly technical corpora where everyone uses the same terms, or conversational data where BM25 adds noise, dense-only can perform equally well. Adaptive Recall says this plainly: hybrid wins on average across benchmarks, but there are domains where it does not. Measure on your data before committing to the added complexity.

    Broken stage one. If the correct chunk is missing from the top 50, no reranker will save you. Fix the retrieval stage first, as described above.

    The recipe

    If you are starting from a dense-only pipeline today, this is the change, in order:

    1. Add BM25 alongside your vector index. Most vector databases now support it natively — Elasticsearch and OpenSearch have it built in, Qdrant since version 1.7, Weaviate with a configurable alpha, Pinecone with sparse-dense vectors, PostgreSQL with pgvector plus full-text search. Pick whichever one your stack already has; do not introduce a new database for this.
    2. Retrieve 50 from each, fuse with RRF at k = 60, weights 50/50. Keep the top 20.
    3. Add a cross-encoder reranker. Start with a managed API (Cohere or Voyage) or FlashRank if you need CPU-only. Rerank the top 20–50, keep the top 3–5.
    4. Measure on your own questions. Build a small eval set — 30 to 50 real questions, with the correct source passage noted for each — and check two numbers before and after: did the right chunk get retrieved at all (recall), and did it rank near the top (precision). Change one thing at a time.

    Three mistakes to avoid, all from production write-ups: use the same tokenization for ingestion and query-time BM25, or the term statistics will not match; strip metadata from chunks before BM25 indexing, or BM25 will match on source markers and section headers instead of content; and do not set the per-method top-k too low — retrieve at least 20 from each side, or the merged set is thinner than you think.

    The short version

    • Vector search misses exact terms. BM25 misses meaning. Run both.
    • Fuse with Reciprocal Rank Fusion, k = 60. It works on ranks, so you never normalise two incompatible scoring systems.
    • Rerank the fused top 20–50 with a cross-encoder. Keep the top 3–5 for the model.
    • Hybrid fixes recall, reranking fixes precision. A reranker cannot save a document retrieval never found.
    • The whole cascade costs roughly 60–220 ms against a 500–3,000 ms generation step. Cheap — unless you are building a typeahead.
    • Measure on your own questions. Keep only what the numbers justify.

    Next up: GraphRAG and LazyGraphRAG — what happens when the answer is not in one chunk, but in the relationship between chunks.

    References

  • What is RAG? A Practical Introduction for Engineers

    After a long break, I am writing again on TechMyTalk — this time about the technology I work on every day: Retrieval-Augmented Generation, or RAG.

    What is RAG?

    A large language model (LLM) is good at language, but it does not know your company’s documents, your tickets, or your runbooks. RAG fixes that. Instead of asking the model to answer from memory alone, you first retrieve the relevant documents, then give them to the model along with the question. The model answers from your documents, not from what it happened to learn in training.

    In short: retrieve first, then generate.

    The three steps

    1. Index — Split your documents into chunks, turn each chunk into a vector embedding, and store them in a vector database such as Elasticsearch.
    2. Retrieve — Turn the user’s question into the same kind of embedding, search for the closest chunks, and rank them. A hybrid search — keyword (BM25) plus vector (kNN) — usually works better than either alone.
    3. Generate — Send the question plus the top retrieved chunks to the LLM, with a simple instruction: answer only from these chunks, and say you do not know if the answer is not there.

    Why it matters

    • Answers are grounded in your own documents, so hallucinations go down.
    • You can update the knowledge by updating the documents — no retraining.
    • You can show the source of each answer, so people can check it.

    Where it gets hard in the enterprise

    The basic idea is simple. Making it work at enterprise scale is not. The hard parts are: keeping access control (a user must only see documents they are allowed to see), handling many languages and many document types, keeping the index fresh, and knowing whether the retrieved chunks actually answer the question.

    I will write about those in the next posts — including GraphRAG, where a knowledge graph is built over the documents to answer questions that plain vector search cannot.

    If you are building RAG, start simple: one document set, hybrid search, and a strict prompt. Measure whether the retrieved chunks answer the question, before you add anything more complex.

  • Apache Kafka Consumer Rebalance

    Consumer rebalance decide which consumer is responsible for which subset of all available partitions for some topic(s). For example, you might have a topic with 20 partitions and 10 consumers; at the end of a rebalance, you might expect each consumer to be reading from 2 partitions. If you shut down 10 of those consumers, you might expect each consumer to have 1 partitions after a rebalance has completed. Consumer rebalance is dynamic partition assignment which can handle automatically by Kafka.

    A Group Coordinator is one of broker responsible to communicate with consumers to achieve rebalances between consumer.In earlier version Zookeeper stored metadata details but latest version it store on brokers.Consumer coordinator receive heartbeat and polling from all consumers of consumer group so he aware about each consumer heartbeat and manager their offset on partitions.

    Group Leader: One of consumer of Consumer Group work as group leader which is chosen by Group coordinator and will responsible for making partition assignment decision on behalf of all consumers in a group.

    Rebalance Scenarios:

    1. Consumer Group subscribes to any topics
    2. A Consumer instance could not able to send heart beat with session.heart.beat time interval.
    3. Consumer long process exceed poll timeout
    4. Consumer of Consumer group through exception
    5. New partition added.
    6. Scaling Up and Down consumer . Added new consumer or remove existing consumer manually for

    Consumer Rebalance

    Consumer rebalance initiated when consumer requests to join a group or leaves a group. The Group Leader receive a list of all active consumer from the Group Coordinator. Group Leader decide partition(s) assigned to each consumer by using PartitionAssigner. Once Group Leader finalize partition assignment it send assignments list to Group Coordinator which send back these information to all consumer. Group only send applicable partitions to their consumer not other consumer assigned partitions. Only Group Leader aware about all consumer and its assigned partitions. After the rebalance is complete, consumers start sending Heartbeat to Group Coordinator that its alive. Consumers send a OffsetFetch request to the Group Coordinator to get last committed offsets for their assigned partitions. Consumers start consuming messaged for newly assigned partition.

    State Management

    While rebalancing, Group coordinator set its state to Rebalance and wait all consumer to re-join the group.

    When the Group start rebalancing , the group coordinator first switches its state to rebalance so that all interacting consumers are notified to rejoin the group. Once rebalance completed Group coordinator create new generation ID and notified to all consumers and group proceed to sync stage where consumers send sync request and go to wait until group Leader finish generating new assign partition.Once consumers received new assigne partition they moved to stable stage.

    enter image description here

    Static Membership

    Thies rebalancing is quite heavy operation as it required to stop all consumer and wait to get new assigned partition. On each rebalance always create new generation id means refresh everything. To solve this overhead Kafka 2.3+ introduced Static Membership to reduce unnecessary Rebalance. KIP-345

    In Static Membership consumer state will persist and on Rebalance the same assignment will get apply. It uses new group.instance.id to persist member identity. So even in worst case scenario member id get reshuffle to assigne new partition but still same consumer instance id will get same partition assignment

    instanceId: A, memberId: 1, assignment: {0, 1, 2}
    instanceId: B, memberId: 2, assignment: {3, 4, 5}
    instanceId: C, memberId: 3, assignment: {6, 7, 8}
    
    

    And after the restart:

    instanceId: A, memberId: 4, assignment: {0, 1, 2}
    instanceId: B, memberId: 2, assignment: {3, 4, 5}
    instanceId: C, memberId: 3, assignment: {6, 7, 8}   
    
    

    Ref:

    1. https://www.confluent.io/blog/kafka-rebalance-protocol-static-membership
    2. https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
  • Kafka – Exactly-Once Semantics

    In distributed environment failure is very common scenario which can be happened any time. In Kafka environment, broker can crash, network failure, failure in processing, failure while publishing message or failure to consume messages etc. These different scenario introduced different kind of data loss and duplication.

    Failure scenarios

    A (Ack Failed): Producer published message successfully with retry>1 but could not received acknowledge due to failure. In that case Producer will retry same message might introduce duplicate.

    B (Producer process failed in batch messages): Producer sending batch of messages it failed with few published success. In that case and once producer will restart it will again republish all message from batch which will introduce duplicate in Kafka.

    enter image description here

    C (Fire & Forget Failed) Producer published message with retry=0(fire and forget). In case of failure published will not aware and send next message this will cause message lost.

    enter image description here

    D (Consumer failed in batch message) A consumer receive a batch of messages from Kafka and manually commit their offset (enable.auto.commit=false). If consumer failed before committing to Kafka , next time Consumer will consume the same records again which reproduce duplicate on consumer side.

    enter image description here

    Exactly-Once semantics

    In this case, even if a producer tries to resend a message, it leads to the message will be published and consume by consumer exactly once.

    To achieve Exactly-Once semantic in Kafka , it uses below 3 property

    1. enable.idempotence=true (address a, b & c)
    2. MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=5(Producer will always have one in-flight request per connection)
    3. isolation.level=read_committed (address d )

    Enable Idempotent(enable.idempotence=true)

    Idempotent delivery enables producer to write message to Kafka exactly once to a particular partition of a topic during the lifetime of a single producer without data loss and order per partition.

    “Note that enabling idempotence requires MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to be less than or equal to 5, RETRIES_CONFIG to be greater than 0 and ACKS_CONFIG be ‘all’. If these values are not explicitly set by the user, suitable values will be chosen. If incompatible values are set, a ConfigException will be thrown”

    To achieve idempotence Kafka uses unique id which is called product id or PID and sequence number while producing messages. Producer keep incrementing sequence number on each message published which map with unique PID. Broker always compare current sequence number with previous one and it reject if new one is not +1 greater than previous one which avoid duplication and same time if more than greater show lost in messages

    enter image description here

    In failure scenario broker will compare sequence number with previous one and if sequence not increased +1 will reject the message.

    enter image description here

    Transaction (isolation.level)

    Transactions give us the ability to atomically update data in multiple topic partitions. All the records included in a transaction will be successfully saved, or none of them will be.It allows you to commit your consumer offsets in the same transaction along with the data you have processed, thereby allowing end-to-end exactly-once semantics.

    Producer doesn’t wait to write message to kafka wherease Producer uses beginTransaction, commitTransaction and abortTransaction(in case of failure) Consumer uses isolation.level either read_committed or read_uncommitted

    • read_committed: Consumer will always read committed data only.
    • read_uncommitted: Read all messages in offset order without waiting for transactions to be committed

    If a consumer with isolation.level=read_committed reaches a control message for a transaction that has not completed, it will not deliver any more messages from this partition until the producer commits or aborts the transaction or a transaction timeout occurs. The transaction timeout is determined by the producer using the configuration transaction.timeout.ms(default 1 minute).

    Exactly-Once in Producer & Consumer

    In normal condition where we have seperate producer and consumer. Producer has to idempotent and same time manage transaction so consumer can use isolation.level to read only read_committed to make whole process as atomic operation. This makes guarantee that producer will always sync with source system. Even producer crash or transaction aborted , it always be consistent and publish message or batch of message as unit once.

    The same consumer will either receive message or batch of message as unit once.

    In Exactly-Once semantic Producer along with Consumer will appeared as atomic operation which will operate as one unit. Either publish and get consumed once at all or aborted.

    Exactly Once in Kafka Stream

    Kafka Stream consume messages from topic A , process and publish message to Topic B and once publish use commit(commit mostly run under cover) to flush all state store data to disk.

    Exactly-once in Kafka Stream is read-process-write pattern which guarantee that these operation will be treated as atomic operation. Since Kafka Stream cater producer , consumer and transaction all together Kafka Stream comes special parameter processing.guarantee which could exactly_once or at_least_once which make life easy not to handle all parameters separately.

    Kafka Streams atomically updates consumer offsets, local state stores, state store changelog topics and production to output topics all together. If any one of these steps fail, all of the changes are rolled back.

    processing.guarantee : exactly_once automatically provide below parameters you no need to set explicetly

    1. isolation.level=read_committed
    2. enable.idempotence=true
    3. MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=5
  • MapReduce Composite Key Operation-Part2

    Composite Key Operation In complex operation where we require multi column operation, basic key may not help. For example if we need to calculate population group by country and state then selection of key matter. If we choose the composite key wisely it could solve problem easily. We can create composite key by implementing WritableComparable interface which make it use like any normal WritableComparable interface object. Below is the composite key with two fields country and state which overwrite compare method to do sorting based on country and state. It provides write() and readfields() method to searlized and de-searlized attributes.

    private static class CompositeGroupKey implements
    WritableComparable<CompositeGroupKey> {
    String country;
    String state;
    public void write(DataOutput out) throws IOException {
    WritableUtils.writeString(out, country);
    WritableUtils.writeString(out, state);
    }
    public void readFields(DataInput in) throws IOException {
    this.country = WritableUtils.readString(in);
    this.state = WritableUtils.readString(in);
    }
    public int compareTo(CompositeGroupKey pop) {
    if (pop == null)
    return 0;
    int intcnt = country.compareTo(pop.country);
    return intcnt == 0 ? state.compareTo(pop.state) : intcnt;
    }
    @Override
    public String toString() {
    return country.toString() + ":" + state.toString();
    }
    }
    

    We are using above composite key to create MapReduce job to count total population group by Country and State. input

    Country State City Population (Mil)
    USA, CA Su 12
    USA, CA SA 42
    USA, CA Fr 23
    USA, MO XY 23
    USA, MO AB 19
    USA, MO XY 23
    USA, MO AB 19
    IND, TN AT 11
    IND, TN KL 10

    output

    Country State Total Population
    IND TN 21
    USA CA 77
    USA MO 84

    Mapper Program Once we define composite key we create the mapper class which use input generated from InputFormat. Input Format split the file and pass to individual Mapper which invoke multiple map tasks. Map task transform input split record into Key-value pair where Key and Value should be implement WritableComparable interface. Writable Interface provides the capabilities to write the data into disk and sort it. The Number of map task will be decided based on InputSplits defined in InputFormat. The split is a logically split not physical. The MapReduce first invoke setup () method of context and then invoke map (Object, Object, Context) for each input split and at last invoke cleanup (Context) method for cleanup activity. We extend Mapper class to basic generic Mapper<Key2, Value1, Key2, Value2> class which indicate the input & out put for key and value(s). Mapper class could be overwrite map () method to process the input data as below

    public void map(LongWritable key, Text value, Context context)
    throws IOException, InterruptedException {
    String line = value.toString();
    String[] keyvalue = line.split(",");
    populat.set(Integer.parseInt(keyvalue[3]));
    CompositeGroupKey cntry = new CompositeGroupKey(keyvalue[0], keyvalue[1]);
    context.write(cntry, populat);
    }
    

    If you see above map method it pass Key as Object and value as Text. Value contains each line of file. Inside map method we split the line and create the Key-value and pass to intermediate area through context. Context will fill the io buffer with key-value mapper and later spills to local disk. The map task output grouped by sorted key and writes to the local disk as intermediate data. The grouping of map output defined by partition that identifies the reducer for each key. MapReduce could also provides local combiner which combine intermediate map output and pass to the reducer to cut down the amount of data transferred from the Map to the Reducer. Reducer: The Reducer copy intermediate map tasks output from local disk through http and pass to individual reducer based on each key. Before invoking individual reduce task, Reducer shuffle, merge and sort key-value. Reducer process collection of values for each key and write to the disk. Below is the reduce method which will be spawned by Reducer for each Key that means each reduce task would be having single key with collection of values

    public void reduce(CompositeGroupKey key, Iterator<IntWritable> values,Context context) throws IOException, InterruptedException {
    int cnt = 0;
    while (values.hasNext()) {
    cnt = cnt + values.next().get();
    }
    context.write(key, new IntWritable(cnt));
    }
    

    Job: For running the MapReduce we have to set the Mapper, Reducers and other property in JobConf. JobConf is the main configuration class, which configure MapReduce parameters such as Mapper, Reducer, Combiner, InputFormat, OutputFormat, and Comparator etc. Below code shows how to create and run the job based on above map and reduce code

    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "GroupMR");
    job.setJarByClass(GroupMR.class);
    job.setMapperClass(GroupMapper.class);
    job.setReducerClass(GroupReducer.class);
    job.setOutputKeyClass(CompositeGroupKey.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.setMaxInputSplitSize(job, 10);
    FileInputFormat.setMinInputSplitSize(job, 100);
    FileInputFormat.addInputPath(job, new Path("/Local/data/Country.csv"));
    FileOutputFormat.setOutputPath(job, new Path("/Local/data/output"));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
    

    This main method calls the MapReduce Job. Before calling the job we need to set the MapperClass, ReducerClass, OutputKeyClass and OutputValueClass. We can also set the FileInputFormat and FileOutputFormat. Running: To run the MapReduce program is quite straight forward, what we need to do package the java application in JAR say hadoopessence.jar and run as below in command line hadoop jar hadoopessence.jar.jar org.hadoopessence.compositesorting /input folder and /output folder in HDFS Download: Click here to download source code Reference Hadoop Essence: The Beginner’s Guide to Hadoop & Hive

  • Hadoop MapReduce Group By Operation – Part1

    Introduction This article is continuity of my previous articles on Hadoop MapReduce. In my previous article from my Book Hadoop Essence, I had discussed about overall MapReduce architecture and how it works plus basic Hello World program. In this article I will be discussing how to write basic MapReduce program to calculate sum based on group by operation.

    You can download full code in here. Click Here to Download source code.

    Input and Output This program read the file country.csv, which contains Country CD, State, City, Population (million) USA, CA, Sunnyvale, 12 USA, CA, SAN JOSE, 42 USA, MO, XY, 23 USA, MO, AB, 19 IND, TN, AT, 11 IND, TN, KL, 10 MapReduce program will process and aggregate the total population group by country and state as below Country CD, State, Total Population (million) IND, TN, 22 USA, CA, 54 USA, MO, 42

    Environment Setup You can refer Setup link to setup to configure Apache Hadoop.

    Project Dependency We have to add Hadoop dependency in POM.xml of Maven project. We can create a maven based Java project and add the below Hadoop core dependency in POM. In this application I am using Hadoop 1.x version.

    <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>
    

    Mapper Program Mapper program split input file and spawned map task for each splits. Map task transform input split record into Key-value pair where Key and Value should be implement WritableComparable interface. Writable Interface provides the capabilities to write the data into disk and sort it. The Number of map task will be decided based on InputSplits defined in InputFormat. The split is a logically split not physical splits. The MapReduce first invoke setup () method of context and then invoke map (Object, Object, Context) for each input split and at last invoke cleanup (Context) method for cleanup activity. We extend Mapper class to basic generic Mapper<Key2, Value1, Key2, Value2> class which indicate the input & out put key and value. Mapper class could be overwrite map () method to process the input data as below

    public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
                String line = value.toString();
                String[] keyvalue = line.split(&amp;amp;quot;,&amp;amp;quot;);
                cntText.set(new Text(keyvalue[0]));
                stateText.set(keyvalue[1]);
                populat.set(Integer.parseInt(keyvalue[3]));
                Country cntry = new Country(cntText, stateText);
                context.write(cntry, populat);
            }
            

    If you see above map method it pass Key as Object and value as Text. Value contains each line of file. Inside map method we split the line and create the Key-value and pass to intermediate area through context. Context will fill the io buffer with key-value mapper out and later spills to local disk. Below is the custom Country key, which I used for sorting the value country wise and then state wise. Below is country key which implements WritableComparable interface

    private static class Country implements WritableComparable&amp;amp;lt;Country&amp;amp;gt; {
    Text country;
            Text state;
            public Country(Text country, Text state) {
                this.country = country;
                this.state = state;
            }
            public Country() {
                this.country = new Text();
                this.state = new Text();
            }
    public void write(DataOutput out) throws IOException {
                this.country.write(out);
                this.state.write(out);
            }
    public void readFields(DataInput in) throws IOException {
                this.country.readFields(in);
                this.state.readFields(in);
            }
    public int compareTo(Country pop) {
                if (pop == null)
                    return 0;
                int intcnt = country.compareTo(pop.country);
                if (intcnt != 0) {
                    return intcnt;
                } else {
                    return state.compareTo(pop.state);
    
                }
            }
    @Override
            public String toString() {
    return country.toString() + &amp;amp;quot;:&amp;amp;quot; + state.toString();
            }
        }
    

    In the above Key I have implemented compare method that will be used MapReduce program to sort the values. Below is the full mapper code

    public  class GroupMapper extends Mapper&amp;amp;lt;LongWritable, Text, Country, IntWritable&amp;amp;gt; {
    Country cntry = new Country();
    Text cntText = new Text();
    Text stateText = new Text();
    IntWritable populat = new IntWritable();
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
    String line = value.toString();
                String[] keyvalue = line.split(&amp;amp;quot;,&amp;amp;quot;);
                cntText.set(new Text(keyvalue[0]));
                stateText.set(keyvalue[1]);
                populat.set(Integer.parseInt(keyvalue[3]));
                Country cntry = new Country(cntText, stateText);
                context.write(cntry, populat);
    
            }
        }
    

    The map task output grouped by sorted key writes to the local disk and intermediate data. The grouping of map output defined by partition that identifies the reducer for each key. MapReduce also provide local combiner which combine intermediate map output and pass to the reducer. It helps to cut down the amount of data transferred from the Map to the Reducer. Here org.apache.hadoop.io.Text class as key, which provides methods to searlized, and compare texts at byte level. The value IntWritable again searlized as it implements writable interface and support sorting as it implements WritableComparable interface. Reducer: The Reducer copy intermediate map tasks output from local disk through http and pass to individual reducer based on each key. Before invoking individual reduce task Reducer shuffle, merge and sort key-value. Reduce task process collection of value for single key. Reducer process collection of values for each key and writ to the disk. Reducer class implements generic Reducer class as below public static class GroupReducer extends Reducer<Country, IntWritable, Country, IntWritable> { The generic Reducer class defined input & output key-value here we are getting input in <Country, IntWritable> and spill the out as <Country, IntWritable> Below is the reduce method which will be spawned by Reducer method for each Key that means each reduce task would be having single key with collection of values.

    public void reduce(Country key, Iterator&amp;amp;lt;IntWritable&amp;amp;gt; values, Context context) throws IOException,
                    InterruptedException {
    int cnt = 0;
                while (values.hasNext()) {
                    cnt = cnt + values.next().get();
                }
                context.write(key, new IntWritable(cnt));
    
            }
            

    Job: For running the MapReduce we have to set the Mapper, Reducers and other property in JobConf. JobConf is the main configuration class, which configure MapReduce parameters such as Mapper, Reducer, Combiner, InputFormat, OutputFormat, and Comparator etc. Below code shows how to create and run the job based on above map and reduce code

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
            FileUtils.deleteDirectory(new File(&amp;amp;quot;/Local/data/output&amp;amp;quot;));
            Configuration conf = new Configuration();
            Job job = Job.getInstance(conf, &amp;amp;quot;GroupMR&amp;amp;quot;);
            job.setJarByClass(GroupMR.class);
            job.setMapperClass(GroupMapper.class);
            job.setReducerClass(GroupReducer.class);
            job.setOutputKeyClass(Country.class);
            job.setOutputValueClass(IntWritable.class);
            FileInputFormat.setMaxInputSplitSize(job, 10);
            FileInputFormat.setMinInputSplitSize(job, 100);
            FileInputFormat.addInputPath(job, new Path(&amp;amp;quot;/Local/data/Country.csv&amp;amp;quot;));
            FileOutputFormat.setOutputPath(job, new Path(&amp;amp;quot;/Local/data/output&amp;amp;quot;));
            System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
    

    This main method calls the MapReduce Job. Before calling the job we need to set the MapperClass, ReducerClass, OutputKeyClass and OutputValueClass. We can also set the FileInputFormat and FileOutputFormat. FileInputFormat also use to decide number of map tasks. We can also set input split size, which take part while deciding number of map task s below. Split size can min between maxSize and blockSizeMath.min(maxSize, blockSize) As per above code, framework will execute the map-reduce job as described in Job.

    Running To run the MapReduce program is quite straight forward, what we need to do package the java application in JAR say hadoopessence.jar and run as below in command line hadoop jar hadoopessence.jar.jar org.techmytalk.hadoopessence. GroupMR /input folder and /output folder in HDFS For testing we can directly run this application as Run As in eclipse.

    Download Source We can download full source using download link

    Summary: In this article I explained basic Map Reduce to compute Group By operation Use download link to download full source code.

    Reference Hadoop Essence: The Beginner’s Guide to Hadoop & Hive

Create your own website, free on WordPress.com.Create your own website.
Start free