explicit vs. implicit context

difference between the two and how they can be gathered better

Converting business requirements given by a stakeholder into technical requirements is a task within itself. So, other than a straight-forward bug or maintenance tasks, it’s always good to have a design jira preceding the actual development jira to dissect, evaluate and expand on the business requirements to come up with the technical requirements that would become the prime point of reference for all the associated development work.

Other than the obvious checking the availability of data, accesses to any necessary APIs etc., it’s highly critical to identify the missing pieces in the business requirements in order to avoid back-and-forth development work and also any additional future maintenance. I recently read a post somewhere which talked about how building a robust system would enable quick turn arounds. An overly simplified example would be to imagine a scenario where you incorporated an additional field in a new data pipeline that was not asked for in the business requirements but you anticipated from your experience that this field could come into play in the very near-future given how it’s an important one. And when the business comes a month or two later requesting for a new metric involving this very field, the turn around would be super quick given how this field already exists in the table and all you have to do is incorporate the business logic to derive the metric. Had you not included this field initially, you would’ve had to go through the whole process of updating the pipeline, table schema, backfilling the historical data which would take much longer.

This isn’t to mean that we should over-optimize everything. There are costs associated with additional work and we need to be smart about the choices and trade-offs we make.

This is where context comes into play and this is how explicit and implicit context differ from each other. Explicit context is comparatively easier to crack and it’s more generalized in a sense whereas implicit context is slightly trickier and many a times needs some solid domain knowledge to identify.

Imagine you are building a pipeline for some orders data. The requirement was to utilize this data to come up with a total sales metric at a store and a product level. Strictly sticking with whatever is asked in the requirements, the fields that can be brought in are - store_id, product_id, sale_amount and the corresponding store and product information from the relevant dimension tables. But say, there is also a field called “status”. Digging deeper you see that there are some records where the status is Cancelled and when you clarify this with the business, they confirm that we indeed need to exclude the cancelled orders from the total sales metric. This is a good example of explicit context where the information is quite straight-forward and easily identifiable given how order status, cancelled orders are something of common knowledge for most engineers irrespective of their seniority.

Implicit context on the other hand is not as obvious or easily identifiable. Implicit context can sometimes be missed even by people who’ve had years of domain knowledge because some details can be hidden so deep in the weeds that we can only react to this missed context when some user points it out. But there would also be cases where with the right amount of domain knowledge and experience, you’d be able to identify something that’s not so obvious. It could be something as simple as identifying an anomalous data point or something very layered where you identify a certain egregious pattern through connecting some select dots.

Getting better at connecting these dots and also knowing what dots to even connect in the first place is what makes you better at identifying implicit context. Though this can’t be taught in a structured way like how you could probably teach someone SQL or data modeling, I believe you can still learn by being observant of how people around you with comparatively better domain knowledge are going about identifying patterns. Every pattern is unique in a sense. It’s more about building that muscle where you keep bettering yourself at finding these patterns.