Advanced Find
IBM Datastage
IBM Datastage
Year
'24
Client
IBM DataStage
Service
Product design
Making complex data findable: The advanced find experience for DataStage
Background
IBM DataStage® is an ETL tool used to integrate and move data between systems — extracting it from sources like databases or cloud services, transforming it per business rules, and loading it into warehouses or analytics platforms.
DataStage lives within IBM Cloud Pak for Data, alongside other connected components like cataloging, governance, and data quality. Any search feature here has to work within that larger ecosystem, not as a standalone tool.
Advanced Find is part of DataStage's move from a legacy desktop tool to a modern platform — and it inherits a specific expectation. Users who came from InfoSphere already trusted its visual query builder. They weren't asking for something new; they wanted something familiar, rebuilt.
In complex projects, changing one piece can affect dozens of connected parts. Next-gen made this painful.
The Problem
Customers needed to:
find their data assets,
see how they connect, and
understand what breaks if they change something.
Without a clear way to trace those dependencies, even routine discovery work became slow, fragmented, and difficult to trust.
Complex asset relations
Not all assets are similar, every asset has different properties, and those properties hold different kinds of values.
Non-uniform search ability
Not all properties are searchable the same way. You can't search a data type by typing arbitrary text; it has to be filtered through predefined options.
The three-layered solution
DataStage's search experience was built in three connected parts: Canvas Find, which locates objects within a single pipeline flow; View Relationship, which shows how assets connect; and Advanced Find, which searches across any asset, anywhere in the platform.
I was responsible for the end-to-end design of Advanced Find from refining an early stakeholder concept through research, visual design, and shipping to production. This is that story.
Canvas Find
Advanced Find
Search assets across the entire platform with Advanced Find
View Relationship
Key Outcomes
~7,000 users
Adopted across DataStage’s user base as the default way to search assets.
Reusable query logic
Saved query patterns made repeat tasks easier to share and repeat.
Fewer SQL escalations
More engineers could complete complex searches independently.
~60% less search time
A guided query replaced manual, column-by-column checking.
Research and Discovery
To validate assumptions and define a user-centric solution, we conducted targeted research over two weeks — user interviews with data engineers from enterprises using both InfoSphere and DataStage, and task flow audits mapping friction points in the existing workflow.
Discovery session with users
Key Insights
InfoSphere familiarity drives expectations — users consistently referenced InfoSphere’s visual query builder; the gap caused task abandonment or dependency on SQL-trained colleagues.
Cognitive load — users were forced to recall schema names or memorize column structures without UI assistance.
No-code flexibility — even technical users preferred a visual interface for building logic-based filters.
Error-prone trial and error — without a live preview, users ran filters blind and had little guidance for recovering from empty results.
Reusability and collaboration — query logic was often reused or shared with team members, with no support for that workflow.
Design Goals
Goal | Description |
|---|---|
Visual Query Construction | Build advanced filters using a no-code, intuitive interface. |
Schema-Awareness | Column/field selection with support for data-type-specific operators. |
Dynamic Logic Handling | Support AND/OR conditions. |
Real-Time Feedback | Preview results instantly as filters are modified. |
Reusability | Allow users to save, reuse, and manage query templates. |
Legacy Workflow Continuity | Preserve familiar InfoSphere paradigms while enhancing usability. |
Key Innovations
Based on our research insights, user interviews, and the clear need for a code-free yet powerful filtering experience, we conceptualised Advanced Find — a dynamic, logic-driven query builder designed for users across technical and non-technical backgrounds.
This solution aimed to bridge the gap between complexity and usability. While users needed the ability to perform deep, multi-condition searches (often involving nested logic), they were often limited by technical constraints or rigid filter tools. Advanced Find addresses this by offering a visual interface that feels intuitive but scales in capability — much like writing SQL, but in plain English and guided form.
A Note on Validation
That was the first instinct from experienced users. But the tension here ran deeper: Legacy's advanced search was genuinely more complex, and users had come to associate that complexity with capability. Holding the line on a simpler design meant trusting that usability and power weren't a trade-off.
The Broader Impact
Advanced Find became the default way non-technical users searched across DataStage, replacing manual schema recall and manual coding with a guided, visual query. What used to require SQL knowledge, or a wait for someone who had it, became something any user could do in minutes.
Background
IBM DataStage® is an ETL tool used to integrate and move data between systems — extracting it from sources like databases or cloud services, transforming it per business rules, and loading it into warehouses or analytics platforms.
DataStage lives within IBM Cloud Pak for Data, alongside other connected components like cataloging, governance, and data quality. Any search feature here has to work within that larger ecosystem, not as a standalone tool.
Advanced Find is part of DataStage's move from a legacy desktop tool to a modern platform — and it inherits a specific expectation. Users who came from InfoSphere already trusted its visual query builder. They weren't asking for something new; they wanted something familiar, rebuilt.
In complex projects, changing one piece can affect dozens of connected parts. Next-gen made this painful.
The Problem
Customers needed to:
find their data assets,
see how they connect, and
understand what breaks if they change something.
Without a clear way to trace those dependencies, even routine discovery work became slow, fragmented, and difficult to trust.
Complex asset relations
Not all assets are similar, every asset has different properties, and those properties hold different kinds of values.
Non-uniform search ability
Not all properties are searchable the same way. You can't search a data type by typing arbitrary text; it has to be filtered through predefined options.
The three-layered solution
DataStage's search experience was built in three connected parts: Canvas Find, which locates objects within a single pipeline flow; View Relationship, which shows how assets connect; and Advanced Find, which searches across any asset, anywhere in the platform.
I was responsible for the end-to-end design of Advanced Find from refining an early stakeholder concept through research, visual design, and shipping to production. This is that story.
Canvas Find
Advanced Find
Search assets across the entire platform with Advanced Find
View Relationship
Key Outcomes
~7,000 users
Adopted across DataStage’s user base as the default way to search assets.
Reusable query logic
Saved query patterns made repeat tasks easier to share and repeat.
Fewer SQL escalations
More engineers could complete complex searches independently.
~60% less search time
A guided query replaced manual, column-by-column checking.
Research and Discovery
To validate assumptions and define a user-centric solution, we conducted targeted research over two weeks — user interviews with data engineers from enterprises using both InfoSphere and DataStage, and task flow audits mapping friction points in the existing workflow.
Discovery session with users
Key Insights
InfoSphere familiarity drives expectations — users consistently referenced InfoSphere’s visual query builder; the gap caused task abandonment or dependency on SQL-trained colleagues.
Cognitive load — users were forced to recall schema names or memorize column structures without UI assistance.
No-code flexibility — even technical users preferred a visual interface for building logic-based filters.
Error-prone trial and error — without a live preview, users ran filters blind and had little guidance for recovering from empty results.
Reusability and collaboration — query logic was often reused or shared with team members, with no support for that workflow.
Design Goals
Goal | Description |
|---|---|
Visual Query Construction | Build advanced filters using a no-code, intuitive interface. |
Schema-Awareness | Column/field selection with support for data-type-specific operators. |
Dynamic Logic Handling | Support AND/OR conditions. |
Real-Time Feedback | Preview results instantly as filters are modified. |
Reusability | Allow users to save, reuse, and manage query templates. |
Legacy Workflow Continuity | Preserve familiar InfoSphere paradigms while enhancing usability. |
Key Innovations
Based on our research insights, user interviews, and the clear need for a code-free yet powerful filtering experience, we conceptualised Advanced Find — a dynamic, logic-driven query builder designed for users across technical and non-technical backgrounds.
This solution aimed to bridge the gap between complexity and usability. While users needed the ability to perform deep, multi-condition searches (often involving nested logic), they were often limited by technical constraints or rigid filter tools. Advanced Find addresses this by offering a visual interface that feels intuitive but scales in capability — much like writing SQL, but in plain English and guided form.
A Note on Validation
That was the first instinct from experienced users. But the tension here ran deeper: Legacy's advanced search was genuinely more complex, and users had come to associate that complexity with capability. Holding the line on a simpler design meant trusting that usability and power weren't a trade-off.
The Broader Impact
Advanced Find became the default way non-technical users searched across DataStage, replacing manual schema recall and manual coding with a guided, visual query. What used to require SQL knowledge, or a wait for someone who had it, became something any user could do in minutes.
Background
IBM DataStage® is an ETL tool used to integrate and move data between systems — extracting it from sources like databases or cloud services, transforming it per business rules, and loading it into warehouses or analytics platforms.
DataStage lives within IBM Cloud Pak for Data, alongside other connected components like cataloging, governance, and data quality. Any search feature here has to work within that larger ecosystem, not as a standalone tool.
Advanced Find is part of DataStage's move from a legacy desktop tool to a modern platform — and it inherits a specific expectation. Users who came from InfoSphere already trusted its visual query builder. They weren't asking for something new; they wanted something familiar, rebuilt.
In complex projects, changing one piece can affect dozens of connected parts. Next-gen made this painful.
The Problem
Customers needed to:
find their data assets,
see how they connect, and
understand what breaks if they change something.
Without a clear way to trace those dependencies, even routine discovery work became slow, fragmented, and difficult to trust.
Complex asset relations
Not all assets are similar, every asset has different properties, and those properties hold different kinds of values.
Non-uniform search ability
Not all properties are searchable the same way. You can't search a data type by typing arbitrary text; it has to be filtered through predefined options.
The three-layered solution
DataStage's search experience was built in three connected parts: Canvas Find, which locates objects within a single pipeline flow; View Relationship, which shows how assets connect; and Advanced Find, which searches across any asset, anywhere in the platform.
I was responsible for the end-to-end design of Advanced Find from refining an early stakeholder concept through research, visual design, and shipping to production. This is that story.
Canvas Find
Advanced Find
Search assets across the entire platform with Advanced Find
View Relationship
Key Outcomes
~7,000 users
Adopted across DataStage’s user base as the default way to search assets.
Reusable query logic
Saved query patterns made repeat tasks easier to share and repeat.
Fewer SQL escalations
More engineers could complete complex searches independently.
~60% less search time
A guided query replaced manual, column-by-column checking.
Research and Discovery
To validate assumptions and define a user-centric solution, we conducted targeted research over two weeks — user interviews with data engineers from enterprises using both InfoSphere and DataStage, and task flow audits mapping friction points in the existing workflow.
Discovery session with users
Key Insights
InfoSphere familiarity drives expectations — users consistently referenced InfoSphere’s visual query builder; the gap caused task abandonment or dependency on SQL-trained colleagues.
Cognitive load — users were forced to recall schema names or memorize column structures without UI assistance.
No-code flexibility — even technical users preferred a visual interface for building logic-based filters.
Error-prone trial and error — without a live preview, users ran filters blind and had little guidance for recovering from empty results.
Reusability and collaboration — query logic was often reused or shared with team members, with no support for that workflow.
Design Goals
Goal | Description |
|---|---|
Visual Query Construction | Build advanced filters using a no-code, intuitive interface. |
Schema-Awareness | Column/field selection with support for data-type-specific operators. |
Dynamic Logic Handling | Support AND/OR conditions. |
Real-Time Feedback | Preview results instantly as filters are modified. |
Reusability | Allow users to save, reuse, and manage query templates. |
Legacy Workflow Continuity | Preserve familiar InfoSphere paradigms while enhancing usability. |
Key Innovations
Based on our research insights, user interviews, and the clear need for a code-free yet powerful filtering experience, we conceptualised Advanced Find — a dynamic, logic-driven query builder designed for users across technical and non-technical backgrounds.
This solution aimed to bridge the gap between complexity and usability. While users needed the ability to perform deep, multi-condition searches (often involving nested logic), they were often limited by technical constraints or rigid filter tools. Advanced Find addresses this by offering a visual interface that feels intuitive but scales in capability — much like writing SQL, but in plain English and guided form.
A Note on Validation
That was the first instinct from experienced users. But the tension here ran deeper: Legacy's advanced search was genuinely more complex, and users had come to associate that complexity with capability. Holding the line on a simpler design meant trusting that usability and power weren't a trade-off.
The Broader Impact
Advanced Find became the default way non-technical users searched across DataStage, replacing manual schema recall and manual coding with a guided, visual query. What used to require SQL knowledge, or a wait for someone who had it, became something any user could do in minutes.
Challenge
IBM DataStage users needed a way to locate specific values buried inside large, complex data assets but existing search offered no way to narrow results by field, type, or context. Users were left scanning manually or relying on guesswork, slowing down workflows that needed precision.
Solution
I designed an advanced find feature with filterable, context-aware search letting users narrow results by asset type and attribute instead of scrolling through everything. Clear result previews and filter states made it easy to refine a search without losing track of what was already applied.
Banking app