Big Data Discovery
Big Data Discovery is a product built on Apache Spark that enables anyone to turn massive amounts of raw data into new insight in minutes. Main functional areas: Explore to learn about the data, Transform for cleaning and wrangling activities and Discover to share insights.
WHAT
We app
WHERE
Oracle
WHEN
Project Overview
There was a company called Endeca, which made faceted search and was bought by Oracle. I have joined them when the concept was defined and we developed multiple features to make that much more robust. I took the challenge to work on the most complex part: Transform, which enabled people to shape their data to be able to visualize.
2015-2017
​
1-2 months long sprints in a 2 years journey
My Contribution
-
I was responsible for a full redesign of Transform workflow, user experience and interface to align more with the user needs
-
I defined Jet standards, which then merged into the Oracle Alta systems
-
I introduced usability plan to get Big Data Discovery to the next level in usability
-
I designed multiple complex functions like aggregation, join data sets, transform tray, bulk editing, replace UI with regular expression support, normalize
-
I introduced iconfonts
-
I created a standard design communication platform
-
I brought a 3rd party solution to measure usage statistics
The Journey
I worked on several big projects, which could be a separate use case each. This story will share some insights about me around different challenges and my approach. I have started my Oracle journey with a complex function called aggregation, which required me to jump to statistics and refresh my dusty university knowledge on the field and learn real quick to be able to deliver a solution. Soon enough I realized that a materialized aggregation could not fit into the actual framework Transform had at that time. It had all the functions hidden in menus for each and every attribute. Some part of it was context dependent, but some not. No bulk operation was possible as well as functions, which changes the whole data table (like aggregations and joins). This is what we had:
The Challenge
As you can see in the pictures above a whole big world was hidden in meaningless hamburger menus, which are hard to explore, learn and more importantly does not support essential functions, like editing the full table. Let's see aggregation: in a typical scenario if you aggregate a table on a specific grain many attributes (columns) become meaningless, so need to be dropped and you will have fewer rows typically, like this:
Bulk editing also does not work with a column-wise hamburger metaphor. Imagine if you need to work on all the orange columns at the same time:
Design Options
I had multiple options, but 2 major metaphors made it to the final round:
-
Ribbon approach
-
Property panel approach
The ribbon is what Microsoft invented. I do know the benefits and pitfalls of that solution. Without detailing all the user research it was done on that approach let me highlight the biggest advantages for my design to that solution. The biggest target audience were the business people, and just the smaller one was data scientists. For the latter, we already had an editor, with a query UI, so the bigger challenge was the business people. They, however, are really great in Excel. With all the disadvantages of the ribbon metaphor, a well know, the almost standard solution is a huge plus.
The so-called property panel approach is more like what Mac has in their software solutions: based on the selection there is one panel on the right which contains column specific options. Bulk is just a situation when one has more columns selected and the property panel can handle that. Some exceptions, which are higher level, like load goes above the table. It would work better with tables, aligned with other BI solutions, like Tableau on the visualization side and could nicely incorporate smart functions like predict.
One important side note here that Product Managers already had a huge list of features they wanted to push based on customer complaints. The MVP plan was to squeeze a button somewhere and it will aggregate. I had to sell real usability vision over chasing features and buy the top level leaders into the mode of stepping forward towards real usability. I had to demonstrate the clear issues in the user flows, support the problem with data (which was based on usability tests I conducted - see later) than present viable concepts.
I had my preference, but to buy all the leaders into this project I presented both solutions equally with pros and cons and I let them make the decision. This way they felt much more attached to that proposal and I assumed they will allocate the right amount of resources to the task.
The Solution
It worked and we could lay out a plan to introduce the functions in separate sprints, which was important from the development point of view as well.
Standards
As you can see from the pictures above I had to deliver redlined documents to developers, which is great, but not really efficient to define colors, sizes in each and every project. This frustration was a strong contributor to another big project I led, which was the Jet standards. Luckily there was an appetite for that from the technical point of view and I got supported by leaders as well because the code organically evolved and the front end developers struggled a lot. I defined a framework and many important pieces, and together with my great UX peers we defined a whole detailed standard documentation for our product:
Icons
Related to the Ribbon and also solving an earlier technical challenge we explored multiple options with the developers and introduced Iconfonts. The benefit was not to generate every state as a separate picture (hover, selected, normal). I had to generate some extensive documentation to my peers to generate those, but we gain a lot on the technical side as well as less labor intense production on the future.
Side Effects
I had another project, which on purpose nicely played together with this Ribbon approach. Every transformation action was collected in a so-called Transform Tray, which was really a container of all the steps anyone did to a specific data set. I had a big vision about the Tray to show a full pipeline of a data set, which realized only in Data Intelligence Platform a year later. We could push through a smaller project, however, which used the icons of the ribbon and with a bit of a visual update I designed, we made the tray lot more readable.
Before
After
Measure Usability
Many usability points are simply hard to make by saying that I am doing this for more than 20 years or calling out any principle, which we went against. Some people will still say that it is good enough. Some people simply do not believe the importance of usability (I do not blame them - it is my job, not theirs). There are so many bad examples out in the world and people can say: this is how "random successful company name with an example" did it, so it is good. I cannot say that that company is successful for other reason, not for their usability. Or even if they are successful in usability you still can find one or two failures they made.
A couple of points can be made only by presenting proof. Have a simple task, users should be able to do, pick someone from the product's target audience and just simply record what they do. And that is what I do.
In the following video, I asked people to rename an attribute. The function did exist, but there were too many compromises in development around this feature, misleading error messages, not in-place editing, so users failed in renaming. I have reported a Jira ticket, but it was prioritized as low (because the feature existed and people wanted to chase other things). I already observed a lot how our target audience worked and I knew that this renaming feature was really an everyday task, but I still had no success in making my point. I could imagine really simple solutions like just change the warning, which is a really low-cost investment, but that time I failed against the decision makers. Luckily my manager supported me to run a usability program (which had many other benefits - it was not focused on rename) and I had rename as a warm-up challenge. This is how people "succeeded":
Usability Program
The goal of the usability program was to bring awareness of usability across the organization. I made presentations to the Chinese developers, QA, and many different departments, presenting the results with the software we actually developed (see below). I grouped the issues by usability measures, which came out from the usability tests and a short questionnaire I had (how frequently they faced a problem in real life examples, could they fix it during the test, and how frustrated they were).
The presentations were really successful and I was asked to show that to the higher level management as well. After I did that we introduced a new label in Jira for usability. With that in the system, developers could fix the rename and introduce in-place editing... Obviously, in this story, the real gain was not fixing that small usability issue but bringing awareness of usability across the whole organization. It was just a funny side effect that we could fix a small, but important issue as well.
Usage Statistics
To do everything manually is great, but automate it is way better. I wanted to know where people click on the product, how they used it to bring data in front of PM. In this case, it was the head of the developers who supported this initiative and I could draft a couple options up to see how we do this. One option was to use our own software for visualizing the usage, we only had to gather the data. It sounds easy, but the code work required more man hours that we were willing to spend on this. So I explored to work with a 3rd party company, who already specialized to this and have their solution with all the statistics. One challenge was to keep the users' privacy, but still learn their usage patterns. But the biggest challenge was to bring a 3rd party company into an extremely complex legal and corporate environment. The success of the story was that Pendo became an official Oracle partner.
Sharing
I believe in the importance of sharing the info with stakeholders. That is why I created our own Confluence space with all important details about our target users, the active project we were working on, resources for designers and links for pooers.
The Team
The design team at Oracle is pretty big, hundreds of people all around the globe. I had a chance to work with many of them, involved in different big projects. On Big Data Discovery project we had a small group sitting in Cambridge, having our manager in Chicago (on the right). I was honored to work with them and sometimes make them move to be more together than individually.
