How Has the Pandemic Impacted Remote Work?

This episode of the Designing Enterprise Platforms podcast from Early Adopter Research (EAR) focuses on a topic that is pandemic related. The episode is about the key internal usability metrics for understanding the productivity impact of the pandemic. EAR’s Dan Woods speaks with Brian Berns, the CEO of KNOA, a company that focuses on tracking metrics about the user experience and usability, and they offer a service that tracks users. They’ve noticed a lot of interesting things going on with the workforce during the pandemic. 

You can listen to the full interview here or read an edited Q&A of the conversation below:

Woods: Could you explain a little bit about your history and about the history of KNOA, along with the background of the product and how it’s given you a window on what’s going on with the workforce during the pandemic?

Berns: Certainly. I’ve been in the software business for a couple decades—a combination of entrepreneurial as well as corporate world. So I’ve founded several companies—thankfully, successfully. And KNOA is a very interesting solution. The company’s been around for about 15 years and we’re New York based. And it’s interesting when you refer to user experience, and we all understand user experience from, actually, our shopping experiences online. We’re all familiar with user experience with our online shopping experience. We all understand that our behavior is being tracked and analyzed on these e-commerce sites. The most typical ones are you put something in your shopping cart and then the vendor will recommend, “You know, people who bought this product also bought that product.” But actually, the analysis and the analytics are much more deep than that. They’ll look at all of your behavior on the site. How did you get to the site? What pages are you on? How are you page—how long are you spending on a page? What is the order of information and content that you’re viewing? And essentially, they’re trying to understand and predict your behavior so they can recommend the most relevant and interesting content or products to you. Now, what we do is something similar in the corporate world. We work with large organizations. You mentioned hundreds of thousands of users. It’s actually millions of users. And what happens in the corporate world is that these organizations use these large enterprise software systems from SAP or Oracle. These systems are essentially running these companies from inventory to billing, order tracking, customer service, call center, finance. And, companies with tens of thousands of users interacting with these systems, you can gain a lot of insight into employee productivity and efficiencies. And that’s what we do. We have some of the largest organizations in the world as our customers, from Unilever, Nestle, Johnson and Johnson, government agencies. We have users around the world and actually SAP resells our product. KNOA User Experience Management is the name of the product. Across industry—so every industry has large numbers of employees who are engaged with these software systems. And the real question is companies, after they invest tens of millions of dollars into these systems, are they effective? Are my employees efficient? How do you know that? How would you know that?

It’s as if KNOA can tell, basically, when you put it in a system, every click that you make, every form that you fill, every navigation move that you make. It gives you a full set of metrics so that you can really understand everything that people are doing with the enterprise software. Is that correct?

That’s correct. You could say screen scraper, but actually we’re looking at the underlying workflow. So we understand the order of screens. So, for example, why does it take Mary 12 screens to ship a product but it takes Mark 18 screens? What’s going on there?  And actually, how would you know that? That’s the important question. You could have employees who are struggling. They may not volunteer that they’re having a challenge. They may be getting a lot of error messages which impact their concentration—their ability to complete a task.

Because you collect all this data, it’s like you have this massive data warehouse of user experience information. Now I know that you’ve got this all segregated so one company can’t see another company’s information, but you’re able to see all of it. You’ve been able to see a lot of things that have been going on during the pandemic. And so this provides a real hands-on, data-based way to get insights about the impact of the pandemic. So what have you seen as the impact of the pandemic on workplace productivity based on your internal dataset?

That’s right. We actually had a front row and have a front row to how organizations respond to the challenge of having people work from home. And I’ll jump ahead—how do you return your employees back to the office? But let’s start with when the pandemic hit. The challenge was companies had little, if any, time to prepare. And so as we all experienced, almost overnight, people were required to work from home. So the question is, number one, could they even access the systems? Number two, do the systems work the same way when accessed remotely as they do when the people are in the office? For example, perhaps due to privacy, compliance, other considerations, certain software components or data are not accessible when you’re accessing them remotely. What we saw is initially, companies scrambled with connectivity—ensuring that their employees had the resources that they needed to be effective and access these systems. We saw across the board—across hundreds of companies—which ones were more prepared than others. Because you could imagine a graph and you see activity and all of a sudden it drops off significantly. And then you could see companies that were able to get the equipment and the connectivity software out to their employees. They were back up more quickly than others. And so we saw some very interesting trends and patterns. And companies continue to analyze the effectiveness and engagement of their employees. 

Did you find any patterns about the nature of the companies that were better prepared or the nature of the companies that stopped the companies from being prepared?

Yes. That’s a good a question. I’ll give you a couple examples. One company I can mention by name is TGS, the largest gas transportation company in Latin America. That organization was prepared and they were able to respond to the user’s need to access these systems. We dealt with a large pharmacy, also in Latin America. Challenges with distribution—enabling the pharmacies to open with support from people from home. We work with a large distributor in the United States and as we all saw, the challenges of getting essential food, medicine, other supplies out to the general population. The companies that, as we say, are essential services—they generally had the contingency plans well-formulated beforehand.

So it was really a matter of planning for this eventuality and having the ability to deliver the key interfaces through other means. And also having the capacity, because one of the things that I noticed happened is a lot of people had VPN infrastructure, but it was only for a small number of people who were expected to use it. And then, when they expanded it to a large number of people, it just was over flooded and everything just stopped working properly. Now, you actually dug in deeper to this data and you have also, as part of KNOA’s general help for people to understand productivity and efficiency, a set of metrics. The metrics that you previously mentioned were idle time, task and transaction completion times, error messages, and engagement. And you said if you look at those four things, you can tell a lot about how well a company’s working. Would you go through the list and explain what each one of them does tell you when you analyze it. 

Sure. Idle time is an interesting metric. Let’s say you have thousands of people looking to complete a transaction—ship a product, for example. There are 10 steps. And we see a pattern that after step four, there’s a pause. So people are clicking, clicking along, and then all of a sudden, there’s a pause after a certain screen. Why did they all pause at the same—in the same point in the transaction? So there’s something that’s affected all these employees. We have to look at the user interface. Perhaps the screen there is not intuitive. Perhaps my users are getting an error at that point. Perhaps the workflow is not efficient. So you can go in and you can look to rectify that issue. And then you use our software to verify that the course of action actually was effective. “Yes, the user interface wasn’t intuitive. We modified it, and now people, after step four, they’re just clicking along to step five, step six, et cetera.” Now let’s say idle time is not consistent. Let’s say I look across my employees and I see that idle time is increasing. What does that tell you? It tells you, perhaps, your employees are just not as engaged. And that’s understandable in the era of the pandemic, where people have different challenges in their environment. It could be a multigenerational home, so you have distractions worrying about your elder relatives, dealing with children, and homeschooling. And so people get distracted. On the other hand, you could have people who are self-isolated—alone. And that has other associated challenges. 

The idea here is that you can look at it both on an individual microsystem level, but then also on a macro level, saying, “Is idle time increasing just generally?” Does anybody actually keep track of this as a pulse of the workforce?

Yes, certainly. Most, if not all of our customers are not looking at the individual employee behavior. This isn’t really spying on employees. You’re not getting the value of the software when you look at individuals. The real return is when you look at the behavior of thousands of employees. And most of our customers anonymize the data to ensure privacy, et cetera. But certainly, idle time is important. If you look at how the world of HR has certainly transitioned over the past couple decades from dealing with recruitment and benefits to now focus on the holistic health of the organization. “Are my people satisfied? Are they stimulated? Are they engaged?” You want to ensure that you retain your people and, you know, if you put your people in front of software that is frustrating, that is clunky, that seems antiquated, you’re going to have a challenge keeping your people because for a lot of employees, the software is their work environment. And you can have a nice café and great benefits, but if they’re sitting for six hours, eight hours in front of software that’s frustrating for them, you will see a lot of idle time. Also, you can look at different work patterns. Some of our customers have noticed increased usage on weekends. With people working from home, there’s some more flexibility with your work time. So people, obviously without the commute, but overlaid with challenges with homeschooling, et cetera, we’re seeing a lot more flex time. People are obviously making that decision themselves. “Oh, I’ll put the kids to bed and I’ll go back and work for a couple hours,” Or, “I’m going to spend three hours on a Saturday catching up on the work.” 

Let’s move on to task and transaction completion times. What does that mean and then how does it actually tell us something?

Basically how long does it take to complete a transaction? We’re looking at workflow and I used the example earlier of shipping a product. Sounds pretty simple. In many cases it’s more complex. If a customer places an order, you want to check the credit of that customer. You want to look at their inventory. You want to look at prior products that they purchased to ensure that what they’re buying is compatible. You want to look at, perhaps, any compliance or safety regulations with the actual shipment, what technology or vendor you can use to ship it, et cetera. So you can appreciate that there can be several steps to simply shipping a product. How long does it take to process that order? How many screens do you have to go through to process that order? And that really looks at the usability, the user interface, the workflow—is the workflow efficient? Is it intuitive? So I know after step one, step two makes sense in terms of consecutive steps that flow. And that impacts productivity. That impacts employee satisfaction. I could tell you how many screens it takes to complete the transaction and how long it takes. We had one customer tell us that one of their sites in Argentina was more efficient than a site in Australia and they were completing the transactions much more quickly. Why was that? Is the design the same? Are there perhaps cultural differences? Were employees not trained properly? Perhaps the language wasn’t translated properly? What’s really interesting is we’re giving you a baseline of all this behavior. When you implement our software, we’re going to give you a rich set of analytics so you can look at your current reality. That certainly helps your organization be more efficient. It helps you with customer satisfaction. Your customers are getting the products shipped more quickly. 

What can you tell from error messages?

Error messages—they could be system messages. They can be user error messages. And just understanding that these messages are popping up and the source of the message will help you again to go in and be proactive. If errors are occurring, and invariably they are, how would you know? Do you have to wait for someone to call into the help desk before you realize that you’re having a problem with errors? Most employees really want to focus on the job at hand. They don’t want to spend time online with the agent—you know, the technician helping them. So, you know, I think by the time they’re calling the help desk, they’ve reach a level of frustration. And obviously that’s something you want to avoid. What we can do is we can identify the errors. We can identify the errors by the software module. We can identify the errors by geography. And organizations can be very proactive in rectifying those errors and ensuring a better user experience.

How do you define engagement and then how do you measure it? 

Using the software. Companies are surprised when they start looking at it. They should be engaged with that software throughout the day and what’s going on here. And you’ll find that employees will perhaps use other applications on the side. For example, they’re in finance and they’re just frustrated and they’ll go and pop up Excel and use Excel to complete a task. Well the challenge is Excel is perhaps not available to other employees who are part of this transaction or whatever workflow the employees engage with. So you want to make sure that the employees are actually using the software. What’s really interesting with engagement is companies invest a lot of resource in upgrades for digital transformation. So they’re migrating from on-premise to the cloud. And you can appreciate these are quite significant projects in terms of resources. They cost a lot of money. There’s a lot of investment. And the question is, post-migration, what benefits are we seeing from these new versions, right? So are my employees using the software more after the migration than before the migration? How would you know that?

It’s so important to utilize enterprise software. If you buy Salesforce but you’re only using Salesforce in a minimal way, you might as well go to some cheapo CRM. Or, if you’re not using certain productivity features or anything, you’re missing out on this huge expense. The second thing is I would love to dig in and see what the difference is in productivity from an on-premise system that has a PC sort of fat client interface where everything’s really snappy and then you go to the cloud version where it’s browser base. I work on both types of software all day long and the cloud stuff just isn’t as snappy or as quick. The keystrokes don’t respond as fast. The user interface is behind some of the things. Many of the things you can do in a Microsoft Word fat client app, you still can’t do even on Office 365 or on Google Docs. And then the time it takes you is longer. But moving on to our next questions, what do you do with all this insight? Prior to our interview we came up with five categories. One was usability improvements—how do you make the software better to use? Another is robotic process automation analysis. Where are there opportunities to do some automation? Another was employee satisfaction. The fourth we just talked about—evaluating the impact of upgrades. And then finally, understanding where you need to do training. Could you talk a little bit about these five ways that getting this data about how your users are using the software can help? 

Companies migrating to the cloud, in our industry, is a priority. And the vendors are prioritizing it and our customers are prioritizing that. The question is, as you said, does the software work as efficiently, as intuitively in the cloud as it did on prem? And again, I come back to the same question—how do you know? IT will roll out a new user interface. Beautiful graphics, beautiful colors. Isn’t this pretty? Well, yes. It looks nice, but is it intuitive? Does it make the software easier to use? That’s the beauty of what our software does. We give you that before and after snapshot. You roll out the new user interface and I’ll tell you immediately whether you’re getting the improvement in usability. Are your employees more engaged? Are they completing the transactions more quickly? Do we have fewer errors? All these metrics meld together to give you a full story of user experience. So what is user experience? I want it to be intuitive. I want as few steps as possible. I don’t want to see any errors. 

From my perspective, robotic process automation or RPA covers this huge range of software. At the lowest end, it’s like keyboard macros you can use to put on top of a system so that in one click, you can do a whole bunch of stuff in a software interface. While a lot of robotic process automation amounts to that, none of the vendors would say that that’s what they do. But practically, that is what they do. Now at a higher level, they do much more complicated stuff where they’re actually intermediating between the internal user or the external user—and then controlling large vast systems underneath and orchestrating large processes so that it makes it much simpler either for the customer or for the end user. That’s what all of them would say they really do. But how does your software help figure that out? 

We provide complete visibility into the front-end processes and the tasks that lend themselves to automation. So our user analytics can be used to identify multiple automation targets. For example, the range of applications and transactions that are used in executing the most common tasks which can be linked through automation. Tasks that are prone to human errors that can be eliminated through automation. And routine tasks—what I would call it low degree of variability where human effort can be entirely offloaded to the robots. We measure the frequency of these robot executed tasks. We compare the productivity of human-driven transactions to the robotic counterparts. We monitor the robots as we do the humans so we can detect errors—error conditions that are encountered by the robots and collect relevant diagnostics. And continually monitor the robot-UI interactions. You’re never going to fully automate these transactions. The question is which ones really lend themselves to automation because they’re more simplistic? 

Now employee satisfaction was the next one.

We touched on that with a few of these examples. Employees are satisfied when they’re productive. When they’re engaged. When they feel a sense of accomplishment and success. If they’re continually frustrated, if they can’t get the job done, if they have to continually go and ask the person in the next cubicle, that impacts people’s satisfaction and the fulfillment in their job. Pretty straightforward. And again, the key factor is they’re using these software systems for hours a day. People can be frustrated very easily. Compound that with the situation with the pandemic where you don’t have access as readily to support resources, it really is an issue. 

The evaluating the impact of upgrades is like a subcase of usability improvements, but you do basically a before or after monitor of the upgrades.

It’s during also, for example, user acceptance testing. These upgrades are quite comprehensive. These are big projects. You want to be able to analyze along the way. You don’t want to have to wait until you do a $10 million, $20 million dollar upgrade for 60,000 employees and, at the end, to find out that, “Wow, this wasn’t designed properly.” With KNOA, you can test throughout this process. Was the upgrade successful? Are transactions being completed more quickly? Fewer errors, fewer steps. But to get to that point, it’s a quite intensive process throughout the migration.

That leads perfectly to the next topic, as one of the things you might find in an upgrade is that maybe a system that is not being used is not being used because people need more training. I’m assuming you can tell where the difference between a user experience where the software is hopeless and a user experience where the people just don’t know where to click.

Right. The last thing you want to do is put people in five-day training classes. Their eyes glaze over and it’s hard to focus and concentrate for anyone, and then compound that with doing it remotely. If we’re designing training, I could tell you which components of the software employees are using so you want to focus on those components. I could tell you where users are struggling. Where they’re abandoning transactions. Where they’re getting the most errors. Where it’s taking an inordinate amount of time to complete a transaction. That’s what you prioritize. So you can be very focused with the training and put somebody in a two hour training class that’s focused on exactly where they’re seeing some challenges. You run the software post-training and see if the training had an impact. And if the training didn’t have an impact, okay, now let’s perhaps revisit the design. This isn’t a one-time analysis that you’re doing. It’s continuous. When you improve the experience, you continually improve efficiencies within your organization. We’ve worked with a third party to come up with some very detailed return on investment of the software. Companies receive a payback in a matter of months. The productivity increase goes right to the bottom line in terms of the efficiency of the organization, of your employees, customer satisfaction. By the way, you have customers which are using these enterprise systems. A lot of customers will use this self-serve software. They want to go in and they want to check availability of a product. They want to check the status of an order. They want to look at technical information. What’s their experience like? The last thing you want is one of your customers to come in and get error messages when they’re trying to get technical spec of a product or to have to click around. 

Let’s now wrap up by talking about the window that you have on the big picture. From our conversation, it’s clear that you have delved into many different aspects of the enterprise software that we really work with day in and day out. What do you think the post-COVID workplace is going to look like?

I think it’s obviously going to be a hybrid environment. I think the industries overall have demonstrated that people can work productively remotely. I think they’ve also pointed out that there are inherent challenges in that. The analysis will continue. Initially, you know, I read articles that companies were surprised at how effective people were working remotely. And then, a few months after that, I saw articles that said, “Wait a minute, there are some real challenges here, and everyone got real focused early on,” but there was something certainly missing in terms of collaboration and having people together. The hybrid environment will focus organizations on understanding how people use the corporate resources. Working on the site versus working from home—does the software work the same? Can I start a project in the office and then work from home for two days and pick up where I left off? Will the software work and be as responsive? There are many important decisions that will have to be made here. For example, which employees do I bring back to the office first? Perhaps you want to bring back the people who are having the greatest challenge working remotely. But wait a minute. You can’t just bring back everyone who’s struggling. Who’s going to support them? So perhaps you want to pair them up with what we’ll call super users—people who have been involved with the software for years, are pros at it, perhaps have been on some of the committees that were involved with the migration projects—perhaps people who are trainers—and pair them with the struggling users.

The idea is that you think with the post-COVID workplace, there will be a permanent shift to remote work and it will be a bigger percentage of things than it was before this period.

I think by definition it will be a bigger percentage. Certainly you’ve seen companies allowing employees to move. Obviously, you’re hearing a lot of stories about Silicon Valley where companies are moving out. We’re allowing employees to move to more affordable locations. Certainly it’s going to be a challenge having those people in an office on a regular basis. So the hybrid model, I think, we’ll see that moving forward. The question is over time, how that will look. There are a lot of benefits in having people work together in terms of collaboration and ideas. One factor that we didn’t really discuss is the whole challenge of managing people. When people worked in the office—if you have 10, 15 direct reports, you’ll have a team meeting and you can look at people and body language and you can just see how they interact. And, you know, someone looks a little off or distracted, you can call them aside at the end of the meeting and have a cup of coffee and see how they’re doing. How does that work when people are remote? And if you have a large number of people who report to you, how do you know who to engage with? So you could see who’s struggling and perhaps focus on those people.

Managing is going to become much more data focused. That brings up the second point that you wanted to make, which was that because there’s probably going to be more data that we can get based on remote working and other ways of collecting data, is it possible that we’re going to have more automation in jobs right now that we don’t expect to have? 

There will be a hybrid of office and remote and also a hybrid of human and robot work, which we touched on earlier when we talked about RPA. There’s a huge industry now of these companies that are deploying these robots. And there’s tremendous value in that. And so, you’ll have the bots taking over the more mundane transactions that require little human intervention and they’ll be paired up with humans. We’re seeing a lot of that already. 

Do you see an era in which even as part of using enterprise software, we use more like an Amazon Echo or a Siri approach to it?

I think certain transactions lend themselves to that. Certainly voice-activated makes a lot of sense. You see that certainly on the shop floor and other places. So 100%, yes.

Thank you. I really appreciate your spending the time here on the Designing Enterprise Platforms podcast of Early Adopter Research, and I hope to talk to you in the future after we’ve distilled this and thought about it a little bit more.

KNOA

Author: Dan Woods

My mission: Find technology for Early Adopters. Follow me on Twitter @danwoodsearly and on LinkedIn www.linkedin.com/in/danwoodsearly/. I am a CTO, writer, and consultant. For tech vendors, I help explain their technology. For users, I help find, select, and deploy new solutions that have explosive business value. I love to speak and share ideas.