Successfully implementing artificial intelligence (AI) in your business requires both expertise in data engineering and analytical acumen. Too often, management teams assume they can leapfrog best practices for data analytics by directly adopting advanced technologies, such as machine learning (ML) — consequently setting themselves up for failure from the get-go.
Enterprises are focusing on innovation to stay competitive by driving digital transformation: the adoption of digital technologies to reinvent business processes and customer experiences to achieve more agility and improved KPIs. Based on my experience in data engineering, here are some best practices for how to implement a solid AI strategy that addresses the question: What is the use of AI in data and data warehousing, infrastructure and storage, as well as business applications?
Innovation is built on solid foundations.
In order to build a culture that embraces data and AI, your first step is creating a stable foundation, a core requirement to enable innovation across your organization. A strong foundation for fluid data access includes the following capabilities:
• Scalable data service to expose all data.
• Centralized policy-based governance and security infrastructure.
• Effective methods to gather all the metadata and make it discoverable.
• Kubernetes and the cloud for elastic disposition of resources.
Scalable, self-service unlocks AI agility.
If you build a strong foundation and provide the right level of well-curated data, and you don’t move data as an architectural pattern, your organization will be in a great place to provide real, scalable self-service data.
Real self-service, in the words of Mao Zedong, means you need to “let a hundred flowers bloom.” Many people using data unencumbered will create opportunities for new products, services, customer touch points, internal processes and, essentially, everything that can be improved with data will be improved.
More experiments and more engagement with data in a safe and governed way will give way to lots of failures — and, most importantly, some successes. In any science-based endeavor, failure is considered to be an “essential prerequisite” for success, so improving the cycle time means more failures and thus the occasional breakthrough.
Consider ‘invisible’ AI.
If you’ve moved from an Android to an iPhone, you have seen that Google’s AI is world-class, and its application touches consumers in ways most don’t even realize. It is not just speech recognition and predictive text capabilities that are so impressive, but also its digital assistant and in-app predictive features that are so seamless it’s easy to forget that it is AI powering these capabilities.
If you aren’t part of the “G-MAFIA” (Google, Microsoft, Amazon, Facebook, Intel, Apple), you may be able to use existing technology — and there is nothing wrong with that. Similar to basic science research, you should consume the output of larger organizations — that is to say, stand on the shoulders of giants. AI as a Service and integrated AI offerings like BQML can alleviate this pain point.
The point being, delivering AI to your consumers does not have to be an in-your-face overt feature; digital transformation is mostly evolution with a bit of revolution sprinkled in.
Diversity yields better AI results.
A McKinsey study shows that diverse teams achieve better outcomes than homogenous ones. Companies that embrace diversity have a 33% greater probability of achieving above-average returns.
Tech and data science have historically been homogeneous industries. This is a problem when data scientists only represent one way of thinking, so they are likely to bring those unintended biases into the collection of data and algorithms used in applications. In addition, it’s a possibility that collected data already represents bias that was applied even before software and algorithms were involved. Simply put, if the data you are using has biases baked in, the machine will learn these same biases.
The following is an example of the type of problem that using data with a bias can have on your model. If you use data that is based on women being underrepresented in the workforce to power your algorithm, the result will be statistical bias. To add balance, equal yet opposite bias can be introduced into the model, balancing gender representation in historic data only if the person preparing the data or algorithms recognizes the preexisting bias. However, in many cases, it can be difficult to be both unbiased and fair.
Consumers and investors need to check claims.
Almost every tech company in the Valley says it is an AI company. A lot of these companies are putting out marketing lipstick on a Mechanical Turk pig, or at best a simple linear regression, which is technically ML. If you can swap in “analytics” for “AI” in a company’s marketing materials, the company is probably not using AI.
It’s difficult to understand technology as complex and mathematical as AI. Stakeholders and consumers alike should pay extra-close attention to claims of computer-based intellect to determine if it’s actually innovating or a sham. If entrepreneurs are legitimately focused on including AI in their company, they should “carefully consider what problems they are trying to solve and how and why they might benefit from AI-enabled solutions before blindly jumping into AI.”
If you are committed to digital transformation driven by data and ML, the technologies you choose should be aligned with that goal. Choose carefully, and evaluate the technologies in conjunction with your data science teammates.
In conclusion, success with AI includes a holistic approach that considers all elements in play, which, for a typical enterprise, are people, process and technology. Concerns for the efficacy and ethics of AI-driven product and service offerings are numerous. No one should be surprised that the actual engineering bits are not necessarily the biggest hurdles to delivering the value you are creating to the marketplace.
