Data Cleansing: Maximising the value of your data investments

Over time, integrated energy organisations generate and collect vast and expanding quantities of data. This data is collected in various formats, generated, shared, uploaded, and duplicated across multiple databases, and inherently involves the input of multiple stakeholders. ​Effectively and efficiently managing this data is both complex and challenging, especially in today’s fast-paced environment.

While energy organisations rely on complete, accurate, and accessible digital data to make timely and safety-focused decisions, knowing where to start and what it takes to cleanse and accurately manage this data can be overwhelming.

In this article, Tina Roberts, our Data Services Lead explores the key data challenges facing energy industry organisations, the steps to undertake to ensure good data preparation, and E&P Consulting’s approach to ensuring data is effectively cleansed and efficiently managed for a successful project outcome.

The digital data challenge

At the heart of any business is the ability to identify opportunities, implement changes, and unlock value. Energy companies need to be able to capture and use increasingly vast quantities of both structured and unstructured data from a wide range of sources. This often poses a real challenge for energy companies operating on a global scale.

When engaging in a project, digital data issues typically arise in five core areas: data deduplication, data analysis, data standardisation, data normalisation, and quality checks.

• Data Deduplication: The energy industry generates massive volumes of data from exploration, production, refining, and distribution activities from which companies can derive real business value and make better business decisions. However, when an enormous amount of structured and unstructured data is generated in real-time, things can quickly get complicated. It’s crucial to overcome the challenges related to the adoption of data and analytics, as it can potentially lead to new business opportunities, enhanced efficiency, and significant savings. 

• Data Analysis: Vast and expanding quantities of data are acquired over time in disparate formats, distributed, loaded, and duplicated repeatedly among different databases. Information may be incomplete, inaccurate, or uncertain, and access to critical project data is too slow for today’s fast-paced decisions. With technology advancing at speed, the temptation is to keep populating new data stores without considering their accuracy state.

• Data Standardisation: The oil and gas industry is subject to strict and ever-increasing regulatory requirements regarding safety, environmental protection, and data reporting. For example, regulations such as the Environmental Protection Agency’s (EPA) Clean Air Act Amendments and the Occupational Safety and Health Administration’s (OSHA) Process Safety Management (PSM) require accurate and timely data reporting. Effective data management is crucial to ensure compliance with these regulations and avoid fines and penalties.

• Data Normalisation: In the current environment of digital transformation, data management technology is changing faster than ever. If the industry is to wholly embrace platforms such as OSDU, a potentially complete solution for efficiently storing, managing, and publishing raw and edited data in one integrated system, then we need to ensure that we get the best possible advantage of what it offers by guaranteeing that the ingested data is complete, accurate, reliable, and accessible.

• Quality Checks: The primary objective of a data management team is to ensure that data assets align with business needs and user requirements. Basing business decisions on actual, tangible data brings many benefits, including the ability to spot trends, challenges, and opportunities before your competition. The most important question is, "How to set up your organisation's data team structure with strategic agility to reap the benefits of these data-informed decisions?"

As oil and gas companies evolve and expand into new alternative energy sectors, the value of legacy and current data is of the upmost importance. The industry has become reliant on digital domain expertise and agile product teams to help plan and implement better data solutions.

With proven expertise in delivering high-quality, interpretation-ready data, our team of consultants can deliver effective data solutions to free data from silos and enable new cross-functional capability.

Why data accuracy is important​

In a complex and challenging industry such as energy, limited knowledge in data management may result in an organisation struggling with below-average data quality. This can impede strategic planning, business functions, and the efficiency of BI, reporting, and data science tools that support decision-making.

This can manifest in various ways, such as errors, inconsistencies, and other data-related problems that hinder operations and analytics outcomes. Moreover, data silos that are isolated hinder users in various departments from obtaining relevant information, making it difficult to extract the necessary business value from datasets.

In addition, exploration and production activities are inherently high-risk and rely on high-quality and precise data to not only guarantee compliance with safety measures, but to reduce the likelihood of accidents and lessens environmental harm.  Ensuring data accuracy is, therefore, paramount in safety-critical decision-making.

To mitigate these common digital data issues, data cleaning must be undertaken regularly. Clean data that is more accurate and organised will then be more reliable and can be stored more effectively and securely.

Key steps to achieving good data preparation

Whether participating in the OSDU ecosystem, migrating to a new storage solution, merging datasets, or embarking on a data clean-up project, the data preparation steps are the same.

Step One: Conducting a Data Inventory

The first step is to conduct a data inventory. This will enable your organisation to gain insights into user interactions, utilisation, and cooperation with the selected system(s), thereby initiating the recognition of pertinent data quality and data management challenges, issues, and factors.

 Step Two: Developing a Data Strategy

Next, a data strategy that aligns with these objectives should be formulated. As the organisation progresses, it is essential to evaluate the team structure and composition periodically to ensure its effectiveness and efficiency. 

To achieve this, specific metrics should be defined to measure the effectiveness of the team model composition accurately, and any necessary adjustments should be made based on these metrics.

 Step Three: Building a Skilled Data Management Squad

A third step that is often underestimated is the role of people. Accurate and reliable data can only truly be achieved by resourcing your data cleansing project with the right people and with the right expertise to ensure your data delivers the right results and real return for your business.

By following these key steps and appointing a solid data management squad, a company can be sure that they are making the most of their data and utilising their resources as efficiently and effectively as possible.

E&P’s approach to data cleansing

When working with integrated energy organisations, E&P follows the key steps for optimising data cleansing, but we also put people at the heart of what we do at every step. By understanding the important role of people and domain expertise in this process, we can confidently deliver relevant data management solutions, ensuring that data quality milestones are fully realised.

We work closely with stakeholders to first define scope and understand the current state of the data, we can then more effectively support the data clean-up based on findings and insights.

Through this approach, a comprehensive consulting Subsurface Data Management solution is provided utilising an experienced and dedicated group of geoscientists and data managers. By appointing the best possible team with the best technical and communication skills, we ensure a solid outcome for data management projects. ​

Having effective communicators who can articulate clearly across all technical levels within the organisation is also key. Providing clear, well-defined processes and documentation will ensure ongoing data maintenance is achieved as the project evolves. The deployment of experienced, skilled subsurface petroleum data managers and geo-technologists in this area is vital to this data quality process.​

Trust and transparency are also important factors in successful outcomes. When working with our clients, we ensure that technical knowledge, process knowledge, and guidelines are passed on from our consultant to the internal teams to improve in-house data management skills and to foster strong working relationships across the team. ​

E&P case study: Building best in class data management squads

Creating an effective data management team involves more than just advanced technology and streamlined processes; it necessitates the collaboration of skilled data subject matter experts who possess a diverse skill set to meet organisational objectives.

E&P Consultancy assists organisations in resolving data cleansing and management issues by putting together proficient teams to manage and enhance their data quality.

The role of the Product Owner

• Engage with the client/business community to understand their vision and to define requirements. ​

• Define a scope of work and agree on the solution based on business requirements. ​

• Recognise what solution works for the desired outcome. ​

• Advise and help you get the most out of your data and help you understand what seamless data stores like OSDU can offer. 

The role of the Geodata Managers

• Strong domain background, ensuring that current and legacy data are identified for review. 

• Utilise powerful scripting skills and visual tools to efficiently identify errors and accommodate complex data formats.

The role of the Geoscientist and Engineers

• Providing integrated Geoscience and/or Engineering skills—identifying erroneous errors in data and providing data validation correction and standardisation.

• Reduce the need for resources from the business.

In this case study example, we demonstrate how E&P Consulting provided the services of a data management squad to one of our clients. The project deliverables included:

• Improve well engineering data completeness

• Automate processes to drive efficiency

• Build API solutions to integrate with other applications

• Effectively migrate data to a cloud-based platform

At E&P, we focus on building or enhancing data management teams, and as the Data Services Lead at E&P, I understand the various critical factors needed for a squad to work effectively together. This includes the required roles, team composition, the ideal timing for expansion, leadership choices, and the alignment of responsibilities with business goals.

As a demonstration of this, we were tasked with assembling a team of data managers to refine well data for an energy major organisation. To meet the project demands, we utilised our extensive network of talented digital professionals to create a specialised team that comprised a Senior Well Engineering Data SME, Well Engineering Data Operational Support, Well Engineering Data Entry and Cleansing, Well Engineering SME Data Specialist, and Well Engineering Data Engineer.

By carefully selecting team members with the right technical expertise and communication skills, we ensured the success of this data management project with significant improvement in data accuracy and availability of trusted data as endorsed by our stakeholders. Most importantly, the team members were able to hit the ground running and integrate seamlessly into the existing team with each digital professional coming with direct domain experience.

Why choose E&P for your next data cleansing project?

The energy business relies on data managers to efficiently handle data, whether integrated into a data management platform, accessible in the cloud, or in open-source formats. Managing this data effectively requires a combination of software tools and best practices – but also skilled people.

As global providers of digital domain expertise for the energy industry, we work closely with our clients to combine technology and skilled people to ensure energy organisations achieve effective and efficient data cleansing and management.

We unequivocally believe that well-executed delivery and clear-thinking result in a satisfied client who considers us as trusted partners to engage for their digital data projects. Through working with E&P to execute a data clean-up project end-to-end, you will see the value in making impactful improvements to the quality and reputability of your data. A solid understanding of the business, deep domain knowledge, and experience will maximise the value of your data investments.

Get in touch with us today to find out more about how E&P Consulting successfully combines people, processes, and technology to enable energy industry organisations to mitigate data cleansing challenges and access the best digital domain expertise in the industry to achieve project success.

__

Author:

Tina Roberts - Data Services Lead, E&P Consulting

Tina Roberts is responsible for leading the Data Services team within E&P Consulting’s Oil and Gas business, and for developing and promoting E&P services to clients in these areas.