{"id":451,"date":"2024-05-19T18:31:35","date_gmt":"2024-05-19T18:31:35","guid":{"rendered":"https:\/\/omaintec.org\/journal\/?post_type=docs&p=451"},"modified":"2024-05-24T22:58:58","modified_gmt":"2024-05-24T22:58:58","password":"","slug":"certification-or-maturity-mentoring-audit-driving-the-asset-management-journey","status":"publish","type":"docs","link":"https:\/\/omaintec.org\/journal\/index.php\/docs\/certification-or-maturity-mentoring-audit-driving-the-asset-management-journey\/","title":{"rendered":"CERTIFICATION OR MATURITY MENTORING AUDIT – DRIVING THE ASSET MANAGEMENT JOURNEY"},"content":{"rendered":"

Authors: CMDR Linda Morris, RAN, CAAM Deputy Director, Maritime System Program Office, Governance<\/em><\/p>\n

Alison Court, CFAM, CAMA Manager, Asset Performance Group, Industry Service Provider<\/em><\/p>\n

Jim Kennedy, CPEng, CFAM, CAMA; Asset Management Specialist, Maritime System Program Office, Governance<\/em><\/p>\n

Abstract<\/strong><\/h2>\n

For many organisations around the globe, ISO 55001 certification has been a strategic component of their Asset Management journey. However, not all organisations consider formal third party certification to be necessary, feasible or in fact desirable for that journey. Other ways of achieving the growth required in asset management capability are available, and for many organisations, may be a more productive journey than ISO Certification alone. This paper will describe the why and how of an Asset Management maturity assessment that drove the improving performance of a support organisation for maritime assets. The role and content of a \u201cmentoring audit\u201d of the Asset Management maturity of the organisation will be described along with the outcomes and lessons learned during the two year journey. Comparisons of the results achieved will be made with another similar program using frameworks drawn from national and international \u201cnot for profit bodies\u201d.<\/p>\n

Keywords: Asset Management, 55001, Certification, Maturity, Audits<\/strong><\/p>\n

1 INTRODUCTION<\/strong><\/h2>\n

ISO Certification is understood by many as demonstration of an organisation\u2019s management system capacity to achieve a particular set of outcomes. These outcomes are often enunciated in the form of principles such as those that underpin Asset Management. These principles become the basis of a formal management system against which organisations may test themselves. This test is like an athlete in a high jump that is concerned only that the qualifying bar was cleared, not with how much the bar was cleared by.<\/p>\n

Without a structured management system an organisation cannot claim an asset management capability. Without a sufficiently mature management system for asset management, the organisation cannot claim a match between the management system capability and the managed asset related risks that is necessary to achieve the organisation\u2019s objectives. In this context, formal certification to the ISO 55001 standard for a management system for asset management is not a guarantee of required management capability. Certification only establishes a single \u201cgo\/no go\u201d threshold for the listed requirements in the standard.<\/p>\n

The Maritime System Program Office (MSPO) involved does not have asset management accountability for the physical assets serviced. Operations belongs to the Royal Australian Navy. Navy outsources the sustainment function to Capability Acquisition and Sustainment Group who, for a variety of reasons, outsource their core management function to an industry service provider (ISP). Two key goals from the scope of work for the ISP were:<\/p>\n

    \n
  • (1.2.3) The Contractor is required to deliver and manage the Services using an asset management approach based on and consistent with ISO 55001:2014 Asset Management<\/em>.<\/li>\n
  • (1.2.4c) The Objectives are to implement a structured asset management framework to maximise the <\/em>value of the Commonwealth\u2019s assets and in doing so minimise the Total Cost of Ownership;<\/em><\/li>\n<\/ul>\n

    As the program progressed, evidence from assessment of Project plans and related independent audits, indicated that the required Asset Management objectives and expectations were not being achieved. A number of critical Asset Management \u2013 Management System Plans (AMSyPs) such as the Enterprise Management Plan had not been approved. While asset management maturity in the organisation was anecdotally considered low, some formal means to assess the level of maturity was required to determine how low that maturity was in the Management System for Asset Management (AMS). An asset management maturity audit of some form was needed.<\/p>\n

    2 THE PROPOSITION<\/strong><\/h2>\n

    A brief to undertake an Asset Management Audit of the ISP was accepted on 3 March 2019. Three alternate approaches to assessing the level of performance of an MSPO in establishing an asset management system framework were:<\/p>\n

      \n
    • An audit process based on the Maintenance Engineering Society of Australia (MESA) original excellence award in maintenance and asset management between 1996 and 2014, which was based on the Australian Business Quality Awards Process. Development would be quick and there is a successful provenance of providing a reasonable coverage of asset management capability.<\/li>\n
    • An AM maturity assessment using the Asset Management Council maturity tool. This approach uses a large question set (158) to test an organisation\u2019s maturity in asset management. Significant written effort and knowledge of asset management is also required to describe the organisation\u2019s current capability. Many workshops would be required and key staff would need to be made available.<\/li>\n
    • An independent gap analysis conducted by a certifying agency such as SAI Global or Bureau Veritas. Such a formal analysis, often quite basic, is meant to be a gap closure activity followed by a full ISO Certification effort. In the context of MSPO many independent and mandatory certification activity was already underway thus challenging both the value and available resourcing for yet another independent formal audit.<\/li>\n<\/ul>\n

      The brief\u2019s recommendation to adopt Approach 1 above was accepted, with the following key points agreed. The audit would:<\/p>\n

        \n
      • adapt the now obsolete MESA assessment tool from the 1990s which would be upgraded to assess the ISO 55000\/1 essentials.<\/li>\n
      • be conducted by existing internal staff with guidance from the MSPO Asset Management Specialist;<\/li>\n
      • establish a baseline for the current state of AM maturity;\n
          \n
        • gain an understanding of the potential gaps in the ISP asset management capability to allow for future AM capability development programs;<\/li>\n
        • assess the shape and size of options to close, over the next three years, the identified AM capability gaps; and<\/li>\n
        • provide a means of assessing the effectiveness of the AM capability defined in a suite of Asset Management System Plans being the plans for the Management System (AMSyPs).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

          3 DEVELOPING AN AUDIT TOOL<\/strong><\/h2>\n

          As attainment of ISO 55001 Certification was not a stated objective for the MSPO, the priority was process development. A process audit tool was developed using the following list of sources:<\/p>\n

            \n
          • ISO 55001:2014 Management system for asset management;<\/li>\n
          • Australian Quality Awards and Bainbridge Awards assessment criteria;<\/li>\n
          • MESA Excellence Awards process 1996 to 2014;<\/li>\n
          • AM Council Asset Management System Model 2015;<\/li>\n
          • AM Council Capability Delivery Model 2005.<\/li>\n<\/ul>\n

            The views of what maturity in Asset Management might look like for an organisation in delivering a process gradually evolved to reflect a combination of the following attributes:<\/p>\n

              \n
            • Is there a defined process <\/strong>to satisfy the posed questions?<\/li>\n
            • Has that process been implemented <\/strong>and at what level?<\/li>\n
            • Are measures <\/strong>of achievement collected that allow measurement of process success? and<\/li>\n
            • Is this information used to continuously improve <\/strong>those processes?<\/li>\n<\/ul>\n

              The developed audit process comprised:<\/p>\n

                \n
              • 101 questions – see example set at Table 1,<\/li>\n
              • across 20 Topic areas such as \u201cContext Setting\u201d \u2013 See example at Table 2 and full Topic listing at Figure 1,<\/li>\n
              • with each question being assessed against four criteria \u2013 See example at Table 3, and<\/li>\n
              • marked on a scale 0 to 9 in 5 pairs (i.e. 0-1, 2-3, 4-5, 6-7, 8-9) \u2013 See example at Table 3.<\/li>\n<\/ul>\n

                The audit tool was used for both the initial audit and the complete follow on Mentoring Audit. The largest difference between the audits was in the extended involvement of both MSPO Governance and ISP over a 15 month period after the initial audit.<\/p>\n

                The initial audit required speed and a broad understanding of our current position. To achieve this requirement the following approach was adopted:<\/p>\n

                  \n
                • The Auditor was to be certified at CFAM level with AM Council and a Certified Asset Management Auditor (CAMA) with the World Partners in Asset Management.<\/li>\n
                • Only half of the 101 questions were posed in the interviews \u2013 questions were not asked if the general consensus was that a score of zero was most likely.<\/li>\n
                • Only 3 of the 4 assessment criteria would be valued as \u201cImprovement\u201d was generally not possible due to the lack of documented processes to formally improve or measurements and results to provide data for improvement action.<\/li>\n
                • Only 15 of 19 Topic areas were assessed as 4 Topic areas were still at the stage of developing initial processes. These 4 Topics were assessed some 6 months later to provide a complete baseline.<\/li>\n
                • Interviews were conducted one on one by the Auditor with the responsible Lead Manager on the basis that results would not be circulated outside the MSPO. An Observer from the MSPO Commonwealth Governance team was present to assure probity.<\/li>\n
                • All statements were assumed as honestly given (and no OQE was requested or assessed).<\/li>\n
                • All interviews were documented and passed to the Interviewees for a fact check.<\/li>\n
                • Lists of Issues and Opportunities for improvement were noted.<\/li>\n
                • A final score for each audit Topic was allocated by the Interviewer in conjunction with the Observer.<\/li>\n
                • The Auditor produced a report on the outcome for the first15 Topics and a second combined report for all Topics.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n\n\n\n
                  Topic 1<\/strong><\/td>\nContext Setting (7 Questions)<\/strong><\/td>\n<\/tr>\n
                  a<\/strong> b<\/strong><\/p>\n

                  c<\/strong><\/p>\n

                  d<\/strong><\/p>\n

                  e<\/strong><\/p>\n

                  f<\/strong><\/p>\n

                  g<\/strong><\/td>\n

                  		Does the organisation have a defined list of stakeholders?<\/td>\n<\/tr>\n
                  Is there an assessment of their importance?<\/td>\n<\/tr>\n
                  Has an associated stakeholder engagement regime been defined?<\/td>\n<\/tr>\n
                  Is there a list of internal and external issues determined from the stakeholder engagement process?<\/td>\n<\/tr>\n
                  Has the asset management portfolio of assets been identified?<\/td>\n<\/tr>\n
                  Is that asset scope clearly identified in a document under quality management control?<\/td>\n<\/tr>\n
                  Is the scope aligned with the SAMP and AM Policy?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                  Table 1 Example Question Set<\/strong><\/p>\n\n\n\n\n\n
                  Topic 01<\/strong><\/td>\nContext Setting<\/td>\n<\/tr>\n
                  Topic 04<\/strong><\/td>\nAsset Management Objectives<\/td>\n<\/tr>\n
                  Topic 05<\/strong><\/td>\nCulture and Leadership<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n\n
                  Topic 07<\/strong><\/td>\nDecision Making<\/td>\n<\/tr>\n
                  Topic 11<\/strong><\/td>\nResourcing<\/td>\n<\/tr>\n
                  Topic 13<\/strong><\/td>\nOutsourcing<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                  Table 2 \u2013 Example Tranche Topic Set<\/strong><\/p>\n\n\n\n\n\n\n\n\n
                  Score<\/strong><\/td>\nProcess<\/strong><\/td>\nImplementation<\/strong><\/td>\nResults<\/strong><\/td>\nImprovement<\/strong><\/td>\n<\/tr>\n
                  0 -1<\/strong><\/td>\nPractices & processes do not apply ISO 55000\/1 intent or meet a moderate amount of the Process Model<\/td>\nThe process is not documented. There is little or no evidence of any systematic <\/em>achievement of purpose.<\/td>\nAd hoc and\/or poor results achieved.Goals are not identified.<\/td>\nNo improvements have been recorded.Improvement goals are not identified.<\/td>\n<\/tr>\n
                  2-3<\/strong><\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\n<\/tr>\n
                  4-5<\/strong><\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\n<\/tr>\n
                  6-7<\/strong><\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\n<\/tr>\n
                  8-9<\/strong><\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\nDescription here<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                  Page 3<\/p>\n

                  Table 3 \u2013 Example Audit Assessment Criteria Set and Scoring Levels<\/strong><\/p>\n

                  A key characteristic of the scoring matrix with its 5 levels, is the non-linearity of the scores. That exact level of exponential growth between scoring blocks e.g. \u201c0-1\u201d and \u201c1-2\u201d has yet to be determined. Rule of thumb would indicate that each block is twice as hard to achieve change to the next block i.e. from \u201c0-1\u201d to \u201c1-2\u201d versus \u201c1-2\u201d to \u201c3-4\u201d. Thus going from a score of 1 to a score of 6 is 4 times as difficult as from a score of 1 to a score of 3. This non linearity, which creates an increasing cost of compliance as the assessment score rises, requires careful consideration of the value to be achieved by increasing levels of maturity.<\/p>\n

                  4 MATURITY SCORES FROM THE INITIAL 2019 AUDIT<\/strong><\/h2>\n

                  The audit using the process described at Section 3 had some urgency. Accordingly, individual interviews of 9 Team Leads and the Head Manager were conducted between 24 April 2019 and 9 May 2019. The results were documented and circulated for fact checking by each interviewee. All interview outcomes were documented and signed off by the interviewee. The final report of 71 pages, was submitted on 19 July.<\/p>\n

                  The initial Asset Management Maturity Audit of the ISP in 2019, indicated the lack of an Asset Management (management) System (AMS) as required by the contract. The performance produced low scores for virtually all Topics assessed as shown at Figure 1. The impact of the MSPO governance function\u2019s asset management responsibilities on a number of the initial scores was noted in the report and it was observed that the Asset Management outcome is a team effort not just the effort of one party.<\/p>\n

                  As shown at Figure 2 achieving ISO certification would likely require at least an average score of about 4.5 out of a maximum of 9 i.e. a mix of scores between 4 and 5 as a minimum across all criteria . Clearly certification would not be possible with the scores at Figure 1 where the average for each Topic was between 0.5 and 3.5. Additionally, as noted in over a decade of AM Council Excellence Awards Audits using similar criteria, the Criteria scores for immature organisations generally degrade across the quality management Plan, Do, Check and Act cycle. This is also clearly evident from the marks at Figure 2 as the audit progresses from assessing \u201cProcess\u201d then \u201cImplementation\u201d then \u201cResults\u201d. As noted \u201cImprovement\u201d was not measured.<\/p>\n

                  \"\"<\/p>\n

                  Figure 1: Radar Plot for Initial AM Maturity Audit Scores by Topic<\/strong><\/p>\n

                  5 INITIAL AUDIT OUTCOME<\/strong><\/h2>\n

                  Using Figure 1 as the performance source and noting that the scores for each Topic go from 0 to a possible 9, the current Asset Management performance was clearly quite immature. As noted, a score of about 4.5 for all Criteria in all Topics would be indicative of a capacity to achieve a formal ISO certification if desired. The outcome did not satisfy the contract requirements to establish an Asset Management Framework and supporting system that achieved the intent of ISO 55001. Something now needed to be done, and fast.<\/p>\n

                  Based on the assessment outcomes, the MSPO leadership (including the ISP) accepted a program of follow up actions focused on rapidly improving their asset management maturity by:<\/p>\n

                    \n
                  • conducting a progressive \u201cMentoring Audit\u201d with MSPO Asset Management Specialist in the role of Mentor, which is intended to encourage the understanding and adoption of a management system for asset management in line with the principles of ISO 55000 and the requirements of the ISO management system 55001;<\/li>\n
                  • extending the coverage of the audit to the MSPO governance function as well the ISP in a joint interview process\u2019<\/li>\n
                  • encouraging engagement by including the first Tranche of the audit (Six selected Topics) in the contract Milestone payment for the next reporting period (a minimum average score of 3 across all criteria was to be achieved), funded from past unclaimed Asset Management milestone payments; and<\/li>\n
                  • designing a low human resource impact audit program with progressive delivery that would grow capability without adversely effecting MSPO output imperatives.<\/li>\n<\/ul>\n

                    6 THE MENTORING AUDIT 2020\/21<\/strong><\/h2>\n

                    The full Asset Management System maturity assessment comprising three Tranches was constrained to the MSPO which was only accountable for producing plans for the materiel system and contracting those plans out for delivery by other Service Providers. Thus, the \u201cMateriel System\u201d comprising maritime physical assets operated by others outside the MSPO, could not be included in the scope of the assessment. Only those aspects, which relate to the management system being the people, processes and information necessary to satisfy agreements with stakeholders for the development and delivery of the technical and management plans were included. Some of these stakeholder agreements are enforceable contracts such as those with the ISP and any outsourced arrangements from the ISP to others. Other internal providers had agreed Memorandums of Understanding or Agreements that are not enforceable at law..<\/p>\n

                    The initial maturity audit outcomes recommended a number of process changes to create a \u201cMentoring Audit\u201d (MA) approach that was focussed on progressive continuous improvement rather than achieving a particular threshold maturity score. The MA concept was new being a combination of an audit with the key mentoring characteristics of:<\/p>\n

                      \n
                    • Preparation: <\/strong>This required the Lead Auditor to meet with the two assessed parties to set expectations for the completion of a pre-audit template. The template captures succinct responses to the questions to be asked and progress with opportunities identified in the Initial Audit.<\/li>\n
                    • Negotiation: <\/strong>During the Topic Audit session with both Governance and ISP parties present and physically together, discuss each of their roles when answering the listed question. Discussion is to include processes applied, measures made, results achieved and improvements identified and progressed. Objective quality evidence is provided and opportunities for improvement sought.<\/li>\n
                    • Enable Growth: <\/strong>Opportunities are listed for all Topics. This produces a large number of potential improvements which must be filtered for duplication, grouping, priority, and development of project portfolios.<\/li>\n
                    • Provide Closure: <\/strong>Recognise what has been achieved. Create a vision of what comes next in the AM journey to a desired maturity.<\/li>\n<\/ul>\n

                      The improved MA process (incorporating recommendations for improvements to the Initial audit progress) was as follows:<\/p>\n

                        \n
                      • The audit comprises three separate tranches of 6 to 7 Topics about 6 months apart. The 20 Topics (including a 9a and 9b) are depicted in the radar plot at Figure 3.<\/li>\n
                      • The lead responsible person for the ISP and for MSPO Governance would respond jointly in the same room at the same time to each question;<\/li>\n
                      • Both lead persons responding to the Topic pre-audit are to provide a written response to the question set of about one page maximum for each question. This is to prepare all parties for the audit response and provide time for the participants to prepare their evidence and any need for additional elaborating questions;<\/li>\n
                      • The pre-audit response template is the basis of the conduct of the audit, with additional questions asked as necessary and the provision of evidence to support claims of conformance during the audit;<\/li>\n
                      • The Tranche audits are conducted in two groups of three Topics, each at weekly intervals, with each group separated by a two week gap as shown at Figure 3. This intent may and did vary based on the availability of key persons and MSPO events;<\/li>\n
                      • The Audit lead shall create a draft report for each topic audit. The draft is verified by the assisting auditors and fact checked by both CoA and ISP participants; and<\/li>\n
                      • The completed individual reports shall be aggregated into a single Tranche report.<\/li>\n<\/ul>\n

                        To encourage engagement, the first Tranche was linked to an existing AM milestone contractual payment by requiring each of the 6 Topics to achieve an average score of 3 or greater. The remaining Tranches audits were not subject to milestone payments. However, the conduct of Tranche 1 had successfully set the scene for the following Tranches with improved understanding of AM processes and the benefits. Financial reward closely linked to a measurable and achievable outcome achieved focus on the task at hand and the indicated the value assigned to AM maturity by the asset owners\/operators and the responsible MSPO Governance function<\/p>\n

                        The program described at Figure 2 was completed in the period allocated with the first audit interview conducted on the 26 June 2020 being Context Setting and the last audit interview conducted on the 14 April and the Final Draft Report submitted on the 3 June 2021.<\/p>\n

                        <\/p>\n

                        Figure 2: 2020\/21 Asset Management Maturity Audit<\/strong><\/p>\n

                        7 MENTORING AUDIT OUTCOMES<\/strong><\/h2>\n

                        The scores achieved for each Topic and each criteria during the three audits are shown diagrammatically in the radar plot at Figure 3. The average score applies to only the first three Criteria to allow a valid comparison between the 2019 Initial Audit and the 2020\/2021 full audit. The reduction in scores as criteria transition from Process defined to Implemented to Measured to Improved are readily evident. Also evident is the modest score of arguably two of the most important Topics after setting Objectives being Decision Making and Risk Management. Without the robust connective tissue between Organisational Objectives and Asset Management Plans an effective asset management capability will never be achieved.<\/p>\n

                        \"\"<\/p>\n

                        Figure 3: Topic Scores for each Criteria and the Topic Average.<\/strong><\/p>\n

                        The initial 2019 audit identified 84 opportunities. The Mentoring Audit produced another 226 opportunities across three Tranches. Many of these opportunities were duplicates effecting multiple Topics. One specific example was the large number of quality management issues effecting multiple Topics as identified in Tranche 1. A summary of the opportunities by Tranche is provided at Table 4 noting that the numbers shown include some duplicates. This table is slightly misleading as Tranche 1 created a set of noted opportunities (100+) that would overwhelm the system both in numbers and our ability to aggregate into a portfolio of projects of repeated in Tranches 2 and 3. Some self-regulation during the subsequent Tranche audits kept the opportunities in Tranches 2 and 3 to manageable bounds.<\/p>\n\n\n\n\n\n\n\n\n
                        Initial Maturity Audit (Issues and Opport)<\/td>\n84<\/strong><\/td>\n<\/tr>\n
                        Tranche 1 Opportunities<\/td>\n101<\/strong><\/td>\n<\/tr>\n
                        Tranche 2 Opportunities<\/td>\n73<\/strong><\/td>\n<\/tr>\n
                        Tranche 3 Opportunities<\/td>\n52<\/strong><\/td>\n<\/tr>\n
                        Tranche Total<\/td>\n226<\/strong><\/td>\n<\/tr>\n
                        Grand Total<\/strong>\u00a0 \u00a0 \u00a0 \u00a0 310<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                        Table 4: Identified Opportunities for Improvement by Tranche<\/strong><\/p>\n

                        Based on the Topic scores and the overall improvements achieved by each Tranche listed at Table 5, the following interpretations for the differences are advanced:<\/p>\n