The AQSCI Method behind CRO Evaluation and Application to
Transcript Of The AQSCI Method behind CRO Evaluation and Application to
PhUSE 2015
Paper PD03
The AQSCI Method behind CRO Evaluation and Application to Pharmaceutical Company Needs
Roy Ward, PPD, Alsace, France James Gallagher, Novartis, Basel, Switzerland
ABSTRACT The high-risk nature of drug development increasingly continues to be a more demanding and costly
adventure. This has led to continuous scrutiny by the financial teams in searching for the correct strategy to maximise return on investment and enable internal teams to focus on value added areas.
This paper will focus on the strategy of outsourcing and discuss a frequently used method in the industry for CRO evaluation and selection called AQSCI. A discussion of how this method is used to produce a quantitative set of statistics which can measure the CRO company strengths and weaknesses will be considered. The two authors both have experience of working in pharmaceutical companies and in the CRO world. They will share personal opinions of what todays companies are looking for in a CRO partner using this method.
INTRODUCTION
The AQSCI evaluation method stands for the following a) Assurance of Supply b) Quality c) Service d) Cost e) Innovation.
The layout for the remainder of this paper is detailed in Table 1.
Table 1: Table of Contents
Section Abstract Introduction Motives for Outsourcing Types of Outsourcing The AQSCI Method of Sponsor selection
Control Techniques
Conclusion. Contact Information
Purpose High Level Overview of the goals of the paper Introduction and structure of the paper. Discussion of why sponsor companies outsource. Outsourcing Models experienced by the authors. Overview of the factors to consider when evaluating a sponsor and also applying this to a scorecard. Processes/considerations which facilitate a successful delivery within the engagement
The views expressed within this paper are entirely that of the authors and do not necessarily reflect the views of PPD or Novartis.
1
PhUSE 2015
MOTIVES FOR OUTSOURCING.
The motives for biometrics departments to outsource statistical reporting activities within pharmaceutical companies have a high degree of variability. Some popular key factors are briefly discussed below. Resource Limitations:
Ultimately, this factor is a key consideration when demand outstrips capacity. Within the dynamic pharmaceutical field, this can occur for various reasons. However, one constraint is that teams and in particular, statistical reporting teams need to have the flexibility to outsource to suitably meet sudden changes in demand.
Cost:
As with all industries, the pharmaceutical companies are trying to reduce costs and outsourcing can play an important role here.
Expertise:
Some organisations can offer expertise in niche areas to statistical reporting groups. Some examples, out of many, include complex statistical methodology, data transparency solutions and the technical implementation to generate CRT’s.
In the author’s experience, it is rare for a pharmaceutical company to exclusively base their decision on one of the above. Typically, there is a combination of factors considered.
TYPES OF OUTSOURCING EXPERIENCED BY THE AUTHORS
Four popular models are discussed below and also a newer model the authors have recently experienced that they are calling the ‘hybrid’ model. FULL SERVICE PRODUCT
This is sometimes called end-to-end outsourcing and unlike the other models discussed below the Biometrics organization is not the sole decision maker in determining the specifics related to an outsourced package. FUNCTIONAL OUTSOURCING (ALSO KNOWN AS FUNCTIONAL SERVICE PROVIDER)
This is where a particular function outsources either all of or part of their component in the broader development chain. For Biometrics organizations this is currently a very popular method. In simple terms, it can be considered as a ‘work-package’ approach.
FULL TIME EQUIVALENT
The Full Time Equivalent model relates to either on-site or off-site “contractors”. Within the industry there is a broad range of responsibilities assigned to contractors. Again, in the author’s experience, the contractors typically act as augmented resource and are integrated with the project team to facilitate the delivery of projects.
HYBRID MODEL
The so-called Hybrid Model is a combination of the above two models namely functional outsourcing and full-time equivalents. It commences as a FTE model and progresses to a work-package model as the project matures. The concept is that FTEs are integrated into the team and immersed in the reporting environment and project standards. As the project matures and new studies commence a suitably skilled FTE is identified and leads a team of his/her colleagues on a package of work but acts as a key interface to the internal team throughout the lifecycle.
2
PhUSE 2015
THE AQSCI METHOD
The AQSCI method is built on five pillars as described below.
ASSURANCE OF SUPPLY
This pillar deals with the topic of supply of resource.
Location of staff.
The location of staff within the CRO is a key consideration for pharmaceutical companies. This brings benefits related to overlapping time zone coverage, costs, local knowledge (e.g. specific regulatory authority expertise) and global scalability.
Quantity of staff.
This is an important indicator related to the magnitude of engagement. If a company is strategically looking to outsource a large package of work then the number of available staff is of high interest. Conversely if a company is looking for niche expertise, then the next metric is of primary interest.
Quality/Education of staff It is important to assess what level of qualifications and experience the company has. For example, how many PhD’s, M.Sc.’s are within the team, average number of years of experience or number of employees with over 10 years of experience? The ratio of staff grades(e.g. junior:senior) should also be considered.
Specific knowledge of staff. If the work being outsourced requires TA knowledge then the CRO should match this experience. Hence, if the package of work being outsourced is from the Oncology sector it is preferential to have this skill embedded within the supplier company. Likewise specialist technical knowledge may be of interest e.g SAS Drug Development.
Turnover rates
All companies wish to have a high level of stability and hence low turnover rates. This avoids re-training costs for the project which are typically absorbed by the vendor. A low turnover rate ensures continuity and mitigates disruption. Questioning what the annual turnover rate is by region and comparing it to internal sponsor rates is required.
Evidence of track record on the company delivering. If a company has consistently delivered for the sponsor company in high quality then this is a good indicator of future trends. In addition this demonstrates that the CRO has experience for the sponsor company.
Evidence of gaining previous awards. Even if the supplier company has little or no experience with the sponsor company, does the company have a track record of being rejected and if so why. This may reveal a problem with competitive rates or a long standing concern teams have with quality.
Experience in the area of phase i, ii, iii an iv, modelling & simulation. The different stages of pharmaceutical development require quite different skills. Phase I is notoriously quick and fast whereas phase III can be very demanding particularly if the study is on a submission critical path. Naturally if the workload being outsourced by the sponsor is phase IV for work example, the supplier should contain phase IV experience
3
PhUSE 2015
QUALITY
This pillar deals with the topic of supply of quality.
Of interest are.
Quality system/processes available What level of quality does the supplier company have in terms of systems and processes? A strict review of SOPs is recommended including a question and answer session with the supplier. This review will highlight any gaps in the processes. Likewise it is imperative that a suitable IT Quality Assurance colleague also contributes to this review.
Health Authority findings. Does the supplier company have a track record of Major findings following a Health Authourity inspection/audit? A key question is what have the supplier company done in order to rectify that finding. This demonstrates whether the company is a learning company or not.
Governance support
It is very important to understand the company’s escalation process and governance structure. Where in the process does the project lead get involved and when do management become informed and expected to act? Is this clear to the team and also is this clear to the sponsor company whom to escalate to. Most relationships have a reporting/governance structure such as an Operating Steering Committee which meets on a regular basis to overview the general running of the projects and review Key Performance Indicators. Executive Steering Committees contain senior members of the relationship and meet less often and discuss any relationship threatening issues and future workload/ strategy of the relationship.
SERVICE
This pillar deals with the topic of supply of service
Of interest are.
Cultural fit Cultural fit is hard to quantify but a general mirroring of expertise, location between the sponsor and supplier is important. Familiarity between the two companies helps the relationship.
Ability to work on host system. Some sponsors wish all of the work to be completed on the sponsor company system. Assuming the technology allows this, suppliers using sponsors systems can save on lengthy data transference discussions and it also allows the results to be seen rapidly and it offers a transparent overview of the status of the biometrical work. Conversely the supplier will require compensation for additional training for its employees to work on a new system rather than the supplier system of which the associates will already be familiar with. Likewise experience of working on the sponsors system may well be of high interest.
Training requirements to get staff up to speed/on-boarding time The time in which a sponsor company needs to get its associates up to speed on a system can vary. This is of importance for costings and also demonstrates the flexibility of the sponsor company to ramp-up the workload if required.
Expertise of a certain system or package e.g SAS, R etc. It is important for the supplier company to mirror certain skills of those at the sponsor. Indeed, as discussed earlier, if the sponsor company is looking to the supplier for expertise it is expected that this skill is better than internal resources.
Experience of outsourcing model being requested(strategic partners). The sponsor company will have a strong idea of the strategic model they are looking to put in place. Even though the supplier company may never have worked with the sponsor company before, the supplier company may have a wealth of experience in this model. Examples are required.
4
PhUSE 2015
Likely overhead costs in terms of internal oversight. How much consultation and support will the supplier require from the sponsor? Ideally the internal oversight is very low as this is the work being outsourced. However internal costs and oversight can mount up resulting in high sponsor costs and supplier dependency of the sponsor.
Management Support & Oversight How much involvement will the supplier’s management give?
COST
This pillar deals with the topic of cost.
Of interest are.
Cost structure competitive. The supplier company may well be top in quality and have an excellent delivery however this may come at a price which may not be competitive to peer suppliers. A financial analyst from within the sponsor company is required to give an opinion on this topic.
Cost structure transparent Is it clear where the final quoted costs are coming from? Where multi-services are involved the sponsor would at least wish to know how much each service is contributing to the final cost. Ideally, a breakdown of the costs for each services component can be supplied. Ideally, this detailed costing can then be used to compare component rates against each of the supplier’s rates.
How Out of Scope costs are dealt with. A “bone of contention” for both sides of the relationship. It is important that this is discussed before the relationship is agreed. It is inevitable that new challenges and changes will arise. How these are dealt with resource wise then becomes a company decision which can be very different between sponsor and supplier.
The financial stability of the company. Many of us who have been in the industry for a while have experienced working with a strong CRO who in 4 to 5 years’ time are no longer in existence or have been bought by a competitor. It is wise to assess and predict the potential outcome of the supplier. Worst case scenario is that the supplier becomes nonfunctional during work execution.
Financial Models Available. How amenable are the supplier to work with financial models which include penalties or rewards?
INNOVATION
This pillar deals with the topic of innovation, of interest are.
Depth of statistical scientist group/innovation of study design. The sponsor may well be looking for niche skills in the supplier such a statistical expertise. Evidence of a scientific group is important to highlight this skills as is attendance and contribution to relevant subject matter conferences.
Demonstration of productivity/process improvements It is important that the supplier demonstrates not only “can do now” skills but also “can do in the future”. This demonstrates that the supplier is moving with the times and has not become complacent with the present.
Demonstration of Technology solutions What new techniques or tools is the sponsor company using? A demonstration of new ideas likely means that the supplier will have ideas to solve your future problems.
Demonstration of Lessons Learned.
5
PhUSE 2015
It is part of human nature and progress that mistakes will occur. It is important that these mistakes are used positively to demonstrate a learning culture of the supplier. So from a negative situation the supplier can demonstrate a positive outcome by demonstrating what has been put in place to prevent that from happening should the same conditions exist in the future. A demonstration of a lessons learned forum with tangible implementations would show this.
Areas of specific company interest e.g signal detection. A sponsor may be looking for a niche skill in which the sponsor has yet to acquire, for example skills in safety signal detection or data transparency.
Visibility in external groups e.g PHUSE! To demonstrate the company is a learning organization it is important that the supplier is visible and contributes to the wider community. Organisations such as PHUSE are ideal grounds in which to assess the level of external engagement and it is an opportunity for them to showcase expertise.
Quantitative Measure
To obtain a quantitative measure the methodology is as follows:
For each pillar, key questions are asked. These are related to relevance to each CRO, hence these questions could be different related to CRO. For example if geographical location on the west coast of America is of importance then the question asked is
Do you have staff in the west coast of America?
A numerical score is given on a constant scale e.g 1-5(where 1 is low and 5 is high)
If the response is “yes we have offices in Chile”, this response is reasonably positive since Chile is in the time-zone of the west coast of America but it is not a perfect match, hence the score for this question would 3. If the answer was “Los Angeles” the score would be 5. If the answer is “Europe only” the score would be one.
Question Do you have staff in the west coast of
America?
Answer Los Angeles
Chile Europe only
Score 5 3 1
Should you also have an interest in staff for Germany another questions would be
Question Do you have staff in Germany?
Answer Berlin Dublin US only
Score 5 3 1
Should you have more interest in the staff in Germany than US you can then weight the scores of the questions.
You can continue to build questions of relevance for this pillar and use the weighting method as appropriate.
Once completed, you will have a score for that pillar, which can be directly compared other CRO ratings.
It is then possible to sum the scores of all the pillars to give an overall score. Weighting can occur within the pillar but this should be agreed upfront and prior to receiving any information from potential vendors.
It is possible to consider the pillars in a pooled fashion but recommendation is to evaluate and compare individually.
To give a qualitative measure comments are captured for that question so the expert in that field can emphasize favorable or non-favorable points as appropriate. It is strongly recommended to give a summary of strengths and weaknesses.
6
PhUSE 2015
CONTROL TECHNIQUES
To Grid or Not to Grid.
Various techniques exist between the sponsor and the supplier in order to communicate and agree on quantity of service required and costs.
A popular tool is the BID GRID. The BID GRID consists of list of services required (e.g. Stats, Programming, Data Management, Project Management) presented by a Work Breakdown Structure (WBS) within that function (e.g SAP, creation of Tables, Listings and Figures). The level of detail of the WBS can vary quite considerably between sponsor and supplier. The supplier may well have a fixed structure for the WBS as this is useful in comparing service costs not only across other CROs but also for comparison with internal resources too.
An example could be as follows, for each line entry it is useful to have # of hours required for each unique task, # of units required by sponsor for task, charge out rate of skill for 1 hour and consequently total cost. However, what is clear is that the methods used by sponsors and supplier are almost always different and a considerable amount of mapping is required between the two. This is possibly an area for standardization!
TASK
SAP Program Table Unique
SKILL
Senior Statistician Junior Programmer
# of hours required for unique task(hours)
50
10
# of units required by sponsor for task
1
30
Charge out rate of skill for 1 hour($, £) 100
80
Total Cost of activity($,£)
5000 24000
Governance & Escalation
A Governance structure needs to be put in place to ensure oversight of the partnership. Many different structures exist and the authors favor two different layers of focus namely an operational and executive layer. The operational layer has a short to medium term focus ensuring the the delivery of projects and line of sight for future incoming work. The executive layer takes a longer term look at the relationship and provides instruction to the operational layer on what continuous improvement needs to occur to be aligned with overall strategic direction. The governance should also highlight escalation pathways. It is imperative that an escalation process is made clear to ensure that the teams escalate appropriately.
Key Performance Indicators (KPIs).
Key Performance Indicators measure whether the study or deliverable is on track on not. KPIs generally circulate around pillars such as Quality and Timelines. It is essential these two factors are not taken in isolation. Hence a poor quality deliverable delivered ahead of time will be considered a failure even if timelines were exceeded. These KPIs can be linked to financial incentives such as bonus’ or penalties.
Work Schedule Demand.
. A regular updated forecast is required to be able to predict resource. High variability is often seen in this space. Studies or projects can be terminated early leaving a gap in work, conversely a high workload may appear at short notice due to internal accelerations or health authority questions. Having visibility and information readily available increases the chances of success.
System Access
A key component to outsourcing is whether the supplier company can work on their own systems or whether the supplier should work on the sponsors system. For the sponsor, the IT systems need to be robust and
7
PhUSE 2015
scalable to be able to take the external workload. In addition external connections to the sponsor system needs to be managed and securely controlled. There are some clear benefits from the sponsor side in that the exchange and transfer of key files is eradicated. Also, having access to key data enables the sponsor to respond in an agile manner to any future request related to the data. One other advantage is this method guarantees full transparency for the sponsor in tracking progress of work. However, it should be noted that challenges also exist. The training of CRO staff is not an activity that should be underestimates. Also, the CRO will find it difficult to derive the same level of efficiency on a ‘foreign’ environment and adhering to unfamiliar processes. In turn, this could result in an increase in costs for the sponsor. Training Operational Manual.
A training and operational manual is created by the sponsor to give more detailed information of the expectations of how to run the project in the confines of the working relationship. It is recommended that this is a living document and updated accordingly with clear communication to users at both parties.
CONCLUSION The AQSCI is one of many methods that can be used to evaluate potential vendors bidding for a new
contract. This paper provides an introduction on key points for consideration. Whilst this method is fundamental in the evaluation stage, it should be revisited during the onboarding and operational stages to include elements based on latest learnings. This will ensure a more robust approach for future engagements.
CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at:
Author Name Roy Ward Company PPD Email: [email protected]
Author Name James Gallagher Company Novartis Email: [email protected]
Brand and product names are trademarks of their respective companies.
8
Paper PD03
The AQSCI Method behind CRO Evaluation and Application to Pharmaceutical Company Needs
Roy Ward, PPD, Alsace, France James Gallagher, Novartis, Basel, Switzerland
ABSTRACT The high-risk nature of drug development increasingly continues to be a more demanding and costly
adventure. This has led to continuous scrutiny by the financial teams in searching for the correct strategy to maximise return on investment and enable internal teams to focus on value added areas.
This paper will focus on the strategy of outsourcing and discuss a frequently used method in the industry for CRO evaluation and selection called AQSCI. A discussion of how this method is used to produce a quantitative set of statistics which can measure the CRO company strengths and weaknesses will be considered. The two authors both have experience of working in pharmaceutical companies and in the CRO world. They will share personal opinions of what todays companies are looking for in a CRO partner using this method.
INTRODUCTION
The AQSCI evaluation method stands for the following a) Assurance of Supply b) Quality c) Service d) Cost e) Innovation.
The layout for the remainder of this paper is detailed in Table 1.
Table 1: Table of Contents
Section Abstract Introduction Motives for Outsourcing Types of Outsourcing The AQSCI Method of Sponsor selection
Control Techniques
Conclusion. Contact Information
Purpose High Level Overview of the goals of the paper Introduction and structure of the paper. Discussion of why sponsor companies outsource. Outsourcing Models experienced by the authors. Overview of the factors to consider when evaluating a sponsor and also applying this to a scorecard. Processes/considerations which facilitate a successful delivery within the engagement
The views expressed within this paper are entirely that of the authors and do not necessarily reflect the views of PPD or Novartis.
1
PhUSE 2015
MOTIVES FOR OUTSOURCING.
The motives for biometrics departments to outsource statistical reporting activities within pharmaceutical companies have a high degree of variability. Some popular key factors are briefly discussed below. Resource Limitations:
Ultimately, this factor is a key consideration when demand outstrips capacity. Within the dynamic pharmaceutical field, this can occur for various reasons. However, one constraint is that teams and in particular, statistical reporting teams need to have the flexibility to outsource to suitably meet sudden changes in demand.
Cost:
As with all industries, the pharmaceutical companies are trying to reduce costs and outsourcing can play an important role here.
Expertise:
Some organisations can offer expertise in niche areas to statistical reporting groups. Some examples, out of many, include complex statistical methodology, data transparency solutions and the technical implementation to generate CRT’s.
In the author’s experience, it is rare for a pharmaceutical company to exclusively base their decision on one of the above. Typically, there is a combination of factors considered.
TYPES OF OUTSOURCING EXPERIENCED BY THE AUTHORS
Four popular models are discussed below and also a newer model the authors have recently experienced that they are calling the ‘hybrid’ model. FULL SERVICE PRODUCT
This is sometimes called end-to-end outsourcing and unlike the other models discussed below the Biometrics organization is not the sole decision maker in determining the specifics related to an outsourced package. FUNCTIONAL OUTSOURCING (ALSO KNOWN AS FUNCTIONAL SERVICE PROVIDER)
This is where a particular function outsources either all of or part of their component in the broader development chain. For Biometrics organizations this is currently a very popular method. In simple terms, it can be considered as a ‘work-package’ approach.
FULL TIME EQUIVALENT
The Full Time Equivalent model relates to either on-site or off-site “contractors”. Within the industry there is a broad range of responsibilities assigned to contractors. Again, in the author’s experience, the contractors typically act as augmented resource and are integrated with the project team to facilitate the delivery of projects.
HYBRID MODEL
The so-called Hybrid Model is a combination of the above two models namely functional outsourcing and full-time equivalents. It commences as a FTE model and progresses to a work-package model as the project matures. The concept is that FTEs are integrated into the team and immersed in the reporting environment and project standards. As the project matures and new studies commence a suitably skilled FTE is identified and leads a team of his/her colleagues on a package of work but acts as a key interface to the internal team throughout the lifecycle.
2
PhUSE 2015
THE AQSCI METHOD
The AQSCI method is built on five pillars as described below.
ASSURANCE OF SUPPLY
This pillar deals with the topic of supply of resource.
Location of staff.
The location of staff within the CRO is a key consideration for pharmaceutical companies. This brings benefits related to overlapping time zone coverage, costs, local knowledge (e.g. specific regulatory authority expertise) and global scalability.
Quantity of staff.
This is an important indicator related to the magnitude of engagement. If a company is strategically looking to outsource a large package of work then the number of available staff is of high interest. Conversely if a company is looking for niche expertise, then the next metric is of primary interest.
Quality/Education of staff It is important to assess what level of qualifications and experience the company has. For example, how many PhD’s, M.Sc.’s are within the team, average number of years of experience or number of employees with over 10 years of experience? The ratio of staff grades(e.g. junior:senior) should also be considered.
Specific knowledge of staff. If the work being outsourced requires TA knowledge then the CRO should match this experience. Hence, if the package of work being outsourced is from the Oncology sector it is preferential to have this skill embedded within the supplier company. Likewise specialist technical knowledge may be of interest e.g SAS Drug Development.
Turnover rates
All companies wish to have a high level of stability and hence low turnover rates. This avoids re-training costs for the project which are typically absorbed by the vendor. A low turnover rate ensures continuity and mitigates disruption. Questioning what the annual turnover rate is by region and comparing it to internal sponsor rates is required.
Evidence of track record on the company delivering. If a company has consistently delivered for the sponsor company in high quality then this is a good indicator of future trends. In addition this demonstrates that the CRO has experience for the sponsor company.
Evidence of gaining previous awards. Even if the supplier company has little or no experience with the sponsor company, does the company have a track record of being rejected and if so why. This may reveal a problem with competitive rates or a long standing concern teams have with quality.
Experience in the area of phase i, ii, iii an iv, modelling & simulation. The different stages of pharmaceutical development require quite different skills. Phase I is notoriously quick and fast whereas phase III can be very demanding particularly if the study is on a submission critical path. Naturally if the workload being outsourced by the sponsor is phase IV for work example, the supplier should contain phase IV experience
3
PhUSE 2015
QUALITY
This pillar deals with the topic of supply of quality.
Of interest are.
Quality system/processes available What level of quality does the supplier company have in terms of systems and processes? A strict review of SOPs is recommended including a question and answer session with the supplier. This review will highlight any gaps in the processes. Likewise it is imperative that a suitable IT Quality Assurance colleague also contributes to this review.
Health Authority findings. Does the supplier company have a track record of Major findings following a Health Authourity inspection/audit? A key question is what have the supplier company done in order to rectify that finding. This demonstrates whether the company is a learning company or not.
Governance support
It is very important to understand the company’s escalation process and governance structure. Where in the process does the project lead get involved and when do management become informed and expected to act? Is this clear to the team and also is this clear to the sponsor company whom to escalate to. Most relationships have a reporting/governance structure such as an Operating Steering Committee which meets on a regular basis to overview the general running of the projects and review Key Performance Indicators. Executive Steering Committees contain senior members of the relationship and meet less often and discuss any relationship threatening issues and future workload/ strategy of the relationship.
SERVICE
This pillar deals with the topic of supply of service
Of interest are.
Cultural fit Cultural fit is hard to quantify but a general mirroring of expertise, location between the sponsor and supplier is important. Familiarity between the two companies helps the relationship.
Ability to work on host system. Some sponsors wish all of the work to be completed on the sponsor company system. Assuming the technology allows this, suppliers using sponsors systems can save on lengthy data transference discussions and it also allows the results to be seen rapidly and it offers a transparent overview of the status of the biometrical work. Conversely the supplier will require compensation for additional training for its employees to work on a new system rather than the supplier system of which the associates will already be familiar with. Likewise experience of working on the sponsors system may well be of high interest.
Training requirements to get staff up to speed/on-boarding time The time in which a sponsor company needs to get its associates up to speed on a system can vary. This is of importance for costings and also demonstrates the flexibility of the sponsor company to ramp-up the workload if required.
Expertise of a certain system or package e.g SAS, R etc. It is important for the supplier company to mirror certain skills of those at the sponsor. Indeed, as discussed earlier, if the sponsor company is looking to the supplier for expertise it is expected that this skill is better than internal resources.
Experience of outsourcing model being requested(strategic partners). The sponsor company will have a strong idea of the strategic model they are looking to put in place. Even though the supplier company may never have worked with the sponsor company before, the supplier company may have a wealth of experience in this model. Examples are required.
4
PhUSE 2015
Likely overhead costs in terms of internal oversight. How much consultation and support will the supplier require from the sponsor? Ideally the internal oversight is very low as this is the work being outsourced. However internal costs and oversight can mount up resulting in high sponsor costs and supplier dependency of the sponsor.
Management Support & Oversight How much involvement will the supplier’s management give?
COST
This pillar deals with the topic of cost.
Of interest are.
Cost structure competitive. The supplier company may well be top in quality and have an excellent delivery however this may come at a price which may not be competitive to peer suppliers. A financial analyst from within the sponsor company is required to give an opinion on this topic.
Cost structure transparent Is it clear where the final quoted costs are coming from? Where multi-services are involved the sponsor would at least wish to know how much each service is contributing to the final cost. Ideally, a breakdown of the costs for each services component can be supplied. Ideally, this detailed costing can then be used to compare component rates against each of the supplier’s rates.
How Out of Scope costs are dealt with. A “bone of contention” for both sides of the relationship. It is important that this is discussed before the relationship is agreed. It is inevitable that new challenges and changes will arise. How these are dealt with resource wise then becomes a company decision which can be very different between sponsor and supplier.
The financial stability of the company. Many of us who have been in the industry for a while have experienced working with a strong CRO who in 4 to 5 years’ time are no longer in existence or have been bought by a competitor. It is wise to assess and predict the potential outcome of the supplier. Worst case scenario is that the supplier becomes nonfunctional during work execution.
Financial Models Available. How amenable are the supplier to work with financial models which include penalties or rewards?
INNOVATION
This pillar deals with the topic of innovation, of interest are.
Depth of statistical scientist group/innovation of study design. The sponsor may well be looking for niche skills in the supplier such a statistical expertise. Evidence of a scientific group is important to highlight this skills as is attendance and contribution to relevant subject matter conferences.
Demonstration of productivity/process improvements It is important that the supplier demonstrates not only “can do now” skills but also “can do in the future”. This demonstrates that the supplier is moving with the times and has not become complacent with the present.
Demonstration of Technology solutions What new techniques or tools is the sponsor company using? A demonstration of new ideas likely means that the supplier will have ideas to solve your future problems.
Demonstration of Lessons Learned.
5
PhUSE 2015
It is part of human nature and progress that mistakes will occur. It is important that these mistakes are used positively to demonstrate a learning culture of the supplier. So from a negative situation the supplier can demonstrate a positive outcome by demonstrating what has been put in place to prevent that from happening should the same conditions exist in the future. A demonstration of a lessons learned forum with tangible implementations would show this.
Areas of specific company interest e.g signal detection. A sponsor may be looking for a niche skill in which the sponsor has yet to acquire, for example skills in safety signal detection or data transparency.
Visibility in external groups e.g PHUSE! To demonstrate the company is a learning organization it is important that the supplier is visible and contributes to the wider community. Organisations such as PHUSE are ideal grounds in which to assess the level of external engagement and it is an opportunity for them to showcase expertise.
Quantitative Measure
To obtain a quantitative measure the methodology is as follows:
For each pillar, key questions are asked. These are related to relevance to each CRO, hence these questions could be different related to CRO. For example if geographical location on the west coast of America is of importance then the question asked is
Do you have staff in the west coast of America?
A numerical score is given on a constant scale e.g 1-5(where 1 is low and 5 is high)
If the response is “yes we have offices in Chile”, this response is reasonably positive since Chile is in the time-zone of the west coast of America but it is not a perfect match, hence the score for this question would 3. If the answer was “Los Angeles” the score would be 5. If the answer is “Europe only” the score would be one.
Question Do you have staff in the west coast of
America?
Answer Los Angeles
Chile Europe only
Score 5 3 1
Should you also have an interest in staff for Germany another questions would be
Question Do you have staff in Germany?
Answer Berlin Dublin US only
Score 5 3 1
Should you have more interest in the staff in Germany than US you can then weight the scores of the questions.
You can continue to build questions of relevance for this pillar and use the weighting method as appropriate.
Once completed, you will have a score for that pillar, which can be directly compared other CRO ratings.
It is then possible to sum the scores of all the pillars to give an overall score. Weighting can occur within the pillar but this should be agreed upfront and prior to receiving any information from potential vendors.
It is possible to consider the pillars in a pooled fashion but recommendation is to evaluate and compare individually.
To give a qualitative measure comments are captured for that question so the expert in that field can emphasize favorable or non-favorable points as appropriate. It is strongly recommended to give a summary of strengths and weaknesses.
6
PhUSE 2015
CONTROL TECHNIQUES
To Grid or Not to Grid.
Various techniques exist between the sponsor and the supplier in order to communicate and agree on quantity of service required and costs.
A popular tool is the BID GRID. The BID GRID consists of list of services required (e.g. Stats, Programming, Data Management, Project Management) presented by a Work Breakdown Structure (WBS) within that function (e.g SAP, creation of Tables, Listings and Figures). The level of detail of the WBS can vary quite considerably between sponsor and supplier. The supplier may well have a fixed structure for the WBS as this is useful in comparing service costs not only across other CROs but also for comparison with internal resources too.
An example could be as follows, for each line entry it is useful to have # of hours required for each unique task, # of units required by sponsor for task, charge out rate of skill for 1 hour and consequently total cost. However, what is clear is that the methods used by sponsors and supplier are almost always different and a considerable amount of mapping is required between the two. This is possibly an area for standardization!
TASK
SAP Program Table Unique
SKILL
Senior Statistician Junior Programmer
# of hours required for unique task(hours)
50
10
# of units required by sponsor for task
1
30
Charge out rate of skill for 1 hour($, £) 100
80
Total Cost of activity($,£)
5000 24000
Governance & Escalation
A Governance structure needs to be put in place to ensure oversight of the partnership. Many different structures exist and the authors favor two different layers of focus namely an operational and executive layer. The operational layer has a short to medium term focus ensuring the the delivery of projects and line of sight for future incoming work. The executive layer takes a longer term look at the relationship and provides instruction to the operational layer on what continuous improvement needs to occur to be aligned with overall strategic direction. The governance should also highlight escalation pathways. It is imperative that an escalation process is made clear to ensure that the teams escalate appropriately.
Key Performance Indicators (KPIs).
Key Performance Indicators measure whether the study or deliverable is on track on not. KPIs generally circulate around pillars such as Quality and Timelines. It is essential these two factors are not taken in isolation. Hence a poor quality deliverable delivered ahead of time will be considered a failure even if timelines were exceeded. These KPIs can be linked to financial incentives such as bonus’ or penalties.
Work Schedule Demand.
. A regular updated forecast is required to be able to predict resource. High variability is often seen in this space. Studies or projects can be terminated early leaving a gap in work, conversely a high workload may appear at short notice due to internal accelerations or health authority questions. Having visibility and information readily available increases the chances of success.
System Access
A key component to outsourcing is whether the supplier company can work on their own systems or whether the supplier should work on the sponsors system. For the sponsor, the IT systems need to be robust and
7
PhUSE 2015
scalable to be able to take the external workload. In addition external connections to the sponsor system needs to be managed and securely controlled. There are some clear benefits from the sponsor side in that the exchange and transfer of key files is eradicated. Also, having access to key data enables the sponsor to respond in an agile manner to any future request related to the data. One other advantage is this method guarantees full transparency for the sponsor in tracking progress of work. However, it should be noted that challenges also exist. The training of CRO staff is not an activity that should be underestimates. Also, the CRO will find it difficult to derive the same level of efficiency on a ‘foreign’ environment and adhering to unfamiliar processes. In turn, this could result in an increase in costs for the sponsor. Training Operational Manual.
A training and operational manual is created by the sponsor to give more detailed information of the expectations of how to run the project in the confines of the working relationship. It is recommended that this is a living document and updated accordingly with clear communication to users at both parties.
CONCLUSION The AQSCI is one of many methods that can be used to evaluate potential vendors bidding for a new
contract. This paper provides an introduction on key points for consideration. Whilst this method is fundamental in the evaluation stage, it should be revisited during the onboarding and operational stages to include elements based on latest learnings. This will ensure a more robust approach for future engagements.
CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at:
Author Name Roy Ward Company PPD Email: [email protected]
Author Name James Gallagher Company Novartis Email: [email protected]
Brand and product names are trademarks of their respective companies.
8