The determination of “outsourcing” requirements and the identification of system “Suppliers” is introduced and covered rather extensively in Chapter 6 (Section 6.3). As part of the supplier selection process, there may be a need to develop a checklist (in a manner similar to the design review checklists in Appendix D) to facilitate the evaluation effort, particularly when two or more suppliers are being considered for a given program activity. The critical issue is to ensure that the right quantity and/or quality of suppliers are included as part of your “organizational” team.
The purpose of this appendix is to provide some typical questions that may be pertinent to your particular organization. Included (see the designated website below) are 30 multi-part questions, grouped within 4 major categories. The objective is to apply these, and other questions as appropriate, to aid in the evaluation of the background, experience, and capabilities of a potential supplier, leading to ultimate selection. While there may be many different questions related to your particular program, it is recommended that you review those questions at the website, and then develop your own checklist(s), tailored” to the program (and its requirements) in question. |
General Criteria
- Has a technical performance specification been prepared covering the product being acquired? Is this specification "supportive" of and "traceable" from the system specification?
- Is the product a commercial off-the-shelf (COTS) item requiring no adaptation, modification, and/or rework for installation?
- Has the COTS item been assessed in terms of effectiveness and life-cycle cost?
- If the product is a COTS item and requires some modification for installation,
- Has the degree of modification been clearly defined and minimized to the extent possible?
- Has the impact of the modification been assessed in terms of effectiveness and life-cycle cost?
- Has the life-cycle cost been minimized to the extent possible?
- Can the modification be accomplished easily and with a minimum of interaction effects?
- Have common and standard parts, reusable software, recycleable material, and so on, been incorporated in the modification/interface package or kit?
- Have alternative sources of supply for the same product been identified?
- If new design is required, has it been justified to the extent that the COTS, modified COTS, and comparable options are not feasible?
Product Design Characteristics
- Technical Performance Parameters
- Does the product fully comply with the functional performance specification (i.e., development, product, process, and/or material specification as applicable)?
- Has the applicable mission scenario (or operational/utilization profile) been defined for the product?
- Are the product's design characteristics responsive to the prioritized technical performance measures (TPMs)? Does the design reflect the most important features?
- Were the design characteristics derived through the use of a quality function deployment (QFD) (or equivalent) approach?
- Are the performance requirements easily traceable from those specified for the system level?
- Are the performance requirements measurable? Can they be verified or validated?
- Technology Applications
- Does the design utilize state-of-the-art and commercially available technologies?
- Do the technologies utilized have a life cycle that is at least equivalent to the product life cycle?
- Have "short-life" technologies been eliminated? If not, have such applications been minimized?
- Has an "open-architecture" approach been utilized in the design such that new technologies can be inserted without causing a redesign of other elements of the product?
- Have alternative sources for each of the technologies being utilized been identified?
- Have the technologies being utilized reached a point of maturity/stability relative to their applications?
- Physical Characteristics
- Is the product both functionally and physically interchangeable?
- Can the product be physically removed and replaced with a like item without requiring any subsequent adjustments or alignments? If not, have such interaction effects been minimized?
- Does the product design comply with the physical requirements in the technical specification (i.e., size, shape, and weight)?
- Effectiveness Factors
- Have the appropriate effectiveness factors been defined and included in the technical specification (i.e., TPMs applicable to the product being acquired)?
- Can the effectiveness requirements be traced back to comparable requirements specified at the system level?
- Has the supplier provided a measure of reliability for the product (e.g., R, failure rate, and/or MTBF)? Is this figure of merit based on actual field experience?
- Have the applicable reliability requirements been considered in the product design?
- Has the supplier provided a measure of maintainability for the product (e.g., MTBM, MLH/OH, Mct, Mpt, MDT, and/or equivalent)? Is this figure or merit based on actual field experience? Refer to Chapter 3 for an explanation of these acronyms.
- Have the applicable maintainability requirements been considered in the product design?
- Have the applicable human-factors requirements been considered in the product design?
- Have the applicable safety and security requirements been considered in the product design?
- Have the applicable supportability/serviceability requirements been considered in the product design?
- Have the applicable quality requirements been considered in the product design?
- Producibility Factors
- Has the product been designed for producibility?
- Is the design data/documentation such that any other supplier with comparable facilities/equipment, capabilities, and experience can manufacture the product in accordance with the specification?
- Disposability Factors
- Has the product been designed for disposability?
- Has the supplier developed the appropriate planning documentation and procedures covering the disposal and/or recycling of the product?
- Environmental Factors
- Has the product been designed with ecological and environmental requirements in mind?
- Has the supplier prepared an environmental impact statement for the introduction of the product?
- Economic Factors
- Has the product been designed with economic considerations in mind?
- Has the supplier conducted a life-cycle cost analysis for the product? Are the results realistic? Refer to Appendix 2.
Product Maintenance and Support Infrastructure
- Maintenance and Support Requirements
- Does the supplier have an established maintenance and support infrastructure in place?
- Has the supplier defined the maintenance concept/plan for the product?
- Have the appropriate supportability "metrics" been established for the product and included in the maintenance concept/plan (i.e., response time, turnaround time, maintenance process time, test equipment reliability and maintainability factors, facility utilization, spare parts demand rates and inventory levels, transportation rates and times, etc.)?
- Does the maintenance concept/plan facilitate or allow for the required degree of responsiveness on the part of the supplier?
- Have the preventive maintenance requirements been established for the product (if any)? Have these requirements been justified through a reliability-centered maintenance (RCM) approach?
- Have the product maintenance and support resource requirements been defined (i.e., spares, repair parts, and associated inventories; personnel quantities, skill levels, and training; test and support equipment; facilities; packaging, transportation and handling; technical data; and computer resources)? Have these requirements been adequately justified through a maintenance engineering analysis (MEA), a supportability analysis (SA), or equivalent?
- Data/Documentation
- Does the supplier have a computerized maintenance management data capability in place? Is this capability being effectively utilized for the purposes of continuous product/process improvement? Does it provide visibility relative to how well the product is performing in the field?
- Does the supplier have in place a reliability data collection, analysis, feedback, and corrective-action process? Are product failures properly recorded and are they traceable to the cause?
- Is the supplier monitoring and measuring the effectiveness of its preventive maintenance program? Where applicable, have the preventive maintenance requirements been revised to reflect a more cost-effective approach?
- Warranty/Guarantee Provisions
- Have product warranties/guarantees been established?
- Have the established warranty provisions been adequately defined through some form of a contractual mechanism?
- Are the warranty provisions consistent with the defined maintenance concept?
- Customer Service
- Does the supplier have an established customer service capability in place?
- Will the supplier provide assistance in the installation and checkout of the product at the producer's site and/or the user's site (if required)?
- Will the supplier provide on-site field service support if required?
- Does the supplier provide operator and maintenance training at the producer's site and/or the user's site when necessary? Is this training available "on call"? Will it be available throughout the product life cycle?
- In support of training activities, will the supplier provide the necessary data, training manuals, software, aids, equipment, simulators, and so on? Will the supplier provide updates/revisions to the training material as applicable?
- Does the supplier have a program for measuring training effectiveness?
- Economic Factors
- Is the product support infrastructure cost-effective?
- Have the requirements been based on life-cycle cost objectives?
Supplier Qualifications
- Planning/Procedures
- Does the supplier have a standard policies and procedures manual/ guide?
- Are the appropriate management procedures properly documented and followed on a day-to-day basis?
- Are the procedures/processes periodically reviewed, evaluated, and revised as necessary for the purposes of continuous process improvement?
- Has the supplier identified the activities and tasks that are essential in the successful accomplishment of system engineering requirements?
- Organizational Factors
- Has the supplier's organization been adequately defined in terms of activities, responsibilities, interface requirements, and so on?
- Does the organizational structure support the overall program objectives for the system? Is it compatible with the producer's organizational structure?
- Has the supplier identified the organizational element responsible for the accomplishment of system engineering tasks (as applicable)?
- Available Personnel and Resources
- Does the supplier have the available personnel and associated resources to assign to the task(s) being contracted? Will these personnel/ resources be available for the duration of the program?
- Do the personnel assigned have the proper background, experience, and training to do the job effectively?
- Design Approach
- Has the supplier implemented the system engineering process in the design of its products?
- Has an effective design database been established, and is it compatible with the system-level database established by the producer (prime contractor)?
- Does the supplier have in place a configuration management program, along with a disciplined change-control process? Has a configuration "baseline" approach been implemented in the development and growth of the product?
- Has the supplier's design process been enhanced through the use of such tools as computer-aided design (CAD), simulation, rapid prototyping, EC applications, and so on?
- Manufacturing Capability
- Does the supplier have a well-defined manufacturing process in place?
- Does the process incorporate the latest technologies and computer-aided methods (i.e., robotics, the use of CAD or computer-integrated manufacturing (CIM) technology, etc.)?
- Is the process flexible, and does it support an "agile" and/or "lean" manufacturing approach?
- Does the supplier utilize materials requirements planning (MRP), capacity planning (CP), shop floor control (SFC), just-in-time (JIT), master production scheduling (MPS), enterprise resource planning (ERP), statistical process control (SPC), Six Sigma, and other such methods in the manufacturing process?
- Has the supplier implemented a formal quality program in accordance with ISO-9000 and ISO-14,000 (or equivalent)? Is the supplier ISO-9000-certified? Does the supplier have a formal procedure in place for correcting deficiencies?
- Has the supplier implemented a total productive maintenance (TPM) program within its manufacturing plant? Has a TPM measure of effectiveness been established (i.e., OEE/overall equipment effectiveness)?
- Test and Evaluation Approach
- Has the supplier developed an integrated test and evaluation plan for the product?
- Have the requirements for testing been derived in a logical manner, and are they compatible with the identified technical performance measures (TPMs) for the system, and as allocated for the product?
- Does the supplier have the proper facilities and resources to support all product testing requirements (i.e., people, facilities, equipment, data)?
- Does the supplier have in place a data collection, analysis, and reporting capability covering all testing activities?
- Does the supplier have a plan for "retesting" if required?
- Management Controls
- Has the supplier incorporated the necessary controls for monitoring, reporting, providing feedback, and initiating corrective action in regard to technical performance measurement, cost measurement, and scheduling?
- Has the supplier implemented a configuration management capability?
- Has the supplier implemented an integrated data management capability?
- Has the supplier developed a risk management plan?
- Experience Factors
- Has the supplier had experience in designing, testing, manufacturing, handling, delivering, and supporting this product before?
- Has the supplier utilized experiences from other projects to help respond to the requirements for this program; that is, the transfer of "lessons learned"?
- Past Performance
- Has the supplier successfully completed similar projects in the past?
- Has the supplier been responsive to all of the requirements for past projects?
- Has the supplier been successful in delivering products in a timely manner and within cost?
- Has the supplier delivered reliable and high-quality products?
- Has the supplier been responsive in initiating any corrective action that has been required to correct deficiencies?
- Has the supplier stood behind all product warranties/guarantees?
- Does the supplier's organization reflect stability, growth, and high quality?
- Is the supplier's business posture good?
- Does the supplier enjoy an excellent reputation?
- Maturity
- Has the supplier established a process for benchmarking?
- Has the supplier implemented an organizational assessment program (i.e., Systems Engineering Capability Model (SECM), Capability Maturity Model Integration (CMMI), or equivalent)?
- Economic Factors
- Has the supplier implemented a life-cycle cost-analysis approach for all of its functions, products, processes, and so on?
- Has the supplier implemented an activity-based costing (ABC) approach with the objective of acquiring full visibility relative to the high-cost contributors and cause-and-effect relationships, and leading to the implementation of improvements for cost-reduction purposes?
|