Skip to main content

Cost of peer mystery shopping to increase cultural competency in community clinics offering HIV/STI testing to young men who have sex with men: results from the get connected trial

A Correction to this article was published on 26 July 2023

This article has been updated



Cultural competency has been identified as a barrier to lesbian, gay, bisexual and transgender (LGBT) populations seeking care. Mystery shopping has been widely employed in the formal health care sector as a quality improvement (QI) tool to address specific client needs. The approach has had limited use in community-based organizations due in part to lack of knowledge and resource requirement concerns. Several mystery shopping initiatives are now being implemented which focus on the LGBT population with the goal of reducing barriers to accessing care. One subset targets men who have sex with men (MSM) to increase uptake of human immunodeficiency virus (HIV) testing. No study investigates the costs of these initiatives. Get Connected was a randomized control trial with the objective of increasing uptake of HIV-prevention services among young men who have sex with men (YMSM) through use of a resource-locator application (App). The initial phase of the trial employed peer-led mystery shopping to identify culturally competent HIV testing sites for inclusion in the App. The second phase of the trial randomized YMSM to test the efficacy of the App. Our objective was to determine the resource inputs and costs of peer-led mystery shopping to identify clinics for inclusion in the App as costs would be critical in informing possible adoption by organizations and sustainability of this model.


Through consultation with study staff, we created a resource inventory for undertaking the community-based, peer-led mystery shopping program. We used activity-based costing to price each of the inputs. We classified inputs as start-up and those for on-going implementation. We calculated costs for each category, total costs and cost per mystery shopper visit for the four-month trial and annually to reflect standard budgeting periods for data collected from September of 2019 through September of 2020.


Recruitment and training of peer mystery shoppers were the most expensive tasks. Average start-up costs were $10,001 (SD $39.8). Four-month average implementation costs per visit were $228 (SD $1.97). Average annual implementation costs per visit were 33% lower at $151 (SD $5.60).


Peer-led, mystery shopping of HIV-testing sites is feasible, and is likely affordable for medium to large public health departments.


While HIV incidence has decreased in the US population, it has remained constant or increased among men who have sex with men (MSM), particularly among younger age groups [1]. Improving rates of HIV testing for this demographic is an essential component of the national plan to prevent cases of HIV and achieve the Centers for Disease Control and Prevention’s (CDC) goal for ending the HIV epidemic [2].

Extensive research and intervention efforts have been aimed at increasing test uptake among young men who have sex with men (YMSM). Lack of cultural competency among providers has been identified as a significant barrier to care engagement for this group [3]. Studies have begun to explore ways to eliminate cultural barriers in the wider lesbian, gay, bisexual and transgender (LGBT) population, such as through peer navigation to promote use of PrEP among transgender women and MSM [4,5,6]. Improving cultural competency for YMSM has yet to be systematically addressed [3].

Public health department quality improvement (QI) has been increasingly promoted to improve service delivery, efficiency and responsiveness to public needs [7]. The National Association of County and City Health Officials now offers a step-by-step guide, while recognizing that QI efforts must be actionable within organizational budgets [8]. Studies have also evaluated pilot public health department projects in select states [9]. One tool yet to be implemented widely is mystery shopping.

Mystery shopping, derived from business marketing, is a tool used in a wide range of settings to evaluate and monitor service performance, and inform QI initiatives to increase customer satisfaction [10]. Mystery shopper visits can range from low-intensity phone calls to high-intensity interventions, such as in-person visits, with findings recorded and reported in near real-time. These visits can provide data for QI regardless of form or setting [11].

The price of mystery shopping is determined by the content and mode of the visit. Costs to undertake mystery shopping include the start-up costs to initiate the program, such as script creation and training in its execution [12]. On-going implementation costs, after the initial investment, reflect the length and type of encounter, the number of encounters per provider, the complexity of the investigation scenario and post visit reporting arrangements [13].

In the formal health care arena, mystery shopping has been used to improve patient experiences in areas such as appointment referral, lab testing, receipt of mammograms, pharmacy delivery, physician waiting rooms and in family planning clinics [14,15,16,17,18].

Recent work in the US and internationally has begun to explore the feasibility of using mystery shopping as a QI tool in community-based HIV-testing sites. Several studies have investigated the level of cultural competency and the nature of provider/client interactions at community-based organizations with the goal of improving elements, such as site friendliness and provider knowledge, as a means to increase test uptake among the YMSM [19,20,21]. A guide for conducting mystery shopping within HIV community-based clinics has also recently been published [22]. However, hesitancy regarding its use exists among community-based organizations due to lack of knowledge about implementation requirements and costs.

The Get Connected trial focused on YMSM at-risk of contracting HIV and employed a QI intervention. The trial objective was to increase uptake of HIV-prevention services in this group through use of a resource-locator App populated with LGBT-friendly clinics. The initial phase of the trial identified culturally competent HIV testing sites for inclusion in the App. The second phase of the trial randomized YMSM to test the efficacy of the App.

In the first phase, high-intensity, peer-led mystery shopping was conducted in clinics recruited from three cities with high-HIV incidence: Atlanta, Houston, and Philadelphia, serving as study sites. Clinics determined to be YMSM-friendly HIV testing sites, based on visits, were included as locations in the mobile App [23]. Research centers in cooperation with public health departments served as implementation hubs to proxy medium to large public health departments or other community-based organizations which might sponsor these initiatives.

The objective of this paper is to investigate the financial sustainability and resource requirements for possible adoption of the mystery shopping model. We documented the resource inputs and used activity-based costing to estimate start-up, implementation and total costs, along with cost per shopper visit, for four-months, an arbitrary period determined by the trial, and annually to reflect expenses over a standard budgeting period [23].


We focused on non-research-related tasks. Based on input from site staff, we created an inventory of activities, materials and equipment required to develop and implement mystery shopping. We used program budgets and activity-based costing to determine program costs for the period September 2019 through September 2020 [24, 25].

Staff reported hours spent on each activity. Labor costs were calculated based on reported hours spent on activities, multiplied by the median hourly wage for each staff category. Activities that were handled centrally due to RCT site responsibilities, such as clinic identification, were allocated equally across sites. Cost values for materials were drawn from program budgets. Peer shoppers visited each clinic twice and were paid $50 per visit, including de-briefing.

For each site we calculated start-up costs or those incurred prior to initiation of mystery shopping which cannot be recouped. We also calculated implementation costs which become a recurrent budget expense if the program is adopted [26]. We then calculated total costs, the sum of the two. We averaged all values the sites.

Using published data on visit numbers, we calculated implementation and total cost per mystery shopper visit for the four-month duration of the trial. We then estimated annual costs by adding eight-month on-going implementation costs, consisting of visit coordination and visits, to the four-month values. We estimated annual cost per shopper visit by assuming the rate of average visits for the four-month period applied through the remainder of the year [27]. Data are available from the author upon request.


Mystery shopping activities, inputs and their associated costs were allocated to start-up and implementation, shown in Table 1. Data are provided for each site in order of size, measured in terms of number of shoppers, and averaged across sites for the trial period. Shopper protocol development, training staff to educate peer shoppers in protocol execution and program management constituted start-up activities. The first two were handled by one site as part of the trial design and thus show little variation. Average four-month start-up cost per site was $10,001 ($39.8 SD) with project management constituting 58% of the total.

Table 1 Activities and Costs of Per-Led Mystery Shopping by Study Sites

Implementation began with identifying clinics to serve as shopping sites. This also handled centrally by one site. With the cooperation of the local departments of public health, study staff used and Google to collect clinic data. Elements included: information on location, services offered, opening hours and contact information. Costs for clinic identification totaled $1398 of which one-third, $466, was allocated to each site.

Other implementation activities included peer recruitment and training of shoppers and coordinating and executing shopper visits, which together constituted two-thirds of four-month average implementation costs of $9995 (SD $2522.8). Costs generally increased with site size with Philadelphia, having the highest costs due to increased costs of shopper recruitment and visit coordination.

Average total four-month costs were $19,995 (SD $2517.8) and were evenly split in percentage terms between start-up and implementation. Implementation costs per visit averaged $228 (SD $2.00) and decreased with site size as did total costs per visit at $469 (SD $55.9), reflecting the costs of increasing visit numbers.

Annual costs are shown at the bottom of Table 1. Average annual total costs, the sum of the initial start-up costs and the estimated 12-month implementation costs, produced an average total of $20,142 (SD $10,297). Average annual implementation cost per visit was $151 (SD $5.6) with an average annual total cost per visit of $232 (SD $13.88).


We identified the activities and resources required to undertake high-intensity, peer-conducted mystery shopping programs to increase cultural competency among community-based clinics offering HIV testing. Our findings indicate that instituting peer mystery shopping to increase HIV-testing among YMSM is feasible given that public health department staff are highly likely to be equipped to undertake mystery shopping activities. Programs were run centrally at study sites in three areas to model conditions for medium to large public health department or community-based organizations. 

Based on the cost of resource inputs, annual average cost per visit was approximately $150. Costs declined with site size, reflecting economies of scale whereby costs decline as start-up costs are spread over an increasing number of visits. Costs varied significantly based on the duration of the program, with costs decreasing as the implementation period increased from four months to one year. Annual values are most relevant for public health departments and community-based organizations, given standard annual budgeting periods. Average total costs fell by over-half from the four-month period to the annual period, while average implementation costs fell by a third, primarily as visit numbers increased, both demonstrating the economies of scale of the mystery shopping programs.

Start-up and implementation costs are important to separate as one indicates the initial investment requirement and the other the on-going cost of the program once it has started. Average start-up costs were around $10,000 per site. These costs appear minimal in comparison with the size of many public health department budgets. For example, in Georgia, location of the Atlanta site, the current state public health department proposed budget for fiscal year 2022, was $298 million dollars [28]. Given its relative costs, mystery shopping may be reasonably integrated into many public health departments, particularly urban areas, as part of their overall QI efforts and expenses.

Average annual implementation cost per visit of $151 (SD $5.6) were at the lower end of published estimates ($50 - $5000) [9]. The intervention evaluated here likely provides an upper bound on mystery shopping costs. The Get Connected visit protocol required two, high-intensity visits per clinic whereas community adoption may only require one, be of lower intensity and have fewer reporting requirements.

Costs presented here reflect the context of the Get Connected trial, conducted in high incidence urban setting and thus may not be generalizable to dissimilar locations. Some values may also have been overestimated. Staff were asked to identify and report time spent on activities related to undertaking mystery shopping only and not research-related costs.

The difficulty of separating these two, particularly in terms of project management hours, may have led overestimation of costs. Salaries to value time reflect those of academic research settings and staff had to have skills related to research in addition to those required to conduct mystery shopping. Thus, wage rates may be higher than those for community staff not engaging in research.

Costs generally increased with site size, with Philadelphia having higher costs in terms of recruitment and training of peer shoppers, coordinating shopper visits, and shopper payments. It should be noted, however, that Philadelphia region was the first site within the clinical trial. As a result, its increased costs may also be linked to unexpected costs associated with being the first site brought on-line.

Other assumptions may have led to an underestimation of costs. We assumed no additional equipment was involved as the App was loaded onto mobile phones already in use by peer shoppers. Reported material costs were minimal reflecting that the intervention relied on mobile devices. No costs were included for acquisition of the App given it is available through open-source software. Annual implementation costs did not account for staff turnover, as none occurred during the four-month period.

Other possible biases are of unknown direction. Costs reported here also reflect the geographical area and features of the local market under investigation, such as ease of access to shopping targets and of shopper recruitment. Staff hours were based on recall which may have led to an over or under-estimate of costs.

Peer-navigator findings from mystery shopping visits were communicated to clinics with the goal of promoting service modification to reduce access barriers to YMSM. This feedback, however, was not communicated formally nor was the degree to which changes were made. This shortcoming needs to be addressed and efforts expanded to help clinics address the cultural barriers identified and subsequent confirmation that service modifications were made.

In addition, user feedback was initially explored through focus groups that provided input during App development. The trial assessed willingness to use and length of use once the App was deployed. Soliciting end-user feedback, however, could provide important information for continuous quality improvement. Feedback could range from including affirmation that the clinic was LBGT-friendly, such as through a “like”, a user clinic rating or qualitative feedback to the App developers. All of which would likely increase value and promote sustainability.

Considering future adoption, it may be possible to institute a lower intensity protocol to identify LGBT-friendly clinics, such as reducing visits to one, or changing the intensity of the visit. Creation of a peer-led, detailed mystery shopper training toolkit, with sample scenarios included, may reduce start-up costs. In addition, the model could be easily modified to include other groups, such as transgender youth, which would further increase its value.


Lack of cultural competency in HIV testing clinics is a known barrier to YMSM, and the wider LGBT population, accessing services in the US [4,5,6]. Implementation of mystery shopping, as a pre-cursor to the Get Connected trial, shows that the approach is feasible and a promising QI tool which can be used to help develop and identify culturally competent HIV/STI testing sites.

Here we investigated the resource and financial requirements to inform possible sustainability for peer-led, high intensity mystery shopping programs aimed at YMSM. Public Health Departments, in high incidence urban area, would be suited to launch such programs. The initiatives appear to be highly affordable and a relatively minor expense for public health department and community-based organizations of this type. However, the approach may not be viable for individual clinics or small public health departments to initiate on their own given centralization is necessary to reap economies of scale or declining average costs per visit.

While the US health care delivery and structure of community-based clinic system is unique, the methodological approach and use of activity-based costing is replicable in other countries.

Given the public health goals of promoting health and responding to community needs, peer-led, mystery shopping offers a promising tool to improve service delivery to an underserved, vulnerable population. Increases in testing can reduce cases of HIV thereby helping achieve the CDC goal of eliminating HIV. In addition, establishing linkage to care can increase the overall health of YMSM.

Data Availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Change history



Centers for Disease Control and Prevention


human immunodeficiency virus




men who have sex with men


post exposure prophylaxis


quality improvement


sexually transmitted infection


young men who have sex with men


  1. Shrestha RK, Chavez PR, Noble M, Sansom SL, Sullivan PS, Mermin JH, MacGowan RJ. Estimating the costs and cost-effectiveness of HIV self-testing among men who have sex with men, United States. JAIDS. 2020; e25445.

  2. Fauci AS, Redfield RR, Sigounas G, Weekhee MD, Giroir BP. Ending the HIV epidemic: a plan for the United States. (2019). JAMA. 2014; 321(9): 844–845.

  3. Philpin MM, Tanner AD, Duva A, et al. Factors affecting linkage to care and engagement in care for newly diagnosed HIV-positive adolescent within fifteen adolescent clinics in the United States. AIDS and Beh. 2014;181501–10.

  4. Wilkerson JM, Rybicki S, Barger CA, et al. Creating a culturally competent clinical environment for LGBT patients. J Gay Lesbian Soc Serv. 2011;2376–394.

  5. Rebeck CJ, Clark KA, Runger D, Fehrenbacher AD. A Promising PrEP Navigation intervention for Transgender Women and Men who have sex with mend experiencing multiple Syndemic Health Disparities. J Community Health. 2019;44:1193–203.

    Article  Google Scholar 

  6. Sevelius JM, Patouhas E, Keatlye JG, Johnson MO. Barriers and facilitators to engagement and retention in care among transgender women in with human immunodeficiency virus. Ann Beh Med. 2014;27(1):5–16.

    Article  Google Scholar 

  7. National Association of Count and City Health Officials (NACCHO). Quality Improvement. Retrieved from: February 1, 2022.

  8. Kan TY, Morgan JW. Armbuster. Developing a Health Department Quality Improvement Plan. Public Health Foundation. Retrieved from: February 1. 2022.

  9. Riley W, Parsons H, McCoy K, Burns D, Anderson D, Suhna L, Sainfort F. Introducing quality improvement to local public health departments: structured evaluation of a statewide pilot project. Heath Serv Res. 2009;45(5 Pt2):1863–79.

    Article  Google Scholar 

  10. Blessing G, Natter M. Do mystery shoppers really predict customer satisfaction and sales performance. J Retail. 2019;95(3):47–62.

    Article  Google Scholar 

  11. Devi SS, Reddy VA. Conceptual study of mystery shopping as an ancillary method for customers surveys. GJMBR. 2016;16(20):11–7.

    Google Scholar 

  12. Brown C. How to Set Up a Mystery Shopping Program: Implementing the right tools to measure customer experience can help your business improve sales, operations, and profits. (2020, October 11). Retrieved from:

  13. Shoppers’. View. No hidden price tag: mystery shopping costs by the numbers. Shoppers’ view blog. 2016, June 7). Retrieved from:

  14. Moriarty H, McLeod D, Dowell A. Mystery shopping in health service evaluation. B J Gen Pract., Wilkinson TA, Rafie S, Clark PD, Carroll AE, Miller E. Evaluating community pharmacy responses about levonorgestrel emergency contraception by mystery caller characteristics. J Adolesc Health. 2018; 63(1), 32–36.

  15. Boodman SM. New Emphasis on Customer Service in Medicine “Secret shoppers” help improve the real patient’s experience. AARP Bulletin. (2010, August 13). Retrieved from:

  16. Sykes S, O’Sullivan K. A ‘mystery shopper’ project to evaluate sexual health and contraceptive services for young people in Croydon. JFRH. 2006;32(1):25–6.

    Google Scholar 

  17. Lazarus A. The secret to improving health care services. Physician Exec. 2010;36(5):42–5.

    PubMed  Google Scholar 

  18. Venkatraman C, Cosima L, Adebayo E, Lundgren IL, Chatterjee S. A systematic review of the use of adolescent mystery clients in assessing the adolescent friendliness of health services in high, middle, and low-income countries. Glob Health Action. 2018; 11(1), 1536412.

  19. Bauermeister JA, Golinkoff JM, Lin WY et al. Testing the testers: are young men who have sex with men receiving adequate HIV testing and counseling services? JAIDS. r. 2019; 82 Suppl 2(2) S133-S141.

  20. Tanner AE, Philbin MM, Duval A, Ellen J, Kapogiannis B, Fortenberry JD. The adolescent trials network for HIV/AIDS Interventions. Youth friendly” clinics: considerations for linking and engaging HIV-infected adolescents into care. AIDS Care. 2014;26(2):199–205.

    Article  PubMed  Google Scholar 

  21. Shah P, Kibel M, Akuyu D, Lobun R, Ayieko J, Keter A, Kamanda A, et al. A pilot study of “peer navigators’ to promote uptake of HIV testing, care and treatment among street-connected children and youth in Eldoret Kenya. AIDS and Beh. 2019;23(4):908–19.

    Article  Google Scholar 

  22. National Association of state and territorial directors. Addressing stigma in healthcare facilities: A secret shopper intervention. June 2019. Retrieved from:

  23. Burmeister JA, Golinkoff JM, Horvath KJ, et al. A multilevel tailored web app-based intervention for linking young men who have sex with men to quality care (get connected): protocol for a randomized controlled trial. JMIR Res Protoc. 2018;7(8):e10444.

    Article  Google Scholar 

  24. Moullin JC, Dickson KS, Stadnick NA et al. Systematic review of the Exploration, Preparation, implementation, sustainment (EPIS) framework. Implement Sci. 2010; 14(1).

  25. Huebschmann A, Trinkley KE, Gritz M, Glasgow RE. Pragmatic considerations and approaches for measuring staff time as an implementation cost in health systems and clinics: key issues and Applied examples. Implement Sci Commun. 2022;3:44.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Haddix AC, Teutsch SM, Corso PS, editors. Prevention effectiveness: a guide to decision analysis and economic evaluation. 2nd edition. Oxford University Press, 2003.

  27. Drummond MF, Sculpher MJ, Torrance GW, O’Brien BJ, Stoddart GL. Methods for the economic evaluation of health c.

  28. Overview HL. 2022 fiscal year budget for the department of public health. Georgia budget and policy institute. Retrieved at: March 1, 2022.

Download references


We thank the Mystery Shoppers across the three cities and project staff from The Children’s Hospital of Philadelphia, Texas Children’s Hospital, Emory University, and the University of Chapel Hill for their assistance with the programmatic input for this project. We thank the iTech staff for their work coordinating research and regulatory requirements.


This work was supported by the National Institutes of Health under cooperative agreement #5U19HD089881.

Author information

Authors and Affiliations



Victoria Phillips (corresponding author): assisted with study design and execution, supervised data collection and analysis; and drafted the manuscript. Ashley Xue: assisted with data collection and analysis. Marné Castillo: served as the site supervisor for Houston and assisted with data collection. Dalia Santiago: served as the site supervisor for Texas and assisted with data collection. Taylor Wimbly: served as the site supervisor for Atlanta and assisted with data collection. Lisa B. Hightow-Wideman, Principal Investigator of iTECH, the umbrella organization overseeing Get Connected: assisted with manuscript preparation. Rob Stephenson, Get Connected Co-Principal Investigator: assisted with study design, implementation and manuscript preparation. José A. Bauermeister, Get Connected Principal Investigator: assisted with study design, implementation and manuscript preparation.

Corresponding author

Correspondence to Victoria L. Phillips.

Ethics declarations

Ethics approval

The study was approved by IRB at University of North Carolina, Chapel Hill (HUM327063), University of Michigan (HUM00127790) and University of Pennsylvania. Shopping sites were verbally consented for participation.

Conflict of competing interests

No author has a conflict of interest in relation to this work.

Consent for Publication

No agency is identified individually in reporting. Agencies consented to publication as part of the Get Connected trial.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original version of this article was revised: the authors corrected the surname of Lisa B. Hightow-Weidman.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Phillips, V.L., Xue, A., Castillo, M. et al. Cost of peer mystery shopping to increase cultural competency in community clinics offering HIV/STI testing to young men who have sex with men: results from the get connected trial. Health Econ Rev 13, 34 (2023).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: