FDA DHT use in clinical trials Draft Guidance comment

90 Canal St., 4th Floor, Boston, MA 02214 | +1-765-234-3463 | www.dimesociety.org

March 19, 2022

Dockets Management Staff (HFA-305)

Food and Drug Administration

5630 Fishers Lane, Rm. 1061

Rockville, MD 20852

To Whom it May Concern:

This letter represents comments submitted on behalf of the Digital Medicine Society

(DiMe) for consideration by the U.S. Food and Drug Administration (FDA) regarding Docket

No. FDA-2021-D-1128 for “Digital Health Technologies for Remote Data Acquisition in

Clinical Investigations Draft Guidance for Industry, Investigators, and Other Stakeholders.”

DiMe is a 501(c)(3) non-proﬁt organization dedicated to advancing digital medicine to

optimize human health. We do this by serving professionals at the intersection of the global

healthcare and technology communities, supporting them in developing digital medicine

through interdisciplinary collaboration, research, teaching, and the promotion of best

practices.

Founded in 2019, DiMe is the ﬁrst professional organization for experts from all disciplines

comprising the diverse ﬁeld of digital medicine. Together, we drive scientiﬁc progress and

broad acceptance of digital medicine to enhance public health.

Many of our DiMe members conduct clinical trials of new medical products or develop digital

health technologies for use in these trials. The following comment leverages the combined

expertise of our members regarding the use of digital health technologies (DHTs) in clinical

trials. We appreciate the opportunity to offer our comments on this draft guidance

document.

We appreciate that this guidance was issued by multiple centers: Oncology Center of

Excellence (OCE), The Center for Devices and Radiological Health (CDRH), the Center for

Drug Evaluation and Research (CDER), and the Center for Biologics Evaluation and Research

(CBER). In particular, we are pleased to see there is a continuing trend in harmonizing

guidance and hope this continues. Reduction in gray areas for those developing digital

health technologies and those using them in clinical investigations will allow regulations to

be more approachable and followable, lessening the burden for all involved.

We applaud FDA’s human-centered approach to this guidance. Recognizing that patients

should be the beneﬁciary of the work done in drug, tool, and measure development, digital

or otherwise, is critical to identifying problems and developing solutions that matter.

We note that the scope of this guidance allows for generalized use of best practices across

technologies and appreciate FDA providing that ﬂexibility.

Table of contents

Comment Structure: Feedback and Resources From the Digital Medicine Community 3

Definitions of and Surrounding Digital Health Technology 3

Comments on Regulatory Considerations and Engagement with the Agency 4

Comments on Considerations When Using Digital Health Technologies in Clinical

Investigations 4

Conclusion 10

Comment Structure: Feedback and Resources

From the Digital Medicine Community

Our comment is structured around feedback and questions that we have received from the

digital medicine community. We do not presume to provide answers, that is the sole domain

of the Agency. Rather, following each question we pose in this comment, we highlight

resources that 1) the digital medicine community is currently referencing, and 2) that we

believe are of high quality.

Deﬁnitions of and Surrounding Digital Health

Technology

Lines 15-18 and 813-818 of the Glossary deﬁne digital health technology (DHT) as a “system

that uses computing platforms, connectivity, software, and/or sensors, for healthcare and

related uses.” We recognize the complexity of DHTs and appreciate the broad deﬁnition and

recommend the further inclusion of language that references a modular stack of hardware

and software components

in this deﬁnition.

Line 23 references hardware and/or software that may comprise a DHT, pointing to a

footnote deﬁning the two terms, as well as ﬁrmware. There is opportunity to clarify these

deﬁnitions to remove ambiguity.

FDA might consider aligning to deﬁnitions provided by ISO/IEEE:

● Hardware - an electronic or mechanical element, including its interface and

documentation

● Software - computer programs, procedures and possibly associated documentation

and data pertaining to the operation of a computer system

● Firmware - combination of a hardware device and computer instructions or computer

data that reside as read-only software on the hardware device

It is worth noting that ISO/IEEE considers the ﬁrmware to be a part of the software, as

opposed to the guidance, which includes the ﬁrmware in the hardware. Aligning to ISO/IEEE

standards may reduce the burden on manufacturers of DHTs as they work to comply with

regulations.

FDA might also consider the addition of language regarding the core operation of the

hardware and the speciﬁc role of the ﬁrmware, including the appropriate regulations for

each.

Comments on Regulatory Considerations and

Engagement with the Agency

The inclusion of language regarding the optional qualiﬁcation regulatory path is welcome

guidance. It provides clarity for the device manufacturer and sponsor. Additional information

provided in the FDA’s webinar covering this draft guidance made a clarifying note that

qualiﬁcation is advantageous when using the same DHT in multiple clinical trials. We

recommend adding this additional language to the guidance as it creates clarity for

following the qualiﬁcation path.

ISO/IEC 12207:2008 Systems and software engineering — Software life cycle processes, 4.14

IEEE 828-2012 IEEE Standard for Conﬁguration Management in Systems and Software Engineering, 2.1

IEEE 1012-2012 IEEE Standard for System and Software Veriﬁcation and Validation, 3.2

Coravos A, Doerr M, Goldsack J, Manta C, Shervey M, Woods B, William A. Wood WA. Modernizing and designing evaluation

frameworks for connected sensor technologies in medicine. npj Digital Medicine (2020) 3:37 ;

https://www.nature.com/articles/s41746-020-0237-3

Lines 110-112 describe devices used in clinical investigations as “exempt from most

requirements applicable to devices.” We request additional clariﬁcation on this

terminology, who is able to make this distinction, and through what mechanism this

distinction should be pursued. Speciﬁcally, whether a medical device clearance is preferred

or irrelevant to the agency would assist our community.

There is an opportunity to deﬁne the boundaries between exempt DHTs, DDT/MDDT,

SaMD, and other qualiﬁcation programs, including speciﬁcs across different centers.

Clarifying these boundaries could allow for harmony across guidances and centers to

produce a guidance library that supports digital health innovation through a shared unifying

language and understanding of processes.

Comments on Considerations When Using Digital

Health Technologies in Clinical Investigations

We recognize that it is challenging to create one guidance that meets the needs of a

diverse audience of readers (manufacturers, sponsors, quality teams, etc.) and appreciate

the care taken to be as broad as possible in this draft guidance.

The DiMe Society and our community have developed relevant resources, such as The

Playbook

that builds a shared foundation for developing and deploying digital clinical

measures using a step-wise approach and Digital Measures that Matter to Patients: A

Framework to Guide the Selection and Development of Digital Measures of Health

that

may be useful to FDA.

The Playbook’s step-wise approach for developing and deploying clinical measures

Manta C, Patrick-Lake B, Goldsack J, C. Digital Measures That Matter to Patients: A Framework to Guide the Selection and

Development of Digital Measures of Health. Digit Biomark 2020;4:69-77. https://www.karger.com/Article/Fulltext/509725#

https://playbook.dimesociety.org/

Through our and our collective digital medicine community’s experiences, we deem

measure development as a precursor to technology selection. We recognize the

discussion of endpoints in Section IV. D and recommend the addition of a section describing

measure development (inclusive of Section IV. D), prior to Section IV. A. Selection of a Digital

Health Technology and Rationale for use in a Clinical Investigation.

Measures collected with DHTs during clinical investigations may be collected to determine

study eligibility, assess efﬁcacy, safety, and/or adherence to protocol.

Digital measures also vary by the type of underlying software/technology. To evaluate the

quality of the digital measurement product, it is valuable to distinguish between those tools

that rely upon sensor-generated data and those that rely on reported, survey data.

We use the term, “biometric monitoring technology” (BioMeT) to describe measurement

tools that process data captured by mobile sensors using algorithms to generate

measures of behavioral and/or physiological function. We suggest that this may be useful

language for the Agency to adopt in future guidance.

Evaluation of a digital measurement product that collects patient-generated data digitally

should be based on the context of use and should consider the type of technology:

● Survey-derived measures (e.g., an ePRO): The evaluation process is already well

described by prior FDA guidelines (e.g., construct, context and content validation)

● Sensor-derived measures (e.g., using a wearable): Evaluation should be based on the

veriﬁcation, analytical validation, and clinical validation (V3)

process detailed further

on page 6 of this comment

As the recently released guidance on Patient-Focused Drug Development: Methods to

Identify What Is Important to Patients Guidance for Industry, Food and Drug Administration

Staff, and Other Stakeholders deals largely with identifying meaningful aspects of health

relevant to patients, citing that guidance would be beneﬁcial to the readers.

As related to the selection of the DHT, we echo the ﬁt-for-purpose deﬁnition (Line 150-152

and Glossary 824-824) of a conclusion that the level of validation associated with a DHT is

sufﬁcient to support its context of use. Our digital medicine community recognizes the need

for clariﬁcation when considering the potential for a mixed technical capability of the

target study population, or at least the need to urge sponsors and investigators to consider

such a perspective when selecting a DHT.

Goldsack JC, Coravos A, Bakker JP, Bent B, Dowling AV, Fitzer-Attas C, Godfrey A, Godino JG, Gujar N, Izmailova E, Manta C.

Veriﬁcation, analytical validation, and clinical validation (V3): the foundation of determining ﬁt-for-purpose for biometric

monitoring technologies (BioMeTs). npj Digital Medicine. 2020 Apr 14;3(1):1-5.

https://www.nature.com/articles/s41746-020-0260-4

We appreciated the ﬂexibility in DHT use in clinical investigation during the COVID-19

pandemic and encourage maintaining a similar approach in the future. This ﬂexibility allows

for participants to have choice when enrolling in a trial and can encourage subject

participation and retention.

Lines 257-258 describe the use of technical speciﬁcations and descriptions provided by

the DHT manufacturer. While these lines indicate that these manufacturer speciﬁcations

and descriptions may be “sufﬁcient,” further guidance is requested to deﬁne the exact

materials needed to meet this sufﬁciency and limit the variety of interpretations possible by

DHT manufacturers and sponsors. Additionally, we suggest the inclusion of clarifying

language on using a DHT in an alternate manner to the DHT manufacturer’s intended use.

In regards to the Veriﬁcation, Validation, and Usability of Digital Health Technologies, we

champion the efforts to properly evaluate and document a digital measure and the DHTs

used to measure it. Our community has developed extensive, high-quality resources to this

end that have been adopted by nearly 100 companies and the European Medicines Agency.

We offer an overview of the resource here.

The evaluation framework for sensor-derived digital measures should encompass both

the product’s components (e.g., hardware, ﬁrmware, and software, including algorithms)

and the intended use of the product. Existing frameworks for new biotechnologies are not

sufﬁciently adaptable, but they can provide meaningful insight for developing new

evaluation frameworks for BioMeTs.

We propose a three-component framework to unite the different disciplinary experts who

should participate in the foundational evaluation of sensor-derived digital measures. This V3

framework includes (1) veriﬁcation, (2) analytical validation, and (3) clinical validation.

V3 are foundational to determine whether a digital medicine tool is ﬁt-for-purpose. An

evaluation of the usefulness and utility is only applicable after gaining evidence and

assurance that the underlying data and predictions are “valid” to answer a given question.

The V3 process is summarized in the ﬁgure below and described in detail in the manuscript,

“Veriﬁcation, analytical validation, and clinical validation (V3): the foundation of determining

ﬁt-for-purpose for Biometric Monitoring Technologies (BioMeTs)”. This manuscript was

developed by a multi-disciplinary group of experts with the goal of identifying

considerations for evaluating and documenting measurement performance of

technologies that generate sensor-derived digital measures. We hope it may be a useful

resource for the Agency.

Utilizing the deﬁnitions shown in the ﬁgure below provides increased ﬂexibility to use parts

of the modular stack of a DHT and not others while maintaining necessary rigor.

The stages of the V3 framework for biometric monitoring technologies

In addition to the offer of this resource, our digital medicine community requests speciﬁcs

about what to include in the veriﬁcation and validation plan, including the level of evidence

that is acceptable. We recognize that the variations in studies, measures, and technologies

makes for a particularly challenging request. Guidance around the general categories that

should be included in the veriﬁcation and validation plan, whether there is an expectation to

reference a gold standard, and what makes for a good reference measure (or examples of

such) are, perhaps, speciﬁc enough to be actionable. We also would encourage providing

examples of the evidence required when a DHT is software alone and when it is a

software/hardware/ﬁrmware combination.

We are pleased that usability is referenced, along with the Applying Human Factors and

Usability Engineering to Medical Devices guidance. We suggest the inclusion of additional

guidance regarding the critical factors that should be included in usability testing for DHTs,

Goldsack JC, Coravos A, Bakker JP, Bent B, Dowling AV, Fitzer-Attas C, Godfrey A, Godino JG, Gujar N, Izmailova E, Manta C.

Veriﬁcation, analytical validation, and clinical validation (V3): the foundation of determining ﬁt-for-purpose for biometric

monitoring technologies (BioMeTs). npj Digital Medicine. 2020 Apr 14;3(1):1-5.

https://www.nature.com/articles/s41746-020-0260-4

such as environmental variables on the sensor performance or human biocompatibility of

the hardware.

We appreciate the inclusion of a section focused on Evaluation of Clinical Endpoints From

Data Collected Using Digital Health Technologies. As mentioned above, we recommend this

section to be moved prior to Selection of a Digital Health Technology and Rationale for Use

in a Clinical Investigation and including guidance on determining a meaningful aspect of

health, identiﬁcation of concept of interest, followed by the development of the digital

endpoint.

We appreciate the approach the Agency takes in the draft guidance and agree that digital

medical product development tools should be no different than non-digital and request

clariﬁcation about how we justify use, speciﬁcally whether using existing biomarker and

COA guidance is sufﬁcient. Should they be sufﬁcient, a reference to these guidances would

be beneﬁcial.

Perhaps out of scope for this guidance, DiMe’s community notes that efforts to develop new

digital clinical measures should return value over and above existing assessments, for

example reduction of participant burden, higher quality data, cost savings, participant

adherence, new information, and appeal to a more diverse population.

As we explore Risk Considerations When Using Digital Health Technologies, we encourage a

risk-based approach, similar to that used by the FDA in other guidances. for other

endpoints.

We suggest the inclusion of language around risks to equity or, if appropriate, references to

other guidances addressing equity and inclusion. The use of DHTs in clinical investigations is

at high risk of excluding populations based on socio-economic status. DiMe has convened a

Digital Health Measurement Collaborative Community (DATAcc) to develop and

demonstrate best practices and advance harmonized approaches to speed the use of digital

health measurement to improve health outcomes, health economics, and health equity.

hope the resources developed and pending release are useful to the readers of this

guidance.

Lines 458-486 discuss Statistical Analysis. There is an opportunity to conform the method

of deﬁning endpoints to those of standard industry guidance. We recognize that the

diversity of sensors, algorithms, and methods make guidance difﬁcult, yet realize the need

for speciﬁcs on the requirements and optional supporting elements for deﬁning an

endpoint. For instance, deﬁning meaningful change, particularly when access to the patient

population is not available, and indicating whether the change in a referenced traditional

http://datacc.dimesociety.org/

endpoint could be used as a benchmark would provide valuable insight to the audience of

this guidance.

We recommend the inclusion of guidance related to missing or changing data due to

software upgrades or technical failures within the Statistical Analysis section. Included in

the required statistical analysis plan should be documentation regarding these missing or

changing data, such as missing timestamps, duplicate timestamps (i.e., during daylight

savings), short and long-term gaps in data, and even partially uploaded ﬁles due to

technical trouble or connectivity issues.

We are pleased to see speciﬁc guidance, such as those around providing instructions to

participants for cleaning the DHT (lines 506-508). Based on our interactions with

individuals who have participated in trials using DHTs, we want to make a note that cleaning

procedures should take into consideration the burden on the trial participants.

Cumbersome, lengthy, or frequent cleaning procedures can result in non-compliance by the

participant, leading to a lack of or altered data collection. DHT manufacturers, sponsors, and

investigators should take this into account and consider evaluating the cleaning protocol as

a part of usability testing.

Lines 515-518 note cybersecurity risks as a part of the Clinical Risks section. FDA may wish

to consider either a separate risks section dedicated to cybersecurity or highlighting the

impact of cybersecurity in each risk category to clarify the extensive reach of

cybersecurity threats.

Lines 536-539, 567-569, and 582-589 discuss important points related to data privacy,

data access, and end-user license agreements. We agree that participants must be made

aware of all entities with access to the data, measures to keep data safe, and that sponsors

and investigators should attempt to work with DHT manufacturers to navigate this process.

We encourage the inclusion of guidance that recommends starting as early as possible

(long before preparing informed consent documents) when working with DHT

manufacturers in understanding the end-user license agreements and their impact on data

privacy and security. We also recommend that relevant highlights from the end-user

license agreements be included in the informed consent document using understandable

language for the participants.

In the Record Protection and Retention section, the guidance states the requirement of

record retention in clinical investigation and emphasizes the need for source data

retention "to reconstruct and evaluate the clinical investigation, and the data should be

available for inspection." There is an opportunity to clarify the deﬁnition of source data in

the context of continuous sensor-based DHT. To derive digital measures from continuous

time series of sensor data, a pipeline composed of signal processing and/or machine

learning steps is typically deployed to transform the raw sensor data to the discrete data

features so that they can be used as endpoints in statistical models for efﬁcacy and safety

analysis. We believe that the retention of the raw sensor data, or its least processed form,

is essential to the use of sensor-based DHT in clinical investigation. This processing

pipeline, or algorithm, is not currently standardized for most sensor-based DHTs and

continues to iterate to keep pace with the rapidly evolving data science and technology

landscape. Processing pipelines can also be proprietary/black box and change during

ﬁrmware and software updates, imposing further challenges to data consistency.

As an example, sensors that use accelerometer data capture gravity (G) changes on multiple

axes. These micro-G measurements are captured many times per second and are the basis

for the resulting endpoints. The micro-G data would be considered the source data, not the

post-processed unit of measurement or calculated endpoint.

There is an opportunity to further clarify the deﬁnition of source data in the context of such

sensor-based DHT and recognize the importance of retaining minimally processed data, as

this offers the ability to investigate missing data, improve data consistency, and potentially

re-evaluate the clinical data when more accurate algorithms become available in the future.

Lines 616-624 discuss details around data outputs for the DHT. With a lack of

standardization around how source data is generated, we recommend that data with the

highest granularity, or least amount of processing, should be the default “source.” This

mitigates the most risk when manufacturers are generating data utilizing different

non-standard methods. The capture of the raw data or least processed data allows for all

other aspects of the data processing to be revisited or audited at any time.

In the DHT Updates and Other Changes section, lines 753-755 discuss the assessment of

updates to a DHT to ensure no signiﬁcant impact on the measurements performed and

data collected. Our digital medicine community requests clariﬁcation on evidence needed

around the impact of updates when assessing for consistency. For instance, is

documentation from the software manufacturer stating backward and/or forward

compatibility sufﬁcient or is additional evidence needed? Further clariﬁcation regarding the

difference between a manufacturer’s software update for something like a security patch

and an update impacting the use of the DHT for the trial purposes is requested in terms of

the evidence needed to support the “no signiﬁcant impact” conclusion. Our community

follows the National Academy of Medicine deﬁnition of high quality data, which is data that

leads to the same conclusion.

We wonder if this aligns to your classiﬁcation of “no

signi

ﬁcant impact” and, if this language might provide clarity to readers.

Lines 762-772 discuss software and operating systems updates. Dedicated solutions are

preferred during active data collection of a DHT. Clear guidance on the types of updates a

manufacturer can push to production will allow sponsors to accept or decline these types of

updates. Certain updates that might be a "feature enhancement" could be opted out by the

sponsor to mitigate risk. While any updates on Security, Data Integrity, and Privacy are

clearly documented and approved by sponsors for "live" clinical trials using a DHT.

Conclusion

We recognize the effort on this draft guidance regarding DHT use in clinical investigations

as an impactful, helpful guidance that instructs and assists DHT manufacturers, sponsors,

investigators, and regulators alike. It is no small feat to provide clarity to a young industry

with a wealth of unknowns. We desire to assist the

ﬁeld of digital medicine by providing

quality resources, best practices and use cases, and forums for valuable connections that

can drive the ﬁeld forward productively and with patients always in mind.

We appreciate the efforts to create a broadly applicable guidance that does not overburden

the industry with regulations and we hope our comments are helpful in improving this

guidance. At DiMe, we are committed to the safe, effective, ethical, and equitable use of

digital technologies in clinical investigations and see through this guidance and our

interactions with FDA that the Agency shares this commitment.

Thank you,

Jennifer Goldsack, MChem, MA, MBA, OLY

CEO

Christina Marasco, PhD

Program Lead

Institute of Medicine (US) Roundtable on Research and Development of Drugs, Biologics, and Medical Devices; Davis JR,

Nolan VP, Woodcock J, et al., editors. Assuring Data Quality and Validity in Clinical Trials for Regulatory Decision Making:

Workshop Report. Washington (DC): National Academies Press (US); 1999. Executive Summary.

https://www.ncbi.nlm.nih.gov/books/NBK224572/