- This topic has 21 replies, 12 voices, and was last updated 2 years, 1 month ago by
.
-
Posts
-
-
Based on the publication by Silenou et al., the newly deployed SOMRAS appeared to have better performance compared to the conventional surveillance system for Human Monkeypox Outbreak in Nigeria. Nevertheless, the authors mentioned that having limited numbers of evaluation attributes was one of the shortcomings of the evaluation.
Propose two additional evaluation indicators that are relevant to this setting. For each indicator, briefly explain the evaluation methods that could have been used in the given scenario.
-
Interesting to explore multiple features of mobile health surveillance system like SORMAS for monitoring Monkeypox outbreaks in Nigeria. After scanning the article, I catch several attributes when the SORMAS had been being evaluated by Silenou et al.:
1. Timeliness of case reporting
2. Response time
3. Assurance
4. FunctionalityApart from the shortcomings of its credits, let me add two more potential attributes which can enrich the quality of surveillance evaluation:
1. Acceptability: the speed of data transfer, function, and confidence from the mHealth surveillance directly led to the user’s acceptability. After system deployment, it is better for the author to evaluate user satisfaction, perception, experience, and barriers through interviews with district surveillance notification officers (DSNO). A qualitative method with snowball sampling can be carried out for interviewing selected DSNOs. If the authors acquire positive feedback, all gathered information is worthwhile to mainstream the importance of SORMAS for disease surveillance.2. Portability: it is better to compare SORMAS with conventional surveillance according to the attributes of system architecture and hardware configuration. The author can conduct system and data review method to explore the strengths of this system. I believe that the gap between both systems is certainly huge, and this knowledge is beneficial to look at the possibility of whether SORMAS platform can be used elsewhere.
-
-
The SORMAS system for
1.rapid case-report visualization and analysis;
2.quickened the rate at which information is updated and daily reports are generated;
3.enhanced the accuracy, timeliness, and utility of the information; and
4.streamlined a number of processes critical for combating monkeypox.Although SORMAS system as a whole was recommended by the author, there are two additional indicators, that should be included in future for evaluation,
1.Integration
There are some shortfall of SORMAS. The availability of data from the conventional system hampered the comparison of data completeness between SORMAS and the conventional system. Due to a lack of integration, dual systems had to be maintained. The full potential of the SORMAS system, which includes continuous surveillance and response management but has not yet been employed consistently, was not available as the outbreak unfolded.Communication
Additional difficulties include DSNOs’ claims that they were not compensated for travel expenses incurred as a result of doing follow-up visits for contact tracing, which could lead to inaccurate data regarding transmission chains. SORMAS may have helped to alleviate it because it created transmission networks that the conventional system lacked but the need to communicate the feature of SORMAS to stakeholders is needed. Health systems need people and buy-in from people is essential for success of a system.-
Thanks for pointing out these additional attributes!
I totally agree that integration is one interesting aspect. Harmonization between two systems or data transfer between the old to the new system is almost always a time-consuming process. While doing so during an outbreak is probably not on the top of our priority list, it needs to be done at some point to free the system from the dual burden.Nicely done – keep it up!
-
-
The Surveillance, Outbreak Response Management and Analysis System (SORMAS) was designed for reporting, analysis, and visualization of the outbreak. In addition to the evaluation indicators that investigators has been assessed, I would like to add some evaluation indicators for this system.
1. Responsiveness for user experience and service quality
Because the success of this system depends on people who use it, there should be prompt service support if the user faces some problems while using the system. We can assess the responsiveness by sending some questionnaire or interviewing some users about system support if there are errors of the system or if they need some instruction to use the system (since there was limited time for introducing the system to the officers).2. Data quality for system quality
We can check the data quality from two components. First, we can ask the users about variables included in the system whether there are too many, too few, or proper number of variables. Second, we should assess functions of the system for preventing data errors. For instance, the system should design the input interface for receiving possible variable, the out-of-range value cannot be entered to the system, all of the required variables should be filled before they submit the report.Thank you.
-
Thanks for pointing out these additional attributes.
Good job pointing out the different facets of evaluation for data quality.
I also agree that they could do further evaluation of the data apart from the completeness.
Evaluating the system from many aspects could give us more insight into the system as a whole!Nicely done – keep it up!
-
-
From my point of view, this SORMAS project is awesome, it can be deployed in only 14 days. they conducted qualitative interviews with the NCDC incident managers about the timeliness, usefulness, and workload of the conventional system compared with SORMAS. Moreover, qualitative evaluation for completeness was also done. If I can add indicators, I would like to add indicators as follows:
1. Data consistency: Since the conventional system used a lot of people and mediums(post and Excel) to send data at each stage, I think there would be a higher error in the data than SORMAS. Verification should be done by checking the NCDC data against the log data from the source to check for errors or duplications.
2. User experience and service quality: Since there are only two days of training for staff who use the SORMAS, it is likely to encounter problems during work. There should be indicators that will be used to improve the system even further such as user satisfaction questionnaires, and a summary of user reports regarding problems of the system.
-
Thanks for your insightful comments on the additional attributes.
Irrefutably, their effort in launching the SORMAS was impressive! Additional evaluation such as the consistency of the data and user/service experience that you mentioned would further refine their evaluation of this digitized system.
-
-
Rather than focus only on information quality for surveillance system. In order to make a successful system implementation, there are also system quality and service quality that matter to users’ satisfaction. The additional evaluation indicators that I would like to propose:
1. Availability for system quality. To see if the system is available at all times for users’ need regardless time and location. Yet, to see whether the system can be accessed from different kind of devices. The availability of the system could potentially effect the successful system implementation.2. Acceptability of user’s experience. Since most of users were familiar with conventional system, acceptability evaluation should conduct to evaluate users’ satisfaction and ask them if there are any barriers or problems that make them don’t want to use the system. In addition, this can be implied the service quality whether the team has to improve for the better system quality.
-
-
The additional evaluation indicators that I would like to propose are usability and responsiveness.
1.Usability for system quality
According to the article, they run both SORMAS and conventional system in parallel. We can assess usability by reviewing the system and data input, checking number of usages by different level of users. From the user interview and survey of user’s satisfaction, the acceptability of user might also represent the usability2.Responsiveness for user experience and service quality
The SORMAS was conducted in 14 days and training session for users even shorter. After implementation, they could have some problem occur and need assistance on the system. We should be available to respond to their questions. So, we need to know whether they receive prompt service when requesting or not. We can assess by conducting in-depth interviewing the manager of each stakeholders such as NCDC, the laboratory center and also sending out the questionnaire surveys to all the users. Besides that, we can look at the user’s reports regarding the system support in terms of user’s satisfaction. -
-
SORMAS system is quite a success from my perspective. It can be deployed with in a short period of time and provided a short training period / low learning curve. I have seen a sample user interface in a provided paper and it is very simple, provides comprehensive visualisation graphics and it has also been used for other diseases as well (as seen in figure 1, there are cholera, dengue, measles, new flu and so on).
Additional indicators for SORMAS in terms of evaluation its effectiveness should be about
(1) User satisfaction; because it has been proven that SORMAS system is superior than the conventional surveillance system. It also has a high potential to be widely implemented in the future for another outbreak to come. By taking user experiences into consideration, the system can improve further.(2) Integration; As the authors stated that they have to deployed SORMAS in the midst of the outbreak without prior plan at the first place, they had to run two separate systems simultaneously. It would be great if we, or anyone, has proposed some kind of “data standard” for outbreak reporting system for interoperability usage regardless of any platform or system they use for outbreak surveillance.
-
-
Thanks for your insight and discussion. I totally agree that the successful implementation of a new system in the middle of an outbreak is already made their mission an impressive one! About the data standard, some case definitions and data collection formats have been proposed by US CDC; however, the implementation could be vastly different in a challenging setting like Nigeria!
FYI:
https://www.cdc.gov/poxvirus/monkeypox/clinicians/case-definition.html
https://www.cdc.gov/poxvirus/monkeypox/health-departments/case-reporting.html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fpoxvirus%2Fmonkeypox%2Fhealthdepts%2Fcase-reporting.html
-
-
From the publication about SOMRAS system, there were a few attributes that had been evaluated. Information quality, Completeness and Timeliness had been measured by percent completeness of data available and average time for data processing respectively. System quality has been described by functions of the system such as availability of dashboard and ability of workload handling. There are other evaluation indicators which could be used to further system improvement. I would suggest two further evaluation methods below:
First, Quality of data based on system quality can be investigated. Even though a higher percent of completeness had been received from using SORMAS compared to conventional system, quality of data sent hasn’t been checked. Quality of data can be checked by many methods such as reviewing previous records. Processing date might not be ordered or out of range date records might be found. Moreover, users interviewing regarding variables collected can be used to ensure the appropriateness of variables collected, staff at healthcare units might have ideas for better data collection which benefits surveillance. These could lead to system improvements such as logical to detect date order, reminder for data input and adjusting suitable variables.
Second, Assurance of users could be measured as well. Because of many reasons such as system expanding, system updates, numbers of staff and staff turnover, it is better to make sure that staff understand and are able to use the system correctly. There could be staff interviewing about the need of training support or quizzes to recheck system understanding. Further, knowledge support could be developed from this assurance evaluation. -
-
SORMAS stands for “Surveillance, Outbreak Response Management and Analysis System”.
According to the article multiple attributes were used to evaluate this system including average time data arrive CDC, average time to update data per case, workload to transfer cases from paper to database, availability of dashboard and statistical module.
In my perspectives this evaluate are mainly focused on the system quality. As this system was implemented for detecting outbreak. I think information quality is one the major attribute to be concerned as well.
Therefore, the completeness of information should be checked. The %missing field or %invalid response could be calculated to ensure that the data quality is maintained.
Apart from the quality, the user experience should be evaluated as well. As mentioned in the article that the system was implemented in very tight timeline with limited time for training. I think there would have the comments and/or concerns from user. The %missing form and/or %form response over time should be measured to ensure that the user was familiar and willing the use this system.-
Thanks for sharing your idea! It is interesting that you mentioned the user experience in relation to the timeline. I also wonder about their opinion on the initial training during the outbreak as well. On one hand, successfully carrying out the training in such a short period of time could be viewed as a success; however, there might be some loopholes that get detected from the user side due to the overtly rushed process as well.
-
-
As reviewed in the publication by Silenou et al., there are some attribute evaluations have been performed for the newly deployed SOMRAS compared to the conventional surveillance system for Human Monkeypox Outbreak in Nigeria, such as timeliness, completeness, and the user’s workload to transfer cases from paper forms to the database. The result seems SOMRAS has better performance. The additional evaluation indicators that I would like to propose are in the User’s experiences and Service Quality domain; including the user’s empathy and acceptability. Both indicators could be evaluated by interviewing the users and providing questionnaires regarding their experiences using the system. The better quality of the system is driven by the users’ feedback. Understanding the users’ obstacles in participation and report completion would help modify the system according to their needs. Thank you.
-
You must be logged in to reply to this topic. Login here