What is happening to the technologically outdated eHealth and why are there disruptions?
The Centre of Registers, which provides maintenance of the eHealth, says that to achieve stability and to completely avoid massive system failures, it is necessary to decompose the current structure of the eHealth system and move to a modern data exchange standard. This would improve the performance and speed of the whole system and ensure its sustainable and efficient development.
The eHealth system, which was developed about fifteen years ago, is the largest and most complex system in the country – it currently connects almost 3,000 healthcare institutions operating in the country, over 1,000 pharmacies, and it handles about 5 million requests per hour from 16,000 different IP addresses. The system currently has more than 500 terabytes of data, and about 400,000 electronic documents, including 100,000 e-prescriptions alone, are added to the system every day.
“These figures show the complexity of the system. Maintenance of any system consists of two parts: development, i.e. deployment of new functionalities, and technical support, including updates. Today, there are around 14 eHealth projects in progress, of which only one focuses on technological upgrade – decomposition of the system. However, even this project is still in the planning stage,” said Antanas Raguotis, IT Director of the Centre of Registers. “Standard system maintenance practice requires that systems are technologically updated at least every 7 years. In the case of eHealth, this has not been done for about 15 years, and the technological architecture of the system is still the same as it has been since its creation. One way to tackle such archaic monolithic systems is to decompose the system down into separate and independent parts.”
According to him, the complex technological set-up of the system, with different integrations of healthcare institutions, pharmacies and other institutions, increases the probability of disruptions, and in case of disruptions, it is quite difficult to identify the cause of the disruption and to resolve it promptly. For example, a disruption in the transmission of data on reimbursable medicinal products could lead to doctors not being able to prescribe the reimbursable medicinal products patients need, or patients not being able to buy them in pharmacies.
Designed for data exchange, but has become a document repository
The Ministry of Health, the owner of eHealth, has focused on the development of the system’s functionalities over the last decade, i.e. creation of patient, specialist and pharmacist portals, various new medical forms, a patient appointment management subsystem, etc. But not enough attention has been paid to technological development.
“The architecture of the eHealth system is very complex, developed about 15 years ago, when probably nobody imagined the level of complexity that the system would reach in a few years’ time. The original idea of the system was to create a data exchange platform, but over time the system has shifted from this purpose and today serves as a repository for data and electronic documents,” said A. Raguotis.
According to him, the eHealth system has a number of integrations, through which data are exchanged between different institutions operating in the healthcare system – hospitals, polyclinics (directly or through external systems used by the institutions), pharmacies, the National Health Insurance Fund, ‘SoDra’, the State Medicines Control Agency and others.
“Data travels between these systems in different ways – some exchanges take place in real time, others take place at night, some institutions need a small amount of data, others need large amounts. Thus, this whole ecosystem of data exchange is quite complex and heterogeneous,” says the IT Director of the Centre of Registers. “Modern systems are built on data, not documents. In contrast, eHealth is document-based, which means that there is very little pure data and the descriptive information contained in documents takes up a lot of space in data depositories.”
Inadequate system size and archaic structure lead to incidents
Currently, the eHealth database contains 530 terabytes of data, which would take three days to restore it from backups. Over 100 million new electronic documents are added to the system per year. In comparison, these numbers were half as high five years ago.
“The amount of data in eHealth has increased drastically recently. The turning point came during the coronavirus pandemic, when the need for electronic health records — tests, vaccinations, certificates, etc. — increased dramatically. In the world, large systems are considered to be those that occupy 10 terabytes, and in this case we are forced to deal with a system 50 times larger. This is a big challenge, as any update or addition to the system can cause very serious technical problems,” added Raguotis.
He said that, in terms of the prospects for eHealth renewal, it is important to note that only the technological decomposition of the system is a key prerequisite for avoiding future disruptions. It is also necessary to move to the latest data exchange standard, as the current standard is significantly outdated and does not allow for optimal data exchange speeds, complicating system development.
Changing wheels on a moving car
The complexity and size of the eHealth system means that any large-scale development work faces significant challenges. The current monolithic system architecture is archaic since its launch, and therefore, a decomposition of the eHealth system is necessary.
“We believe that decomposition of the whole system – breaking down individual parts of the system into separate modules, such as e-prescription, patient appointment management, laboratory tests, etc., would help to avoid massive system disruptions and would improve the system’s performance and speed. This would significantly reduce critical incidents, as a failure in one module would not affect other modules,” noted the IT Director of the Centre of Registers.
According to him, global practice shows that decomposed systems provide greater reliability, speed of development and cost savings.
“With the current monolithic system architecture, introducing new functionality while keeping the system up and running poses risks of disruption and performance issues. It is nearly the same as changing the wheels on a moving car. In order to overhaul the system and achieve optimal performance, we need to refrain from introducing new functionalities for a while and prepare for the decomposition of the system,” said A. Raguotis.
What happened?
Two weeks ago, large-scale changes were made to the system, including both technical improvements and the introduction of new functionalities. This implementation has taken several years of preparation, hundreds of different scenarios have been tested, involving healthcare institutions, developers of the information systems serving them and pharmacies.
“Nevertheless, it is not possible to reproduce any scenarios in the test environment that would allow us to test the functionality in real-life conditions. That is why in the first few days after the upgrade we encountered problems with the system’s performance and speed – the large number of requests increased the database load, which slowed down the system’s operation and, in some moments, caused a complete disruption,” he added.
Preparations were made for such a scenario and at the first signs of system slowdown, a team of several dozen specialists was activated, external partners were brought in, and consultations were initiated with representatives of the international company Oracle, whose database management system is used for the eHealth system.
“After identifying the most resource-intensive queries and functionalities in the system, we immediately implemented software changes. Unfortunately, not all of them have had the desired result, and this monitoring and improvement work is continuing. It is normal for such major system upgrades to be followed by a period of several weeks of intensive maintenance, during which some glitches may still occur. However, we assure you that in cooperation with the companies operating the systems of healthcare institutions, the medical community itself and representatives of pharmacies, we will continue these maintenance works and will strive to optimise the work of the system as soon as possible,” ensured the representative of the Centre of Registers.
According to him, immediately after the large-scale upgrades, more intensive communication with the companies maintaining the healthcare information systems and pharmacies was started, with three daily updates on the situation regarding the exchange of data between them, identifying problems and finding solutions to them.
It is important to note that the large-scale upgrades carried out by the Centre of Registers were not limited to the central eHealth system. At the same time, one of the integration systems, Foxus, developed by Softdent, was also upgraded. It has been noted that the most complaints are from doctors using this integration system. The eHealth system involves integration of the information systems of the country’s healthcare institutions, so the uninterrupted operation of the whole system depends not only on the central eHealth system, but also on the maturity level of the integration of the information systems of the healthcare institutions in the broad network of eHealth users.
“At the moment, we are working as resuscitation specialists in the maintenance of the eHealth system – we are constantly trying to keep the system alive. However, a qualitative leap and new work standards will only be achieved through long-term technological solutions, which we must make now,” said Raguotis.