Robotic Systems in Operating Theaters: New Forms of Team–Machine Interaction in Health Care

Abstract

Background Health information systems have developed rapidly and considerably during the last decades, taking advantage of many new technologies. Robots used in operating theaters represent an exceptional example of this trend. Yet, the more these systems are designed to act autonomously and intelligently, the more complex and ethical questions arise about serious implications of how future hybrid clinical team–machine interactions ought to be envisioned, in situations where actions and their decision-making are continuously shared between humans and machines.

Objectives To discuss the many different viewpoints—from surgery, robotics, medical informatics, law, and ethics—that the challenges of novel team–machine interactions raise, together with potential consequences for health information systems, in particular on how to adequately consider what hybrid actions can be specified, and in which sense these do imply a sharing of autonomous decisions between (teams of) humans and machines, with robotic systems in operating theaters as an example.

Results Team–machine interaction and hybrid action of humans and intelligent machines, as is now becoming feasible, will lead to fundamental changes in a wide range of applications, not only in the context of robotic systems in surgical operating theaters. Collaboration of surgical teams in operating theaters as well as the roles, competencies, and responsibilities of humans (health care professionals) and machines (robotic systems) need to be reconsidered. Hospital information systems will in future not only have humans as users, but also provide the ground for actions of intelligent machines.

Conclusions The expected significant changes in the relationship of humans and machines can only be appropriately analyzed and considered by inter- and multidisciplinary collaboration. Fundamentally new approaches are needed to construct the reasonable concepts surrounding hybrid action that will take into account the ascription of responsibility to the radically different types of human versus nonhuman intelligent agents involved.

Keywords

robots - health information systems - operating theater - team–machine interaction - hybrid action - health care

Introduction

How conscious and “free” are decisions in a pervasively digitized work environment like a robotic operating theater? How will self-perception and self-conception change when decisions and physical actions of health care professionals heavily rely on robotic assistants? Is it empirically feasible and normatively justifiable to refer to a particular physician as “having the last word” in such a team–machine interaction? What ethical and legal consequences for self-conception, responsibility, and liability arise? Within our current age of digitization, which is bringing enormous changes to health care,[1] [2] [3] [4] such questions have to be raised also in the context of how this will impact and be impacted by the design of health information systems (HISs).

HISs have seen incredibly rapid development during the past decades. For comparison we can refer the reader to the seminal paper of Peter Reichertz[5] on hospital information systems, their architectures, functionalities, and challenges in the 1980s, to Haux[6] for developments up to the first decade of the 21st century, and to Arnrich et al[7] Friedman et al[8] and Winter et al[9] for more current observations with a focus on pervasiveness, strategic management, and learning. In Haux's work,[6] a seventh important line of development for HIS is noted as “the steady increase of new technologies to be included, now starting to include ubiquitous computing environments and sensor-based technologies,” using terms like “active environments” and “sensor-based ICT.”[6]

Outstanding examples in this trend of intensively including new technologies are operating theaters, where robotic systems are increasingly supporting surgical teams, way beyond providing simple tools. The more these robotic systems act autonomously and intelligently (meaning, among other things, providing the basis for human decisions), the more questions need to be raised about the implications for clinical workflow and for revising team work that includes machines as well as humans.

In this article, we want to address one of the key questions that needs to be answered in this context: Does this new hybrid action between humans and machines imply a sharing of autonomous decisions between (teams of) humans on the one hand and machines on the other hand and if so, what consequences can be foreseen from this?

Underlying questions with important consequences include: which work processes will change through the use of robots in operating theaters? What do patients expect and will they benefit? Will there be new requirements for surgeons' and how they deploy their skills to work successfully with robots? What additional technical know-how will be needed for them and for nurses in surgical teams? How do roles, responsibilities, and communication styles change? Who is authorized to include the robots in surgery? With additional data, coming from robots, are new criteria for quality management needed, and if so, set by whom and to whose advantage will that be? That is, who determines what is “good surgery” and who are “good surgeons,” and what norms and deviations from norms are professionally, legally, and ethically tolerable?[10]

Objective and Structure

Discussing “highly original and relevant research fields for biomedical and health informatics”[11] and stimulating “multidisciplinary communication on research that is devoted to high-quality, efficient health care”[12] is one of the major goals of Methods of Information in Medicine.[6] [13] [14] [15] [16] [17] [18] [19] [20] New forms of team–machine interaction in health care with robotic systems in operating theaters are an example of highly original and relevant human–machine interaction.

Our objective is to discuss from several viewpoints the challenges of such new team–machine interactions and their consequences for HISs, in particular on how to adequately take into account hybrid action. We will present our thoughts on this topic by using robotic systems in operating theaters, and, as far as we know, this is the first time that this topic is addressed in a multidisciplinary way.

The mentioned challenges are presented from surgical, robotics, information systems, legal, and ethical points of view. The present contribution elaborates on the authors' presentations at a recent related symposium.[21]

On the authors' contributions to this manuscript, J.S. and D.F. respectively initiated and catalyzed our discussions on the topic. R.H. is responsible for “the information systems viewpoint” section and for the overall organization of the manuscript. D.F. is responsible for “the surgical viewpoint” section, J.S. for “the robotics viewpoint” section, S.B. for “the legal viewpoint” section and A.M. for “the ethical viewpoint” section. For all other sections, the authors are jointly responsible.

The Surgical Viewpoint

Team interaction is highly important in operating theaters. Every team member has his or her own work assignments, but communication is required for a good interacting team[22] to obtain the best possible surgical results and outcomes for the patient.[23] [24] When a robotic system is introduced in this established situation, significant changes in professional roles, interactions, and communication can be expected,[24] [25] furthermore the former team interaction is radically changed into a team–machine interaction.[22] [26] Possibly, there will also be changes in surgical procedures, due to the requirements of robotic systems.[27] Because of this team–machine interaction, all coworkers will have to learn new skills, as well as to integrate machines as team members.[22] [24]

Physicians, especially surgeons, believe that good doctors need to accumulate a lot of experience to become a high-level practitioner.[28] They perceive their profession as an art, a challenging craft that requires a lot of creativity, courage, and determination. When analyzing complicated situations within a surgery procedure, including cooperation requirements in operating theaters, much beyond precision alone is required. There are situations that need nothing less than creativity. At the moment, we solely attribute creativity to humans.[29] Thus, if machines are able to substitute surgical work, if only in part, this fundamentally questions the self-conception of surgeons. It will likely deeply influence the professional behaviors, self-conceptions, self-perceptions,[30] self-efficacy, and the teamwork structures in health care.[31]

The medical profession and its fields of action will change dramatically, but so will the work profiles of other professions, associated with physicians, especially in operating theaters.[22] These professions seemed largely irreplaceable until now, because of their specialization. But general conditions in health care systems and the requirements in operating theaters in particular will likely change considerably and continuously in the near future. New professions may be forming and well-known ones may disappear. In part, they may be replaced by intelligent technology,[32] i.e., algorithms and machines that have control functions and take over responsibilities for reporting and evaluation. Such machines may be more precise in measuring and acting than humans are[33] and they can perform in a more reproducible manner. Furthermore, by using machine learning or artificial intelligence (AI), they may resolve problems more efficiently than humans can, and they should be designed so as to not harm patients (precautionary principle).[34] [35] But, can we really count on this kind of seamless embedding of machines in the surgery team work or is this only a simplified projection of our technological wishes? Much data must still be collected and new educational environments and scenarios will have to be developed to fulfill these visions as “next decade medical solution.”[36]

Furthermore, it is necessary that the use of machines is based on good evidence,[22] [24] even more than with any other new surgical technique.[37] Therefore, new techniques including robotic or other digital devices need to prove superiority over, or at least equivalence to, established conventional processes and procedures. Claimed benefits of robotic surgery include not only miniaturization, but also the “virtual reality” ability of giving the surgeon the feeling of being within the body of the patient, e.g., the abdomen or chest etc., to provide navigation in small spaces with the best possible view, and also improved overall control from consoles and through communication techniques.[27] For now, however, a first and most basic necessity is to conduct studies on when and how machines should be used in operating theaters.[22] [26]

As of today, most people do not want to be treated by fully automated systems, nor are surgeons that confident in delegating their existing responsibilities to autonomous machines. Patients mostly require the contact with a “real-life” physician carrying out the decision-making processes in their individual case.[38] To date, the majority of people in European countries, where health care systems in general work well, are not willing to be treated for major surgery just by robots instead of human surgeons.[38] Nevertheless, the use of information and robotic assistants is increasing and more widespread and, in the future, one might speculate that it might even be rated as the better practitioner. But it is important to define what we mean by a robot: is it a fully autonomous system or an integrated robotic system, interacting with humans and with other systems (electronic patient files, electronic hospital information system, complication registry, searchbots for integrating best publications, data resources on surgical/medical experience, etc.) to optimize the surgical result, through interaction processes, as a kind of smart assistant?[38]

Thus, introducing robotics and smart information processing in operating theaters rises at least the following basic questions:

• How will responsibilities change with the use of robots in operating theaters?

• How and when might it be safer for patients, if robots do the work?

• Is it necessary that robots be supervised by surgeons or even new health care professionals?

• Who will do surgical trouble-shooting for the many possible complications of surgery, if surgeons will only care for machines? Do we need specialized teams, trained in open surgery to intervene in case of problems?[39] A dramatic situation that has to be managed is, if machines make fatal errors, e.g., system breakdowns,[35] [40] operating crews could be completely blocked, and patients could be in great danger. In this case, all staff should be able to react by de-docking of robots and continue and finish the operation using conventional operating techniques,[23] e.g., traditional microinvasive surgery or open surgery. How can surgery teams be trained to prepare for such situations, especially when decreasing expertise in conventional surgery might follow from increasing robotic surgery?

• What about other professions surrounding the surgeon: anesthetists—should not robots also do anesthesia? Nursing in operating theaters—should not robotic systems, for example, change instruments automatically? And furthermore: intensive care unit staff—could not they be replaced by robotic systems in performing surveillance?

• Who will work in future in operating theaters? Operators, technicians, assistants, supervisors—will they need to have medical as well as computing or technical experience?[22]

• Is it promising to work with robotic systems as training partners or as educational tools[36] in a more controlled context? This will only happen if robotics is developed for conventional surgical techniques, rather than only microinvasive surgery[27] as is predominant today.[38] But this would broaden all the previously discussed problems over all disciplines of surgery.

• On the level of data processing, the digital nature of the robot's system control functions generates detailed data about and within the procedure. It creates highly sensitive electronic traces of the patient on the one hand and of the surgical team on the other hand ([Table 1]). This raises many open questions such as: who will be the owner of what parts of the data, what will be done with them, and will they be used with beneficial or detrimental intentions, and for whom?[22] The discussion of these questions cannot only consider economic evaluations[41] which, in the worst case, could lead to limitations in the quality of patient care and security.

Table 1 Electronic data traces from robotic surgical procedures: patients and surgical teams. Topics that should to be discussed on the background of introducing robotic systems in operating theaters

Patient	Surgical team
Controllability
Intraoperative actions	Efficiency of surgical procedure, time management, etc.
Quality of work (e.g., suture, etc.)	Selection of approved surgeons
Treatment improvement in conjunction with outcome parameters	Improvement of training of skills, speed of surgical performance
Accessibility
Quality of surgical outcome: organ functioning, mobilization, social/familial reintegration, etc.	Quality of surgical outcomes and complications: wound healing, pain, bleeding, etc.
Evaluation of the result: activity of daily living, function, aesthetics, satisfaction, etc.	Evaluation of the result: organ functioning, functional limitations
Recovery/convalescence	Time management efficiency
Resource consumption
Who will be treated and who will benefit from the procedure?	Material, time, staff

To summarize, robots may be beneficial, if studies prove their advantages in operating theaters.[42] These studies ought to yield tools that evaluate if the technology improves patient care and how to avert harm from surgical teams.[22] A new structure of professions with new skills is most likely to arise. But for the case of dangerous situations for patients, teams should still be trained[43] in conventional surgical techniques.[23] Economic aspects are important.[28] Nevertheless, they are not the most important ones considering the introduction of new team–machine robotic systems in operating theaters.[29] Medical, technical, social, political, ethical, and legal aspects as well as patients' views and wishes are very important and need to be discussed and analyzed. But a major overarching question in this discussion is, who will define quality criteria and for whom?

The Robotics Viewpoint

The operating theater is a particularly complex, interesting, and relevant domain to investigate the interplay of robotics and advanced information technology and their interaction with the different types of users, which range from the surgeon to the hospital's management level. Evidently, the digitalized operating theater is a highly sensitive environment and relevant in terms of the implications of novel technologies.[44] It has a history of using robots in the spirit of full automation[45] [46] and robotic surgery is a very active field of technical research.[47] Thus there is a richer literature on the impact of technology in the operating theater than for other fields of medicine. There are increasingly powerful assistive systems for all levels of physical action, information processing, and decision-making, both in the surgical suite and beyond. It has, for instance, already been argued that a high degree of pervasion with robotics and AI makes the assignment of responsibility in the operating theater difficult,[48] when decisions are prepared, supported, or taken through data processing systems. The increasing emphasis on using machine learning and other AI methods which are difficult to understand strongly contributes to this challenge. It has also been argued that there is the trend away from fully automatic robot operation to continuous guidance and assistance,[46] which we believe will change the nature of the human–machine interaction significantly. In physical assistance, continuous interaction takes place, where the robot suggests, guides, or corrects the human's action in an ongoing manner over longer time spans. In decision-making, continuous assistance occurs through the real-time processing of the relevant data for conducting the procedures. The division of labor between machines and humans is even less clear when work is performed in a team like in the operation theater where cooperation and communication is mediated by information displays and where roles, hierarchies, and the value of specialized expertise change. By providing the technical communication framework and real-time information, the technology then also strongly contributes to a common ground for teamwork. Overall, this can strongly blur the boundaries between the human and the machine and we denote this type of continuous team–machine interaction as hybrid action.

We here focus on the use of the DaVinci system, although similar considerations hold for other advanced robotics as well. The computer-based application system, displays, and the robot arms of the DaVinci system provide assistance to the surgeon and the team, and de facto also strongly influence the structure and organization of all surgery teamwork and the procedure itself. Nevertheless, the vendor claims that the surgeon is solely and at all times in full control of the robot, which thus is regarded as an advanced but teleoperated tool, which also provides the basis for its certification. If one follows this account, there is no problem in assigning responsibility and roles between the robot and the operating human. This standard treatment, however, falls short of accounting for the complexity of hybrid action in the operating theater.

We here argue that in a broader context including the teamwork in the operating room, the picture is not that simple. Hybrid action as we see it rather blurs the border between humans and machines and is in the vein of earlier notions of hybrid systems[49] and hybrid agency.[50] It does not fit in the scales of the standard and much discussed notion of robot autonomy,[51] [52] which assumes a clear distinction between the machine (i.e., automation system, robot, software agent, decision assistant) and the human, with both sides carrying out their tasks mostly independently. Typically, the human is required to set the overall goals and often is supposed to be ultimately in charge of the final decision, i.e., to have the so-called last word. Notions of mixed initiative control,[53] shared autonomy,[54] or supervised autonomy[55] have been introduced to denote the increasing amount of cooperation between artificial agents and humans about how and which parts of a task shall be executed and by whom, but they still assume a clear and strict distinction between the robot and the human. Shared control and shared autonomy are also standard concepts in telerobotics,[56] mostly denoting the superposition of control signals from human and the robot, which also is the case for the DaVinci system. The latter is regarded noncritical as long as the human still sets the goals. In this context, the robot in the operating theater falls in the class of assisted teleoperation, which implies a very low degree of autonomy.

This ignores, however, the increasing pervasion of the operation theater with the respective information systems,[44] where information systems are used in a broad sense to comprise both dedicated expert systems as well as the even more pervasive background hospital information and data processing systems, including those accompanying an assistive robot. Note that this comprehensive embedding of the physical technical system, here the robot, in the networked information systems and its enhancement with powerful tools from AI is in general seen as necessary and desired to realize a specific “intelligent function” of the robot. And we expect that this will hold all the more for future more advanced AI systems that include even more decision-making and will routinely touch all parts of the “sense–plan–act” cycle. It is therefore not an option to limit ourselves to simpler systems without such a function. Thus, it stands to reason that it is necessary to consider what it means to receive continuous assistance for the human and what the respective hybrid action implies in particular for competences and capacities.[57]

The interplay between robotics and advanced data processing enables also a meta-level for introducing machine learning and other advanced data analytics. It can, for example, safely be assumed that through a larger number of robotic installations scaling effects appear that allow for aggregation of data across HISs, robots, and local data infrastructure. Some trends include, for instance, the collection of robotic and haptic data from different robots and surgeons to investigate automatic skill evaluation[10] by means of data-driven learning systems.[58] The latter is an example of the trend to develop novel quantitative measures of manual skills through assistive robotics. This may have far-reaching consequences, as it may become feasible to provide online evaluation and case-by-case feedback in an unprecedented manner. It can be assumed that there will be much interest from stakeholders to use automatic skill evaluation to reduce costs and possibly to introduce quality-based payment schemes. In another vein, too much assistance may lead to skill deprivation and loss of competences.[57] Automatic and assistive systems implicitly set norms and it will be difficult to ensure that the inevitable mistakes that novices and apprentices make will be tolerated in a highly assisted environment.

In summary, we find that the notion of autonomy is not appropriate to fully embrace the cooperative, shared, and implicit nature of the human–technology relationship which we denote by hybrid action. From a robotics and data-processing point of view, continuous assistance in physical action and decision-making like in the surgery room creates a complex mutual dependency between the single human, the team, and the robotic system which should be considered more when introducing such systems. Networking and possible scaling effects for big-data applications may introduce novel means to quantitatively assess performance in the operation theater, implying far reaching consequences and creating novel risks like skill deprivation, which is well known in automation.

The Information Systems Viewpoint

From an information systems viewpoint, the temptation might be to regard robotic systems in operating theaters just as a new but mainly similar type of modality within hospitals as, e.g., X-ray machines in radiology departments. This would imply that there are no real new consequences for information system architectures and for respective information management strategies, as dealing with such modalities is well known. In our opinion, this view is much too simplistic and completely inadequate. To explain this, we need first to explain how information systems personnel (e.g., chief information officers of hospitals or medical informatics researchers focusing on HISs) view information systems and their management. For this view it is better to consider hospitals as a whole, with operating theaters constituting parts of hospitals.

Let us start with a common definition on hospital information systems (HOISs), which themselves are important instances of HISs: “A hospital information system is the socio-technical subsystem of a hospital, which comprises all information processing … . Typical components of hospital information systems are enterprise functions, business processes, application components, and physical data processing systems.”[59] Literature on HOISs, its architectures, and its information management strategies spans many decades.[5] [6] [60] [61] [62] In larger hospitals, HOISs consist today of thousands of connected computers (including modalities with embedded computers), with about 100 functionally comprehensive and connected computer-based application systems implemented on these computers. These application systems support users (health care professionals, i.e., physicians, nurses, etc.) in a variety of functions for patient care, hospital management, and biomedical research. Important services are to appropriately store data, in particular patient data, to have these data available when needed, and to support decisions. How HOISs can at best provide adequate services, in particular for health care professionals, is an ongoing challenge.[63]

One of the pioneers of medical informatics, François Grémy, summarized the situation as follows: “Any technology sets a relationship between human beings and their environment, ... This is especially relevant when dealing with large automatic information systems, developed to contribute to the management and integration of large organizations, such as hospitals. In such a context, the environment is mainly made up of humans.”[64] In other words, HISs are setting relationships between human beings, here mainly between health care professionals, who themselves are caring for patients. For health care professionals it is most important to know from their HOISs about a patient's diagnoses, her or his medications, and about the existence of problems such as allergies.

Why should robotic systems and X-ray machines not be regarded as mainly similar types of modalities? Modalities like today's X-ray machines act passively. They produce images with additional information, which are then viewed and which are also stored in picture archiving and communication systems for, maybe, further use and further analyses. The functionality of these machines is limited compared with the functionality of robotic systems. In contrast, robotic systems in operating theaters are increasingly active participants. This is why team–machine interaction and hybrid action of humans and machines have to be considered, as discussed in the sections above. Through increased digitization of health care, with sensors and actors, denoted, e.g., as assistive systems for pervasive health[65] or as health-enabling technologies,[66] these robotic systems are not the only machines, or “nonliving entities,” respectively, which can be viewed in this way. We nowadays also have “intelligent” beds or “intelligent” rooms (e.g., in wards) which behave similarly and can know a lot about patients, their behavior, and their health. Similar developments can be expected for future cars (patient logistics, etc.) or for future implants like, e.g., pace makers. It is important that these nonliving entities can adequately communicate with other entities, in particular with “living entities” like health care professionals or patients and their relatives. This might be denoted as “seamless interactivity with automated data capture and storage for patient care, and beyond (from perception to high-level semantic concepts, related to human–human, machine–machine, as well as human–machine interaction; “beyond” in the meaning of not being restricted to certain disease episodes).”[67]

Similarly, as for living entities it might be also helpful for nonliving entities to have access to patients' data, and may be even to get support in making decisions. For robotic systems in operating theaters as well as for intelligent beds in wards, etc. information on their patients' diagnoses, their medication, and the presence of allergies may also be of importance to appropriately doing their services. From this point of view, robotic systems in operating theaters as well as related nonliving entities in hospitals may no more be regarded simply as modalities of HOISs. In analogy to health care professionals, they should be viewed as HOISs users.

HIS architectures, with HOISs as important instances, as well as information management strategies have to consider this dramatic change, caused by a significantly increased hybrid action of humans and machines in hospitals. Putting it into a nutshell with reference to the quoted text of François Grémy: HOISs should set a relationship between both human beings (living entities) and “intelligent” machines (nonliving entities) as well as with their environment.

The Legal Viewpoint

Law sets a societally constraining framework for technological developments. Testing and using robotic systems in medical contexts is only possible if the persons involved—producer, programmer, user, etc.—know about legal risks and how to avoid them. If the danger of being (individually) responsible for damages is too high, one might restrain oneself from developing the technology further. Thus, it is important to understand the legal circumstances that might pertain to new technologies such as robotic systems in operating theaters.

First, one has to analyze what is special and new about these systems compared with the machines we are already using[52]: the machines lead to distance between doctors and patients, create new potentials for violating the patient or mistreatment (such as malfunctioning machines or wrongful usage by the doctors). Traditional education is not sufficient to know how to deal with these machines and knowledge of traditional operations might get lost with the use of these systems which is known as skill deprivation in automation. The systems will collect huge amounts of data and will be connected to platforms and other machines. And, most importantly, some of these machines might act (partly) autonomously, preparing or even making decisions and taking part in actions.

All these new characteristics will produce new challenges for the legal system in different areas.

One relevant legal area is administrative law, as these machines have to be licensed, tested in hospitals, approved by ethics committees, etc.[68] [69] [70] In this complex legal field, there are many laws applicable to our robotic systems, such as, for example, the Directive 93/42/EWG about Medical Devices, the Medical Devices Act, or on a nonstate level, the DIN EN ISO 14155 or the Declaration of Helsinki (about research on humans). These laws are meant to ensure the safety of the patients and therefore set the framework of acceptable actions and treatments. One has to keep in mind, though, that these laws are not designed for autonomous systems or AI. Testing products, balancing advantages and risks, involving ethic committees, and explaining the device to the patient all these become more complex and difficult in the case of a device developing over time, being connected to other machines, being trained by different people, etc. Therefore, it is necessary to adapt the administrative laws to the technological developments and balance the interests—and the ethical as well as legal responsibilities defined for the people involved.

In civil law, one can find regulations on the relations between a doctor (clinic) and the patient, as well as between a producer and a consumer (clinic/doctor). The contract between doctors and patients in Germany is regulated in §§ 630a et seqq. BGB.[71] The doctor is obligated to act according to his professional standards, to explain all circumstances to the patient so that she or he can give fully informed consent.[72] As robotic systems in operating theaters are partly new and not fully tested systems, the doctor also has to inform the patient that he is not using traditional methods but new systems. Especially if we are confronted with AI (but not only then), the risks for all persons involved, especially for the patient, are not fully known. The system might develop in unpredictable ways and not be fully understandable and truly controllable. This has to be included in the explanation of the operation.[72] One also has to be aware that, until now, there are no specific standards for the contractual obligations of all people involved—for the doctor, the clinic, but also for the producer: the novelty of these machines means that one does, so far, not fully know what to expect of the product, of the producer, as well as of the user. Therefore, the obligations—including the contractual obligations of the producer—are not clear and should be clarified by the interpreter of the law as well as the lawgiver.

These missing standards could be one of many problems for the liability and responsibility of the people involved: the programmer, the producer, the hospital management, the doctor, and the nursing staff could all, in general, be liable for damages resulting from using the machines.[73] [74] Generally, one only is liable in the case of negligent behavior, except in some cases of strict liability. The producer, for example, could be liable for defects of the products even if he did not act negligently. The liability of the doctor, for example, does depend on his acting wrongfully, meaning to transgress the generally accepted professional standards. In this context, the doctor might not only face liability in terms of having to pay damages, but even individual criminal responsibility. The clinic could be responsible for either installing machines too dangerous for the patients or, at some point, not installing machines although they might be provably more efficient than traditional machines and/or human beings. In the case of new robotic systems, we will be faced with, as mentioned, missing standards and other changes of the existing liability/responsibility regime.[72] [75] Not only is it difficult to prove what went wrong in the case of such new machines, it also is problematic to talk about misconduct or negligence if there are no clear and specific standards about how to act correctly. One will have to distribute the liability between the people involved—not only because of the difficulties to prove where something went wrong, but because the end result, the treatment, is in fact the result of the hard-to-untangle collaboration of different participants. For the distribution of responsibilities, again, so far there are no established rules. Especially for the patient, it would be preferable if he would not have to prove who made a mistake, who acted negligently, but to be able to address the collective directly. For this undertaking, it is discussed to introduce the so-called “electronic person.”[76] This construct allows, analogous to the already existing legal person, to address the machine as a symbol of the people involved. The machine would have its own monetary account, which could then be used to pay legally determined economic damages. These accounts would have to be negotiated, legally justified, and value-assigned by the humans involved. The machine could be registered and addressed from the moment of registration onwards. This construct does carry its own problems which are too complex to be discussed in detail here—just as an example: the construct only addresses a specific machine, while, in reality, most of these machines will be connected and act on information from these connections, leading to implausibility of one of these machines being liable for damages stemming from these networks. But there will be more problems, and one will have to consider if the advantages of such a construct outweigh its disadvantages. Regardless of how this question is answered, one would also need to develop other mechanisms, such as mandatory insurances, social liability, etc. The most important aspect should be that the patient, if (s)he becomes a victim, or bearer of the harms or damages, should not face the difficulties of the new technology alone but should be insured so that the damages should be paid by the ones who draw the largest profits from the new technologies.

One of the biggest problems in the context of responsibility, though, will be the responsibility of the “human in the loop.” In many contexts of rising AI, it is demanded that there should be a human in the decision loop to ensure the inclusion of human values and human capacities in the decisions.[77] It might even be questionable if this does, in all cases, increase the quality of the decision. But even if it did, this does not answer the question if this demand actually is excessive compared with what the human being is able to achieve: if the car is driving, the human driver will not pay enough attention to overtake another car at any given second of time, but leave it to the automatic system. If the computer suggests a specific diagnosis, the human doctor might be strongly influenced by this, and thus forget other potential diagnoses he might have thought of by himself. This is unavoidable and should, therefore, not be imposed on the human individual who often did not even choose to use the machine. Altogether, it becomes clear that we have to find new solutions for liability and responsibility in the context of robotics and AI.

Finally, our focus is directed toward a specific problem of the machines we are analyzing here: the protection of data. On the one hand, the further development of AI depends on the possibility of using huge amounts of data (Big Data), on the other hand, the data in our context are very sensitive and have to protected with special care.[68] [69] The current data protection regime is focusing strongly on the protection of the individual; especially if one regards anonymization or pseudonymization as almost impossible nowadays, it becomes difficult to feed AI programs with enough data to develop them further. This is especially the case for data protected even by criminal law such as the data of patients. The strong protection is not necessarily problematic, but one should be aware of the problems it might cause for the technology discussed here. It might be necessary to find new balances between the different interests in the future.

To conclude, it cannot be doubted that robotics and AI pose enormous challenges for the traditional legal regime. This will also be the case for the context of the operating theaters. The interpreter of the existing laws and especially the law giver are challenged to adapt the legal situation to the reality we are facing—not in a few years, but now.

The Ethical Viewpoint

Dealing with ethical implications of new technologies like robotics in operating theaters means, first of all, to explain what an ethical point of view is. To put it in a nutshell: ethics is a way of reflecting methodologically on questions with which human beings decide upon the way they are living together and what it means to be a human being who behaves with integrity or ethical standards on his or her own and in relation to all others. Furthermore, questions like what are we to do? and what should we refrain from? are part of ethical deliberation. However, before coming to a point of arriving at answers and concrete recommendations, we should set out some more general considerations about technology and human–machine interaction.

It is more than a laconic stance, that it is difficult or impossible to precisely foresee the future. In terms of technology there is a very specific problem that we, human beings, try to avoid unwanted side effects of any technology deployed as long as we cannot estimate these side effects being acceptable when compared with the expected benefits. But, how to assess this ratio with respect to a technology that is not yet established? On the other hand, we do know quite well about unwanted side effects once a technology has been deployed, but then, it is very hard to retract this technology bearing more harm than benefit. This “dilemma of control” has been named after David Collingridge, a chemist and philosopher of technology,[78] and still is challenging our thinking when establishing a new technology.

So is the case with robots and expert systems in the operating theater. We can imagine that this technology will impact the number and structure of employment in respective teams and companies, as new technology did in any industrial sector in the past centuries. We may suppose that it will change the professional roles and hierarchies. Will it be for good or for harm, or indifferent? Will the up-skill and down-skill effects on employees which are already observable in other sectors working with robots have a balanced score and will it tend to have negative[79] or positive[80] effects on employment? There is no clear answer yet. Whom should we trust: the prophets of “creative destruction”[81] telling us that the rate of employment and innovation will far outweigh the loss of former jobs? Or should we listen to those warning against a new caste of “redundant people”[82] caused by digitization and robotics in every branch profitable for automation? Obviously, these are not technical questions, nor are they issues, which can be addressed in economic utility terms alone, but it is a vital socio-political question and an ethical one in the sense that we decide upon how we will live and work together in the near future.

Another open question to be considered is how interaction of human beings and machines can be conceptualized and the direction it is supposed to take in surgical and clinical practice. There is no doubt that robots equipped with AI and high-precision handling features already have left behind the state of being a mere instrument. In the literature, they are called “cooperative partners,” “companions,” or “co-workers,” raising several questions concerning the technical, juridical, societal, and ethical aspects of cooperation. Referring to the ethical ones, it is still unclear whether any or which moral status these machines should be assigned to. Even if European politicians think about an “electronic-personhood”[83] as a status of responsibility and liability for most sophisticated autonomous robots, ethical concerns are not covered by this proposal. Ethical reflections in this field have to address the social and moral status of these machines and the consequences derived from it. One crucial issue involves the distribution of roles between humans and machines. Is the human still in control of the machine, or do we have to acknowledge a superiority of the machine being smarter, more efficient, and more precise? Are surgeon's teams allowed to deviate from the expert system's recommendation on how to treat a patient, when the expert system has retrieved data from all relevant databases to determine the “best” treatment options for this particular patient—in a period of time never achievable for humans? If these teams were not allowed to do so due to reasons of evidence, standards, and potential claims for compensation, the question is whether the human would increasingly be driven into dependency on the machines? Furthermore, how would this impact the self-conception of human beings as moral subjects, i.e. as beings who decide upon their actions with free will and a rationale combining ends and means with their own personal ethical viewpoint on what are acceptable tradeoffs in responsibilities and actions. It looks like we may be asked to revise strong convictions about a specific superiority of mankind in terms of decision-making efficiency and purported rationality—now based in AI on “limited” constraints that are, in practice, almost always based on economic values from utility theory. Of course, one may ask whether the fact of being dependent on a faster and better calculating system will contest human agency and responsibility at all. This is going to be not only an empirical question, but an ethical—in the sense of analytical—question as well. How do we, human beings, perceive the way we form an opinion and shape our actions accordingly? What does it mean to be autonomous in a pragmatic and a moral sense, and how does this autonomy is compatible with the autonomy of the robot and the expert system? How can we assert the supremacy of human autonomy theoretically and pragmatically? No doubt, this is a pressing issue as the committee on rules for robots of the European Parliament stated: “The Commission is called upon to establish criteria for the classification of robots that would need to be registered. They also considered it essential, in the development of robotics and AI, to guarantee that humans have control over intelligent machines at all times and that special attention should be paid to the possible development of an emotional connection between humans and robots—particularly in vulnerable groups (children, the elderly, and people with disabilities),”[84] emphasized by the author. What does it mean to have the control over machines at all times? And how shall we achieve it? The report suggested that advanced robots should be equipped with a “black box” which records data on every transaction carried out by the machine, including the logic that contributed to its decisions[84]. Would that help to (re-)gain control over the machine, taking into account that no human being would be able to retrace the logical combinations executed by self-learning algorithms in massive amounts of data? Norbert Wiener made a hint towards the systematic problem behind this demand[85] in the early 1960s already. Philosopher of technology, Christoph Hubig, pleads for a parallel communication on three layers informing the human beings about what is happening at the intersection of human and machine.[86]

In any case, the cooperation of humans and machines in general challenges our social capabilities as well as our mindset to establish new practices complemented by new theoretical insights into traditional concepts of autonomy, agency, responsibility, and decision-making. Working on these questions will mean on the one hand investigating or co-evolving research on empirical questions (e.g., how does the team work changes in operating theaters when deploying a robotic and AI system?) and theoretical questions (e.g., what is cooperation between humans and machines in terms of agency, outcome, and modus operandi? what is the difference between human and artificial autonomy, intelligence, or agency?). On the other hand, ethical deliberation can never only refer to empirical data. Ethics first and foremost deals with the question of what should be considered as ethical and moral norms, so that benefits and demands against others to make the world a better place can be considered. This leads to more specific ethical questions: what are the criteria for good ethical guidance? The above-mentioned paper of the European Parliament referred to bioethical principles, i.e., beneficence, nonmaleficence, autonomy, and justice. Furthermore, researchers and designers were asked to act responsibly and bear in mind the need to respect, dignity, privacy, and human safety.[87] What does this mean in effect for concrete interaction in the operating theater, and for the new staff composition?[57] This is the empirical part of research, that ethical deliberation cannot afford to cast aside. On the other hand, ethical thinking has to stimulate the process of designing and constructing socio-technical arrangements that meet the criteria mentioned above. In doing so, ethics has to go into details, exploring and explicating the meaning of, e.g., privacy with respect to the patient, the surgeon, or any other people involved. Will the monitoring of every movement of the team in the operating theater lead to a benchmarking with other teams (or other days) and will this lead to standardization and a demand for continuous improvement on the basis of objective data? How should technical artifacts be constructed, how should social practices be established in order to avoid unwanted consequences and constellations provoking rejection from an ethical point of view?

Finally, the debate on this issue may not be done by experts only, but should be opened to the public as a whole, because what is at stake is a question of public concern, and thus a very pressing one as the European Parliament states: “Humankind stands on the threshold of an era when ever more sophisticated robots, bots, androids, and other manifestations of AI seem to be poised to unleash a new industrial revolution, which is likely to leave no stratum of society untouched. The development of robotics and AI raises legal and ethical issues that require a prompt intervention at EU level.”[87]

Consequences

Although coming from different scientific disciplines we all agree that the questions, raised in the Introduction section, are timely and highly relevant and that fundamentally new approaches are needed to construct a reasonable concept of hybrid action and of an according ascription of responsibility.

We are convinced that new hybrid actions between humans and machines imply at least to a certain extent the mentioned shared decision-making between (teams of) humans on the one hand and machines on the other hand. Collaboration of surgical teams in operating theaters as well as roles, competencies, and responsibilities of humans (health care professionals) and machines (robotic systems) needs to be reconsidered. Hospital information systems will in the future not only have humans as users, but also intelligent machines. Both will need access to patient data, ask for decision support and for appropriate communication facilities. These tremendous changes have to be mapped into legal as well as professional, societal, and economic negotiated frameworks among others for appropriately considering responsibilities. For all these matters, ethical considerations have to be the basis.

Although team–machine interaction, and hybrid actions of humans and machines, has been discussed here in the context of robotic systems in operating theaters, it became evident that these issues are relevant for health care in general and even beyond. We are convinced that the expected significant changes in the relationship of humans and machines can only be appropriately dealt in inter- and multidisciplinary collaboration. We fully agree with Jan van Bemmel's statement that “Interdisciplinary research is not a category of research but a consequence of addressing a complex problem in society, involving the collaboration between and methods drawn from multiple disciplines. Because research is people, personal interactions are critical for interdisciplinary research.”[88]

Conflict of Interest

None declared.

Acknowledgments

As mentioned in the Objective and Structure section, this article elaborated on the authors' presentations in a Bioethics Symposium on “Robots in Operating Theaters,” organized by the Braunschweig Scientific Society (BWG[21]). We cordially want to thank BWG President Otto Richter and BWG Vice President Klaus Gahl for their strong support and encouragement in both selecting this topic for the 2018 Bioethics Symposium and for motivating us to continue to work on it, which, among others, led to this manuscript. Our cordial thanks go also to Casimir Kulikowski, Rutgers University, New Jersey. He kindly edited an earlier version of this manuscript. Through his edits, he not only turned our text into a much better readable English style, being a pioneer in AI in medicine, he also helped us to better shape our thoughts.

• References

• 1 Gardner RM. Clinical information systems - from yesterday to tomorrow. Yearb Med Inform 2016; (Suppl. 01) S62-S75
• 2 Lun KC. The datafication of everything - even toilets. Yearb Med Inform 2018; 27 (01) 234-236
• 3 Martin-Sanchez FJ, Lopez-Campos GH. The new role of biomedical informatics in the age of digital medicine. Methods Inf Med 2016; 55 (05) 392-402
• 4 Scheer AW, Wahlster W. Saarbrücken Declaration for a digitization push in Germany (Saarbrücker Manifest für einen Digitalisierungsruck in Deutschland). November 2016 (in German). Available at: https://www.scheer-group.com/Scheer/uploads/2016/11/Scheer_Saarbrücker-Manifest.pdf . Accessed August 14, 2018
• 5 Reichertz PL. Hospital information systems--past, present, future. Int J Med Inform 2006; 75 (3–4): 282-299
• 6 Haux R. Health information systems - past, present, future. Int J Med Inform 2006; 75 (3–4): 268-281
• 7 Arnrich B, Ersoy C, Mayora O, Dey A, Berthouze N, Kunze K. Wearable therapy - detecting information from wearables and mobiles that are relevant to clinical and self-directed therapy. Methods Inf Med 2017; 56 (01) 37-39
• 8 Friedman CP, Rubin JC, Sullivan KJ. Toward an information infrastructure for global health improvement. Yearb Med Inform 2017; 26 (01) 16-23
• 9 Winter A, Takabayashi K, Jahn F. , et al. Quality requirements for electronic health record systems*. A Japanese-German information management perspective. Methods Inf Med 2017; 56 (07) e92-e104
• 10 Vedula SS, Ishii M, Hager GD. Objective assessment of surgical technical skill and competency in the operating room. Annu Rev Biomed Eng 2017; 19: 301-325
• 11 Haux R, Kulikowski CA, Bakken S. , et al. Research strategies for biomedical and health informatics. some thought-provoking and critical proposals to encourage scientific debate on the nature of good research in medical informatics. Methods Inf Med 2017; 56 (Open): e1-e10
• 12 McCray AT, Gefeller O, Aronsky D. , et al. The birth and evolution of a discipline devoted to information in biomedicine and health care. As reflected in its longest running journal. Methods Inf Med 2011; 50 (06) 491-507
• 13 Ainsworth J, Buchan I. Combining health data uses to ignite health system learning. Methods Inf Med 2015; 54 (06) 479-487
• 14 Detmer DE. At last! A working model of a data ecosystem for continuous learning in the evolving health noosphere. Methods Inf Med 2015; 54 (06) 477-478
• 15 Denaxas S, Friedman CP, Geissbuhler A. , et al. Discussion of “Combining health data uses to ignite health system learning”. Methods Inf Med 2015; 54 (06) 488-499
• 16 Shortliffe EH. Digital medicine and biomedical informatics: what's in a name?. Methods Inf Med 2016; 55 (05) 389-391
• 17 Al-Shorbaji N, Bellazzi R, Gonzalez Bernaldo de Quiros F. , et al. Discussion of “The new role of biomedical informatics in the age of digital medicine”. Methods Inf Med 2016; 55 (05) 403-421
• 18 Gonzalez Bernaldo de Quiros F, Dawidowski AR, Figar S. Representation of people's decisions in health information systems.* A complementary approach for understanding health care systems and population health. Methods Inf Med 2017; 56 (Open): e13-e19
• 19 Ring D, Tierney WM. Health information systems supporting health and resiliency through improved decision-making. Methods Inf Med 2017; 56 (Open): e11-e12
• 20 Al-Shorbaji N, Borycki EM, Kimura M. , et al. Discussion of “Representation of people's decisions in health information systems: a complementary approach for understanding health care systems and population health”. Methods Inf Med 2017; 56 (Open): e20-e29
• 21 Braunschweig Scientific Society. 14th Bioethics Symposium “Robots in Operating Theatres”. (Braunschweigische Wissenschaftliche Gesellschaft. 14. Bioethik-Symposium “Roboter im Operationssaal”), February 7, 2018 (in German). Available at: https://publikationsserver.tu-braunschweig.de/servlets/MCRFileNodeServlet/dbbs_derivate_00045303/Jahrbuch-BWG-2018.pdf . Accessed August 14, 2018
• 22 Catchpole KR, Hallett E, Curtis S, Mirchi T, Souders CP, Anger JT. Diagnosing barriers to safety and efficiency in robotic surgery. Ergonomics 2018; 61 (01) 26-39
• 23 Sharma B, Mishra A, Aggarwal R, Grantcharov TP. Non-technical skills assessment in surgery. Surg Oncol 2011; 20 (03) 169-177
• 24 Cunningham S, Chellali A, Jaffre I, Classe J, Cao CGL. Effects of experience and workplace culture in human-robot team interaction in robotic surgery: a case study. Int J Soc Robot 2013; 5: 75-88
• 25 Randell R, Honey S, Hindmarsh J. , et al. A realist process evaluation of robot-assisted surgery: integration into routine practice and impacts on communication, collaboration and decision-making. Health Serv Del Res 2017 5(20):
• 26 Lai F, Entin E. Integrating surgical robots into the next medical toolkit. Stud Health Technol Inform 2006; 119: 285-287
• 27 Vitiello V, Lee SL, Cundy TP, Yang GZ. Emerging robotic platforms for minimally invasive surgery. IEEE Rev Biomed Eng 2013; 6: 111-126
• 28 Bhatti NI, Cummings CW. Competency in surgical residency training: defining and raising the bar. Acad Med 2007; 82 (06) 569-573
• 29 Szasz P, Louridas M, Harris KA, Aggarwal R, Grantcharov TP. Assessing technical competence in surgical trainees: a systematic review. Ann Surg 2015; 261 (06) 1046-1055
• 30 Dietz A. Ändert sich mit der Digitalisierung des Operationssaals das Berufsbild des Chirurgen? Beispiel Kopf-Hals-Onkologie. In: Niederlag W, Lemke HU, Strauß G, Feußner H. , eds. Der digitale Operationssaal. Berlin: De Gruyter; 2014: 219-231
• 31 Kluge EW. Health information professionals in a global ehealth world: ethical and legal arguments for the international certification and accreditation of health information professionals. Int J Med Inform 2017; 97: 261-265
• 32 SARAS. Smart autonomous robotic assistant surgeon im Rahmen des EU Framework Programme for Research and Innovation. Available at; https://saras-project.eu/ . Accessed December 23, 2018
• 33 TMCapital. The Next Generation of Medicine: Artificial Intelligence and Machine Learning. Industry Spotlight. New York, NY: TMCapital; 2017: 1-23
• 34 Beyer P. The Future Directive On Environmental Liability - A Tool to Implement the Precautionary Principle?. Berlin: Ecologic Institute; 2004
• 35 Veruggio G. EURON Roboethics Roadmap. Genova: Scuola di Robotics; 2006
• 36 Abboudi H, Khan MS, Aboumarzouk O, Guru KA, Challacombe B, Dasgupta P, Ahmed K. Current status of validation for robotic surgery simulators - a systematic review. BJU Int 2013; 111 (02) 194-205
• 37 Roman H, Marpeau L, Hulsey TC. Surgeons' experience and interaction effect in randomized controlled trials regarding new surgical procedures. Am J Obstet Gynecol 2008; 199 (02) 108.e1-108.e6
• 38 PwC. What doctor? Why AI and robotics will define New Health. 2017. Available at: https://www.pwc.at/de/publikationen/branchen-und-wirtschaftsstudien/healthcare-ai-new-health.pdf . Accessed September 17, 2018
• 39 Zattoni F, Guttilla A, Crestani A. , et al. The value of open conversion simulations during robot-assisted radical prostatectomy: implications for robotic training curricula. J Endourol 2015; 29 (11) 1282-1288
• 40 Lucas SM, Pattison EA, Sundaram CP. Global robotic experience and the type of surgical system impact the types of robotic malfunctions and their clinical consequences: an FDA MAUDE review. BJU Int 2012; 109 (08) 1222-1227 , discussion 1227
• 41 Walters C, Webb PJ. Maximizing efficiency and reducing robotic surgery costs using the NASA task load index. AORN J 2017; 106 (04) 283-294
• 42 Ross SB, Downs D, Saeed SM, Dolce JK, Rosemurgy AS. Robotics in surgery: is a robot necessary? For what?. Minerva Chir 2017; 72 (01) 61-70
• 43 Chikwe J, de Souza AC, Pepper JR. No time to train the surgeons. BMJ 2004; 328 (7437): 418-419
• 44 Niederlag W, Lemke HU, Strauß G, Feußner H. , eds. Der digitale Operationssal, 2nd ed. Berlin: De Gruyter; 2014. (in German).
• 45 Da Rosa CC. Operationsroboter in Aktion: kontroverse Innovationen in der Medizintechnik. Bielefeld: Transcript Verlag; 2014
• 46 Beasly RA. Medical robots: current systems and research directions. J Robot 2012; 401613
• 47 Taylor RH, Menciassi A, Fichtinger G, Fiorini P, Dario P. Medical robotics and computer-integrated surgery. In: Siciliano B, Khatib O. , eds. Springer Handbook of Robotics. 2nd ed. Heidelberg: Springer; 2016: 1657-1684
• 48 Manzeschke A. Digitales Operieren und Ethik. In: Niederlag W, Lemke HU, Strauß G, Feußner H. , eds. Der digitale Operationssal. 2nd ed. Berlin: De Gruyter; 2014. (in German): 227-49
• 49 Hubig H. Mensch-Maschine-Interaktion in hybriden Systemen. In: Hubig H, Koslowski P. , eds. Maschinen, die unsere Brüder werden. Mensch-Maschine-Interaktion in hybriden Systemen. Paderborn: Fink; 2008. (in German): 9-17
• 50 Rammert W. Technik-Handeln-Wissen. Wiesbanden: VS Verlag für Sozialwissenschaften; 2007. (in German)
• 51 Beer JM, Fisk AD, Rogers WA. Toward a framework for levels of robot autonomy in human-robot interaction. J Hum Robot Interact 2014; 3 (02) 74-99
• 52 Beck S. The problem of ascribing legal responsibility in the case of robotics. AI Soc 2016; 31: 473-481
• 53 Murphy R, Casper J, Micire M, Hyams J. Mixed-initiative control of multiple heterogeneous robots for usar. Technical Report CRASAR-TR2000–11. Tampa, FL: Center for Robot Assisted Search and Rescue, University of South Florida; 2000
• 54 Schilling M, Kopp S, Wachsmuth S. , et al. Towards a multidimensional perspective on shared autonomy. Paper presented at: Proc. AAAI Fall Symposium Series: Shared Autonomy in Research and Practice. Paolo Alto, CA: AAAI; 2016 :338–344
• 55 Cheng G, Zelinsky A. Supervised autonomy: a framework for human-robot systems development. Auton Robots 2001; 10: 251-266
• 56 Niemeyer G, Preusche C, Hirzinger G. Telerobotics. In: Siciliano B, Khatib O. , eds. Springer Handbook of Robotics. Berlin: Springer; 2008: 741-757
• 57 Gransche B. Wir assistieren uns zu Tode. Leben mit Assistenzsystemen zwischen Kompetenz und Komfort. In: Biniok P. , ed. Assistive Gesellschaft. Wiesbaden: Springer VS; 2017. (in German): 77-97
• 58 Krishnan S, Garg A, Patil S. , et al. Transition state clustering: unsupervised surgical trajectory segmentation for robot learning. Int J Robot Res 2017; 36: 1595-1618
• 59 Winter A, Haux R, Ammenwerth E, Brigl B, Hellrung N, Jahn F. Health Information Systems - Architectures and Strategies. London: Springer; 2011
• 60 Ball MJ. An overview of total medical information systems. Methods Inf Med 1971; 10 (02) 73-82
• 61 Lorenzi NM, Riley RT. Organizational Aspects of Health Informatics - Managing Technological Change. New York, NY: Springer; 1995
• 62 Kuhn KA, Giuse DA. From hospital information systems to health information systems. Problems, challenges, perspectives. Methods Inf Med 2001; 40 (04) 275-287
• 63 Islam MM, Poly TN, Li YJ. Recent advancement of clinical information systems: opportunities and challenges. Yearb Med Inform 2018; 27 (01) 83-90
• 64 Fessler JM, Grémy F. Ethical problems in health information systems. Methods Inf Med 2001; 40 (04) 359-361
• 65 Mayora O, Lukowicz P, Marschollek M. Evolving pervasive health research into clinical practice. Methods Inf Med 2014; 53 (05) 380-381
• 66 Haux R, Koch S, Lovell NH, Marschollek M, Nakashima N, Wolf KH. Health-enabling and ambient assistive technologies: past, present, future. Yearb Med Inform 2016; 25 (Suppl. 01) S76-S91
• 67 Haux R. Medical informatics: past, present, future. Int J Med Inform 2010; 79 (09) 599-610
• 68 Igl G, Welti F. Gesundheitsrecht. Eine systematische Einführung. Munich: Vahlen; 2014. (in German)
• 69 Ortner R, Daubenbüchel F. Medizinprodukte 4.0. NJW. 2016: 2918-2924 (in German)
• 70 Federal Institute for Drugs and Medical Devices. Available at: https://www.bfarm.de/EN/MedicalDevices/_node.html https://www.bfarm.de/EN/MedicalDevices/MarketAccess/_node.html . Accessed June 20, 2018
• 71 Wagner G. In: Säcker FJ, Rixecker R, Oetker H, Limperg B. , eds. Münchener Kommentar zum Bürgerliche Gesetzbuch (Band 4). 2016. ; § 630a recital 1–2 (in German). Munich: Beck.
• 72 German Federal Supreme Court of Justice. Judgment of June 13, 2006 - VI ZR 323/04 - BGHZ 168, 103–12 (in German)
• 73 Ratzel R, Lissel PM. Handbuch des Medizinschadensrechts. 2013; § 1 recital 100 ff.; § 3 recital 172 ff. (in German)
• 74 Weimer T. Straf- und zivilrechtliche Haftung der Anwender und Betreiber von Medizinprodukten - Teil 3. MPR 2007; 119-123 (in German)
• 75 Eidenmüller H. The rise of robots and the law of humans. ZEuP 2017; 765-777 , recital 776
• 76 Delvaux-Stehres M. Draft report with recommendations to the Commission on Civil Law Rules on Robotics (2015/2103(INL)) to the European Parliament's Legal Affairs Committee. Available at: http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//NONSGML%2BCOMPARL%2BPE-582.443%2B01%2BDOC%2BPDF%2BV0//EN . Accessed June 20, 2018
• 77 Phillips-Wren G, Ichalkaranje N, Jain LC. Intelligent Decision Making: An AI-Based Approach. Heidelberg: Springer; 2008
• 78 Collingridge D. The Social Control of Technology. Milton Keynes: Open University Press; 1981
• 79 Frey CB, Osborne MA. The future of employment: how susceptible are jobs to computerisation? Available at: https://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf . Accessed September 23, 2018
• 80 Graetz G, Michaels G. Robots at work. CEPR Discussion Paper 1335, March 2015. Available at: http://cep.lse.ac.uk/pubs/download/dp1335.pdf . Accessed September 23, 2018.
• 81 Schumpeter JA. Der Prozeß der schöpferischen Zerstörung. In: Herdzina K. , ed. Wettbewerbstheorie. Cologne: Kiepenheuer & Witsch; 1975. (in German)
• 82 Ricardo D. On the principles of political economy and taxation. In: The Works and Correspondence of David Ricardo. Vol I. Cambridge: Cambridge University Press; 1981
• 83 Hern A. Give robots 'personhood' status, EU committee argues. The Guardian. January 12, 2017
• 84 Committee report on rules for robotics, 2015/2103(INL) - 27/01/2017. Committee report tabled for plenary, single reading. Available at: http://www.europarl.europa.eu/oeil/popups/summary.do?id=1473044&t=d&l=en . Accessed September 23, 2018
• 85 Wiener N. Some moral and technical consequences of automation. Science 1960; 131 (3410): 1355-1358
• 86 Hubig C. Mensch-Maschine-Interaktion in hybriden Systemen. In: Hubig C, Koslowski P. , eds. Maschinen, die unsere Brüder werden. Mensch-Maschine-Interaktion in hybriden Systemen. Munich: Fink; 2008. (in German): 9-17
• 87 European Parliament on rules for robotics, 2015/2103(INL) - 16/02/ 2017 . Text adopted by Parliament, single reading. Available at: http://www.europarl.europa.eu/oeil/popups/summary.do?id=1477231&t=d&l=en . Accessed September 23, 2018; High-Level Expert Group on Artificial Intelligence set up by the EU Commission: ETHICS GUIDELINES FOR TRUSTWORTHY AI, Brussels 2019. Available at: https://ec.europa.eu/digital-single-market/en/high-level-expert-group-artificial-intelligence
• 88 van Bemmel JH. Medical informatics is interdisciplinary avant la lettre. Methods Inf Med 2008; 47 (04) 318-321