Over 10 years in medical technology. What have I learned here, and what recommendations can I offer? Goran Madzar approached me with this question and asked if I'd be interested in writing a guest blog. He was happy to respond. It took a while, but the post is now available.
My name is Heiko Schmidt, and I have worked in the medical technology field for over 10 years. I started as a software developer and, after four years, took over management of software development. Later, I also took on the role of head of system verification.
I'd like to share a brief glimpse of my experiences with you today. I'm happy to discuss these or other topics with you. Just get in touch!
Everything is perfect
I switched to the medical industry because I thought everything would be more structured and better there. My first question mark was when I wanted to discuss with the QA department what modern software development requires and how it could be done. When I asked various questions about regulatory requirements, I received the answer: "And you want to do all of that? Well, have fun!"
It's not always about wanting something. There are, after all, some standards and guidelines that require certain things from software development. My understanding was that most of the things stated there make sense and should be adhered to.
What I learned quite quickly from such experiences is that you usually don't get a concrete answer as to how to do something. Hence my recommendation to you:
You have to work out the answer yourself and put it into practice in a practical way. Look at the relevant standards and guidance documents and derive the requirements and procedures for yourself from them.
Create a simple concept for meeting these requirements, including a mapping, and present this approach to your quality department. This will create a solid template, and objections will be rare, as you are the expert and know how software development works and what is practical.
experts
I've met many experts over the past 10 years. There are many experts who can tell you what's in every textbook, all very theoretical and abstract. I have this knowledge too, and it doesn't help me. When the questions become more specific, things often get dark, and many experts are gone.
Solutions are often proposed that are correct but impractical in practice. Implementing them would require twice the number of employees, which you don't have.
If you ask three experts, you'll get at least four opinions. This also shows that standards and guidance have a certain level of abstraction and give you the freedom to interpret and implement things in different ways.
There is not just one right way!
Therefore, my recommendation: Listen to experts. It certainly broadens your horizons, and you often hear very good things. However, you should also consider your own expertise and trust your experience! Trust your skills and your knowledge of best practices in software development.
Establish a course of action that you believe in, that fits your business, and that is practical, show backbone, and defend your approach to auditors and critics.
Stand by it, after all, you are also a expert in your field.
Architecture / Components / Units / Interfaces
Time is short, the requirements aren't yet clear, so we're already starting to code. Who hasn't experienced this? I know it too, and I've practiced it! I'm fully committed; everything can't be done purely sequentially. Waiting until the requirements are perfect and signed isn't realistic either and only works in theory. But starting without a rough architecture, without having defined the system context, and without having broken the system down into components, it's extremely difficult.
Defining components and units retrospectively with artificial boundaries is completely unacceptable. You'll realize this at the very latest when you want to set up your component and unit tests and discover how many interfaces and dependencies exist.
My recommendation, therefore, is to create an architecture with clear naming conventions, clear interfaces, and clear statements about who develops what and what SOUP is. Invest sufficient time in the architecture; it's worth the investment! Start early, identify your software systems and the underlying components, and represent them graphically. You'll notice that you now have something concrete, a picture that serves as the basis for technical discussions. You and your colleagues are now talking about the same thing, you're discussing technical issues, and now have the opportunity to refine the architecture in ever greater detail where necessary and uncover any contradictions. At the same time, you share knowledge within the team.
This architecture doesn't have to be a 100-page book; even a few drafts with brief descriptions of the purpose, naming, interfaces, etc. are extremely helpful! Get started. In my opinion, the architecture, the boundaries, and the clear naming are among the most important things for everyone involved, not just for new colleagues! This stage will determine whether you will win the game!
Don't forget to document important decisions. It won't take half a year before you'll be asked why you made one decision and not another. Include the decisions in your architectural documentation. Look at www.arc42.de There are some excellent templates there that you can adapt to your needs. Also, be sure to get the accompanying book (e.g., arc42 in Action).
A little promotional advice on the side: Get a moderator expert who has an outside perspective on your project and is much better able to ask the right questions, e.g. Mr. Madzar!
Units and their number or definition
The topic of units has haunted me for a long time and sparked many discussions. What is a unit, is a unit a module, how are units specified, is a function a unit, is a C file a unit, etc.
You can discuss and research this topic endlessly, and you will never find the right answer. 62304 states: A software unit is a software component that is not subdivided into further components (can't)!
So you decide what a software unit is for you, no one else!
What have I learned from this, and what are my recommendations: At the beginning of your project, clearly define what you consider units. Make the units clearly visible in your software architecture, so you have a quick overview of how many units you have and can easily check whether all units have been specified, implemented, and verified. Don't get too granular. If you define every C file as a unit, you'll quickly end up with several hundred to a thousand software units. If you then want to implement unit tests for this number of units, you probably won't be able to do it due to the effort involved.
Once you have identified your units, do the following for each unit:
- Name your units clearly and meaningfully
- Define the purpose of the unit,
- Specify your units, what are the inputs, what are the outputs and how do the outputs behave in relation to the inputs.
The unit interfaces should have clear names, units, and scopes. Don't forget to define the behavior in case of errors! Once you've done all of this, you're in a great position to write a suitable unit test.
Naming and filing
What I've seen time and again throughout my professional practice is the issue of how things are named and filed. Everyone is a little different about what's good and right. Everyone names things differently and has a different mental structure. This is where the problem begins. People talk past each other, and over time, they look for things and lose track. It starts with the name of a project and what the specific content of the project is and what doesn't belong to it. Put it in writing and don't accept anything else!
If the focus or content changes, no problem, just adapt it, but do so in writing! What are the names of your software systems, the names of the software components, and the software units? Define names in the architecture for each software system, each software component, and each software unit.
Don't forget the interfaces; name them clearly too! Once everything has been given a name, the next step is: Use the names, and only use the specified names, nothing else!
It is a disaster when you are sitting in an audit and have to explain why component A corresponds to component B, or why the test report X represents the verification of unit Y.
Name your specifications, test specifications, deliverables, unit tests, code reviews, etc. with the name of the corresponding software system, component, or unit.
The following image shows an example: A LedController unit has been identified via the architecture. All necessary documentation has this identification in its naming. This allows you to quickly identify any gaps.
Define a directory structure that defines where things are stored. There's no one-size-fits-all approach here either; what's important is that you have a clear definition, that everyone adheres to this definition, and that, if you have multiple projects, everything is done the same way.
You'll quickly notice how you can find things yourself and not always have to ask your colleagues. New colleagues will also be delighted; they only need to have the system explained to them once, and then they'll be able to run independently pretty quickly! Be consistent with this and encourage your colleagues to follow it!
It's important to define it at the beginning of a project. You won't have a chance to correct the confusion surrounding naming and filing later!
Tracking and traceability
This topic shouldn't be underestimated, and without the right tools, it's almost impossible to tackle efficiently. Forget Excel spreadsheets and other unsuitable tools for this task. Get a system that supports you. But don't assume that the tool will do everything for you and solve all your problems for you!
I've often been confronted with questions that actually sound very simple, but aren't always easy to answer. Here are some examples:
- Have all requirements been verified and where are the test results?
- Where is the requirement reflected in the architecture?
- In which unit was the requirement implemented?
- Where in the source code is the requirement implemented?
- There is a bug ticket with number 1234. Has it been fixed, in which release, where in the source code?
- ….
In my opinion, these are all simple, legitimate questions. Can you answer them?
At the beginning of the project, think about the necessary tools and how they work together to minimize manual work and thus the sources of error.
Tooling
In the medical industry, it can be very difficult and time-consuming to launch and consistently use modern tools. Arguments are repeatedly raised that the tools are unreliable, that they need to be validated first, etc. So, people often stick with Word and Excel.
Word and Excel, of course, have their place, no question. If your system is very simple and only has 20 requirements, then yes, then stick with those tools. Otherwise, I can only recommend using the appropriate tools and not misusing Excel as a database!
Yes, the tools have to work reliably. Yes, you have to think about what you actually want to achieve with the tools beforehand. Most tools are significantly more reliable and efficient than Word or Excel.
Therefore, my recommendation: Think about your toolchain well in advance and determine what you need—not too much, not too little. Document your toolchain and show what is used in each phase of development. Once you know what you need, acquire the tools. Validate the tools with the right eye for detail and use them.
Adapt the validation effort to the use of the tool. Tools used at the beginning of a development process are usually less critical than those used at the end of development (e.g., during verification). Only then will they work efficiently and reproducibly.
Project management
Do you know where you stand in the project? Can you explain on a slide where you stand, what the topics are, and what's next?
I've often experienced our management legitimately wanting to know the status, what project risks exist, and what the top issues are. At this point, the big search began. You had to ask many colleagues about the status and quickly put together a slide that looked different each time. Often, the resulting statements were vague, leading to a very vague picture of the status. This gave our management the right impression!
Since we broke the project down into work packages, it suddenly became much clearer where we stood. It was clear, for example, that 40% of the work packages had been completed. There was a person responsible for each work package, and the expected outcome of the work package (definition of done) was also defined.
From these work packages, you can generate very good burndown charts, which are ideal for a management slide and give a quick impression of whether we are on schedule and on track with our activities.
Therefore, my clear recommendations:
- Structure your project into clear work packages with clear tasks, assign responsible persons and define what the result should be.
- Create a simple cockpit that can be updated with little effort.
You can then look forward to the next management request with confidence! You'll shine and exude competence, demonstrating that you're in control of the project, not the project in control of you!
My favorite tools
Here are some tools that I have come to know and love in my 10 years.
Bugzilla
This was our first tool, besides Excel! Setting something like this up was the best decision we ever made! I talked a lot about names, clarity, and traceability, and how important that is. The same applies to bugs found or requested changes; everyone names a problem or change differently.
The Bugzilla tool assigns a unique number to each entry. This has the huge advantage of eliminating any discussions about what exactly is being discussed. Everyone refers to a bug number, and it's unique; you can easily track the status and generate queries. For me, it was and still is a fantastic tool!
At our company, all code changes were checked in with just a bug number, which provided excellent traceability. Even years later, colleagues from other departments would come to me and ask if and when a bug had been fixed or a change implemented. Nothing could be easier. I searched for the bug number or description in Bugzilla and was thus able to quickly, using version control, drill down to the line of code to see what was done when and in which release the change was included.
Our motto has always been: No code change without bug reporting!
By the way: Bugzilla is just one example. You can achieve the same effect with Jira or similar tools (not Excel!)!
Doxygen
The documentation must match the source code—a phrase many people probably know. We all know how quickly documentation and source code can no longer match. Make a quick change, document it later! And just like that, it's happened!
It's very helpful to use a tool like Doxygen and document it accordingly in the source code. With this tool, you can clearly specify your interfaces, define the input ranges, and define the return values. You can document your units in no time!
To ensure this is done consistently, file templates, which should be used for all projects, help. These predefine the structure of each file, which in turn leads to consistent documentation across multiple projects. Integrate this whole process into your coding guidelines, and you'll see that the effort becomes manageable.
Jenkins
What good is Doxygen documentation if it's not up to date?
Anything that needs to be done regularly but is started manually has a chance of being forgotten at some point. Set up a Jenkins build server. This system can automatically do things for you at regular intervals. Start small! Have Jenkins check out the current source code every day. Have Doxygen generate the documentation and translate your project. This way, you can quickly determine whether a translation-ready version of your software is in the version control system and whether the documentation is complete.
You can then expand the whole thing as desired. In the next step, add static code analysis to the system. Then, run dynamic smoke tests, triggered by Jenkins, every day!
You can already see the advantages: Every day, you can see immediately whether you have a stable system or whether something has been changed that has unwittingly led to negative side effects. Once you've set up such a system, you'll benefit from it every day, completely automatically! Therefore, my clear recommendation: Use such automated systems for recurring tasks, start small and expand iteratively as needed! Get in touch if you'd like to learn more about optimizing your test environment!
Conclusion
All of the points mentioned are based on my personal experience. Many of them have nothing to do with medical technology, but are simply best practices!
I hope I've been able to give you some insights and inspire you to think about certain things. If you have specific questions about specific topics or would like to hear my opinion, please let me know.
Yours, Heiko Schmidt