What is our primary use case?
One use case is when a development team finishes, or even in the middle of, development. They run Checkmarx, which shows potential vulnerabilities. If they don't understand something, they consult with me.
I explain what Checkmarx is highlighting, why it's "shouting" as we say, the specific vulnerability, and the problem it found in the code. Then, together, we explore the code and decide if it's a valid issue requiring a fix.
We also discuss how to fix it, or if it's a false positive because, in their environment, the problem either cannot exist or doesn't exist in the way they use their software.
We also have another use case. When a software company, like an integration company, does a project for us, we request them to run their code through Checkmarx. If they don't have their own tool, we run it on our Checkmarx and provide them with the report. We request, or rather insist, that they fix most, if not all, of the problems Checkmarx finds.
These might be issues they didn't consider, but we put it in the contract that they have to submit their software to a "code check," meaning they can use Checkmarx or another approved tool. If they don't have a tool or refuse, then it's okay. The key is to have it in the contract and signed.
Otherwise, fixing the software later becomes difficult, especially when the project is nearing completion. That's why we do it when the integration begins, so there's still time to address the issues. If you wait until the very end, it's too late.
How has it helped my organization?
The solution improved the efficiency of our code security reviews. It helps tremendously because it finds hundreds of potential problems sometimes. When the development teams fix them, or even some of them, it significantly enhances the security of the software.
For example, we had a project, an outsourced one, that provided code written in PHP and included dozens of open-source utilities, libraries, and the like. Their server-side code was in PHP, and their client-side was in JavaScript. Both sides also used many libraries and utilities.
When we ran Checkmarx, it found numerous problems in both their code and the third-party software, including hundreds of high- and medium-severity issues in the PHP code. I didn't dig into the specifics; I just said, "Look, it found hundreds of high and medium problems. You need to reduce them. Before testing starts, you need to provide us the code again, and we'll run it again."
They started fixing it, and while I didn't follow up on the specific fixes, perhaps they removed some libraries. As long as the number of high and medium problems in the Checkmarx report decreased, it meant they were making progress. They hadn't finished yet, though.
After they fixed about half of the problems, we allowed them to start integration. However, they still need to fix the remaining issues, and hopefully, they will.
What is most valuable?
The most valuable feature is that Checkmarx specifies the exact line of code where it finds the problem. They show it in the report, the exact line or two lines. They also show where the problem starts and where it's used.
Even if it's used later in routines or messages during the computation, they show both sides. For example, they show the user input and where it's being used, even if it's saved in a different file.
They follow the code, the function code, the method code, and all the calls until it's used because they have all the code mapped. So, they show where it starts, where it's being used, and they say it hasn't been checked all the way. They prove it, not just say it, by showing exactly where the issue is.
Even if you don't know the software, like third-party software you want to fix or modify, you know where to start looking in the code.
As for the UI, it's okay. You give it the code, it runs, and it's pretty good.
What needs improvement?
There's one thing Checkmarx can maybe fix, actually two things.
First, when we first ran it on a big project, there wasn't enough memory on the computer. It originally ran with eight gigabytes, and now it runs with 32. The software stopped at some point, and while I don't think it said it ran out of memory, it just said "stopped" and something else.
We had to go to the logs and send them to the integrator, and eventually, they found a memory issue in the logs and recommended increasing the memory. We doubled it once, and it didn't seem enough. We doubled it again, and it helped.
So, even if the software reaches capacity on the computer, even though it writes it in the logs, it should also give an indication in the GUI to the person running it, saying "not enough memory" or "not enough disk space."
Another problem is that when it's scanning and it has an internal problem, for example, it cannot check something, or an internal bug or internal problem, it's being found in the logs, but there's no indication to the user. Now, this is good for them because the user runs it, gets a report, everything's fine.
But in a way, it's not good for them because the user doesn't know there's a problem since they don't check the logs. Because mostly, only the manager looks at the logs and only if there's a problem being reported. You run a process, get a report, but in the logs, there might be an indication that it couldn't check several files or understand something. There's a problem, an internal problem that can be fixed, but nobody knows about it because we don't look at the code. The user doesn't look at the logs; only the business manager does, but they don't know because the user doesn't report it, because the user doesn't know.
So, my suggestion for them is this: if they have problems, they should say, 'Here is the report,' but also indicate to the user somewhere, perhaps in the GUI, not necessarily in the report itself, 'We found 100 problems while looking at your code. Please provide us the logs so we can try to fix those.' Then they can ask if the user has any problems. This way, users would know to send them their logs, and they could improve their software, meaning fix the problems.
Now, they may not want to do this because they'll get flooded with millions of responses and millions of problems from all over the world. They would have to fix them, and people might get angry, asking why they provided a report when there were hidden problems. People might say, 'How come you gave me a report with seven or eight problems when analyzing it, there were internal problems with your code? So it's not a perfect report.'"
So, these internal issues are logged but not communicated to the user through the Checkmarx interface (GUI) or report.
The solution also has a few false positives. So, if they had an easier way for users to send an email directly, instead of just opening a ticket. Because when we open a ticket, they want all the logs and everything, and it becomes a hassle.
Perhaps they could implement an easier system where users can send a snippet of the code, along with an explanation of why they believe it's a false positive, referencing the specific report.
This way, Checkmarx could analyze the information and the development team could potentially fix the product in those areas. It wouldn't require them to necessarily respond to the user, but I'm not sure if that's feasible for most companies.
For how long have I used the solution?
I have been using it for one year.
What do I think about the stability of the solution?
What do I think about the scalability of the solution?
If you have enough memory, it's scalable. You need a lot of memory for it to be scalable.
Once you have enough memory, it is stable and scalable, and there are one or two parameters you can modify to make it even more scalable. Scalability is relatively fine.
For the scanning option, the default is to use only one main language, but you can request multiple languages. It's scalable.
Nowadays, nearly all the developers, when they finish development, either they or the team leader runs it, and they have to fix the problems.
How are customer service and support?
The customer service and support are okay because the thing is, we spoke with the integrator, so we didn't reach Checkmarx tech support.
How would you rate customer service and support?
What about the implementation team?
The setup was done by an integration company.
What other advice do I have?
I would definitely recommend it. It's an excellent solution.
Overall, I would rate the solution a nine out of ten because there is always room for improvement.
Checkmarx could perhaps give more examples of solutions in the reports. It's very good, but sometimes the solutions they give are not necessarily relevant to the code or how it's written.
So, Checkmarx should give more examples of solutions. Although, it's not that bad because they give a few, one or two. And if you want more, you can look online. But it would help if they could refine it and give additional options for solutions.
Disclosure: My company does not have a business relationship with this vendor other than being a customer.
