Our company suffered a violent and terrible computer attack, which could have destroyed it. But, thanks in particular to the relentlessness of her computer team, she is still alive, and without sequelae. This article is a feedback that could be used by other companies.
March 19: Saint Joseph
7:55 am: a technician (aspiring administrator) told me that he could not connect to one of our virtual servers in RDP (Remote Desktop Protocol). Immediately, I interrupted what I was doing and connected to this machine directly, via vSphere (VmWare).
Seeing the skull, I knew we had been attacked and immediately I suspended all the virtual servers. Because, in such circumstances, it is necessary to freeze the situation and collect evidence (time-stamped files, software used, files injected).
The IT Director wanted to have a quick estimate of the extent of the damage. The backups seemed correct. Indeed, another technician, aspiring administrator as the first, who often makes relevant observations, too, said that the backups were good because we received the automatic email at the end of backups and it was all green ( no anomalies).
But by checking directly on the backup servers, I quickly realized that, apparently, there was not much left because many files were missing and the files present all appeared encrypted. So, it seemed we had nothing: no local backup, no backup on the remote site, no replicas of virtual servers ... nada! Apparently, there was nothing but smoking ashes. This physical machine seemed inapplicable, apparently, but we kept the hope of recovering something, enough to revive the business. And also, we still had to see what was possible to recover on the other physical machine, the backup server of the site PRA (Plan of resumption of activities), that we had to bring back to the headquarters of the business as soon as possible.
Thus, according to preliminary findings, the hackers had destroyed everything, immediately after the automatic sending of the email "report of the automatic backups".
At that moment, I told myself that the day was going to be long, very long. This was the case. The company's employees were worried, and rightly so, and several people came on the news. We had to isolate ourselves a bit, the time to recover the data to save the company and, consequently, all the jobs.
And there was a communication in this direction:
That night, I came back very late. And even if she knows that I never divulge the confidential information, my wife wanted to know at least if it was serious: "- You who always find a lot of computer solutions, you can not do anything? - Yes, I do the maximum ... I'll see what I can do ... I'm going to sleep a bit and I'm going back. " Like everyone else, she knew that the attack had been very violent and well prepared by pirates who knew what they were doing. We first had to put the company back in working order and then, in a second step, analyze the traces to identify the origin of the attack.
The next day, I returned to the "war", with the encouragement of my wife who asked me to do everything possible, and even the impossible. The days were long: they started early in the morning and ended late, often in the middle of the night.
Sur le trajet du retour à la maison : "balade" dans la ville, en pleine nuit... que les journées ont été longues !
To make up for a little fatigue, you needed several good doses of coffee or tea ...
All the IT team mobilized, spontaneously:
the administrators the administrator (that is me, because the other administrator had resigned a few weeks before), the technicians, the developers, as well as the colleagues of the cartography, and even an external IT provider ... Everyone participated in the battle ... even the DSI (Information System Manager). We worked so much together, him and me, day and night, that during this period, I spent more time with him than with my wife ;-)
While I was dealing with the virtual servers with the DSI, colleagues scanned all the users' physical machines with a USB stick containing SOPHOS, a software that could detect one of the identified noise programs (ZEUS).
The backup server that was on the PRA site was reported to the company headquarters and put in the server room to get everything we could. Because the pirates had prepared well. And they waited until the end of automatic backup operations to, then, almost destroy everything: local backups, remote backups, and even replicas of virtual servers. It was shocking, especially since other companies were attacked the same night, and we know today that some of them have not recovered.
As a security measure, I can not go into too much detail and disclose sensitive information. However, I can share some information that might be useful to other companies.
Thus, without detailing how the hackers have entered (although I am not concerned), I can say that they have used several programs including mimikatz, a program that Benjamin Delpy (aka Gentil Kiwi), Head of the Research Center & Development in Security of the Bank of France (according to the different articles about it), would have created "to learn the language C and to make some experiments with the security of Windows".
So, a so-called "amateur" developer would have found several major security flaws in Windows, the most used operating system in the world! Just that, it's already remarkable! In addition, the nice amateur programmer ("Gentil Kiwi") would have "the kindness" to share his findings by freely distributing the source codes of his programs on the Internet ... since 2007! And these programs would still be virulent today, in 2019, on recent Windows systems, potentially compromising, globally, the smooth operation of many companies or public bodies including, for example, hospitals (see non-exhaustive list at the end of the article). It's a joke? But no ... This is not a "fake news", if you have to believe what is mentioned on the site of the author, which exposes the said security flaws and indicates, even, the instructions for use of its programs:
Plus on partage, plus on possède. Voilà le miracle ! (Leonard Nimoy)
Le blog de Gentil Kiwi
"Rendu public en 2007"... et toujours fonctionnel en 2019, sur les version récentes de Windows !... Please, Micro$oft... Wake up !
"mimikatz is a tool I've made to learn C and make somes experiments with Windows security." (gentilkiwi)
In fact, initially, with Mimikatz, Benjamin Delpy wanted to demonstrate the vulnerability of Microsoft's authentication protocols. Then, he expanded his "package" ... which was recovered by the pirates. The revelations of "Kind Kiwi" are staggering and his work is remarkable. However, one could wonder about the advisability of distributing this kind of software on the Internet, moreover freely accessible, so at the mercy of unscrupulous people who, like the pirates of the digital world, use them to try to extract money by "kidnapping" data. Mimikatz is especially interesting from a didactic point of view and, if we have to believe Mr. Delpy himself, Microsoft would have secured his system in the meantime. It's hard to believe when we know that our servers have been vulnerable, recently, even though they enjoy the latest updates from Microsoft.
And, at home, hackers took care to erase most of their traces in the logs of our Windows servers. But we did not find many clues during the analyzes:
une trace mimikatz
quelques programmes utilisés par les pirates (mimilove fait partie du package "mimikatz")
les mots de passe décodés par les pirates : ils ne sont plus d'actualité dans notre entreprise ;-)
Mimikats makes it possible to reveal all the passwords. And, with a password "administrator", the hackers did what they wanted, without being worried, not even by our anti-virus server they neutralized. The "dark forces" did a lot of damage that night. But we faced.
For all intents and purposes, I specify that these passwords are no longer used in our company (which I will not mention the name, by excess of security). In addition, the DSI has charged me to control all the rules of our firewalls. The security has been reinforced: some old rules (which were there even before my arrival in the company) have been modified or removed. The new Manager, who will take office in June, will probably be more receptive when I report a security issue from now on (better management of password files, more restricted access rights, etc ...).
During this period, our server room, which is usually very tidy, had become a battlefield. We recovered all that was possible to recover in the disks to rebuild our servers (data, account, payroll, etc) from the few remaining elements, including "snapshots" (kind of backup for virtual machines) which can help rebuild the servers. And, little by little, we recovered a lot, including the main data server (DATA) that had been declared "completely irrecoverable" by the specialist providers. However, I noticed an inconsistency in vSphere (manager of VmWare): for this server, the manager indicated that there was no snapshot. But, curiously, he proposed to "delete all snapshots". So, to remove the doubts, we made a low-level connection (SSH, direct connection without going through vSphere) on the Veeam physical server that contained backups deleted by hackers. And we found a snapshot of this server ... the Holy Grail, almost ... Hallelujah!
He was happy to do this check which allowed to recover this important virtual server DATA because, after the necessary refurbishment of the physical server "Veeam" (complete reinstallation of the machine and the backup system), we would have nothing could recover. To fully understand what happened, you should know that Veeam automatically, before the backup, a snapshot of each virtual machine to backup. This snapshot is automatically removed by Veeam at the end of the backup. As the hackers destroyed the Veeam server immediately after the backup, the snapshot could not be removed by Veeam. And we got it back. If your company is attacked, consider doing this check which could allow you to recover some items.
And, after verifying that the DATA server was saved and that all the data was recoverable (unencrypted), I was able to announce the good news to the employees as well as to the members of the Management, who had questioned me a few days before . :
On hearing this good news, everyone was relieved and we received several messages (it's warm to the heart):
These testimonials of satisfaction have already been a great reward. And that was not all ! Indeed, a feast was prepared in honor of the computer team, which triumphed the nasty pirates. We feasted!
And, in addition, the management invited the entire team to lunch in a renowned gourmet restaurant. Again, it was delicious!
... With a letter of thanks from Management, in addition to the bonus. It is a pleasure to work for grateful leaders.
Chaleureux remerciements de la Direction de notre belle entreprise familaile, à dimension internationale
This experience has shown that one is never cautious: whatever the level of safety, there are always pests that try to break into the systems. But, if it is true that "what does not kill makes you stronger," then something good will come out of that experience. And our company still lives ... and gives life to its many employees.
So, if your business is under attack, do not give up. Let your computer experts ... that you have rigorously selected ... proactively. And know how to keep them!
JM + MP + TS = Semper fidelis, Semper paratus
Here are some news articles about computer attacks suffered by other companies:
Some funny quotes:
- Règle informatique n°1 : Si tout va bien, ne touchez à rien !
- L'urgent est fait, l'impossible est en cours, pour les miracles prévoir un délai...
- Article 1 : Le chef a toujours raison. Article 2 : Si le chef a tort, appliquer l'article 1.
- Nos clients sont nos meilleurs beta testeur. (Microsoft ?)
- Si les ouvriers construisaient les bâtiments comme les développeurs écrivent leurs programmes, le premier pivert venu aurait détruit toute civilisation (Gerald Weinberg)
- Aujourd’hui, la programmation est devenue une course entre le développeur, qui s’efforce de produire de meilleures applications à l’épreuve des imbéciles et l’univers, qui s’efforce de produire de meilleurs imbéciles. Pour l’instant, l’univers a une bonne longueur d’avance (Rich Cook)
- Si on peut utiliser l'Iphone 5 d'une seule main, c'est parce qu'il coûte déjà l'autre bras.
- To do : Créer un navigateur et l’appeler Christophe Colomb.
- No keyboard present, press F1 to resume...
- Computers are like air conditioners - They stop working properly when you open Windows.
- A bus station is where bus stops... a train station is where train stops... My computer is a workstation...
- Un PC devient lent et difficile à utiliser dès que celui d’un des autres employés du service a été remplacé par un neuf.
- Grâce à l’ordinateur, on peut faire plus rapidement des choses qu’on n’aurait pas eu besoin de faire sans ordinateur.
- Un disque dur qui foire, ça n’arrive jamais, sauf quand ça arrive...
- La probabilité d’un crash du disque dur augmente de manière exponentielle avec l’âge de la dernière sauvegarde complète.
- Le bug se trouve parfois entre la chaise et le clavier.
- L'homme est toujours l'ordinateur le plus extraordinaire de tous. (John F. Kennedy)
- Une entreprise dans laquelle il n'y a pas d'ordre est incapable de survivre ; mais une entreprise sans désordre est incapable d'évoluer. (Bernard Nadoulek)
- Le vrai courage ne se laisse jamais abattre. (Fénelon/Télémaque)
- L’intégrité est une composante essentielle de la sécurité. Et pas seulement en informatique ! (Didier Hallépée)
- En informatique, la miniaturisation augmente la puissance de calcul. On peut être plus petit et plus intelligent. (Bernard Werber)
- Avec une montre, on connaît l'heure. Avec deux, on est jamais sûr (Murphy)
- Dans toute organisation, il y a toujours une personne qui sait ce qui se passe. Elle doit être virée. (Murphy)
- Celui qui sourit lorsque les choses vont mal, a déjà pensé à celui qui portera le chapeau. (Murphy)
- Quel que soit le nombre de preuves démontrant la fausseté d'une chose, il se trouve toujours quelqu'un pour croire qu'elle est vraie. (Murphy)
- Faire du mal me serait trop pénible, j'aime bien mieux le supporter. (Jean Frain du Tremblay)
- Spéciale dédicace à Molon/Gioria/Klein : La justice de Dieu est sans appel. Les forces du mal n'ont pas d'avenir. La vérité triomphe toujours.
- Dieu récompensera la vertu et punira le vice, dans ce monde ou dans l'autre. (Benjamin Franklin)
- Quand on fait le mal, c'est en vain que l'on brûle de l'encens et qu'on offre des sacrifices. (Kao-Tong-Kia)
- En un mot, contrairement à bien des humains qui sont malfaisants, si les bêtes tuent, c'est pour se nourrir, et non pour le plaisir, comme le font certains hommes. (Edgar Fruitier)
Nota Bene: NB: None of our Linux servers were impacted by cryptovirus, of course ;-)
Esprit de corps
Je dédie cet article à toutes celles et tous ceux qui partagent les belles valeurs et, en particulier, à l'adjudant qui commandait le troisième peloton du troisième escadron du troisième régiment de Hussards lorsque j'étais sous les drapeaux, un homme droit et généreux.
Le troisième hussard, "il en vaut plus d'un" (devise du 3e RH) et "rien ne l'effraie" (devise du troisième escadron).
Les pirates ?... même pas peur ! Avec un AMX ou avec un clavier... Semper paratus, Semper fidelis