Forking is a term used in Linux, to indicate executing another process (loosely).
In java world we hear fork VM in run or fork VM in test. This implies to Java VM. In this case, a forked Java VM is a child process used to isolate your unit tests by class or method, ensuring that no state pollution occurs between multiple tests.
To make the understanding a little more clear, we have many build tools like maven, sbt, gradle. More or less these tools are built with Java and would be employing the underlying JVM to run themselves.
Some tools that are involved in various aspects of compilation and testing (e.g. Maven) are written in Java and use the JVM to run themselves.
If you run unit tests for your application without forking the VM, Maven will run those tests within the same VM as Maven is running. Therefore, it may be affected by certain VM-wide settings (e.g. some system properties).
To avoid side-effects due to Maven, it’s possible to run the tests in a forked VM, that is, in a completely separate VM running as a different process in the OS.
(This can apply to other tools, Maven is just an example.)
Crashing a forked VM at least allows you to come back to the other Java application that started and orchestrated these unit tests. If you were running these tests within the same VM, you would also crash the application that launched your tests (and thus get very little information in return).