Ok, hold on guys. This isn't an issue about vRAM. vRAM doesn't even get utilized until a world is actually being generated for rendering. All the texture stitching is being done within system RAM.
In fact, this issue isn't strictly related to RAM at all. This issue is related to the way Java handles memory and the way things are being built. Simply put, with the right tuning there is no reason you shouldn't be able to launch this.
First off, I'm running a similar setup today.
System: Ryzen 2700x, GTX 1080Ti, & 32GB RAM.
Mods: DDSS mod pack + Millenaire + JourneyMap
Resources: 256x PureBD, 3D Blocks Addon, 3D Items Addon, Tech Addon, ExOf Addon, 128x DDSS Sphax, 128x Millenaire Sphax
Resolution: 4K
I've had only one issue (mentioned earlier in this thread), but I've been able to work around that and still put in nearly 100 hours of game time thus far. So what you are trying to do is certainly possible, but the issue you are having is a familiar one. Allow me to explain...
The memory you allocate in your JVM args is for the heap only. There is additional memory being allocated for other things on top of that. I've found a number of things that people typically say works well will not work at all when loading a massive amount of things like this. For example, I could never find any settings to launch using the G1 garbage collector.
There is also the issue of allocating too much memory. When texture stitching is complete, the memory usage skyrockets. If the usage jumps too quick, the JVM shuts down as it appears you encountered a memory leak. Java is greedy for memory, so if you allocate it, it will try to consume it (again, we are not strictly talking about the heap here).
I found that the args -Xms18G -Xmx18G worked the best for me. If I upped the heap settings to 20+, it would never launch and give me the same error you are seeing (again, the error just says it failed at that point).
To illustrate a few of these points, I'll share the memory usage graphs for both my system and video RAM. Take note of the call outs I made so you know where in the process it is.
You'll see that video RAM isn't touched until a world is actually rendered, so if you can't get to the title screen then it has nothing to do with vRAM.
Systems can be different in many ways and we are riding the edge on what is possible with Java here, so YMMV. But I'd try the following:
Use JVM args: -Xms18G -Xmx18G
Download process explorer and watch your memory curve. Post it up here if you fail to launch. I can help troubleshoot further with that info.
You can also try a different version of Java. You are using the built in version right now which is Java 8 build 51. I'd try the latest build of Java 8 (231). I've been more successful with this build. Openj9 could also work, but you'll need some more tuning arguments.
I also want to call out what Han Fox mentioned too. Mipmapping expands the size of the texture atlas, and in all likely hood, you won't be able to have mipmapping & all these textures, so you should probably set that to 0 as he suggested.