For the NEOM underwater digital twin of Yuba island, a dynamic fish behavior and population system based off of on-location surveys was needed. To solve that, I used UE5 Niagara Graphs in combination with the Procedural Content Graph(PCG).
The method was:
1. On-location surveys.
1. On-location surveys.
Species data would specify size, distance from reef, depth, speed, habitat, etc. This was added to a spreadsheet for all the ~100 fish documented, by the marine biologists on site. At the same time, video and photo of the species were recorded as references for the reconstruction phase.
2. Divide up the data in to 2 categories, dynamic and non-dynamic.
Dynamic data was kept as text-file metadata for each scanned location, so that we could easily add new fish data to any arbitrary location. This would include species name, density and area specifications.
Non-dynamic data was added to an Unreal Data Table, and would be read from the PCG population system or the Niagara particle systems. Some data would be directly translated to either hardcoded animation or particle data.
3. Population of fish through a PCG Graph.
For each scanned location dataset, a new PCG graph was generated at runtime, in order to easily add, remove or optimize data externally. The PCG graph picks up the metadata text file, the geometry from the incoming tileset, and the species specifications in the data table. Additional data was also grabbed from the unreal scene, such as water height, optimization multipliers and splines generated from metadata areas.
Using this information, I determine where a specific species should be distributed, and output a temporary instanced mesh. At runtime, I replace this mesh with the actor holding the Niagara fish particle system depending on the distance from camera.
Using this information, I determine where a specific species should be distributed, and output a temporary instanced mesh. At runtime, I replace this mesh with the actor holding the Niagara fish particle system depending on the distance from camera.
4. Fish modelling and base animation.
I reconstructed the fish through traditional 3d-modelling and rigging, based off of the recorded references. As we wanted performance to be very scalable, I decided to use vertex animation textures(VATs) for all base animation rather than bones. Base animation would include pelvic, pectoral and dorsal fins. On some species, eye movement was also added.
The tail fin and bending on turns was animated through the Niagara system, in order for me to have more control of the animation based on the direction or speed of the fish.
5. Niagara System.
The Niagara system that drives the movement of the fish was built from a lot of stacked modules. Some modules would send data over to the shader in order to bend fish on turns, animate speed and strength of tail fin, modify colors of fish based on depth or modulate the incoming base animation.
Credit: Niagara system was inspired by Ghislain Girardot's excellent boids system, even though Grid 3D sampling would have been too expensive for this project performance-wise.
At a late request from the client, I added a fish gallery where you could view the catalogue of fish and learn more about them.