MPI_ERR_TRUNCATE during halo construction after periodic joining (v9.0.0)

code_saturne usage
1

Hello,

I am running a periodic channel case with code_saturne 9.0.0 and the preprocessing stage aborts with an MPI error immediately after the joining operation. Attachments are restricted to images on this forum, so I am pasting the relevant log extracts below.

Setup

  • code_saturne 9.0.0, GCC 11.1.0, Open MPI 4.0.1 (UCX 1.12), PT-SCOTCH 7.0.4
  • Mesh: CGNS exported from Fluent 2025 R1, 4 300 241 polyhedral cells, 21 861 201 interior faces, 15 729 561 vertices
  • Translation periodicity of 0.46125 m along z, selection criterion all[]
  • Single node, 52 MPI ranks (also tested with 24)

End of run_solver.log (with periodicity)

The joining completes and reports success, and this is where the log stops:

  Joining operation is non-conforming.

  Global number of new vertices to create:       8309

  Equivalences broken for 0 edges.

  Pre-merge for 23525 global element couples.

  Max size of a merge set of vertices: 5

  Excessive transitivity for 0 set(s) of vertices.

  Edge removed for 0 faces (global).
  Join mesh cleaning done.

  Merge vertices and mesh update done.

  Global number of faces after splitting:    1257975

  Global configuration after the joining operation:
     Global number of border faces to add:     624134

  Mesh cleaning done for degenerate faces.
    Global number of cleaned interior faces:        0
    Global number of cleaned border faces:          0

  After joining
     Number of cells:          4300241
     Number of interior faces: 21872039
     Number of boundary faces: 624134
     Number of vertices:       15736754

  Joining  1 completed (3.45 s).

  All joining operations successfully finished:

    Wall clock time:                  3.45

Nothing is printed after this: Composing periodicities never appears.

Job stderr

*** An error occurred in MPI_Waitall
*** reported by process [1567358977,39]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_TRUNCATE: message truncated
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
solver script exited with status 15.

(preceded only by repeated UCX VM_UNMAP warnings from all 52 ranks). No error* files are produced.

Same mesh, same 52 ranks, periodicity removed

The whole preprocessing completes without any error:

 Halo construction with extended neighborhood
 ============================================

 Face interfaces creation
 Vertex interfaces creation
 Halo creation
 Halo definition
    Local halo definition
    Distant halo creation
    Updating the faces -> cells connectivity
 Extended neighborhood structures definition

All mesh quality checks pass (0% bad cells on all five criteria). The failure is also independent of the number of ranks: identical behaviour with 24 and 52.

Possible clue

The two periodic surfaces do not have the same number of faces: inlet-Pg has 2150 and outlet-Pg has 2143. The joining is reported as non-conforming and creates 8309 new vertices. The mesh has a refined boundary layer with a first cell thickness of 0.03 mm, so the joining tolerances on the periodic surfaces are very small.

Could you help me understand what is causing this? Is the mismatch between the two periodic surfaces likely to be the problem, or should I look elsewhere?

Thanks in advance,
Nicolò

2

Hello,

I would first look at your selection criterion. With all[],the periodicity is not applied to your two periodic groups but to the entire boundary: your log shows 1.2M faces after splitting, roughly twice your 624 000 boundary faces, while only 2150 + 2143 faces are actually periodic.

So I would retry with only two groups as criterion : inlet-Pg or outlet-Pg, and see whether the preprocessing goes through.

You can also have a look at this tutorial for the setup of a streamwise-periodic case : Tutorials periodic case

Regards,
Florian

3

Hello Florian,

Thank you very much, that was indeed the problem. Changing the selection criterion from all[] to inlet-Pg or outlet-Pg solved it: the joining now works on the 4293 periodic faces only and the preprocessing completes without any error.

Thanks also for the tutorial link, very useful.

Best regards,
Nicolò

1 Like