Step 6: Error 9#
Error Message#
1$ pylith step06_twofaults.cfg
2
3 >> software/pylith-debug/lib/python3.12/site-packages/pylith/apps/PyLithApp.py:76:main
4 -- info (application-flow)
5 -- Running on 1 process(es).
6 >> src/cig/pylith/libsrc/pylith/utils/PetscOptions.cc:251:static void pylith::utils::_PetscOptions::write(pythia::journal::info_t&, const char*, const pylith::utils::PetscOptions&)
7 -- info (application-flow)
8 -- Setting PETSc options:
9 dm_reorder_section = true
10 dm_reorder_section_type = cohesive
11 ksp_atol = 1.0e-7
12 ksp_converged_reason = true
13 ksp_error_if_not_converged = true
14 ksp_gmres_restart = 100
15 ksp_guess_pod_size = 8
16 ksp_guess_type = pod
17 ksp_rtol = 1.0e-14
18 mg_fine_ksp_max_it = 5
19 mg_fine_pc_type = vpbjacobi
20 pc_type = gamg
21 snes_atol = 5.0e-7
22 snes_converged_reason = true
23 snes_error_if_not_converged = true
24 snes_monitor = true
25 snes_rtol = 1.0e-14
26 ts_error_if_step_fails = true
27 ts_exact_final_time = matchstep
28 ts_monitor = true
29 ts_type = beuler
30 viewer_hdf5_collective = true
31
32 >> src/cig/pylith/libsrc/pylith/meshio/MeshIOPetsc.cc:204:virtual void pylith::meshio::MeshIOPetsc::_read()
33 -- info (application-flow)
34 -- Component 'meshiopetsc.reader': Reading finite-element mesh from 'mesh_tri.msh'.
35 >> src/cig/pylith/libsrc/pylith/meshio/MeshIO.cc:76:void pylith::meshio::MeshIO::read(pylith::topology::Mesh*, bool)
36 -- info (application-flow)
37 -- Component 'meshiopetsc.reader': Domain bounding box:
38 (-100000, 100000)
39 (-100000, 0)
40 >> src/cig/pylith/libsrc/pylith/problems/TimeDependent.cc:316:virtual void pylith::problems::TimeDependent::verifyConfiguration() const
41 -- info (application-flow)
42 -- Component 'timedependent.problem': Verifying problem configuration.
43 >> software/pylith-debug/lib/python3.12/site-packages/pylith/problems/Problem.py:238:_printInfo
44 -- info (application-flow)
45 -- Scales for nondimensionalization:
46 Length scale: 2500*m
47 Displacement scale: 1*m
48 Time scale: 3.15576e+09*s
49 Rigidity scale: 1e+10*m**-1*kg*s**-2
50 Temperature scale: 1*K
51 >> src/cig/pylith/libsrc/pylith/problems/TimeDependent.cc:342:virtual void pylith::problems::TimeDependent::initialize()
52 -- info (application-flow)
53 -- Component 'timedependent.problem': Initializing problem.
54[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
55[0]PETSC ERROR: Error in external library
56[0]PETSC ERROR: Could not find values for initiation_time at ( -21860.3 -27621) in spatial database 'Fault rupture for main fault'.
57[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!
58[0]PETSC ERROR: Option left: name:-ksp_atol value: 1.0e-7 source: code
59[0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: code
60[0]PETSC ERROR: Option left: name:-ksp_error_if_not_converged (no value) source: code
61[0]PETSC ERROR: Option left: name:-ksp_gmres_restart value: 100 source: code
62[0]PETSC ERROR: Option left: name:-ksp_guess_pod_size value: 8 source: code
63[0]PETSC ERROR: Option left: name:-ksp_guess_type value: pod source: code
64[0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-14 source: code
65[0]PETSC ERROR: Option left: name:-mg_fine_ksp_max_it value: 5 source: code
66[0]PETSC ERROR: Option left: name:-mg_fine_pc_type value: vpbjacobi source: code
67[0]PETSC ERROR: Option left: name:-pc_type value: gamg source: code
68[0]PETSC ERROR: Option left: name:-snes_atol value: 5.0e-7 source: code
69[0]PETSC ERROR: Option left: name:-snes_converged_reason (no value) source: code
70[0]PETSC ERROR: Option left: name:-snes_error_if_not_converged (no value) source: code
71[0]PETSC ERROR: Option left: name:-snes_monitor (no value) source: code
72[0]PETSC ERROR: Option left: name:-snes_rtol value: 1.0e-14 source: code
73[0]PETSC ERROR: Option left: name:-ts_error_if_step_fails (no value) source: code
74[0]PETSC ERROR: Option left: name:-ts_exact_final_time value: matchstep source: code
75[0]PETSC ERROR: Option left: name:-ts_monitor (no value) source: code
76[0]PETSC ERROR: Option left: name:-ts_type value: beuler source: code
77[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
78[0]PETSC ERROR: PETSc Development Git Revision: v3.25.1-168-g70613f6caab Git Date: 2026-05-21 15:15:28 +0000
79[0]PETSC ERROR: software/pylith-debug/bin/mpinemesis with 1 MPI process(es) and PETSC_ARCH arch-pylith-debug on igskci164warlng.gs.doi.net by baagaard Tue May 26 14:20:23 2026
80[0]PETSC ERROR: Configure options: --PETSC_ARCH=arch-pylith-debug --with-debugging=1 --with-clanguage=c --with-mpi-compilers=1 --with-shared-libraries=1 --with-64-bit-points=1 --with-large-file-io=1 --with-lgrind=0 --download-parmetis=1 --download-metis=1 --download-triangle --download-ml=1 --download-superlu=1 --with-fc=0 --download-f2cblaslapack --with-hdf5=1 --with-hdf5-dir=software/pylith-debug --with-zlib=1 CFLAGS+=-g
81[0]PETSC ERROR: #1 static PetscErrorCode pylith::topology::FieldQuery::queryDBPointFn(PylithInt, PylithReal, const PylithReal*, PylithInt, PylithScalar*, void*)() at src/cig/pylith/libsrc/pylith/topology/FieldQuery.cc:320
82[0]PETSC ERROR: #2 DMProjectPoint_Func_Private() at /software/baagaard/petsc-dev/src/dm/impls/plex/plexproject.c:143
83[0]PETSC ERROR: #3 DMProjectPoint_Private() at /software/baagaard/petsc-dev/src/dm/impls/plex/plexproject.c:545
84[0]PETSC ERROR: #4 DMProjectLocal_Generic_Plex() at /software/baagaard/petsc-dev/src/dm/impls/plex/plexproject.c:1083
85[0]PETSC ERROR: #5 DMProjectFunctionLocal_Plex() at /software/baagaard/petsc-dev/src/dm/impls/plex/plexproject.c:1114
86[0]PETSC ERROR: #6 DMProjectFunctionLocal() at /software/baagaard/petsc-dev/src/dm/interface/dm.c:8361
87[0]PETSC ERROR: #7 void pylith::topology::FieldQuery::queryDB()() at src/cig/pylith/libsrc/pylith/topology/FieldQuery.cc:227
88Fatal error. Calling MPI_Abort() to abort PyLith application.
89Traceback (most recent call last):
90 File "software/pylith-debug/lib/python3.12/site-packages/pylith/apps/PetscApplication.py", line 55, in onComputeNodes
91 self.main(*args, **kwds)
92 File "software/pylith-debug/lib/python3.12/site-packages/pylith/apps/PyLithApp.py", line 85, in main
93 self.problem.initialize()
94 File "software/pylith-debug/lib/python3.12/site-packages/pylith/problems/Problem.py", line 212, in initialize
95 ModuleProblem.initialize(self)
96 File "software/pylith-debug/lib/python3.12/site-packages/pylith/problems/problems.py", line 165, in initialize
97 return _problems.Problem_initialize(self)
98 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
99RuntimeError: Error detected while in PETSc function.
100C++ traceback (13 frames):
101 [0] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith8topology10FieldQuery7queryDBEv+0x292) [0x7c7f835a1782]
102 [1] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith10feassemble16AuxiliaryFactory15setValuesFromDBEv+0x2ca) [0x7c7f834025e4]
103 [2] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith6faults6KinSrc10initializeERKNS_8topology5FieldERKNS_6scales6ScalesEPKN11spatialdata9geocoords8CoordSysE+0x51d) [0x7c7f833880cd]
104 [3] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith6faults16FaultCohesiveKin20createAuxiliaryFieldERKNS_8topology5FieldERKNS2_4MeshE+0xa51) [0x7c7f8336f305]
105 [4] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith10feassemble10Integrator10initializeERKNS_8topology5FieldE+0x29f) [0x7c7f833a7693]
106 [5] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith10feassemble19IntegratorInterface10initializeERKNS_8topology5FieldE+0x3d6) [0x7c7f833ca87c]
107 [6] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith8problems7Problem10initializeEv+0x400) [0x7c7f8351d846]
108 [7] software/pylith-debug/lib/libpylith.so.0(_ZN6pylith8problems13TimeDependent10initializeEv+0x26f) [0x7c7f8352d011]
109 [8] software/pylith-debug/lib/python3.12/site-packages/pylith/problems/_problems.so(+0x15c30) [0x7c7f7cfeec30]
110 [9] software/pylith-debug/bin/mpinemesis(+0x15a51) [0x5887f81e9a51]
111 [10] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7c7f9fc29d90]
112 [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7c7f9fc29e40]
113 [12] software/pylith-debug/bin/mpinemesis(+0x5805) [0x5887f81d9805]
114
115--------------------------------------------------------------------------
116MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
117 Proc: [[18481,1],0]
118 Errorcode: -1
119
120NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
121You may or may not see output from other processes, depending on
122exactly when Open MPI kills them.
123--------------------------------------------------------------------------
124--------------------------------------------------------------------------
125prterun has exited due to process rank 0 with PID 0 on node igskci164warlng calling
126"abort". This may have caused other processes in the application to be
127terminated by signals sent by prterun (as reported here).
128--------------------------------------------------------------------------
129software/pylith-debug/bin/nemesis: mpiexec: exit 255
130software/pylith-debug/bin/pylith: software/pylith-debug/bin/nemesis: exit 1
Troubleshooting Strategy#
We still get an error about not being able to find an initiation time for a point.
This suggests there are still one or more errors in our spatial database file related to interpolation.
We examine the header and data points for errors.
We notice that our deepest point has a y coordinate of -25 km, but PyLith is looking for values at a point with a y coordinate of -27.621 km.
We need to add an additional point to our spatial database.
This explains why we had num-locs=4 when we started!
Resolution#
num-locs = 4
...
0.0 99.0 -2.0 0.0 0.0
0.0 -20.0 -2.0 0.0 0.0
0.0 -25.0 0.0 0.0 0.0
0.0 -99.0 0.0 0.0 0.0
Our simulation now runs without errors and the output looks correct.