It was an ordinary afternoon when our Solr 9.7.0 core, emailIndex, crashed in the middle of a heavy write cycle. We restarted the container and expected Solr to recover on its own. Instead, the core refused to come back up.
In the end it took us about five hours to bring it back. The recovery was more surgical than we’d hoped — we rebuilt the index’s internal map file by hand instead of re-indexing from scratch. This post is about what went wrong, how we misread it at first, and what actually fixed it.
The Misleading Error Link to heading
We pulled the logs expecting a run-of-the-mill file truncation error. What we found looked far worse:
cq-solr | org.apache.solr.common.SolrException: Unable to create core [emailIndex]
...
cq-solr | Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(ByteBufferIndexInput(path="/var/solr/data/emailIndex/data/index/segments_8fx"))): 0 (needs to be between 1071082519 and 1071082519).
The first thing that stands out: the message says the index format is “too old.” But we hadn’t downgraded Solr. This server had been running the same version for months, so the error didn’t make sense at first.
Why It Actually Happened Link to heading
Solr is built on Lucene, and Lucene organizes data into segments. A special file — the segments_N file — acts as a master map, kind of like a table of contents for all those segments. When Solr reads that file, it checks a magic number in the header to confirm the version.
Our server had crashed at the exact moment Solr was updating that map file. The filesystem zeroed the file out. When Solr booted back up, it read a literal 0 where the version number should have been.
Since 0 is older than the minimum supported version for Lucene 9, the engine threw IndexFormatTooOldException. It was a red herring. Our index wasn’t old or downgraded — the map file was just blank. The actual data was still on disk, intact. It simply didn’t have a table of contents tying it together anymore.
Why We Didn’t Use CheckIndex Link to heading
Our first instinct was Lucene’s built-in repair tool, CheckIndex. We set it aside fairly quickly.
The problem is that CheckIndex treats a fatal header error as a strong signal that data is damaged, and it tends to discard whole segments to be safe. In our case the header of the map file was broken, but that didn’t mean the underlying segments were corrupt. Our data was healthy — it just wasn’t referenced anymore. Using CheckIndex risked throwing away large chunks of perfectly good historical data.
We decided to ignore the broken map file entirely. We could scan the raw data blocks on disk, identify which ones were intact, and write a brand-new map file from scratch.
Building a Fresh Map File Link to heading
There was one complication: every Lucene segment carries a unique 16-byte identifier in its header. If you get even one byte of that ID wrong, the engine treats the segment as corrupt and refuses to use it. So our recovery script had to read each segment’s real ID directly from its file header and feed it into the new map.
We wrote a small Java program called SolrStitcher to do this. It opens the index directory, loops through the physical segment files, reads their IDs, and commits a fresh segments_N file.
One practical note: production images rarely include a javac compiler, so we pulled a portable Java compiler into the container’s /tmp directory to build and run it.
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.store.IOContext;
import org.apache.lucene.store.IndexInput;
import org.apache.lucene.index.SegmentInfos;
import org.apache.lucene.index.SegmentInfo;
import org.apache.lucene.index.SegmentCommitInfo;
import org.apache.lucene.codecs.Codec;
import java.nio.file.Paths;
public class SolrStitcher {
public static void main(String[] args) throws Exception {
if (args.length < 1) {
System.out.println("Usage: java SolrStitcher <path_to_index_dir>");
return;
}
String dirPath = args[0];
FSDirectory directory = FSDirectory.open(Paths.get(dirPath));
// The definitive list of physical data blocks found on disk
String[] segments = {
"_41m", "_43", "_58", "_5b", "_5l", "_5wd", "_7jv", "_7r9", "_8hh", "_8oi",
"_8u7", "_98p", "_9hr", "_9ky", "_9lk", "_9ll", "_9lm", "_9ln", "_9lo",
"_9w", "_c8", "_dx", "_fp", "_hm", "_kq", "_my", "_q4", "_qo", "_r8",
"_s", "_sf", "_sr", "_sw", "_t4", "_t7", "_t9"
};
SegmentInfos sis = new SegmentInfos(9);
for (String seg : segments) {
String siFile = seg + ".si";
try {
// Read the unique 16-byte id from the segment file header
IndexInput input = directory.openInput(siFile, IOContext.READ);
input.readInt();
input.readString();
input.readInt();
byte[] id = new byte[16];
input.readBytes(id, 0, 16);
input.close();
// Read the full metadata for this segment
SegmentInfo si = Codec.getDefault().segmentInfoFormat().read(directory, seg, id, IOContext.READ);
// Explicitly assign the codec to avoid a NullPointerException
si.setCodec(Codec.getDefault());
// Wrap it in a clean commit and add it to our new map
SegmentCommitInfo commitInfo = new SegmentCommitInfo(si, 0, 0, -1L, -1L, -1L, id);
sis.add(commitInfo);
System.out.println("Re-stitched segment: " + seg);
} catch (Exception e) {
System.err.println("Skipping segment " + seg + ": " + e.getMessage());
}
}
// Write the fresh segments_1 file directly to disk
sis.commit(directory);
System.out.println("\nDone. Wrote a fresh 'segments_1' file.");
directory.close();
}
}
The Traps We Had to Avoid Link to heading
Critical warning: make sure the core is completely offline before running anything like this. If Solr is still running and tries to write while you’re rebuilding the map, you’ll trigger a file lock and make things worse.
When we ran the script, it confirmed our suspicion:
Skipping segment _9ls due to error: codec footer mismatch (file truncated?)
_9ls was the exact block Solr was writing when the crash happened. The filesystem had zeroed it out, so we skipped it. That was the right call — we lost only the small number of documents that were mid-write at that exact second. Everything else, including millions of historical documents, was recovered.
Bringing the Core Back Online Link to heading
After the new map file was written, we finished with three simple steps:
-
Remove the corrupt map so Solr wouldn’t try to read it again:
rm segments_8fx -
Fix permissions so Solr could access the new file:
chown solr:solr segments_1 -
Reload the core via Solr’s CoreAdmin API:
curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=emailIndex"
The API returned a success, Solr mounted the new map, and our search functionality was back online in seconds.
Lessons Learned Link to heading
If you manage search infrastructure, a crash like this doesn’t have to mean rebuilding your index from scratch.
-
Use Solr’s backup API, not raw disk copies. Copying the files with
tarorrsyncwhile Solr is running risks capturing a partial write. Solr’s built-in replication backup handles this correctly. -
A leader-follower setup helps a lot. If the writer node crashes, a follower can take over, then stream a fresh copy to the broken node. That’s a much easier path than manual surgery.
-
Don’t automatically trust the stack trace. A “version too old” error when you haven’t downgraded anything is worth digging into. A zeroed file looks like an empty file to us, but it reads as “version 0” to the computer.
Trust your architecture, verify the raw bytes, and don’t be afraid to do a little surgery on your data when the logs are pointing you in the wrong direction.