Thread model:
- replace the IOState CAS state machine with a StampedLock: region IO
(RegionFileStorage.write/flush, now both guarded via @WrapMethod) holds
the read lock, backups hold the write lock; the old witness-restore
pattern could wedge the state permanently
- run saveEverything on the main thread (was called from the backup
executor), with shutdown-aware waiting; skip autoSave/saveEverything
while a backup holds the write lock so the main thread never blocks
- broadcast player messages from the main thread, honor `broadcast`
Backup correctness:
- watcher reported the parent directory instead of the changed file, so
every incremental backup was empty; change sets are now deduplicated,
atomically drained and merged back on failure; OVERFLOW forces a full
backup; watch roots are re-registered when their inode changes (world
format conversion orphaned the watches)
- fix the region reassembly pipeline: zlib was "decompressed" with
DeflaterInputStream, writeChunk dropped the 5-byte chunk header,
the length field was off by one, coordinates were transposed, and
sector padding was wrong on 4096-byte multiples — every reassembled
chunk in old bundles is unrecoverable garbage
- full backups now collect the file list after saving, off-thread
New features:
- reflink (FICLONE) staging via FFM: on CoW filesystems the write lock
is only held while cloning, packing/uploading runs unlocked
- chained restore: restoring an incremental automatically applies its
full base and every incremental in between, in order
- generation-based rotation implementing keepBackups; whole generations
are deleted so an incremental never loses its base
- Backup records carry subject/createdAt/incremental for chaining
Reliability:
- atomic index persistence (tmp + ATOMIC_MOVE), ConcurrentHashMap for
the tracked-backup index, fsync bundles before storing
- BundleReader: overlay extraction for chains, zip-slip guard, fix
reversed createSymbolicLink args, EOF instead of infinite loop on
truncated bundles
- LocalBackupStrategy.deleteBackup compared the wrong field and could
never delete; S3 retries now catch SdkException (retry was dead code),
drop ifNoneMatch for idempotent retries
- ordered shutdown: SERVER_STOPPING awaits in-flight backups before the
final world save; SERVER_STOPPED runs restore, closes storage and
cleans the temp dir with isolated error handling
- fix /backup suspend being registered under /backup create
Tests: region round-trip byte-comparison, BundleReader overlay/zip-slip/
symlink/truncation, backup chain resolution and rotation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Key changes:
- Renamed RegionFileReassembler to RegionParser for better semantics
- Refactored RegionParser to use ByteBufAllocator injection and improved memory management
- Added thread-local buffer reuse in BundleWriter to reduce allocations
- Implemented Zstd dictionary training functionality for region files
- Added region file statistics collection for training optimization
- Improved error handling and resource management
Detailed changes:
BundleReader/BundleWriter:
- Updated to use RegionParser instead of RegionFileReassembler
- Modified RegionParser constructor to accept ByteBufAllocator
- Changed writeReassembled() method signature to use injected allocator
- Added thread-local buffer reuse to reduce memory allocation overhead
- Improved logging with better formatted output
Main class:
- Added dictionary training support with --train, --train-sample-size, and --train-dict-size options
- Implemented region file statistics collection with --stat option
- Added proper validation for training parameters
- Enhanced help text and error messages
RegionParser:
- Improved file reading using FileChannel for better performance
- Added size validation to prevent processing overly large files
- Implemented readChunkDecompress() method for training purposes
- Better resource management with proper buffer allocation/release
New features:
- ZstdRegionTrainer class for training zstd dictionaries on region files
- Statistics collection for chunk size distribution analysis
- Dictionary training workflow integrated into main application
Performance improvements:
- Reduced memory allocations through buffer reuse
- Better resource management with proper close() implementations
- Improved error handling for corrupted region files
Testing:
- Updated test cases to reflect RegionParser rename
- Add fastutil and lz4-java dependencies for efficient data structures and compression
- Implement VarInt/VarLong serialization for more compact bundle headers
- Redesign bundle header format to include timestamps and use variable-length integers
- Replace GZIP-based compression with region file reassembly for better compression of chunk data
- Add RegionFile and RegionFileReassembler classes for handling Minecraft region files
- Implement configurable compression types (GZIP, ZLIB, LZ4, NONE) for region files
- Add timestamp filtering and duplicate handling to BundleReader
- Improve parallel processing capabilities with configurable worker threads
- Add JUnit 5 testing support and configure shadowJar minimization
- Update file extensions from .jpack to .jbp.zst for clarity
- Enhance error handling and logging throughout the codebase
- Add comprehensive region file manipulation including chunk-level recompression
- Implement memory-efficient region file parsing using Netty ByteBuf
- Add support for dictionary-based Zstd compression
- Update backup worker to use new bundle format
- Various performance optimizations and bug fixes
- Convert project to multi-module structure with 'bundler' subproject
- Extract bundler functionality into separate module with proper dependencies
- Improve IO state management with atomic reference for better synchronization
- Enhance bundler performance with parallel processing support
- Add CLI interface to bundler with various compression options
- Fix file handling issues in bundle reader/writer
- Update compression configuration and remove redundant options
- Improve region file synchronization with proper state management
- Add dictionary support for better compression ratios
- Refactor backup strategies to use new bundler API
- Update dependencies and build configuration
- Fix path resolution issues in bundle operations
`
- Remove s3-lite submodule and replace with AWS SDK and shadow plugin
- Implement comprehensive backup commands with list, delete, create, restore, and suspend functionality
- Add bundle format with compression support using Zstd and gunzip optimization
- Implement local and S3 backup strategies with configurable options
- Add backup tracking system with JSON persistence
- Improve world directory handling and file exclusion logic
- Add standalone bundler tool for bundle creation/extraction
- Update Gradle configuration with shadow jar support and dependency management
- Enhance configuration system with additional options for compression and temporary directories
- Add zstd-jni dependency for compression support
- Define Backup record with name, key, incremental, and size fields
- Implement BackupManager with backup listing, deletion, and creation methods
- Create BackupWorker to handle backup operations with server state coordination
- Refactor WorldDir to simplify dimension handling and file listing
- Introduce bundling system with SerialBundlerReader^[[1;5P/Writer for efficient storage
- Add BackupStrategy interface with LocalBackupStrategy implementation
- Support both compressed and uncompressed backup formats
- Add config option for parallel backup operations
- Remove old backup implementation from JustBackupMod
- Update backup file format with metadata footer and magic number validation
iceBear67