Compare commits

...
10 Commits
Author SHA1 Message Date
iceBear67andClaude Opus 5 045770958b port to Minecraft 26.3
build / build (push) Canceled after 0s
No source changes were needed — nothing JustBackup touches moved in 26.3.
Version bumps only:

- minecraft 26.2 -> 26.3, loader 0.19.3 -> 0.19.5
- fabric-api 0.154.2+26.2 -> 0.160.5+26.3
- fabric.mod.json depends: minecraft ~26.2 -> ~26.3,
  fabricloader >=0.19.3 -> >=0.19.5
- README version line

Verified with `./gradlew build` against 26.3 (clean).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 15:53:27 +00:00
iceBear67 70c2c23522 update gradle wrapper configuration 2026-07-22 15:06:57 +08:00
iceBear67 e1fec8c3ce update license 2026-07-22 15:04:12 +08:00
iceBear67 b21d0f7f61 remove awssdk 2026-07-17 22:45:25 +08:00
iceBear67 ea92ae5e2b fix: record delete operations 2026-07-17 20:13:13 +08:00
iceBear67 f9de570f25 fix timestamp sign problem 2026-07-17 20:02:59 +08:00
iceBear67 cf3049624e update readme 2026-07-17 19:38:56 +08:00
iceBear67 34f7d15566 overhaul thread model and backup reliability
Thread model:
  - replace the IOState CAS state machine with a StampedLock: region IO
    (RegionFileStorage.write/flush, now both guarded via @WrapMethod) holds
    the read lock, backups hold the write lock; the old witness-restore
    pattern could wedge the state permanently
  - run saveEverything on the main thread (was called from the backup
    executor), with shutdown-aware waiting; skip autoSave/saveEverything
    while a backup holds the write lock so the main thread never blocks
  - broadcast player messages from the main thread, honor `broadcast`

  Backup correctness:
  - watcher reported the parent directory instead of the changed file, so
    every incremental backup was empty; change sets are now deduplicated,
    atomically drained and merged back on failure; OVERFLOW forces a full
    backup; watch roots are re-registered when their inode changes (world
    format conversion orphaned the watches)
  - fix the region reassembly pipeline: zlib was "decompressed" with
    DeflaterInputStream, writeChunk dropped the 5-byte chunk header,
    the length field was off by one, coordinates were transposed, and
    sector padding was wrong on 4096-byte multiples — every reassembled
    chunk in old bundles is unrecoverable garbage
  - full backups now collect the file list after saving, off-thread

  New features:
  - reflink (FICLONE) staging via FFM: on CoW filesystems the write lock
    is only held while cloning, packing/uploading runs unlocked
  - chained restore: restoring an incremental automatically applies its
    full base and every incremental in between, in order
  - generation-based rotation implementing keepBackups; whole generations
    are deleted so an incremental never loses its base
  - Backup records carry subject/createdAt/incremental for chaining

  Reliability:
  - atomic index persistence (tmp + ATOMIC_MOVE), ConcurrentHashMap for
    the tracked-backup index, fsync bundles before storing
  - BundleReader: overlay extraction for chains, zip-slip guard, fix
    reversed createSymbolicLink args, EOF instead of infinite loop on
    truncated bundles
  - LocalBackupStrategy.deleteBackup compared the wrong field and could
    never delete; S3 retries now catch SdkException (retry was dead code),
    drop ifNoneMatch for idempotent retries
  - ordered shutdown: SERVER_STOPPING awaits in-flight backups before the
    final world save; SERVER_STOPPED runs restore, closes storage and
    cleans the temp dir with isolated error handling
  - fix /backup suspend being registered under /backup create

  Tests: region round-trip byte-comparison, BundleReader overlay/zip-slip/
  symlink/truncation, backup chain resolution and rotation.

  Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 19:35:43 +08:00
iceBear67 592c79ed90 bump version to 26.2 2026-07-17 18:11:09 +08:00
iceBear67 c1836635fb migrate thread cleanup from server.stopped to runtime hook 2026-02-21 18:33:21 +08:00
38 changed files with 2195 additions and 507 deletions

No files matched your search

+17 -117
View File
@@ -1,121 +1,21 @@
Creative Commons Legal Code
MIT License
CC0 1.0 Universal
Copyright (c) 2024 SaltedFish Club
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
HEREUNDER.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
Statement of Purpose
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
The laws of most jurisdictions throughout the world automatically confer
exclusive Copyright and Related Rights (defined below) upon the creator
and subsequent owner(s) (each and all, an "owner") of an original work of
authorship and/or a database (each, a "Work").
Certain owners wish to permanently relinquish those rights to a Work for
the purpose of contributing to a commons of creative, cultural and
scientific works ("Commons") that the public can reliably and without fear
of later claims of infringement build upon, modify, incorporate in other
works, reuse and redistribute as freely as possible in any form whatsoever
and for any purposes, including without limitation commercial purposes.
These owners may contribute to the Commons to promote the ideal of a free
culture and the further production of creative, cultural and scientific
works, or to gain reputation or greater distribution for their Work in
part through the use and efforts of others.
For these and/or other purposes and motivations, and without any
expectation of additional consideration or compensation, the person
associating CC0 with a Work (the "Affirmer"), to the extent that he or she
is an owner of Copyright and Related Rights in the Work, voluntarily
elects to apply CC0 to the Work and publicly distribute the Work under its
terms, with knowledge of his or her Copyright and Related Rights in the
Work and the meaning and intended legal effect of CC0 on those rights.
1. Copyright and Related Rights. A Work made available under CC0 may be
protected by copyright and related or neighboring rights ("Copyright and
Related Rights"). Copyright and Related Rights include, but are not
limited to, the following:
i. the right to reproduce, adapt, distribute, perform, display,
communicate, and translate a Work;
ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person's image or
likeness depicted in a Work;
iv. rights protecting against unfair competition in regards to a Work,
subject to the limitations in paragraph 4(a), below;
v. rights protecting the extraction, dissemination, use and reuse of data
in a Work;
vi. database rights (such as those arising under Directive 96/9/EC of the
European Parliament and of the Council of 11 March 1996 on the legal
protection of databases, and under any national implementation
thereof, including any amended or successor version of such
directive); and
vii. other similar, equivalent or corresponding rights throughout the
world based on applicable law or treaty, and any national
implementations thereof.
2. Waiver. To the greatest extent permitted by, but not in contravention
of, applicable law, Affirmer hereby overtly, fully, permanently,
irrevocably and unconditionally waives, abandons, and surrenders all of
Affirmer's Copyright and Related Rights and associated claims and causes
of action, whether now known or unknown (including existing as well as
future claims and causes of action), in the Work (i) in all territories
worldwide, (ii) for the maximum duration provided by applicable law or
treaty (including future time extensions), (iii) in any current or future
medium and for any number of copies, and (iv) for any purpose whatsoever,
including without limitation commercial, advertising or promotional
purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
member of the public at large and to the detriment of Affirmer's heirs and
successors, fully intending that such Waiver shall not be subject to
revocation, rescission, cancellation, termination, or any other legal or
equitable action to disrupt the quiet enjoyment of the Work by the public
as contemplated by Affirmer's express Statement of Purpose.
3. Public License Fallback. Should any part of the Waiver for any reason
be judged legally invalid or ineffective under applicable law, then the
Waiver shall be preserved to the maximum extent permitted taking into
account Affirmer's express Statement of Purpose. In addition, to the
extent the Waiver is so judged Affirmer hereby grants to each affected
person a royalty-free, non transferable, non sublicensable, non exclusive,
irrevocable and unconditional license to exercise Affirmer's Copyright and
Related Rights in the Work (i) in all territories worldwide, (ii) for the
maximum duration provided by applicable law or treaty (including future
time extensions), (iii) in any current or future medium and for any number
of copies, and (iv) for any purpose whatsoever, including without
limitation commercial, advertising or promotional purposes (the
"License"). The License shall be deemed effective as of the date CC0 was
applied by Affirmer to the Work. Should any part of the License for any
reason be judged legally invalid or ineffective under applicable law, such
partial invalidity or ineffectiveness shall not invalidate the remainder
of the License, and in such case Affirmer hereby affirms that he or she
will not (i) exercise any of his or her remaining Copyright and Related
Rights in the Work or (ii) assert any associated claims and causes of
action with respect to the Work, in either case contrary to Affirmer's
express Statement of Purpose.
4. Limitations and Disclaimers.
a. No trademark or patent rights held by Affirmer are waived, abandoned,
surrendered, licensed or otherwise affected by this document.
b. Affirmer offers the Work as-is and makes no representations or
warranties of any kind concerning the Work, express, implied,
statutory or otherwise, including without limitation warranties of
title, merchantability, fitness for a particular purpose, non
infringement, or the absence of latent or other defects, accuracy, or
the present or absence of errors, whether or not discoverable, all to
the greatest extent permissible under applicable law.
c. Affirmer disclaims responsibility for clearing rights of other persons
that may apply to the Work or any use thereof, including without
limitation any person's Copyright and Related Rights in the Work.
Further, Affirmer disclaims responsibility for obtaining any necessary
consents, permissions or other rights required for any use of the
Work.
d. Affirmer understands and acknowledges that Creative Commons is not a
party to this document and has no duty or obligation with respect to
this CC0 or use of the Work.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+134 -14
View File
@@ -1,19 +1,139 @@
# JustBackup
An ultra-fast, space-efficient backup mod written by iceBear67.
A fast, space-efficient backup mod for Fabric servers, written by iceBear67.
## Features
- Designed with speed in mind.
JustBackupMod utilizies Zstd compression algorithm, which is way faster(80% faster) than algorithms used in our competitors without sacrifising compression rate. (even smaller)
- Chunk reassembler (Optional Feature)
JustBackupMod supports reassembling your chunks to a more space-efficient form. In detail, it tries to recompress your region files using Zstd for better compression rate, while empty areas are also stripped for smaller bundle sizes.
- Incremental.
While JustBackupMod is already fast at full backups, you can enable incremental backups to gain more speed. By default, JustBackupMod perform a full backup on server startup and keep tracking for files modified and needed to make a backup.
- S3 Support.
JustBackupMod also supports uploading your backups to S3 to release spaces for your minecraft world.
- **Minecraft 26.3** · Fabric Loader ≥ 0.19.5 · Java 25
- Consistent backups: no torn region files, even while the server is running
- Incremental backups with **chained restore** (a full base + its incrementals restore as one unit)
- **Reflink (CoW) fast path**: on btrfs/XFS/ZFS the world is only "frozen" for milliseconds
- Local directory or **S3-compatible** storage, generation-based rotation
- Zstd compression with optional trained dictionaries and a chunk reassembler
## Roadmap
- [x] Implement all features
- [ ] Further testing and a serious benchmark.
- [ ] Publish to modrinth
## How it works
On each backup, JustBackup:
1. saves the world on the main thread and drains the chunk IO workers, so everything is complete on disk;
2. takes an exclusive lock over region IO (chunk writes park on a read-write lock — the main thread itself never blocks: autosaves and `/save-all` issued during a backup are skipped and logged);
3. snapshots the files — on CoW filesystems by reflink-cloning them into a staging directory (metadata-only, milliseconds, no extra disk space), then releases the lock immediately; without CoW support it keeps the lock while packing;
4. packs everything into a zstd-compressed bundle (`.swb.zst`) and hands it to the storage backend.
Changed files are tracked by a filesystem watcher, so incremental backups only bundle what actually changed. If the OS drops watch events (overflow, directory replaced), the next backup automatically upgrades itself to a full one — the mod never silently backs up less than you think.
### Reflink requirements
The fast path needs Linux and a filesystem supporting `FICLONE` (btrfs, XFS with reflink, OpenZFS ≥ 2.2), and `temporaryBackupDir` must be on the **same filesystem** as the backup subject. Anything else (ext4, overlayfs in containers, cross-mount setups) falls back to packing under the lock — correct, just with a longer pause of chunk IO. The mod logs which mode it uses.
## Configuration
`config/just_backup.json` (created with defaults on first run):
```json
{
"bundleFullAtStartup": true,
"incremental": true,
"broadcast": true,
"backupIntervalMinutes": 60,
"keepBackups": 5,
"compressionLevel": 3,
"compressionWorkerThreads": 4,
"allowGunzip": true,
"temporaryBackupDir": "./.backup_cache",
"zstdDictPath": null,
"backupSubjects": { "world": "world" },
"option": { "type": "local", "saveDir": "backups", "diskSizeQuotaBytes": 0 }
}
```
| Key | Meaning |
|---|---|
| `bundleFullAtStartup` | Take a full backup when the server starts. Recommended: incrementals need a full base to restore. |
| `incremental` | Scheduled backups only bundle changed files. Full backups are still taken at startup and on demand. |
| `broadcast` | Announce backup progress/results to players (always logged to console). |
| `backupIntervalMinutes` | Scheduled backup interval. Skipped while the server is idle (no ticks advanced). |
| `keepBackups` | How many **generations** to keep per subject. A generation = one full backup plus all incrementals after it; old generations are deleted whole, so an incremental never loses its base. |
| `compressionLevel` | Zstd level (must be > 0). |
| `compressionWorkerThreads` | Zstd multithreading; `0`/`1` = single-threaded. |
| `allowGunzip` | Chunk reassembler: recompress region files as a whole for a better ratio (see below). |
| `temporaryBackupDir` | Scratch space for bundles and reflink staging. Put it on the same filesystem as your world to enable the CoW fast path. |
| `zstdDictPath` | Optional zstd dictionary trained with the bundler CLI. |
| `backupSubjects` | Named directory trees to back up, e.g. the world folder. `@all` is reserved. |
| `option` | Storage backend, see below. |
**Local storage** — `{ "type": "local", "saveDir": "backups", "diskSizeQuotaBytes": 0 }`. `diskSizeQuotaBytes` > 0 refuses new backups once the backup dir exceeds the quota (`0` = unlimited).
**S3 storage** —
```json
{
"type": "s3",
"region": "us-east-1",
"accessKey": "...",
"secretKey": "...",
"endpoint": "https://s3.example.com",
"bucket": "my-backups",
"prefix": "myserver",
"enforcePathStyle": true,
"retryAmount": 3
}
```
Uploads retry with exponential backoff up to `retryAmount` times.
## Commands
All commands require permission level 4 (owners).
| Command | Effect |
|---|---|
| `/backup help` | Show help. |
| `/backup list` | List tracked backups with `FULL`/`INCR`, size and creation time, plus clickable delete/restore shortcuts. |
| `/backup create <incremental> <subject\|@all>` | Take a backup now. |
| `/backup restore <backupKey>` | Schedule a restore for the next shutdown. Restoring an **incremental** automatically applies its full base and every incremental in between, in order — the resolved chain is printed for confirmation. |
| `/backup cancelrestore` | Cancel a scheduled restore. |
| `/backup delete <backupKey>` | Delete one backup, no confirmation. |
| `/backup suspend` | Toggle the automatic backup scheduler until next restart. |
### How restore works
The restore runs during server shutdown: the current world directory is moved aside to `<world>_bak_<timestamp>` (nothing is deleted), then each bundle in the chain is extracted over the same destination, later bundles overwriting earlier ones. If anything fails midway, your original world is intact in the `_bak_` directory — move it back to undo.
## The chunk reassembler (`allowGunzip`)
Region files store each chunk individually zlib-compressed, which compresses poorly as a whole. With the reassembler enabled, backups decompress chunks and store them raw inside the bundle so zstd can compress across chunk boundaries (optionally with a trained dictionary); restores rebuild standard `.mca` files with zlib, byte-identical chunk data, vanilla-compatible layout. Region files that fail to parse are stored as-is instead of failing the backup.
## Bundler CLI (`jpack`)
The `bundler` subproject builds a standalone tool for working with `.swb.zst` bundles outside the server:
```console
# extract a bundle
$ java -jar bundler.jar --in backup_world_123.swb.zst --out ./restored_world
# pack a directory
$ java -jar bundler.jar --in ./world --out world.swb.zst --level 9
# train a zstd dictionary on your region files
$ java -jar bundler.jar --train ./world/dimensions --stat
$ java -jar bundler.jar --train ./world/dimensions --train-sample-size <n> --train-dict-size <n>
```
Run with `--help` for all options.
## Known limitations
- File deletions are not tracked: a chained restore may resurrect files deleted between two backups. Minecraft worlds rarely delete files, so in practice this is cosmetic.
- While a backup holds the IO lock (non-CoW fallback), chunk saves queue up in memory and autosaves are skipped; nothing is lost, but on huge worlds prefer a CoW filesystem to keep the pause short.
- Bundles created by pre-1.0 development builds contain incorrectly reassembled region data and cannot be restored. Take a fresh full backup after upgrading.
## Building
```console
$ JAVA_HOME=/path/to/jdk-25 ./gradlew build
```
The distributable mod jar (with dependencies shaded) is `build/libs/justbackup-<version>.jar`; the bundler CLI is under `bundler/build/libs/`. Run `./gradlew runServer` for a dev server in `run/`.
## License
CC0-1.0
+26 -12
View File
@@ -1,5 +1,5 @@
plugins {
id 'fabric-loom' version "${loom_version}"
id 'net.fabricmc.fabric-loom' version "${loom_version}"
id 'maven-publish'
id("io.freefair.lombok") version "9.0.0-rc2"
id("com.gradleup.shadow") version '9.1.0'
@@ -15,6 +15,11 @@ allprojects {
archivesName = project.archives_base_name
}
// Lombok 1.18.40+ is required for JDK 25 (class file major version 69).
lombok {
version = "1.18.46"
}
repositories {
// Add repositories to retrieve artifacts from in here.
// You should only use this when depending on other mods because
@@ -46,33 +51,37 @@ loom {
// If you remove this line, sources will not be generated.
withSourcesJar()
sourceCompatibility = JavaVersion.VERSION_21
targetCompatibility = JavaVersion.VERSION_21
sourceCompatibility = JavaVersion.VERSION_25
targetCompatibility = JavaVersion.VERSION_25
}
tasks.withType(JavaCompile).configureEach {
it.options.release = 21
it.options.release = 25
}
}
dependencies {
// To change the versions see the gradle.properties file
// Minecraft 26.1+ is unobfuscated, so no mappings are required.
minecraft "com.mojang:minecraft:${project.minecraft_version}"
mappings loom.officialMojangMappings()
modImplementation "net.fabricmc:fabric-loader:${project.loader_version}"
implementation "net.fabricmc:fabric-loader:${project.loader_version}"
// Fabric API. This is technically optional, but you probably want it anyway.
modImplementation "net.fabricmc.fabric-api:fabric-api:${project.fabric_version}"
// https://mvnrepository.com/artifact/software.amazon.awssdk/s3
implementation "net.fabricmc.fabric-api:fabric-api:${project.fabric_version}"
shadow project(":bundler")
shadow 'software.amazon.awssdk:s3:2.34.0'
testImplementation 'org.junit.jupiter:junit-jupiter:5.13.4'
testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}
test {
useJUnitPlatform()
}
configurations.implementation.extendsFrom(configurations.shadow)
shadowJar {
configurations = [project.configurations.shadow]
exclude("META-INF")
dependencies {
exclude(dependency("software.amazon.awssdk:aws-xml-protocol:.*"))
exclude(dependency("org.lz4:lz4-java:.*"))
exclude(dependency("it.unimi.dsi:fastutil:.*"))
exclude(dependency("io.netty:netty-all:.*"))
@@ -89,16 +98,21 @@ processResources {
}
}
jar {
// Minecraft 26.1+ is unobfuscated, so there is no remapJar step; the shaded
// jar is the distributable mod. Keep the thin jar under a separate classifier.
archiveClassifier = 'dev'
inputs.property "archivesName", project.base.archivesName
from("LICENSE") {
rename { "${it}_${inputs.properties.archivesName}" }
}
}
remapJar {
input = tasks.shadowJar.archiveFile
shadowJar {
archiveClassifier = ''
}
assemble.dependsOn shadowJar
// configure the maven publication
publishing {
publications {
@@ -16,5 +16,7 @@ public record BundleEntry(
) {
public static short ATTR_REASSEMBLE = 1;
public static short ATTR_SYMLINK = 2;
/** Tombstone: the file was deleted since the base backup; the reader removes it on restore. */
public static short ATTR_DELETE = 4;
public static short MAGIC = (short) 0xA2E3;
}
@@ -6,8 +6,11 @@ import io.ib67.sfcraft.bundler.region.RegionParser;
import io.netty.buffer.ByteBufAllocator;
import lombok.Builder;
import java.io.EOFException;
import java.io.IOException;
import java.io.InputStream;
import java.nio.channels.FileChannel;
import java.nio.file.DirectoryNotEmptyException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
@@ -19,21 +22,29 @@ public class BundleReader {
protected final boolean linkSymbol;
private final byte[] buffer = new byte[4096];
protected final boolean skipOldEntry;
protected final boolean ignoreDeletions;
protected final ByteBufAllocator allocator;
protected final byte[] dict;
@Builder
public BundleReader(long maxTime, long minTime, boolean linkSymbol, boolean skipOldEntry, ByteBufAllocator allocator, byte[] dict) {
public BundleReader(long maxTime, long minTime, boolean linkSymbol, boolean skipOldEntry, boolean ignoreDeletions, ByteBufAllocator allocator, byte[] dict) {
this.maxTime = maxTime == 0 ? Long.MAX_VALUE : maxTime;
this.minTime = minTime;
this.linkSymbol = linkSymbol;
this.skipOldEntry = skipOldEntry;
this.ignoreDeletions = ignoreDeletions;
this.allocator = allocator == null ? ByteBufAllocator.DEFAULT : allocator;
this.dict = dict;
}
/**
* Extracts a bundle into {@code destination}. Existing files are overwritten, which is
* what makes chained restores work: apply the full base first, then each incremental in
* order — later bundles win.
*/
public void extract(Path bundle, Path destination) throws IOException {
var readMaps = new HashMap<String, Long>();
var destRoot = destination.toAbsolutePath().normalize();
try (var zstd = new ZstdInputStreamNoFinalizer(Files.newInputStream(bundle, StandardOpenOption.READ));
var in = new EntryInputStream(zstd)) {
if (dict != null) zstd.setDict(dict);
@@ -42,62 +53,128 @@ public class BundleReader {
var time = entry.time();
if (time < minTime || time > maxTime) {
System.err.println("Skipping entry " + entry.name() + " (too old or too new)");
skipExactly(in, entry.length());
continue;
}
var lastSeen = readMaps.get(entry.name());
if (lastSeen != null && skipOldEntry && entry.time() < lastSeen) {
System.err.println("Skipping entry " + entry.name() + " (older than )");
System.err.println("Skipping entry " + entry.name() + " (older than seen)");
skipExactly(in, entry.length());
continue;
}
readMaps.put(entry.name(), time);
var path = destination.resolve(entry.name()); // todo prevent path underflow
var path = resolveSafely(destRoot, entry.name());
if ((entry.attribute() & BundleEntry.ATTR_DELETE) != 0) {
// tombstone from an incremental: mirror the deletion instead of leaving a
// resurrected file from the base backup. Deleting a path not present in the
// base is a harmless no-op. When ignoreDeletions is set the tombstone is
// skipped, so files deleted after the base are kept on restore.
if (!ignoreDeletions) {
try {
Files.deleteIfExists(path);
} catch (DirectoryNotEmptyException e) {
// a directory tombstone; its contained files are removed by their own
// tombstones and an empty leftover directory is harmless
}
}
skipExactly(in, entry.length());
continue;
}
var parentDir = path.getParent();
if (Files.notExists(parentDir)) Files.createDirectories(parentDir);
if ((entry.attribute() & BundleEntry.ATTR_SYMLINK) != 0) {
var toRead = (int) entry.length();
var strArr = new byte[toRead];
while ((toRead -= in.read(strArr, (int) (entry.length() - toRead), toRead)) > 0) ;
var target = new String(strArr);
var targetPath = Path.of(target);
Files.createSymbolicLink(targetPath, path);
var strArr = new byte[(int) entry.length()];
readExactly(in, strArr);
var targetPath = Path.of(new String(strArr));
Files.deleteIfExists(path);
Files.createSymbolicLink(path, targetPath);
} else if ((entry.attribute() & BundleEntry.ATTR_REASSEMBLE) != 0) {
var entryData = allocator.buffer();
try {
var toRead = entry.length();
var buffer = this.buffer;
while (toRead > 0) {
var read = in.read(buffer, 0, (int) Math.min(buffer.length, toRead));
if (read < 0) throw new EOFException("Truncated bundle while reading " + entry.name());
toRead -= read;
entryData.writeBytes(buffer, 0, read);
}
try (var reassembler = new RegionParser(entryData, allocator)) {
var result = reassembler.writeReassembled(RegionFile.CompressType.ZLIB);
try (var fc = FileChannel.open(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
result.readBytes(fc, 0, result.readableBytes());
try (var fc = FileChannel.open(path, StandardOpenOption.CREATE,
StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING)) {
// readBytes(FileChannel, ...) does a single channel write and
// may write fewer bytes than requested; loop or the region file
// is silently truncated.
long pos = 0;
int remaining = result.readableBytes();
while (remaining > 0) {
int written = result.readBytes(fc, pos, remaining);
if (written <= 0)
throw new IOException("Short write restoring " + entry.name() + " to " + path);
pos += written;
remaining -= written;
}
} finally {
result.release();
}
} catch (Exception e) {
try (var ch = FileChannel.open(Path.of("dump.mca"), StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
var dump = Files.createTempFile("justbackup_dump_", ".mca");
try (var ch = FileChannel.open(dump, StandardOpenOption.WRITE)) {
entryData.readerIndex(0);
entryData.readBytes(ch, 0, entryData.readableBytes());
}
throw new IOException("Error while reassembling " + entry.name(), e);
throw new IOException("Error while reassembling " + entry.name() + ", dumped to " + dump, e);
}
} finally {
entryData.release();
}
} else {
writePlain(path, entry, in);
}
}
}
}
/**
* Resolves an entry name inside the destination, refusing names that escape it
* (absolute paths or {@code ..} traversal).
*/
private static Path resolveSafely(Path destRoot, String entryName) throws IOException {
var resolved = destRoot.resolve(entryName).normalize();
if (!resolved.startsWith(destRoot)) {
throw new IOException("Entry " + entryName + " escapes the destination directory, refusing to extract");
}
return resolved;
}
private static void readExactly(InputStream in, byte[] target) throws IOException {
int off = 0;
while (off < target.length) {
int read = in.read(target, off, target.length - off);
if (read < 0) throw new EOFException("Truncated bundle");
off += read;
}
}
private void skipExactly(InputStream in, long length) throws IOException {
var buffer = this.buffer;
while (length > 0) {
int read = in.read(buffer, 0, (int) Math.min(buffer.length, length));
if (read < 0) throw new EOFException("Truncated bundle");
length -= read;
}
}
private void writePlain(Path path, BundleEntry entry, EntryInputStream in) throws IOException {
var buffer = this.buffer;
try (var fc = Files.newOutputStream(path, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE)) {
try (var fc = Files.newOutputStream(path, StandardOpenOption.CREATE,
StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING)) {
var toRead = entry.length();
while (toRead > 0) {
var read = in.read(buffer, 0, (int) Math.min(buffer.length, toRead));
if (read < 0) throw new EOFException("Truncated bundle while reading " + entry.name());
toRead -= read;
fc.write(buffer, 0, read);
}
@@ -12,9 +12,12 @@ import lombok.Builder;
import lombok.SneakyThrows;
import java.io.*;
import java.nio.channels.Channels;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.*;
import java.util.concurrent.*;
import java.util.function.UnaryOperator;
@@ -131,14 +134,57 @@ public class BundleWriter implements Closeable {
Iterable<Path> path,
UnaryOperator<BundleWriterBuilder> config
) throws IOException {
createBundle(pathToBundle, path, List.of(), config);
}
/**
* @param deletedRelPaths bundle-relative paths of files deleted since the base backup;
* written as tombstones so a chained restore mirrors deletions
* instead of resurrecting files. Empty for full backups.
*/
public static void createBundle(
Path pathToBundle,
Iterable<Path> path,
Iterable<String> deletedRelPaths,
UnaryOperator<BundleWriterBuilder> config
) throws IOException {
try (var channel = FileChannel.open(pathToBundle,
StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING)) {
try (var writer = config.apply(BundleWriter.builder())
.outputStream(Files.newOutputStream(pathToBundle))
.outputStream(new NonClosingOutputStream(Channels.newOutputStream(channel)))
.build()) {
for (var p : path) {
if (!Files.isRegularFile(p) || Files.size(p) == 0) continue;
var s = p.toString();
writer.write(p, (s.contains("region") && s.endsWith(".mca")) || s.endsWith(".dat"));
}
for (var rel : deletedRelPaths) {
writer.writeTombstone(rel);
}
}
// the zstd epilogue is flushed by writer.close(); fsync before closing so a crash
// right after "backup success" cannot leave a hollow bundle behind
channel.force(true);
}
}
/**
* Lets the zstd stream be closed (writing its epilogue) without closing the underlying
* channel, so the caller can fsync it afterwards.
*/
private static final class NonClosingOutputStream extends FilterOutputStream {
NonClosingOutputStream(OutputStream out) {
super(out);
}
@Override
public void write(byte[] b, int off, int len) throws IOException {
out.write(b, off, len);
}
@Override
public void close() throws IOException {
out.flush();
}
}
@@ -156,6 +202,14 @@ public class BundleWriter implements Closeable {
writePlain(path);
}
/**
* Records that {@code relPath} was deleted since the base backup. Carries no payload; on
* restore the reader removes the file so a chained restore mirrors the deletion.
*/
public void writeTombstone(String relPath) throws IOException {
outputStream.writeEntryHeader(relPath, BundleEntry.ATTR_DELETE, 0, System.currentTimeMillis());
}
private void writeSymbol(Path path) throws IOException {
if (!Files.isSymbolicLink(path)) return;
var target = Files.readSymbolicLink(path);
@@ -191,8 +245,10 @@ public class BundleWriter implements Closeable {
result.readBytes(out, length);
result.release();
// all done
} catch (IOException e) {
System.err.println("Failed to parse region " + path + ": " + e);
} catch (Exception e) {
// any parse failure (not only IOException) must fall back to a plain copy instead
// of failing the whole backup
System.err.println("Failed to parse region " + path + ", storing it as-is: " + e);
finalResult.release();
writePlain(path);
return;
@@ -205,7 +261,8 @@ public class BundleWriter implements Closeable {
private void writePlain(Path path) throws IOException {
var rPath = relativeRoot.toAbsolutePath().relativize(path.toAbsolutePath());
var outputStream = this.outputStream;
outputStream.writeEntryHeader(rPath.toString(), (short) 0, Files.size(path), 0);
outputStream.writeEntryHeader(rPath.toString(), (short) 0, Files.size(path),
Files.getLastModifiedTime(path).toMillis());
var read = 0;
try (var fs = Files.newInputStream(path)) {
var buffer = BundleWriter.buffer.get();
@@ -25,8 +25,12 @@ public class EntryInputStream extends FilterInputStream {
public BundleEntry readEntry() throws IOException {
var magic = (short) (in.read() << 8 | in.read()); // u16
if (magic != BundleEntry.MAGIC) throw new IOException("Invalid magic number: "+Long.toHexString(magic));
// every byte must be widened to long before shifting: an int shift such as
// `in.read() << 24` goes negative when the byte is >= 0x80 and sign-extends into the
// high 32 bits, corrupting the timestamp (which then makes the entry look "too old"
// and get silently skipped on restore).
var time = (long) in.read() << 56 | (long) in.read() << 48 | (long) in.read() << 40 | (long) in.read() << 32 |
(long) in.read() << 24 | in.read() << 16 | in.read() << 8 | in.read();
(long) in.read() << 24 | (long) in.read() << 16 | (long) in.read() << 8 | (long) in.read();
var attribute = VarInts.readVarInt(in);
var segmentLen = VarInts.readVarLong(in);
var lenName = VarInts.readVarInt(in);
@@ -45,31 +45,31 @@ public class RegionFile implements Closeable {
return (short) ((z & 0b111111) << 6 | (x & 0b111111));
}
// this retains the input bytebuf.
public void writeChunk(int chunkX, int chunkZ, CompressType type, ByteBuf data) throws IOException {
if (chunkX > 32 || chunkZ > 32 || chunkX < 0 || chunkZ < 0)
if (chunkX >= 32 || chunkZ >= 32 || chunkX < 0 || chunkZ < 0)
throw new IllegalArgumentException("Invalid chunkX+chunkZ: " + chunkX + " " + chunkZ);
var key = getChunkKey(chunkX, chunkZ);
var buf = allocator.buffer(5 + data.readableBytes());
ByteBuf payload;
if (type == CompressType.NONE) {
buf.writeInt(data.readableBytes());
buf.writeByte(type.index);
buf.writeBytes(data);
payload = allocator.buffer(data.readableBytes());
payload.writeBytes(data);
} else {
try (var out = findCompressor(type, new ByteBufOutputStream(buf))) {
while (data.readableBytes() > 0) data.readBytes(out, data.readableBytes());
var head = allocator.buffer(5);
head.writeInt(buf.readableBytes());
head.writeByte(type.index);
var composite = allocator.compositeBuffer();
composite.addComponent(true, head);
composite.addComponent(true, buf);
payload = allocator.buffer();
try (var out = findCompressor(type, new ByteBufOutputStream(payload))) {
data.readBytes(out, data.readableBytes());
} catch (IOException e) {
buf.release();
payload.release();
throw e;
}
}
regionData.put(key, buf);
var buf = allocator.buffer(5 + payload.readableBytes());
// vanilla region format: the length field also counts the compression type byte
buf.writeInt(payload.readableBytes() + 1);
buf.writeByte(type.index);
buf.writeBytes(payload);
payload.release();
var previous = regionData.put(key, buf);
if (previous != null) previous.release();
}
private OutputStream findCompressor(CompressType type, OutputStream out) throws IOException {
@@ -82,7 +82,7 @@ public class RegionFile implements Closeable {
}
public void updateChunkTime(int chunkX, int chunkZ, int timestamp) {
if (chunkX > 32 || chunkZ > 32 || chunkX < 0 || chunkZ < 0)
if (chunkX >= 32 || chunkZ >= 32 || chunkX < 0 || chunkZ < 0)
throw new IllegalArgumentException("Invalid chunkX+chunkZ: " + chunkX + " " + chunkZ);
var index = (chunkX + 32 * chunkZ) * 4;
this.timestamp.setInt(index, timestamp);
@@ -105,7 +105,7 @@ public class RegionFile implements Closeable {
var chunk = entry.getValue();
var size = chunk.readableBytes();
var sizeSectors = Math.ceilDiv(size, 4096);
var zeros = 4096 - (size % 4096);
var zeros = (4096 - (size % 4096)) % 4096;
for (int i = 0; i < zeros; i++) {
chunk.writeByte(0);
}
@@ -10,8 +10,8 @@ import java.nio.channels.FileChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.zip.DeflaterInputStream;
import java.util.zip.GZIPInputStream;
import java.util.zip.InflaterInputStream;
public class RegionParser implements Closeable {
protected final ByteBuf source;
@@ -28,7 +28,9 @@ public class RegionParser implements Closeable {
try (var fc = FileChannel.open(file, StandardOpenOption.READ)) {
var toRead = size;
while (toRead > 0) {
toRead -= source.writeBytes(fc, 0L, (int) size);
var read = source.writeBytes(fc, (long) (size - toRead), toRead);
if (read < 0) throw new EOFException("Region file " + file + " shrank while reading");
toRead -= read;
}
}
}
@@ -98,8 +100,9 @@ public class RegionParser implements Closeable {
var key = chunks.getLong(i);
var chunkXZ = readChunkXZ(key);
var entry = readEntry(key);
short x = (short) (chunkXZ & 0x3F);
short z = (short) ((chunkXZ >>> 6) & 0x3F);
// packed as (x << 6) | z in readAllValidChunks
short x = (short) ((chunkXZ >>> 6) & 0x3F);
short z = (short) (chunkXZ & 0x3F);
var compressType = readChunk(entry, compressed);
try (var decompressor = getDecompressorByType(compressType, compressed)) {
while (uncompressed.writeBytes(decompressor, 4096) != -1) ;
@@ -116,7 +119,7 @@ public class RegionParser implements Closeable {
private InputStream getDecompressorByType(int compressionType, ByteBuf in) throws IOException {
return switch (compressionType) {
case 1 -> new GZIPInputStream(new ByteBufInputStream(in));
case 2 -> new DeflaterInputStream(new ByteBufInputStream(in));
case 2 -> new InflaterInputStream(new ByteBufInputStream(in));
case 3 -> new ByteBufInputStream(in);
case 4 -> new LZ4BlockInputStream(new ByteBufInputStream(in));
// user defined compression algorithm, not going to support.
@@ -0,0 +1,182 @@
package io.ib67.bundler;
import com.github.luben.zstd.ZstdOutputStreamNoFinalizer;
import io.ib67.sfcraft.bundler.BundleReader;
import io.ib67.sfcraft.bundler.BundleWriter;
import io.ib67.sfcraft.bundler.EntryOutputStream;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.EOFException;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.junit.jupiter.api.Assertions.*;
public class TestBundleReader {
@Test
public void testChainedOverlayRestore(@TempDir Path tmp) throws IOException {
var src = Files.createDirectories(tmp.resolve("src"));
Files.writeString(src.resolve("a.txt"), "version-1-that-is-long");
Files.writeString(src.resolve("b.txt"), "untouched");
var fullBundle = tmp.resolve("full.swb.zst");
BundleWriter.createBundle(fullBundle, List.of(src.resolve("a.txt"), src.resolve("b.txt")),
c -> c.relativeRoot(src));
// shorter content verifies the overlay truncates instead of leaving a stale tail
Files.writeString(src.resolve("a.txt"), "v2");
var incrBundle = tmp.resolve("incr.swb.zst");
BundleWriter.createBundle(incrBundle, List.of(src.resolve("a.txt")),
c -> c.relativeRoot(src));
var dest = tmp.resolve("dest");
var reader = BundleReader.builder().build();
reader.extract(fullBundle, dest);
reader.extract(incrBundle, dest);
assertEquals("v2", Files.readString(dest.resolve("a.txt")));
assertEquals("untouched", Files.readString(dest.resolve("b.txt")));
}
@Test
public void testTombstoneRemovesFileOnRestore(@TempDir Path tmp) throws IOException {
var src = Files.createDirectories(tmp.resolve("src"));
Files.writeString(src.resolve("keep.txt"), "keep");
Files.writeString(src.resolve("gone.txt"), "gone");
var full = tmp.resolve("full.swb.zst");
BundleWriter.createBundle(full, List.of(src.resolve("keep.txt"), src.resolve("gone.txt")),
c -> c.relativeRoot(src));
// gone.txt was deleted; the incremental carries only a tombstone for it
var incr = tmp.resolve("incr.swb.zst");
BundleWriter.createBundle(incr, List.of(), List.of("gone.txt"), c -> c.relativeRoot(src));
var dest = tmp.resolve("dest");
var reader = BundleReader.builder().build();
reader.extract(full, dest);
assertTrue(Files.exists(dest.resolve("gone.txt")), "base restore should lay down the file first");
reader.extract(incr, dest);
assertTrue(Files.exists(dest.resolve("keep.txt")), "unrelated files must survive the tombstone");
assertFalse(Files.exists(dest.resolve("gone.txt")),
"a tombstoned file must be removed on chained restore, not resurrected");
}
@Test
public void testIgnoreDeletionsKeepsTombstonedFile(@TempDir Path tmp) throws IOException {
var src = Files.createDirectories(tmp.resolve("src"));
Files.writeString(src.resolve("keep.txt"), "keep");
Files.writeString(src.resolve("gone.txt"), "gone");
var full = tmp.resolve("full.swb.zst");
BundleWriter.createBundle(full, List.of(src.resolve("keep.txt"), src.resolve("gone.txt")),
c -> c.relativeRoot(src));
var incr = tmp.resolve("incr.swb.zst");
BundleWriter.createBundle(incr, List.of(), List.of("gone.txt"), c -> c.relativeRoot(src));
var dest = tmp.resolve("dest");
var reader = BundleReader.builder().ignoreDeletions(true).build();
reader.extract(full, dest);
reader.extract(incr, dest);
assertTrue(Files.exists(dest.resolve("gone.txt")),
"with ignoreDeletions the tombstone is skipped and the file is kept");
assertEquals("gone", Files.readString(dest.resolve("gone.txt")));
assertEquals("keep", Files.readString(dest.resolve("keep.txt")));
}
@Test
public void testTombstoneForUnknownPathIsHarmless(@TempDir Path tmp) throws IOException {
var src = Files.createDirectories(tmp.resolve("src"));
Files.writeString(src.resolve("a.txt"), "a");
var bundle = tmp.resolve("b.swb.zst");
// tombstone a nested path that was never in the base backup
BundleWriter.createBundle(bundle, List.of(src.resolve("a.txt")),
List.of("region/r.0.0.mca"), c -> c.relativeRoot(src));
var dest = tmp.resolve("dest");
BundleReader.builder().build().extract(bundle, dest);
assertEquals("a", Files.readString(dest.resolve("a.txt")));
}
@Test
public void testTombstoneCannotEscapeDestination(@TempDir Path tmp) throws IOException {
var outside = Files.writeString(tmp.resolve("outside.txt"), "precious");
var bundle = tmp.resolve("evil.swb.zst");
try (var zstd = new ZstdOutputStreamNoFinalizer(Files.newOutputStream(bundle), 3);
var out = new EntryOutputStream(zstd)) {
out.writeEntryHeader("../outside.txt", io.ib67.sfcraft.bundler.BundleEntry.ATTR_DELETE, 0,
System.currentTimeMillis());
}
var dest = tmp.resolve("dest");
assertThrows(IOException.class, () -> BundleReader.builder().build().extract(bundle, dest));
assertTrue(Files.exists(outside), "a tombstone must not delete files outside the destination");
}
@Test
public void testZipSlipIsRejected(@TempDir Path tmp) throws IOException {
var bundle = tmp.resolve("evil.swb.zst");
var payload = "owned".getBytes();
try (var zstd = new ZstdOutputStreamNoFinalizer(Files.newOutputStream(bundle), 3);
var out = new EntryOutputStream(zstd)) {
out.writeEntryHeader("../evil.txt", (short) 0, payload.length, System.currentTimeMillis());
out.write(payload);
}
var dest = tmp.resolve("dest");
var ex = assertThrows(IOException.class,
() -> BundleReader.builder().build().extract(bundle, dest));
assertTrue(ex.getMessage().contains("escapes"), ex.getMessage());
assertFalse(Files.exists(tmp.resolve("evil.txt")));
}
@Test
public void testSymlinkIsCreatedAtLinkPath(@TempDir Path tmp) throws IOException {
var src = Files.createDirectories(tmp.resolve("src"));
var outside = Files.writeString(tmp.resolve("outside.txt"), "outside");
var link = src.resolve("link");
Files.createSymbolicLink(link, outside);
var bundle = tmp.resolve("sym.swb.zst");
BundleWriter.createBundle(bundle, List.of(link), c -> c.relativeRoot(src));
var dest = tmp.resolve("dest");
BundleReader.builder().build().extract(bundle, dest);
assertTrue(Files.isSymbolicLink(dest.resolve("link")),
"the symlink itself must be recreated at the link's path, not at its target");
}
@Test
public void testTimestampWithHighBitSurvivesRoundTrip(@TempDir Path tmp) throws IOException {
// 1798761600000 has 0xCE at bits 24-31; an int shift on that byte sign-extends and
// makes the decoded time negative, which used to make extract() skip the entry as
// "too old". ~half of all real-world dates land in such a band.
var time = 1798761600000L;
assertTrue(((time >> 24) & 0xFF) >= 0x80, "test vector must exercise the sign-extension path");
var bundle = tmp.resolve("hibit.swb.zst");
var payload = "kept".getBytes();
try (var zstd = new ZstdOutputStreamNoFinalizer(Files.newOutputStream(bundle), 3);
var out = new EntryOutputStream(zstd)) {
out.writeEntryHeader("world.dat", (short) 0, payload.length, time);
out.write(payload);
}
var dest = tmp.resolve("dest");
BundleReader.builder().build().extract(bundle, dest);
assertEquals("kept", Files.readString(dest.resolve("world.dat")),
"entry must not be skipped just because its timestamp byte has the high bit set");
}
@Test
public void testTruncatedEntryThrowsInsteadOfLooping(@TempDir Path tmp) throws IOException {
var bundle = tmp.resolve("truncated.swb.zst");
try (var zstd = new ZstdOutputStreamNoFinalizer(Files.newOutputStream(bundle), 3);
var out = new EntryOutputStream(zstd)) {
out.writeEntryHeader("short.bin", (short) 0, 100, System.currentTimeMillis());
out.write(new byte[10]); // 90 bytes missing
}
var dest = tmp.resolve("dest");
assertThrows(EOFException.class,
() -> BundleReader.builder().build().extract(bundle, dest));
}
}
@@ -0,0 +1,91 @@
package io.ib67.bundler;
import io.ib67.sfcraft.bundler.region.RegionFile;
import io.ib67.sfcraft.bundler.region.RegionParser;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import io.netty.buffer.Unpooled;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.*;
/**
* Simulates the full backup->restore cycle for region files: vanilla mca -> reassembled
* uncompressed (what goes into the bundle) -> reassembled ZLIB (what restore writes back),
* then verifies every chunk's uncompressed NBT is byte-identical and lives in the same
* header slot as in the original file.
*/
public class TestRegionRoundTrip {
@Test
public void roundTripPreservesEveryChunk() throws IOException {
roundTrip("region/r.9.-5.mca");
roundTrip("region/r.-1.-1.mca");
}
private static void roundTrip(String resource) throws IOException {
byte[] originalBytes;
try (var in = TestRegionRoundTrip.class.getClassLoader().getResourceAsStream(resource)) {
originalBytes = in.readAllBytes();
}
var alloc = ByteBufAllocator.DEFAULT;
// backup side: vanilla -> NONE bundle payload
ByteBuf intermediate;
try (var parser = new RegionParser(Unpooled.wrappedBuffer(originalBytes), alloc)) {
intermediate = parser.writeReassembled(RegionFile.CompressType.NONE);
}
// restore side: bundle payload -> ZLIB region written to disk
ByteBuf restored;
try (var parser = new RegionParser(intermediate, alloc)) {
restored = parser.writeReassembled(RegionFile.CompressType.ZLIB);
} finally {
intermediate.release();
}
try (var originalParser = new RegionParser(Unpooled.wrappedBuffer(originalBytes), alloc);
var restoredParser = new RegionParser(restored.readerIndex(0).retain(), alloc)) {
var originalChunks = collect(originalParser);
var restoredChunks = collect(restoredParser);
assertFalse(originalChunks.isEmpty(), resource + " should contain chunks");
assertEquals(originalChunks.keySet(), restoredChunks.keySet(),
"restored file must populate exactly the same chunk slots (no transposition)");
for (var entry : originalChunks.entrySet()) {
var origNbt = decompress(originalParser, entry.getValue(), alloc);
var restNbt = decompress(restoredParser, restoredChunks.get(entry.getKey()), alloc);
try {
assertEquals(origNbt, restNbt, "chunk " + entry.getKey() + " must survive the round trip unchanged");
} finally {
origNbt.release();
restNbt.release();
}
}
// timestamp table is copied verbatim
var origTimes = Unpooled.wrappedBuffer(originalBytes).slice(4096, 4096);
var restTimes = restored.slice(4096, 4096);
assertEquals(origTimes, restTimes, "chunk timestamps must survive the round trip");
} finally {
restored.release();
}
}
/** chunkXZ -> header entry */
private static Map<Short, Integer> collect(RegionParser parser) {
var result = new HashMap<Short, Integer>();
var chunks = parser.readAllValidChunks();
for (int i = 0; i < chunks.size(); i++) {
var key = chunks.getLong(i);
result.put(RegionParser.readChunkXZ(key), RegionParser.readEntry(key));
}
return result;
}
private static ByteBuf decompress(RegionParser parser, int entry, ByteBufAllocator alloc) throws IOException {
var buf = alloc.buffer();
parser.readChunkDecompress(entry, buf);
return buf;
}
}
BIN
View File
Binary file not shown.
+7 -5
View File
@@ -2,12 +2,14 @@
org.gradle.jvmargs=-Xmx4G
org.gradle.parallel=true
# IntelliJ IDEA is not yet fully compatible with configuration cache, see: https://github.com/FabricMC/fabric-loom/issues/1349
org.gradle.configuration-cache=false
# Fabric Properties
# check these on https://fabricmc.net/develop
minecraft_version=1.21.8
yarn_mappings=1.21.8+build.1
loader_version=0.16.14
loom_version=1.11-SNAPSHOT
minecraft_version=26.3
loader_version=0.19.5
loom_version=1.17-SNAPSHOT
# Mod Properties
mod_version=1.0.0
@@ -15,4 +17,4 @@ maven_group=io.ib67.sfcraft
archives_base_name=justbackup
# Dependencies
fabric_version=0.129.0+1.21.8
fabric_version=0.160.5+26.3
+1 -1
View File
@@ -1,6 +1,6 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.14.2-bin.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-9.6.1-bin.zip
networkTimeout=10000
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
+27 -3
View File
@@ -2,17 +2,41 @@ package io.ib67.sfcraft;
import net.minecraft.network.chat.Component;
import java.time.Instant;
import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
/**
* @param subject name of the backup subject this bundle belongs to; null on entries created
* by older versions (those are treated as standalone full backups)
* @param createdAt epoch millis of creation; 0 on legacy entries
*/
public record Backup(
String name,
String backupKey,
String type,
String from,
String subject,
boolean incremental,
long createdAt,
long sizeTotal
) {
private static final DateTimeFormatter TIME_FORMAT =
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss").withZone(ZoneId.systemDefault());
public boolean legacy() {
return subject == null || createdAt <= 0;
}
public Component toText() {
return Component.nullToEmpty("sizeTotal: "+Math.round(((sizeTotal / 1024.0) / 1024.0) * 100) * 0.01 + "MiB, "+
(incremental ? "incremental" :"") + ", type: "+type+"."
);
var size = Math.round(((sizeTotal / 1024.0) / 1024.0) * 100) * 0.01;
var sb = new StringBuilder()
.append(incremental ? "INCR" : "FULL")
.append(", ").append(size).append("MiB")
.append(", type: ").append(type);
if (createdAt > 0) {
sb.append(", at ").append(TIME_FORMAT.format(Instant.ofEpochMilli(createdAt)));
}
return Component.nullToEmpty(sb.toString());
}
}
+108 -21
View File
@@ -7,20 +7,22 @@ import com.mojang.brigadier.arguments.StringArgumentType;
import com.mojang.brigadier.context.CommandContext;
import com.mojang.brigadier.suggestion.Suggestions;
import com.mojang.brigadier.suggestion.SuggestionsBuilder;
import io.ib67.sfcraft.strategy.BackupChains;
import lombok.extern.log4j.Log4j2;
import net.minecraft.commands.CommandBuildContext;
import net.minecraft.commands.CommandSourceStack;
import net.minecraft.commands.Commands;
import net.minecraft.network.chat.ClickEvent;
import net.minecraft.network.chat.Component;
import java.awt.*;
import java.io.IOException;
import java.lang.ref.WeakReference;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import static net.minecraft.commands.Commands.argument;
import static net.minecraft.commands.Commands.literal;
@@ -28,15 +30,18 @@ import static net.minecraft.commands.Commands.literal;
@Log4j2
public class BackupCommands {
protected final JustBackupMod mod;
protected Runnable restoreIssued;
protected volatile Runnable restoreIssued;
protected final List<String> helpMessage = List.of(
"__ Help messages of JustBackup __",
" /backup help ",
" -- Show this message",
" /backup list ",
" -- List backups",
" /backup restore <backupKey>",
" /backup restore [keepdeleted] <backupKey>",
" -- Restore the backup on next server restart.",
" Restoring an incremental backup automatically applies its full base",
" and every incremental in between, in order.",
" keepdeleted: skip deletions, keeping files removed after the base backup.",
" Can be cancelled by /backup cancelrestore",
" /backup delete <backupKey>",
" -- Delete a backup. This does not ask for a confirmation.",
@@ -58,45 +63,101 @@ public class BackupCommands {
CommandDispatcher<CommandSourceStack> dispatcher,
CommandBuildContext registry, Commands.CommandSelection env) {
dispatcher.register(literal("backup")
.requires(it -> it.hasPermission(4))
.requires(Commands.hasPermission(Commands.LEVEL_OWNERS))
.then(literal("help").executes(this::cmdHelp))
.then(literal("list").executes(this::cmdList))
.then(literal("cancelrestore").executes(this::cmdCancelRestore))
.then(literal("suspend").executes(this::cmdSuspend))
.then(literal("delete")
.then(argument("backupKey", StringArgumentType.greedyString())
.suggests(this::suggestBackups)
.executes(this::cmdDelete)))
.then(literal("restore")
.then(literal("keepdeleted")
.then(argument("backupKey", StringArgumentType.greedyString())
.suggests(this::suggestBackups)
.executes(this::cmdRestore)))
.executes(ctx -> cmdRestore(ctx, true))))
.then(argument("backupKey", StringArgumentType.greedyString())
.suggests(this::suggestBackups)
.executes(ctx -> cmdRestore(ctx, false))))
.then(literal("create")
.then(argument("incremental", BoolArgumentType.bool())
.then(argument("subject", StringArgumentType.greedyString())
.executes(this::createBackupBySubject))
.then(literal("@all").executes(this::fullBackup)))
.then(literal("suspend").executes(this::cmdSuspend))
));
.then(literal("@all").executes(this::fullBackup)))));
}
private int createBackupBySubject(CommandContext<CommandSourceStack> context) {
var incremental = BoolArgumentType.getBool(context, "incremental");
var subject = StringArgumentType.getString(context, "subject");
var s = context.getSource();
if (!mod.backupSubjects.containsKey(subject)) {
s.sendSystemMessage(Component.literal("Unknown backup subject: '" + subject + "'.")
.withColor(Color.RED.getRGB()));
var configured = mod.backupSubjects.keySet();
if (configured.isEmpty()) {
s.sendSystemMessage(Component.literal(
"No backup subjects are configured. Set \"backupSubjects\" (name -> world directory) "
+ "in config/just_backup.json and restart the server.").withColor(Color.YELLOW.getRGB()));
} else {
s.sendSystemMessage(Component.literal("Configured subjects: " + String.join(", ", configured)
+ " (or use \"@all\").").withColor(Color.YELLOW.getRGB()));
}
return 0;
}
s.sendSystemMessage(Component.literal("The issued backup has been scheduled.").withColor(Color.CYAN.getRGB()));
mod.issueBackup(subject, incremental);
reportOnCompletion(s, mod.issueBackup(subject, incremental));
return Command.SINGLE_SUCCESS;
}
/**
* Reports the outcome of a single backup back to the command source. Completion runs on the
* backup executor, so the message is delivered on the main thread; the source is held weakly
* (the player may have left) and falls back to the server console source.
*/
private void reportOnCompletion(CommandSourceStack source, CompletableFuture<Backup> future) {
var weakRef = new WeakReference<>(source);
future.whenComplete((backup, throwable) -> {
var server = mod.server;
if (server == null || !server.isRunning()) return;
server.submit(() -> {
var ref = weakRef.get();
if (ref == null) ref = server.createCommandSourceStack();
if (throwable != null) {
ref.sendSystemMessage(Component.literal("Backup failed: " + rootMessage(throwable))
.withColor(Color.RED.getRGB()));
} else {
ref.sendSystemMessage(Component.literal("Backup finished: ").append(backup.toText()));
}
});
});
}
private static String rootMessage(Throwable t) {
while (t instanceof CompletionException && t.getCause() != null) t = t.getCause();
var msg = t.getMessage();
return msg != null ? msg : t.toString();
}
private int fullBackup(CommandContext<CommandSourceStack> context) {
var incremental = BoolArgumentType.getBool(context, "incremental");
var s = context.getSource();
if (mod.backupSubjects.isEmpty()) {
s.sendSystemMessage(Component.literal(
"No backup subjects are configured, nothing to back up. Set \"backupSubjects\" in "
+ "config/just_backup.json and restart the server.").withColor(Color.RED.getRGB()));
return 0;
}
s.sendSystemMessage(Component.literal("The issued backup has been scheduled.").withColor(Color.CYAN.getRGB()));
var weakRef = new WeakReference<>(s);
mod.backupAll(incremental).thenAccept(result -> {
var server = mod.server;
if (server == null || !server.isRunning()) return;
// completion runs on the backup executor; command sources must be used on the main thread
server.submit(() -> {
var ref = weakRef.get();
if (ref == null) {
ref = mod.server.createCommandSourceStack();
ref = server.createCommandSourceStack();
}
ref.sendSystemMessage(Component.literal("-- BACKUP RESULT SUMMARY --"));
for (var entry : result.entrySet()) {
@@ -107,6 +168,7 @@ public class BackupCommands {
ref.sendSystemMessage(Component.literal("GENERATED FROM: " + v.from()).withColor(Color.GRAY.getRGB()));
}
});
});
return Command.SINGLE_SUCCESS;
}
@@ -123,7 +185,7 @@ public class BackupCommands {
return Command.SINGLE_SUCCESS;
}
private int cmdRestore(CommandContext<CommandSourceStack> context) {
private int cmdRestore(CommandContext<CommandSourceStack> context, boolean ignoreDeletions) {
var s = context.getSource();
if (restoreIssued != null) {
s.sendSystemMessage(Component.literal("Another restore request is already present.").withColor(Color.RED.getRGB()));
@@ -136,31 +198,56 @@ public class BackupCommands {
s.sendSystemMessage(Component.literal("Invalid backup " + backupKey + ". Note: Use backupKey instead of backupName!"));
return 0;
}
final List<Backup> chain;
try {
chain = BackupChains.resolveChain(mod.tracker.getTrackedBackups().values(), backup);
} catch (IllegalStateException e) {
s.sendSystemMessage(Component.literal(e.getMessage()).withColor(Color.RED.getRGB()));
return 0;
}
s.sendSystemMessage(Component.literal("The following " + chain.size() + " bundle(s) will be applied in order:"));
for (var b : chain) {
s.sendSystemMessage(Component.literal(" -> [" + (b.incremental() ? "INCR" : "FULL") + "] " + b.backupKey())
.withColor(Color.GRAY.getRGB()));
}
if (ignoreDeletions) {
s.sendSystemMessage(Component.literal("Deletions will be ignored: files removed after the base backup are kept.")
.withColor(Color.YELLOW.getRGB()));
}
restoreIssued = () -> {
log.warn("EXTRACTING BACKUP " + backup + " TO " + backup.from());
log.warn(" == STEP 1 == Make a backup for the destination.");
var destination = Path.of(backup.from());
var movedDst = destination.resolveSibling(destination.getFileName().toString() + "_bak_" + System.currentTimeMillis());
log.warn("EXTRACTING {} BUNDLE(S) TO {}", chain.size(), destination);
log.warn(" == STEP 1 == Make a backup for the destination.");
try {
Files.move(destination, movedDst);
log.info("{} has been moved to {}", destination, movedDst);
} catch (IOException e) {
log.error(e);
log.error("CANNOT MOVE "+destination+" TO "+movedDst);
log.error("BACKUP INTERRUPTED. Here are some tips helping you out:");
log.error("CANNOT MOVE {} TO {}", destination, movedDst, e);
log.error("RESTORE INTERRUPTED. Here are some tips helping you out:");
log.error(" -- A. Merge already moved files");
log.error(" Try this command in your server directory: (linux)");
log.error(" $ cp ./"+movedDst+"/* ./"+destination);
log.error(" $ cp ./{}/* ./{}", movedDst, destination);
log.error(" -- B. Remove them all and use external unbundler tools");
log.error(" You may want to copy them elsewhere first (see kind A)");
log.error(" $ rm -r ./"+destination+" ./"+movedDst);
log.error(" $ rm -r ./{} ./{}", destination, movedDst);
log.error(" Visit https://github.com/saltedfishclub/justbackup for the usage of bundler tools");
log.error(" ------- ERROR END -------");
return;
}
log.info(" == STEP 2 == Recovering backup "+backup);
mod.tracker.recoverBackup(backup, destination);
log.info(" == STEP 2 == Applying {} bundle(s)", chain.size());
try {
for (int i = 0; i < chain.size(); i++) {
var b = chain.get(i);
log.info("Applying {} ({}/{}){}", b.backupKey(), i + 1, chain.size(),
ignoreDeletions ? " [ignoring deletions]" : "");
mod.tracker.recoverBackup(b, destination, ignoreDeletions);
}
log.info("Backup has been recovered.");
} catch (Exception e) {
log.error("RESTORE FAILED midway. Your original world is intact at {}", movedDst, e);
log.error("Move it back to {} to undo the restore.", destination);
}
};
s.sendSystemMessage(Component.literal("Backup restoration task has been scheduled!"));
s.sendSystemMessage(Component.literal("Restart your server to take changes."));
@@ -180,7 +267,7 @@ public class BackupCommands {
source.sendSystemMessage(Component.nullToEmpty("Backup has been deleted."));
} catch (Exception e) {
source.sendSystemMessage(Component.nullToEmpty("Backup has not been deleted successfully. ERROR: " + e));
log.error(e);
log.error("Cannot delete backup {}", backupKey, e);
}
return Command.SINGLE_SUCCESS;
}
@@ -1,11 +1,92 @@
package io.ib67.sfcraft;
import java.nio.file.Path;
import java.util.List;
import java.util.HashSet;
import java.util.Set;
public record BackupSubject(
String name,
Path root,
List<Path> changedFiles
) {
/**
* A directory tree being backed up, plus the changes the watcher saw since the last
* incremental backup: files that were created/modified, and files that were deleted. All
* access goes through synchronized methods so the watcher's events and the worker's drain can
* never interleave badly (no lost events, no ConcurrentModificationException while bundling).
*/
public final class BackupSubject {
/** A drained snapshot of changes: touched files to bundle, deleted files to tombstone. */
public record Changes(Set<Path> changed, Set<Path> deleted) {
}
private final String name;
private final Path root;
private final Set<Path> changedFiles = new HashSet<>();
private final Set<Path> deletedFiles = new HashSet<>();
private boolean forceFullNext;
public BackupSubject(String name, Path root) {
this.name = name;
this.root = root;
}
public String name() {
return name;
}
public Path root() {
return root;
}
/** A create/modify supersedes any pending deletion of the same path. */
public synchronized void addChanged(Path file) {
deletedFiles.remove(file);
changedFiles.add(file);
}
/** A deletion supersedes any pending create/modify of the same path. */
public synchronized void addDeleted(Path file) {
changedFiles.remove(file);
deletedFiles.add(file);
}
/**
* Atomically takes ownership of the accumulated changes, leaving empty sets behind.
* If the backup fails, give them back via {@link #mergeBack(Changes)}.
*/
public synchronized Changes drain() {
var changed = new HashSet<>(changedFiles);
var deleted = new HashSet<>(deletedFiles);
changedFiles.clear();
deletedFiles.clear();
return new Changes(changed, deleted);
}
/**
* Restores drained changes after a failed backup, without clobbering newer events: a path
* the watcher has since seen change is not re-marked deleted, and vice versa.
*/
public synchronized void mergeBack(Changes changes) {
for (var f : changes.changed()) {
if (!deletedFiles.contains(f)) changedFiles.add(f);
}
for (var f : changes.deleted()) {
if (!changedFiles.contains(f)) deletedFiles.add(f);
}
}
/**
* Marks the next backup as full, used when the watcher reports an OVERFLOW (events were
* dropped by the OS, so the change set can no longer be trusted).
*/
public synchronized void forceFullNext() {
forceFullNext = true;
}
public synchronized boolean consumeForceFull() {
var v = forceFullNext;
forceFullNext = false;
if (v) {
// a full backup snapshots the live filesystem, so pending deltas are moot
changedFiles.clear();
deletedFiles.clear();
}
return v;
}
}
+150 -17
View File
@@ -6,21 +6,46 @@ import lombok.SneakyThrows;
import lombok.extern.log4j.Log4j2;
import net.minecraft.server.MinecraftServer;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.function.Supplier;
import java.util.function.UnaryOperator;
import java.util.stream.Stream;
import static io.ib67.sfcraft.Globals.BACKUP_LOCK;
import static io.ib67.sfcraft.Globals.WORLD_IO_LOCK;
/**
* Runs on the backup executor thread. Flow:
* <ol>
* <li>Save the world on the main thread (saveEverything is not thread-safe anywhere else)
* with flush=true, draining the IOWorkers so region files are complete on disk.</li>
* <li>Acquire the write side of {@link Globals#WORLD_IO_LOCK}, parking all region IO.</li>
* <li>Reflink-clone the files to a staging dir and release the lock early (CoW
* filesystems), or keep holding it while packing (fallback).</li>
* <li>Hand the bundle to the storage strategy.</li>
* </ol>
* The main thread never blocks on the lock — autoSave/saveEverything are skipped while it is
* write-locked (see MixinMinecraftServer) — so waiting for main-thread tasks while holding
* the write lock cannot deadlock.
*/
@Log4j2
public record BackupWorker(
MinecraftServer server,
BackupStrategy strategy,
Path tmpDir,
BackupSubject subject,
boolean incremental,
List<Path> excludedRoots,
UnaryOperator<BundleWriter.BundleWriterBuilder> config
) implements Supplier<Backup> {
private static final long LOCK_TIMEOUT_SECONDS = 60;
@SneakyThrows
public BackupWorker {
@@ -33,30 +58,138 @@ public record BackupWorker(
if (server == null) {
throw new IllegalStateException("MinecraftServer is not initialized yet, not doing backup");
}
server.saveEverything(true, true, false);
while (!BACKUP_LOCK.compareAndSet(IOState.IDLE, IOState.BACKUP)) {
log.info("Server is saving world while performing backup! waiting...");
Thread.sleep(2000);
var createdAt = System.currentTimeMillis();
var full = !incremental || subject.consumeForceFull();
runOnMainThread(() -> server.saveEverything(true, true, false), "world save");
long stamp = WORLD_IO_LOCK.tryWriteLock(LOCK_TIMEOUT_SECONDS, TimeUnit.SECONDS);
if (stamp == 0) {
throw new IllegalStateException("World IO is still busy after " + LOCK_TIMEOUT_SECONDS
+ "s, aborting backup of " + subject.name());
}
// performing backup is now true
var tmpBundle = tmpDir.resolve("backup_" + subject.name() + "_" + System.currentTimeMillis() + ".swb.zst");
var tmpBundle = tmpDir.resolve("backup_" + subject.name() + "_" + createdAt + ".swb.zst");
BackupSubject.Changes changes = null;
List<String> tombstones = List.of();
Path stagingRoot = null;
try {
// tmp file path
log.info("Creating bundle {}", tmpBundle);
BundleWriter.createBundle(tmpBundle, subject.changedFiles(), config);
// Barrier: a main-thread save that started before we locked has finished by now.
runOnMainThread(() -> {
}, "main-thread barrier");
List<Path> files;
if (full) {
files = collectFullFileList();
} else {
changes = subject.drain();
files = changes.changed().stream().filter(this::eligible).toList();
tombstones = resolveTombstones(changes.deleted());
}
var packRoot = subject.root();
stagingRoot = ReflinkCloner.tryStage(subject.name(), packRoot, files,
tmpDir.resolve("staging").resolve(subject.name() + "_" + createdAt));
if (stagingRoot != null) {
var absRoot = packRoot.toAbsolutePath().normalize();
var absStaging = stagingRoot;
files = files.stream()
.map(f -> absStaging.resolve(absRoot.relativize(f.toAbsolutePath().normalize())))
.toList();
packRoot = stagingRoot;
WORLD_IO_LOCK.unlockWrite(stamp);
stamp = 0;
log.info("Reflinked {} files to staging; world IO resumed, packing without the lock.", files.size());
}
log.info("Creating {} bundle {} ({} files, {} deletions)",
full ? "full" : "incremental", tmpBundle, files.size(), tombstones.size());
var finalPackRoot = packRoot;
BundleWriter.createBundle(tmpBundle, files, tombstones, c -> config.apply(c).relativeRoot(finalPackRoot));
} catch (Exception e) {
if (changes != null) subject.mergeBack(changes);
Files.deleteIfExists(tmpBundle);
throw e;
} finally {
// Though we won't access the state within backup progress, there are certain cases
// that this state will be changed to STORAGE_SYNC.
// Also take a look at MixinRegionBasedStorage.
while (!BACKUP_LOCK.compareAndSet(IOState.BACKUP, IOState.IDLE)) {
log.error("Expect BACKUP state but got {}", BACKUP_LOCK.get());
Thread.sleep(2000);
if (stamp != 0) WORLD_IO_LOCK.unlockWrite(stamp);
if (stagingRoot != null) {
try {
FsUtil.deleteRecursively(stagingRoot);
} catch (IOException e) {
log.warn("Cannot clean up staging dir {}", stagingRoot, e);
}
}
var result = strategy.createBackup(subject.name(), subject.root(), tmpBundle);
}
try {
return strategy.createBackup(subject.name(), subject.root(), tmpBundle, !full, createdAt);
} catch (Exception e) {
if (changes != null) subject.mergeBack(changes);
throw e;
} finally {
Files.deleteIfExists(tmpBundle);
}
}
/**
* Turns absolute deleted paths into bundle-relative tombstone names. A path that exists
* again at snapshot time (deleted then re-created) is dropped — it is bundled as content
* instead — as is anything we would not have backed up (session.lock, excluded roots, or
* a path outside the subject).
*/
private List<String> resolveTombstones(Set<Path> deleted) {
var absRoot = subject.root().toAbsolutePath();
var result = new ArrayList<String>();
for (Path p : deleted) {
var abs = p.toAbsolutePath();
if (Files.exists(abs)) continue;
if (!abs.startsWith(absRoot)) continue;
if (!eligible(abs)) continue;
result.add(absRoot.relativize(abs).toString());
}
return result;
}
/**
* Runs a task on the main thread, but never hangs: once the server leaves its tick loop
* (shutdown) submitted tasks are no longer processed, so we poll and abort the backup
* instead of blocking forever.
*/
private void runOnMainThread(Runnable task, String what) {
var future = server.submit(task);
while (true) {
try {
future.get(1, TimeUnit.SECONDS);
return;
} catch (TimeoutException e) {
if (!server.isRunning()) {
future.cancel(false);
throw new IllegalStateException("Server is shutting down, aborting backup while waiting for " + what);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while waiting for " + what);
} catch (ExecutionException e) {
throw new IllegalStateException(what + " failed", e.getCause());
}
}
}
@SneakyThrows
private List<Path> collectFullFileList() {
try (Stream<Path> s = Files.walk(subject.root())) {
return s.filter(Files::isRegularFile)
.filter(this::eligible)
.toList();
}
}
private boolean eligible(Path path) {
if (path.getFileName().toString().equals("session.lock")) return false;
var abs = path.toAbsolutePath().normalize();
for (Path excluded : excludedRoots) {
if (abs.startsWith(excluded.toAbsolutePath().normalize())) return false;
}
return true;
}
}
@@ -3,87 +3,191 @@ package io.ib67.sfcraft;
import lombok.SneakyThrows;
import lombok.extern.log4j.Log4j2;
import java.io.IOException;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.TimeUnit;
import java.util.function.BiConsumer;
import java.util.stream.Collectors;
import java.util.stream.Stream;
@Log4j2
public class FileWatcherThread extends Thread {
protected final Map<Path, String> watchSubject;
protected final BiConsumer<String, Path> eventListener;
private static final long VALIDATE_INTERVAL_MILLIS = 10_000;
public FileWatcherThread(
Map<String, Path> watchSubject,
BiConsumer<String, Path> eventListener
) {
this.watchSubject = watchSubject.entrySet()
.stream()
// todo support for nonexist dir
.flatMap(it -> walkThrough(it.getValue()).map(r -> Map.entry(r.toAbsolutePath(), it.getKey())))
.filter(it->Files.isDirectory(it.getKey()))
.collect(Collectors.toMap(
Map.Entry::getKey,
Map.Entry::getValue
));
this.eventListener = eventListener;
setName("JustBackup/FileWatcher");
setDaemon(true);
public interface Listener {
void onChange(String subject, Path file);
/**
* A path under this subject was deleted. The path is already gone, so it may have been
* a file or a directory; the backup layer reconciles against the live filesystem and
* only tombstones paths that are still absent at snapshot time.
*/
void onDelete(String subject, Path file);
/**
* Events for this subject were lost (OS queue overflow, or its directory tree was
* replaced underneath us); the accumulated change set can no longer be trusted and
* the next backup should be a full one.
*/
void onOverflow(String subject);
}
/** subject name -> root directory (absolute) */
protected final Map<String, Path> subjectRoots = new HashMap<>();
/** subject name -> fileKey (inode) of the root at registration time */
protected final Map<String, Object> subjectRootKeys = new HashMap<>();
/** watched directory (absolute) -> subject name */
protected final Map<Path, String> watchedDirs = new HashMap<>();
protected final Map<WatchKey, Path> keyToDir = new HashMap<>();
protected final Listener listener;
private WatchService watcher;
private long lastValidation;
@SneakyThrows
private static Stream<Path> walkThrough(Path path) {
if(Files.notExists(path)){
log.error("{} not found. Creating directory for it..", path);
Files.createDirectories(path);
public FileWatcherThread(
Map<String, Path> watchSubject,
Listener listener
) {
for (var entry : watchSubject.entrySet()) {
var root = entry.getValue();
if (Files.notExists(root)) {
log.error("{} not found. Creating directory for it..", root);
Files.createDirectories(root);
}
return Files.walk(path);
subjectRoots.put(entry.getKey(), root.toAbsolutePath());
}
this.listener = listener;
setName("JustBackup/FileWatcher");
setDaemon(true);
}
@Override
@SneakyThrows
public void run() {
try (var watcher = FileSystems.getDefault().newWatchService()) {
var keyToParent = new HashMap<WatchKey, Path>();
for (Path value : watchSubject.keySet()) {
var key = value.register(
watcher,
StandardWatchEventKinds.ENTRY_CREATE,
StandardWatchEventKinds.ENTRY_MODIFY
);
keyToParent.put(key, value);
try (var watchService = FileSystems.getDefault().newWatchService()) {
this.watcher = watchService;
for (var entry : subjectRoots.entrySet()) {
registerSubject(entry.getKey(), false);
}
log.info("Watching {} directories for {} subjects", keyToDir.size(), subjectRoots.size());
lastValidation = System.currentTimeMillis();
for (; ; ) {
var key = watcher.poll(10, TimeUnit.SECONDS);
if (key == null) continue;
if (Thread.interrupted()) return;
if (System.currentTimeMillis() - lastValidation >= VALIDATE_INTERVAL_MILLIS) {
validateRoots();
}
if (key == null) continue;
var dir = keyToDir.get(key);
var subject = dir == null ? null : watchedDirs.get(dir);
for (var ev : key.pollEvents()) {
var kind = ev.kind();
if (kind == StandardWatchEventKinds.OVERFLOW)
continue;
var parent = keyToParent.get(key);
var subject = watchSubject.get(parent);
if (subject == null) {
log.warn("Ignoring changes in "+parent);
log.warn("Ignoring changes in unknown watch key {}", dir);
continue;
}
var path = parent.resolve(((WatchEvent<Path>) ev).context());
if (Files.isRegularFile(path)) {
eventListener.accept(subject, parent);
var kind = ev.kind();
if (kind == StandardWatchEventKinds.OVERFLOW) {
log.warn("Watch events for {} overflowed; forcing a full backup on next run.", subject);
listener.onOverflow(subject);
continue;
}
var path = dir.resolve((Path) ev.context());
log.debug("event {} on {} ({})", kind, path, subject);
if (kind == StandardWatchEventKinds.ENTRY_DELETE) {
listener.onDelete(subject, path);
} else if (Files.isRegularFile(path)) {
listener.onChange(subject, path);
} else if (Files.isDirectory(path) && kind == StandardWatchEventKinds.ENTRY_CREATE) {
var nK = path.register(watcher, StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_MODIFY);
watchSubject.put(path, subject);
keyToParent.put(nK, path);
// report contained files too: they may have been created before we
// managed to register and would never produce their own events
registerTree(path, subject, true);
}
}
key.reset();
if (!key.reset()) {
log.debug("Watch key for {} is no longer valid, dropping it", dir);
keyToDir.remove(key);
if (dir != null) watchedDirs.remove(dir);
}
}
} catch (InterruptedException interruptedException) {
log.info("FileWatcherThread is quitting!");
}
}
/**
* Watch keys silently go stale when a watched root is replaced rather than modified —
* e.g. a world format conversion moves the directory aside and recreates it, leaving us
* watching an orphaned inode. Compare the root's fileKey (inode) against what we
* registered; on mismatch, re-register the whole subject and force a full backup since
* events were lost in between.
*/
private void validateRoots() {
lastValidation = System.currentTimeMillis();
for (var entry : subjectRoots.entrySet()) {
var subject = entry.getKey();
var root = entry.getValue();
Object currentKey = fileKeyOf(root);
if (java.util.Objects.equals(currentKey, subjectRootKeys.get(subject))) continue;
log.warn("Root of subject {} ({}) was replaced or recreated; re-registering watches and forcing a full backup.",
subject, root);
registerSubject(subject, true);
}
}
private static Object fileKeyOf(Path path) {
try {
return Files.readAttributes(path, BasicFileAttributes.class).fileKey();
} catch (IOException e) {
return null;
}
}
/**
* (Re-)registers a subject's whole directory tree. Existing watches of the subject are
* cancelled first; when {@code lostEvents} is set, the subject is flagged for a full
* backup because changes may have happened while we were not watching.
*/
private void registerSubject(String subject, boolean lostEvents) {
keyToDir.entrySet().removeIf(e -> {
if (subject.equals(watchedDirs.get(e.getValue()))) {
e.getKey().cancel();
return true;
}
return false;
});
watchedDirs.values().removeIf(subject::equals);
var root = subjectRoots.get(subject);
subjectRootKeys.put(subject, fileKeyOf(root));
if (Files.isDirectory(root)) {
// no need to report existing files: the startup full backup (initial call) or the
// forced full backup (lostEvents) already covers them
registerTree(root, subject, false);
} else {
log.error("Root of subject {} ({}) does not exist, nothing is watched for it.", subject, root);
}
if (lostEvents) listener.onOverflow(subject);
}
private void registerTree(Path root, String subject, boolean reportFiles) {
try (Stream<Path> walk = Files.walk(root)) {
for (var p : walk.toList()) {
if (Files.isDirectory(p)) {
var abs = p.toAbsolutePath();
var key = abs.register(watcher,
StandardWatchEventKinds.ENTRY_CREATE,
StandardWatchEventKinds.ENTRY_MODIFY,
StandardWatchEventKinds.ENTRY_DELETE);
watchedDirs.put(abs, subject);
keyToDir.put(key, abs);
} else if (reportFiles && Files.isRegularFile(p)) {
listener.onChange(subject, p);
}
}
} catch (IOException e) {
log.warn("Cannot watch directory tree {}; forcing a full backup on next run.", root, e);
listener.onOverflow(subject);
}
}
}
+31
View File
@@ -0,0 +1,31 @@
package io.ib67.sfcraft;
import java.io.IOException;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
public final class FsUtil {
private FsUtil() {
}
public static void deleteRecursively(Path root) throws IOException {
if (Files.notExists(root)) return;
Files.walkFileTree(root, new SimpleFileVisitor<>() {
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
Files.deleteIfExists(file);
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
if (exc != null) throw exc;
Files.deleteIfExists(dir);
return FileVisitResult.CONTINUE;
}
});
}
}
+16 -8
View File
@@ -6,18 +6,26 @@ import io.ib67.sfcraft.config.StorageOption;
import io.ib67.sfcraft.config.serializer.StorageOptionSerializer;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.StampedLock;
public class Globals {
/**
* for api users: DO NOT MUTATE THIS STATE IN WHATEVER CASE SINCE THIS IS CAUTIOUSLY ALIGNED WITH THE CURRENT THREAD MODEL
* AND DOING SO WILL BREAK YOUR WORLD
* This state indicates:
* 1. A minecraft auto-save is in-progress.
* 2. The backup is in-progress
* 3. Some chunks are writing to disk
* Guards on-disk world files during backups.
* <p>
* Chunk IO ({@code RegionFileStorage.write/flush}, running on the per-dimension IOWorker
* threads) holds the read lock; the backup worker holds the write lock while it snapshots
* files. The main thread never blocks on this lock: {@code autoSave} and manual
* {@code saveEverything} calls are skipped while the write lock is held (see
* {@code MixinMinecraftServer}), which is what makes it safe for the backup thread to
* wait on main-thread tasks while holding the write lock.
* <p>
* A {@link StampedLock} is used instead of a ReentrantReadWriteLock because the write
* stamp may be released from a different code path than the one that acquired it
* (early release once files are cloned to staging). None of the guarded sections are
* reentrant.
*/
public static final AtomicReference<IOState> BACKUP_LOCK = new AtomicReference<>(IOState.IDLE);
public static final StampedLock WORLD_IO_LOCK = new StampedLock();
public static final Gson GSON = new GsonBuilder().setPrettyPrinting()
.registerTypeAdapter(StorageOption.class, new StorageOptionSerializer(
Map.of("s3", StorageOption.S3.class,
@@ -1,8 +0,0 @@
package io.ib67.sfcraft;
public enum IOState {
SAVING_WORLD,
BACKUP,
STORAGE_SYNC,
IDLE
}
+161 -62
View File
@@ -4,6 +4,7 @@ import com.github.luben.zstd.Zstd;
import com.google.gson.reflect.TypeToken;
import io.ib67.sfcraft.config.JustBackupConfig;
import io.ib67.sfcraft.config.StorageOption;
import io.ib67.sfcraft.strategy.BackupChains;
import io.ib67.sfcraft.strategy.BackupStrategy;
import io.ib67.sfcraft.strategy.BackupTracker;
import io.ib67.sfcraft.strategy.LocalBackupStrategy;
@@ -25,14 +26,16 @@ import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class JustBackupMod implements ModInitializer {
public class JustBackupMod implements ModInitializer, FileWatcherThread.Listener {
public static final String MOD_ID = "justbackup";
public static final Logger LOGGER = LoggerFactory.getLogger(MOD_ID);
protected final Map<String, BackupSubject> backupSubjects = new HashMap<>();
@@ -44,6 +47,8 @@ public class JustBackupMod implements ModInitializer {
protected JustBackupConfig config;
protected BackupTracker tracker;
protected Path backupIndexPath;
protected Path tmpDir;
protected List<Path> excludedRoots;
protected FileWatcherThread watcherThread;
@Override
@@ -62,33 +67,45 @@ public class JustBackupMod implements ModInitializer {
@SneakyThrows
private void doInitialize() {
LOGGER.info("Loading backup configuration");
Path config1 = Path.of("config");
backupIndexPath = config1.resolve("just_backup_index.json");
var configPath = config1.resolve("just_backup.json");
Path configDir = Path.of("config");
backupIndexPath = configDir.resolve("just_backup_index.json");
var configPath = configDir.resolve("just_backup.json");
if (Files.notExists(configPath.getParent()))
Files.createDirectories(configPath.getParent());
if (Files.notExists(configPath))
saveConfig(configPath);
if (Files.notExists(backupIndexPath))
Files.createFile(backupIndexPath);
config = Globals.GSON.fromJson(Files.readString(configPath), JustBackupConfig.class);
config.backupSubjects().forEach((k, v) -> {
var subjectPath = Path.of(v);
if (Files.notExists(subjectPath)) {
LOGGER.error("Backup _subject {} does not exist.", v);
LOGGER.error("Backup subject {} does not exist.", v);
return;
}
backupSubjects.put(k, new BackupSubject(k, subjectPath, new ArrayList<>()));
backupSubjects.put(k, new BackupSubject(k, subjectPath));
});
if (config.zstdDictPath() != null && !config.zstdDictPath().isEmpty()) {
zstdDict = Files.readAllBytes(Path.of(config.zstdDictPath()));
}
tmpDir = Path.of(config.temporaryBackupDir());
excludedRoots = new ArrayList<>();
excludedRoots.add(tmpDir);
if (config.option() instanceof StorageOption.Local local) {
excludedRoots.add(Path.of(local.saveDir()));
}
var strategy = createStrategy();
this.tracker = new BackupTracker(strategy, Globals.GSON.fromJson(Files.readString(backupIndexPath), new TypeToken<>() {
}));
Map<String, Backup> initialIndex = Files.exists(backupIndexPath)
? Globals.GSON.fromJson(Files.readString(backupIndexPath), new TypeToken<Map<String, Backup>>() {
})
: null;
this.tracker = new BackupTracker(strategy, initialIndex, backupIndexPath);
LOGGER.info("Configuration has been loaded. Using storage option: {}", config.option().type());
scheduledBackupExecutor = Executors.newSingleThreadScheduledExecutor();
scheduledBackupExecutor = Executors.newSingleThreadScheduledExecutor(r -> {
var t = new Thread(r, "JustBackup/Worker");
t.setDaemon(false);
return t;
});
registerWatchService();
registerHandlers();
}
@@ -98,20 +115,30 @@ public class JustBackupMod implements ModInitializer {
config.backupSubjects().entrySet().stream()
.map(it -> Map.entry(it.getKey(), Path.of(it.getValue())))
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)),
this::handleFileChanges
this
);
ServerLifecycleEvents.SERVER_STOPPED.register(server -> {
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
if (watcherThread.getState() != Thread.State.TERMINATED) watcherThread.interrupt();
});
}));
watcherThread.start();
}
private void handleFileChanges(String _subject, Path path) {
synchronized (backupSubjects) {
var subject = backupSubjects.get(_subject);
if (subject != null)
subject.changedFiles().add(path); // check null
@Override
public void onChange(String subjectName, Path file) {
var subject = backupSubjects.get(subjectName);
if (subject != null) subject.addChanged(file);
}
@Override
public void onDelete(String subjectName, Path file) {
var subject = backupSubjects.get(subjectName);
if (subject != null) subject.addDeleted(file);
}
@Override
public void onOverflow(String subjectName) {
var subject = backupSubjects.get(subjectName);
if (subject != null) subject.forceFullNext();
}
@SneakyThrows
@@ -130,96 +157,168 @@ public class JustBackupMod implements ModInitializer {
}
lastBackupServerTicks = server.getTickCount();
if (!suspend) backupAll(config.incremental()).thenAccept(it -> {
LOGGER.info("Backup success!");
it.forEach((k, v) -> LOGGER.info(k + ": " + v));
LOGGER.info("Scheduled backup finished.");
it.forEach((k, v) -> LOGGER.info("{}: {}", k, v));
});
},
config.backupIntervalMinutes(), config.backupIntervalMinutes(), TimeUnit.MINUTES);
});
ServerLifecycleEvents.SERVER_STOPPED.register(server -> {
try {
Files.deleteIfExists(Path.of(config.temporaryBackupDir()));
} catch (IOException e) {
throw new RuntimeException(e);
}
});
var command = new BackupCommands(this);
CommandRegistrationCallback.EVENT.register(command::registerCommand);
ServerLifecycleEvents.SERVER_STOPPED.register(server -> {
LOGGER.info("Shutting down the executor");
// Runs at the head of stopServer, before the final world save: in-flight backups must
// release the world IO lock before that save, or the flush would park forever.
ServerLifecycleEvents.SERVER_STOPPING.register(server -> {
suspend = true;
scheduledBackupExecutor.shutdown();
if (command.restoreIssued != null) {
command.restoreIssued.run();
try {
if (!scheduledBackupExecutor.awaitTermination(15, TimeUnit.SECONDS)) {
LOGGER.info("Waiting up to 10 minutes for the in-flight backup to finish...");
if (!scheduledBackupExecutor.awaitTermination(10, TimeUnit.MINUTES)) {
LOGGER.error("In-flight backup did not finish in time; its bundle may be incomplete.");
}
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
tracker.close();
});
ServerLifecycleEvents.SERVER_STOPPED.register(server -> {
if (command.restoreIssued != null) {
try {
command.restoreIssued.run();
} catch (Exception e) {
LOGGER.error("Backup restoration failed", e);
}
}
try {
tracker.close();
} catch (Exception e) {
LOGGER.error("Error while closing the backup storage", e);
}
cleanTemporaryDir();
if (watcherThread != null && watcherThread.getState() != Thread.State.TERMINATED) {
watcherThread.interrupt();
}
});
}
/**
* Removes only artifacts we created (staging clones, leftover bundles, temp downloads).
* Never deletes the directory itself: a misconfigured temporaryBackupDir pointing at the
* server root must not wipe the server.
*/
private void cleanTemporaryDir() {
try {
FsUtil.deleteRecursively(tmpDir.resolve("staging"));
} catch (IOException e) {
LOGGER.warn("Cannot clean staging dir", e);
}
if (Files.notExists(tmpDir)) return;
try (Stream<Path> files = Files.list(tmpDir)) {
files.filter(Files::isRegularFile)
.filter(p -> {
var n = p.getFileName().toString();
return (n.startsWith("backup_") && n.endsWith(".swb.zst")) || n.startsWith("s3_");
})
.forEach(p -> {
try {
Files.deleteIfExists(p);
} catch (IOException e) {
LOGGER.warn("Cannot delete leftover {}", p, e);
}
});
} catch (IOException e) {
LOGGER.warn("Cannot clean temporary backup dir", e);
}
}
/**
* Logs the message and, when enabled and a server is up, broadcasts it to players from
* the main thread (PlayerList is not safe to touch from the backup executor).
*/
public void broadcast(Component message) {
LOGGER.info(message.getString());
if (!config.broadcast()) return;
var s = server;
if (s == null || !s.isRunning()) return;
s.submit(() -> s.getPlayerList().broadcastSystemMessage(message, false));
}
public CompletableFuture<Map<String, Backup>> backupAll(boolean incremental) {
if (backupSubjects.isEmpty()) return CompletableFuture.completedFuture(Map.of());
var map = new HashMap<String, CompletableFuture<Backup>>();
server.getPlayerList().broadcastSystemMessage(
Component.literal(" BACKUP >> ").withColor(Color.RED.getRGB()).withStyle(it -> it.withBold(true))
.append(Component.nullToEmpty("The server is performing a backup, you may experience some lag.")),
false
);
broadcast(Component.literal(" BACKUP >> ").withColor(Color.RED.getRGB()).withStyle(it -> it.withBold(true))
.append(Component.nullToEmpty("The server is performing a backup, you may experience some lag.")));
for (var entry : backupSubjects.entrySet()) {
map.put(entry.getKey(), issueBackup(entry.getKey(), incremental));
}
return CompletableFuture.allOf(map.values().toArray(new CompletableFuture[0]))
.thenApply(it -> {
var _map = new HashMap<String, Backup>();
var results = new HashMap<String, Backup>();
map.forEach((k, v) -> {
var result = v.join();
LOGGER.info("Backup information of {}: {}", k, result);
_map.put(k, result);
results.put(k, result);
});
return _map;
return results;
});
}
public CompletableFuture<Backup> issueBackup(String subjectName, boolean incremental) {
var subject = backupSubjects.get(subjectName);
if (subject == null)
return CompletableFuture.failedFuture(new IllegalArgumentException("Unknown backup _subject: " + subjectName));
if (!incremental) {
try (var s = Files.walk(subject.root())) {
subject = new BackupSubject(subject.name(), subject.root(), s.filter(Files::isRegularFile)
.filter(it -> !it.getFileName().toString().equals("session.lock"))
.toList());
} catch (IOException e) {
return CompletableFuture.failedFuture(new IllegalArgumentException("Cannot perform full backup for " + subjectName, e));
}
}
var finalSubject = subject;
return CompletableFuture.failedFuture(new IllegalArgumentException("Unknown backup subject: " + subjectName));
if (config.allowGunzip()) LOGGER.info("Chunk reassembler is enabled!");
var worker = new BackupWorker(server, tracker, Path.of(config.temporaryBackupDir()),
subject, c ->
var worker = new BackupWorker(server, tracker, tmpDir, subject, incremental, excludedRoots, c ->
c.allowGunzip(config.allowGunzip())
.relativeRoot(finalSubject.root())
.allocator(ByteBufAllocator.DEFAULT)
.maxWorkers(config.compressionWorkerThreads())
.compressionLevel(config.compressionLevel())
.dictionary(zstdDict)
);
return CompletableFuture.supplyAsync(worker, scheduledBackupExecutor).whenComplete(this::onBackupComplete);
return CompletableFuture.supplyAsync(() -> {
if (!tracker.isAvailable()) {
throw new IllegalStateException("Backup storage is not available (quota exceeded?), skipping backup of " + subjectName);
}
return worker.get();
}, scheduledBackupExecutor).whenComplete(this::onBackupComplete);
}
@SneakyThrows
private void onBackupComplete(@UnknownNullability Backup backup, @UnknownNullability Throwable throwable) {
if (throwable != null) {
server.getPlayerList().broadcastSystemMessage(Component.literal("Backup failed. For administrators, please check your server console."), false);
LOGGER.error(throwable.getMessage(), throwable);
broadcast(Component.literal("Backup failed. For administrators, please check your server console."));
LOGGER.error("Backup failed", throwable);
} else {
Files.writeString(backupIndexPath, Globals.GSON.toJson(tracker.getTrackedBackups()));
server.getPlayerList().broadcastSystemMessage(Component.literal("Backup success. " + backup), false);
broadcast(Component.literal("Backup success. ").append(backup.toText()));
rotate(backup.subject());
}
}
/**
* Deletes whole generations (a full backup plus its incrementals) beyond the newest
* {@code keepBackups} ones. Entries without chain metadata are never rotated away.
*/
private void rotate(String subjectName) {
if (subjectName == null) return;
try {
var victims = BackupChains.rotationVictims(
tracker.getTrackedBackups().values(), subjectName, config.keepBackups());
for (var victim : victims) {
LOGGER.info("Rotating out old backup {} ({})", victim.backupKey(), victim.incremental() ? "INCR" : "FULL");
try {
tracker.deleteBackup(victim);
} catch (Exception e) {
LOGGER.warn("Cannot delete old backup {}", victim.backupKey(), e);
}
}
} catch (Exception e) {
LOGGER.warn("Backup rotation failed for {}", subjectName, e);
}
}
private BackupStrategy createStrategy() {
return switch (config.option()) {
case StorageOption.Local local -> new LocalBackupStrategy(local);
case StorageOption.S3 s3 -> new S3BackupStrategy(s3, Path.of(config.temporaryBackupDir()));
case StorageOption.S3 s3 -> new S3BackupStrategy(s3, tmpDir);
};
}
@@ -0,0 +1,163 @@
package io.ib67.sfcraft;
import lombok.extern.log4j.Log4j2;
import java.lang.foreign.Arena;
import java.lang.foreign.FunctionDescriptor;
import java.lang.foreign.Linker;
import java.lang.foreign.MemoryLayout;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.StructLayout;
import java.lang.foreign.ValueLayout;
import java.lang.invoke.MethodHandle;
import java.lang.invoke.VarHandle;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Locale;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
/**
* Clones files via the Linux {@code FICLONE} ioctl (reflink). On CoW filesystems
* (btrfs, XFS with reflink, OpenZFS >= 2.2) this is a metadata-only operation: the clone
* shares extents with the source, so staging a whole backup takes milliseconds and no
* extra disk space, letting the backup release the world IO lock before compressing.
* <p>
* Unsupported setups (non-Linux, ext4, overlayfs, staging dir on a different mount) are
* detected on first use and cached per subject; callers then fall back to packing while
* holding the world IO lock.
*/
@Log4j2
public final class ReflinkCloner {
private static final long FICLONE = 0x40049409L;
private static final int O_RDONLY = 0;
private static final int O_WRONLY = 1;
private static final int O_CREAT = 0x40;
private static final int O_TRUNC = 0x200;
private static final int ENOENT = 2;
private static final int EXDEV = 18;
private static final MethodHandle OPEN;
private static final MethodHandle IOCTL;
private static final MethodHandle CLOSE;
private static final StructLayout CAPTURE_LAYOUT;
private static final VarHandle ERRNO;
private static final boolean AVAILABLE;
private static final Set<String> UNSUPPORTED_SUBJECTS = ConcurrentHashMap.newKeySet();
static {
MethodHandle open = null, ioctl = null, close = null;
StructLayout capture = null;
VarHandle errno = null;
boolean ok = false;
try {
if (System.getProperty("os.name", "").toLowerCase(Locale.ROOT).contains("linux")) {
var linker = Linker.nativeLinker();
var lookup = linker.defaultLookup();
var captureErrno = Linker.Option.captureCallState("errno");
capture = Linker.Option.captureStateLayout();
errno = capture.varHandle(MemoryLayout.PathElement.groupElement("errno"));
open = linker.downcallHandle(lookup.findOrThrow("open"),
FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.ADDRESS, ValueLayout.JAVA_INT, ValueLayout.JAVA_INT),
captureErrno);
ioctl = linker.downcallHandle(lookup.findOrThrow("ioctl"),
FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.JAVA_INT, ValueLayout.JAVA_LONG, ValueLayout.JAVA_INT),
captureErrno, Linker.Option.firstVariadicArg(2));
close = linker.downcallHandle(lookup.findOrThrow("close"),
FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.JAVA_INT));
ok = true;
} else {
log.info("Reflink fast path is Linux-only; backups will pack while holding the world IO lock.");
}
} catch (Throwable t) {
log.info("Reflink fast path unavailable ({}); backups will pack while holding the world IO lock.", t.toString());
}
OPEN = open;
IOCTL = ioctl;
CLOSE = close;
CAPTURE_LAYOUT = capture;
ERRNO = errno;
AVAILABLE = ok;
}
private ReflinkCloner() {
}
/**
* Clones {@code files} (which must live under {@code subjectRoot}) into
* {@code stagingRoot}, preserving their relative structure. Files that vanished since
* being listed are skipped.
*
* @return the staging root on success, or null if reflink is unsupported here — in that
* case any partial staging has been cleaned up and the subject is remembered as
* unsupported so future backups skip the attempt.
*/
public static Path tryStage(String subjectName, Path subjectRoot, List<Path> files, Path stagingRoot) {
if (!AVAILABLE || UNSUPPORTED_SUBJECTS.contains(subjectName)) return null;
var absRoot = subjectRoot.toAbsolutePath().normalize();
try {
Files.createDirectories(stagingRoot);
for (Path file : files) {
var src = file.toAbsolutePath().normalize();
if (!Files.isRegularFile(src)) continue;
var dst = stagingRoot.resolve(absRoot.relativize(src));
Files.createDirectories(dst.getParent());
int err = clone(src, dst);
if (err == ENOENT) continue; // deleted between listing and cloning
if (err != 0) {
markUnsupported(subjectName, src, err);
FsUtil.deleteRecursively(stagingRoot);
return null;
}
}
return stagingRoot;
} catch (Throwable t) {
log.warn("Reflink staging failed; falling back to packing under the world IO lock.", t);
UNSUPPORTED_SUBJECTS.add(subjectName);
try {
FsUtil.deleteRecursively(stagingRoot);
} catch (Exception cleanup) {
log.warn("Cannot clean up staging dir {}", stagingRoot, cleanup);
}
return null;
}
}
private static void markUnsupported(String subjectName, Path src, int err) {
UNSUPPORTED_SUBJECTS.add(subjectName);
if (err == EXDEV) {
log.warn("Reflink not possible for subject {}: {} and the temporary backup dir are on " +
"different filesystems. Put temporaryBackupDir on the same filesystem as the " +
"backup subject to enable the CoW fast path. Falling back to packing under the world IO lock.", subjectName, src);
} else {
log.info("Reflink unsupported for subject {} (errno {} on {}); " +
"falling back to packing under the world IO lock.", subjectName, err, src);
}
}
private static int clone(Path src, Path dst) throws Throwable {
try (var arena = Arena.ofConfined()) {
var capture = arena.allocate(CAPTURE_LAYOUT);
int srcFd = (int) OPEN.invokeExact(capture, arena.allocateFrom(src.toString()), O_RDONLY, 0);
if (srcFd < 0) return errno(capture);
try {
int dstFd = (int) OPEN.invokeExact(capture, arena.allocateFrom(dst.toString()),
O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (dstFd < 0) return errno(capture);
try {
int result = (int) IOCTL.invokeExact(capture, dstFd, FICLONE, srcFd);
return result == 0 ? 0 : errno(capture);
} finally {
var ignored = (int) CLOSE.invokeExact(dstFd);
}
} finally {
var ignored = (int) CLOSE.invokeExact(srcFd);
}
}
}
private static int errno(MemorySegment capture) {
return (int) ERRNO.get(capture, 0L);
}
}
@@ -1,14 +1,22 @@
package io.ib67.sfcraft.mixin;
import io.ib67.sfcraft.Globals;
import io.ib67.sfcraft.IOState;
import io.ib67.sfcraft.JustBackupMod;
import net.minecraft.server.MinecraftServer;
import org.spongepowered.asm.mixin.Mixin;
import org.spongepowered.asm.mixin.Shadow;
import org.spongepowered.asm.mixin.injection.At;
import org.spongepowered.asm.mixin.injection.Inject;
import org.spongepowered.asm.mixin.injection.callback.CallbackInfo;
import org.spongepowered.asm.mixin.injection.callback.CallbackInfoReturnable;
/**
* Skips main-thread saves while a backup holds the write lock. The main thread must never
* block on {@link Globals#WORLD_IO_LOCK}: a save request would join the IOWorkers, which
* are parked on the read lock, wedging the main thread until the backup finishes (and
* tripping the watchdog). The world was saved right before the backup acquired the lock,
* so skipping loses nothing.
*/
@Mixin(MinecraftServer.class)
public abstract class MixinMinecraftServer {
@@ -19,23 +27,21 @@ public abstract class MixinMinecraftServer {
protected abstract int computeNextAutosaveInterval();
@Inject(method = "autoSave", at = @At("HEAD"), cancellable = true)
private void backup$checkSaving(CallbackInfo ci) {
// is already backing up
private void backup$skipAutoSaveDuringBackup(CallbackInfo ci) {
if (Globals.WORLD_IO_LOCK.isWriteLocked()) {
// vanilla resets this at the head of autoSave; replicate it on the cancel path
// so the autosave doesn't re-trigger every tick
this.ticksUntilAutosave = computeNextAutosaveInterval();
IOState witness;
while ((witness = Globals.BACKUP_LOCK.compareAndExchange(IOState.IDLE, IOState.SAVING_WORLD)) == IOState.STORAGE_SYNC) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
System.err.println("Error while waiting for chunk sync. " + e);
JustBackupMod.LOGGER.info("Backup in progress, skipping this auto-save.");
ci.cancel();
}
}
if (witness == IOState.BACKUP) ci.cancel();
}
@Inject(method = "autoSave", at = @At("RETURN"))
private void backup$disableProtect(CallbackInfo ci) {
Globals.BACKUP_LOCK.setRelease(IOState.IDLE);
@Inject(method = "saveEverything", at = @At("HEAD"), cancellable = true)
private void backup$skipManualSaveDuringBackup(boolean suppressLog, boolean flush, boolean force, CallbackInfoReturnable<Boolean> cir) {
if (Globals.WORLD_IO_LOCK.isWriteLocked()) {
JustBackupMod.LOGGER.warn("Backup in progress, skipping this save request. The world was already saved when the backup started.");
cir.setReturnValue(false);
}
}
}
@@ -1,62 +1,38 @@
package io.ib67.sfcraft.mixin;
import com.llamalad7.mixinextras.injector.wrapmethod.WrapMethod;
import com.llamalad7.mixinextras.injector.wrapoperation.Operation;
import io.ib67.sfcraft.Globals;
import io.ib67.sfcraft.IOState;
import it.unimi.dsi.fastutil.longs.Long2ObjectLinkedOpenHashMap;
import lombok.SneakyThrows;
import net.minecraft.world.level.chunk.storage.RegionFile;
import net.minecraft.nbt.CompoundTag;
import net.minecraft.world.level.ChunkPos;
import net.minecraft.world.level.chunk.storage.RegionFileStorage;
import org.spongepowered.asm.mixin.Final;
import org.spongepowered.asm.mixin.Mixin;
import org.spongepowered.asm.mixin.Overwrite;
import org.spongepowered.asm.mixin.Shadow;
@Mixin(RegionFileStorage.class)
public abstract class MixinRegionBasedStorage {
@Shadow
@Final
private Long2ObjectLinkedOpenHashMap<RegionFile> regionCache;
/**
* @author iceBear67
* @reason to wrap the saving process with an exception handler.
* Holds the read side of {@link Globals#WORLD_IO_LOCK} around region file writes and flushes.
* Multiple IOWorker threads (one per dimension and storage kind) may run concurrently under
* the read lock; while the backup worker holds the write lock, all region IO parks here and
* resumes untouched afterwards.
*/
@Overwrite
@SneakyThrows
public void flush() {
// allowed states:
// STORAGE_SYNC, AUTO_SAVE, IDLE
// we try
var witness = Globals.BACKUP_LOCK.getAndSet(IOState.STORAGE_SYNC);
if (witness == IOState.BACKUP) {
// yield out and keep blocking.
// The backup worker will wait for us setting this back to backup.
int i =0;
Globals.BACKUP_LOCK.set(IOState.BACKUP);
while (Globals.BACKUP_LOCK.getAcquire() == IOState.BACKUP) {
if(i==0){
System.out.println("Server is performing backup, ChunkIO will wait for it.");
}
if(i++ > 3000){
System.err.println("WARNING: BACKUP_LOCK is holding for 10 minutes!");
System.err.println("ChunkIO will not wait for backup anymore, writting to disk...");
System.err.println("You may adjust this behaviour in configuration");
break;
}
Thread.sleep(2000);
}
}
// program point: IDLE, AUTOSAVE, STORAGESYNC
@Mixin(RegionFileStorage.class)
public abstract class MixinRegionBasedStorage {
@WrapMethod(method = "write")
private void backup$guardWrite(ChunkPos pos, CompoundTag tag, Operation<Void> original) {
long stamp = Globals.WORLD_IO_LOCK.readLock();
try {
for (RegionFile value : this.regionCache.values()) {
value.flush();
}
original.call(pos, tag);
} finally {
// we assert that there are no other threads calling sync()
// so we can just jump back to the witness value.
// witness: AUTO_SAVE or IDLE
Globals.BACKUP_LOCK.compareAndExchange(IOState.STORAGE_SYNC, witness);
Globals.BACKUP_LOCK.setRelease(witness);
Globals.WORLD_IO_LOCK.unlockRead(stamp);
}
}
@WrapMethod(method = "flush")
private void backup$guardFlush(Operation<Void> original) {
long stamp = Globals.WORLD_IO_LOCK.readLock();
try {
original.call();
} finally {
Globals.WORLD_IO_LOCK.unlockRead(stamp);
}
}
}
@@ -0,0 +1,81 @@
package io.ib67.sfcraft.s3;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.security.MessageDigest;
import java.util.HexFormat;
import java.util.SortedMap;
import java.util.StringJoiner;
/**
* Minimal AWS Signature Version 4 signer for S3.
*
* @see <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html">
* Authenticating Requests: Using the Authorization Header (AWS Signature Version 4)</a>
*/
final class AwsV4Signer {
// SHA-256 of a zero-length payload
static final String EMPTY_PAYLOAD_SHA256 =
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
private static final HexFormat HEX = HexFormat.of();
private AwsV4Signer() {
}
/**
* @param encodedPath already URI-encoded absolute path, exactly as sent on the wire
* @param headers headers to sign; keys must be lowercase, values trimmed.
* Must include host, x-amz-date and x-amz-content-sha256.
*/
static String authorization(String method, String encodedPath, String canonicalQuery,
SortedMap<String, String> headers, String payloadHash,
String amzDate, String region, String service,
String accessKey, String secretKey) {
var canonicalHeaders = new StringBuilder();
var signedHeaders = new StringJoiner(";");
for (var header : headers.entrySet()) {
canonicalHeaders.append(header.getKey()).append(':').append(header.getValue()).append('\n');
signedHeaders.add(header.getKey());
}
var canonicalRequest = method + '\n'
+ encodedPath + '\n'
+ canonicalQuery + '\n'
+ canonicalHeaders + '\n'
+ signedHeaders + '\n'
+ payloadHash;
var dateStamp = amzDate.substring(0, 8);
var scope = dateStamp + "/" + region + "/" + service + "/aws4_request";
var stringToSign = "AWS4-HMAC-SHA256\n" + amzDate + "\n" + scope + "\n"
+ sha256Hex(canonicalRequest.getBytes(StandardCharsets.UTF_8));
var signingKey = hmac(hmac(hmac(hmac(
("AWS4" + secretKey).getBytes(StandardCharsets.UTF_8), dateStamp), region), service), "aws4_request");
return "AWS4-HMAC-SHA256 Credential=" + accessKey + "/" + scope
+ ",SignedHeaders=" + signedHeaders
+ ",Signature=" + HEX.formatHex(hmac(signingKey, stringToSign));
}
static String sha256Hex(byte[] data) {
return HEX.formatHex(sha256().digest(data));
}
static MessageDigest sha256() {
try {
return MessageDigest.getInstance("SHA-256");
} catch (GeneralSecurityException e) {
throw new IllegalStateException("JVM lacks SHA-256", e);
}
}
private static byte[] hmac(byte[] key, String data) {
try {
var mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(key, "HmacSHA256"));
return mac.doFinal(data.getBytes(StandardCharsets.UTF_8));
} catch (GeneralSecurityException e) {
throw new IllegalStateException("JVM lacks HmacSHA256", e);
}
}
}
@@ -0,0 +1,21 @@
package io.ib67.sfcraft.s3;
import java.io.IOException;
/**
* A non-2xx response from the S3 endpoint. Network failures surface as plain
* {@link IOException}s instead.
*/
public final class S3Exception extends IOException {
private final int statusCode;
public S3Exception(String request, int statusCode, String errorBody) {
super(request + " -> HTTP " + statusCode + (errorBody == null || errorBody.isBlank()
? "" : ": " + errorBody.strip()));
this.statusCode = statusCode;
}
public int statusCode() {
return statusCode;
}
}
@@ -0,0 +1,170 @@
package io.ib67.sfcraft.s3;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.HexFormat;
import java.util.Locale;
import java.util.Objects;
import java.util.TreeMap;
/**
* A tiny synchronous S3 client on top of {@link HttpClient}, covering only what
* this mod needs: headBucket, putObject, getObject and deleteObject. Requests
* are authenticated with AWS Signature Version 4 and work against any
* S3-compatible endpoint, in both path-style and virtual-hosted style.
*/
public final class SimpleS3Client implements AutoCloseable {
private static final DateTimeFormatter AMZ_DATE =
DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'", Locale.ROOT).withZone(ZoneOffset.UTC);
private final HttpClient http;
private final String scheme;
// authority as HttpClient will put it into the Host header: default ports stripped
private final String hostPort;
private final String region;
private final String accessKey;
private final String secretKey;
private final boolean pathStyle;
public SimpleS3Client(String endpoint, String region, String accessKey, String secretKey, boolean pathStyle) {
var uri = URI.create(endpoint);
this.scheme = Objects.requireNonNull(uri.getScheme(), "endpoint has no scheme: " + endpoint);
Objects.requireNonNull(uri.getHost(), "endpoint has no host: " + endpoint);
var defaultPort = "https".equals(scheme) ? 443 : 80;
this.hostPort = uri.getPort() == -1 || uri.getPort() == defaultPort
? uri.getHost()
: uri.getHost() + ":" + uri.getPort();
this.region = Objects.requireNonNull(region);
this.accessKey = Objects.requireNonNull(accessKey);
this.secretKey = Objects.requireNonNull(secretKey);
this.pathStyle = pathStyle;
this.http = HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.followRedirects(HttpClient.Redirect.NEVER)
.connectTimeout(Duration.ofSeconds(10))
.build();
}
/**
* Throws {@link S3Exception} if the bucket does not exist or the credentials
* cannot access it.
*/
public void headBucket(String bucket) throws IOException, InterruptedException {
var target = target(bucket, null);
var request = signed("HEAD", target, AwsV4Signer.EMPTY_PAYLOAD_SHA256)
.HEAD().build();
var response = http.send(request, HttpResponse.BodyHandlers.discarding());
ensureSuccess("HEAD " + target.uri(), response.statusCode(), null);
}
public void putObject(String bucket, String key, Path file, String contentType)
throws IOException, InterruptedException {
var target = target(bucket, key);
var request = signed("PUT", target, sha256Hex(file))
.header("content-type", contentType)
.PUT(HttpRequest.BodyPublishers.ofFile(file))
.build();
var response = http.send(request, HttpResponse.BodyHandlers.ofString());
ensureSuccess("PUT " + target.uri(), response.statusCode(), response.body());
}
/**
* Returns the object body; the caller is responsible for closing it.
*/
public InputStream getObject(String bucket, String key) throws IOException, InterruptedException {
var target = target(bucket, key);
var request = signed("GET", target, AwsV4Signer.EMPTY_PAYLOAD_SHA256)
.GET().build();
var response = http.send(request, HttpResponse.BodyHandlers.ofInputStream());
if (response.statusCode() / 100 != 2) {
String body;
try (var in = response.body()) {
body = new String(in.readNBytes(4096), StandardCharsets.UTF_8);
}
throw new S3Exception("GET " + target.uri(), response.statusCode(), body);
}
return response.body();
}
public void deleteObject(String bucket, String key) throws IOException, InterruptedException {
var target = target(bucket, key);
var request = signed("DELETE", target, AwsV4Signer.EMPTY_PAYLOAD_SHA256)
.DELETE().build();
var response = http.send(request, HttpResponse.BodyHandlers.ofString());
ensureSuccess("DELETE " + target.uri(), response.statusCode(), response.body());
}
@Override
public void close() {
http.close();
}
private record Target(URI uri, String host, String encodedPath) {
}
private Target target(String bucket, String key) {
var host = pathStyle ? hostPort : bucket + "." + hostPort;
var path = new StringBuilder();
if (pathStyle) {
path.append('/').append(URLEncoder.encode(bucket, Charset.defaultCharset()));
}
if (key != null) {
for (var segment : key.split("/", -1)) {
path.append('/').append(URLEncoder.encode(segment, Charset.defaultCharset()));
}
}
if (path.isEmpty()) {
path.append('/');
}
return new Target(URI.create(scheme + "://" + host + path), host, path.toString());
}
/**
* Host is signed but never set explicitly — HttpClient forbids that and
* derives it from the URI; {@link #hostPort} mirrors that derivation.
*/
private HttpRequest.Builder signed(String method, Target target, String payloadHash) {
var amzDate = AMZ_DATE.format(Instant.now());
var headers = new TreeMap<String, String>();
headers.put("host", target.host());
headers.put("x-amz-content-sha256", payloadHash);
headers.put("x-amz-date", amzDate);
var authorization = AwsV4Signer.authorization(method, target.encodedPath(), "",
headers, payloadHash, amzDate, region, "s3", accessKey, secretKey);
return HttpRequest.newBuilder(target.uri())
.header("x-amz-content-sha256", payloadHash)
.header("x-amz-date", amzDate)
.header("authorization", authorization);
}
private static void ensureSuccess(String request, int statusCode, String body) throws S3Exception {
if (statusCode / 100 != 2) {
throw new S3Exception(request, statusCode, body);
}
}
private static String sha256Hex(Path file) throws IOException {
var digest = AwsV4Signer.sha256();
try (var in = Files.newInputStream(file)) {
var buffer = new byte[64 * 1024];
int read;
while ((read = in.read(buffer)) != -1) {
digest.update(buffer, 0, read);
}
}
return HexFormat.of().formatHex(digest.digest());
}
}
@@ -0,0 +1,84 @@
package io.ib67.sfcraft.strategy;
import io.ib67.sfcraft.Backup;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Comparator;
import java.util.List;
import java.util.Objects;
/**
* Pure functions over the backup index: resolving restore chains (full base + following
* incrementals) and grouping backups into generations for rotation. Legacy entries
* (created before subject/createdAt existed) are excluded from both — they are restorable
* standalone but never chained or rotated away.
*/
public final class BackupChains {
private BackupChains() {
}
/**
* Resolves the list of bundles that must be extracted, in order, to restore {@code target}.
* A full backup restores alone; an incremental needs its most recent full base and every
* incremental in between.
*
* @throws IllegalStateException if the target is an incremental without a reachable base
*/
public static List<Backup> resolveChain(Collection<Backup> tracked, Backup target) {
if (!target.incremental()) return List.of(target);
if (target.legacy()) {
throw new IllegalStateException("Backup " + target.backupKey()
+ " lacks chain metadata (created by an older version) and cannot be restored incrementally.");
}
var chain = new ArrayList<Backup>();
tracked.stream()
.filter(b -> !b.legacy() && Objects.equals(target.subject(), b.subject()))
.filter(b -> b.createdAt() < target.createdAt())
.sorted(Comparator.comparingLong(Backup::createdAt))
.forEach(b -> {
if (!b.incremental()) chain.clear(); // a newer full base supersedes older history
chain.add(b);
});
chain.add(target);
if (chain.getFirst().incremental()) {
throw new IllegalStateException("No full backup found before " + target.backupKey()
+ "; cannot restore an incremental backup without its full base.");
}
return List.copyOf(chain);
}
/**
* Groups a subject's backups into generations: each full backup starts a new generation
* that collects the incrementals after it. Incrementals older than the first full backup
* (orphans) belong to no generation.
*/
public static List<List<Backup>> generations(Collection<Backup> tracked, String subject) {
var result = new ArrayList<List<Backup>>();
List<Backup> current = null;
var sorted = tracked.stream()
.filter(b -> !b.legacy() && Objects.equals(subject, b.subject()))
.sorted(Comparator.comparingLong(Backup::createdAt))
.toList();
for (var b : sorted) {
if (!b.incremental()) {
current = new ArrayList<>();
result.add(current);
}
if (current != null) current.add(b);
}
return result;
}
/**
* @return the backups that fall out of the newest {@code keepGenerations} generations and
* should be deleted. Whole generations only — an incremental is never separated from its base.
*/
public static List<Backup> rotationVictims(Collection<Backup> tracked, String subject, int keepGenerations) {
var generations = generations(tracked, subject);
if (generations.size() <= keepGenerations) return List.of();
return generations.subList(0, generations.size() - keepGenerations).stream()
.flatMap(List::stream)
.toList();
}
}
@@ -6,12 +6,24 @@ import java.nio.file.Path;
public interface BackupStrategy {
/**
* create a backup
* @return a key to this backup
* Stores the finished bundle.
*
* @param subject name of the backup subject
* @param source directory the bundle was created from
* @param bundle the bundle file to store; the caller deletes it afterwards
* @param incremental whether the bundle only contains changed files
* @param createdAt epoch millis the backup was taken at
* @return metadata describing the stored backup
*/
Backup createBackup(String subject, Path source, Path bundle);
Backup createBackup(String subject, Path source, Path bundle, boolean incremental, long createdAt);
void recoverBackup(Backup backup, Path restorePath);
/**
* Extracts {@code backup} into {@code restorePath}.
*
* @param ignoreDeletions when true, tombstone entries are skipped, so files deleted since
* the base backup are kept instead of removed on restore
*/
void recoverBackup(Backup backup, Path restorePath, boolean ignoreDeletions);
void deleteBackup(Backup backup);
@@ -1,40 +1,78 @@
package io.ib67.sfcraft.strategy;
import io.ib67.sfcraft.Backup;
import io.ib67.sfcraft.Globals;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.extern.log4j.Log4j2;
import java.io.IOException;
import java.nio.file.AtomicMoveNotSupportedException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.HashMap;
import java.nio.file.StandardCopyOption;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
/**
* Wraps a storage strategy with an in-memory index of known backups, persisted atomically
* to {@code indexPath} after every mutation. The map is concurrent: the backup executor
* writes while the main thread reads it for commands and tab completion.
*/
@Log4j2
public class BackupTracker implements BackupStrategy {
protected final BackupStrategy underlying;
@Getter
protected final Map<String, Backup> trackedBackups;
protected final Map<String, Backup> trackedBackups = new ConcurrentHashMap<>();
protected final Path indexPath;
public BackupTracker(BackupStrategy underlying, Map<String, Backup> trackedBackups) {
public BackupTracker(BackupStrategy underlying, Map<String, Backup> initial, Path indexPath) {
this.underlying = Objects.requireNonNull(underlying);
this.trackedBackups = trackedBackups == null ? new HashMap<>() : trackedBackups;
this.indexPath = Objects.requireNonNull(indexPath);
if (initial != null) {
initial.forEach((k, v) -> {
if (k != null && v != null) trackedBackups.put(k, v);
});
}
}
@Override
public Backup createBackup(String subject, Path from, Path pathToBundle) {
var result = underlying.createBackup(subject, from, pathToBundle);
public Backup createBackup(String subject, Path source, Path bundle, boolean incremental, long createdAt) {
var result = underlying.createBackup(subject, source, bundle, incremental, createdAt);
trackedBackups.put(result.backupKey(), result);
persist();
return result;
}
@Override
public void recoverBackup(Backup backup, Path restorePath) {
underlying.recoverBackup(backup, restorePath);
public void recoverBackup(Backup backup, Path restorePath, boolean ignoreDeletions) {
underlying.recoverBackup(backup, restorePath, ignoreDeletions);
}
@Override
public void deleteBackup(Backup backup) {
underlying.deleteBackup(backup);
trackedBackups.remove(backup.backupKey());
persist();
}
/**
* Writes the index to a temp file first and atomically moves it into place, so a crash
* mid-write can never leave a corrupt index behind (which would disable the mod on the
* next startup).
*/
protected synchronized void persist() {
try {
var tmp = indexPath.resolveSibling(indexPath.getFileName() + ".tmp");
Files.writeString(tmp, Globals.GSON.toJson(trackedBackups));
try {
Files.move(tmp, indexPath, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
} catch (AtomicMoveNotSupportedException e) {
Files.move(tmp, indexPath, StandardCopyOption.REPLACE_EXISTING);
}
} catch (IOException e) {
log.error("Cannot persist backup index to {}", indexPath, e);
}
}
@Override
@@ -25,33 +25,32 @@ public class LocalBackupStrategy implements BackupStrategy {
@Override
@SneakyThrows
public Backup createBackup(String subject, Path from, Path pathToBundle) {
public Backup createBackup(String subject, Path source, Path bundle, boolean incremental, long createdAt) {
var parent = backupParentRoot.resolve(subject);
var fileName = bundle.getFileName().toString();
var result = new Backup(
pathToBundle.getFileName().toString(),
Path.of(subject).resolve(pathToBundle.getFileName()).normalize().toString(),
"local", from.toString(), false, Files.size(pathToBundle));
fileName,
Path.of(subject).resolve(fileName).normalize().toString(),
"local", source.toString(), subject, incremental, createdAt, Files.size(bundle));
if (Files.notExists(parent)) {
Files.createDirectories(parent);
}
//todo new name
Files.move(pathToBundle, parent.resolve(pathToBundle.getFileName()));
Files.move(bundle, parent.resolve(fileName));
return result;
}
@Override
@SneakyThrows
public void recoverBackup(Backup backup, Path restorePath) {
// check if it is compressed
public void recoverBackup(Backup backup, Path restorePath, boolean ignoreDeletions) {
var backupFile = backupParentRoot.resolve(backup.backupKey());
if (Files.notExists(backupFile)) throw new IllegalArgumentException("Backup " + backupFile + " does not exist");
BundleReader.builder().build().extract(backupFile, restorePath);
BundleReader.builder().ignoreDeletions(ignoreDeletions).build().extract(backupFile, restorePath);
}
@SneakyThrows
@Override
public void deleteBackup(Backup backup) {
Validate.isTrue("local".equals(backup.backupKey()));
Validate.isTrue("local".equals(backup.type()), "Not a local backup: %s", backup.backupKey());
var bundlePath = backupParentRoot.resolve(backup.backupKey());
Files.deleteIfExists(bundlePath);
}
@@ -61,14 +60,14 @@ public class LocalBackupStrategy implements BackupStrategy {
public boolean isAvailable() {
if (option.diskSizeQuotaBytes() == 0) return true;
try (var f = Files.walk(backupParentRoot)) {
if (f.mapToLong(it->{
if (f.filter(Files::isRegularFile).mapToLong(it -> {
try {
return Files.size(it);
} catch (IOException e) {
throw new RuntimeException(e);
}
}).sum() >= option.diskSizeQuotaBytes()) {
log.warn("Have no space to set another backup! ");
log.warn("Backup disk quota ({} bytes) exceeded, refusing to create new backups!", option.diskSizeQuotaBytes());
return false;
}
}
@@ -3,41 +3,30 @@ package io.ib67.sfcraft.strategy;
import io.ib67.sfcraft.Backup;
import io.ib67.sfcraft.bundler.BundleReader;
import io.ib67.sfcraft.config.StorageOption;
import io.ib67.sfcraft.s3.SimpleS3Client;
import lombok.SneakyThrows;
import lombok.extern.log4j.Log4j2;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import java.io.IOException;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicBoolean;
@Log4j2
public class S3BackupStrategy implements BackupStrategy {
protected final AtomicBoolean uploading = new AtomicBoolean(false);
protected final StorageOption.S3 option;
protected final S3Client s3;
protected final SimpleS3Client s3;
protected final Path temporaryDownloadPath;
@SneakyThrows
public S3BackupStrategy(StorageOption.S3 option, Path tempDownloadPath) {
this.option = option;
var credential = AwsBasicCredentials.create(option.accessKey(), option.secretKey());
this.s3 = S3Client.builder()
.region(Region.of(option.region()))
.forcePathStyle(option.enforcePathStyle())
.endpointOverride(URI.create(option.endpoint()))
.credentialsProvider(StaticCredentialsProvider.create(credential))
.build();
this.s3 = new SimpleS3Client(option.endpoint(), option.region(),
option.accessKey(), option.secretKey(), option.enforcePathStyle());
log.info("Testing S3 credentials");
// throws exception if the bucket neither inaccessible nor not exist
s3.headBucket(b -> b.bucket(option.bucket()));
s3.headBucket(option.bucket());
this.temporaryDownloadPath = Objects.requireNonNull(tempDownloadPath);
if (Files.notExists(temporaryDownloadPath))
Files.createDirectories(temporaryDownloadPath);
@@ -45,46 +34,41 @@ public class S3BackupStrategy implements BackupStrategy {
@Override
@SneakyThrows
public Backup createBackup(String subject, Path from, Path pathToBundle) {
if (!uploading.compareAndSet(false, true)) {
throw new IllegalStateException("Last upload is not done yet! keep waiting...");
}
int i = 0;
do {
public Backup createBackup(String subject, Path source, Path bundle, boolean incremental, long createdAt) {
// bundle names embed the creation timestamp, so keys are unique and retries idempotent
var object = option.prefix() + "/" + bundle.getFileName();
int attempt = 0;
while (true) {
try {
var object = option.prefix() + "/" + pathToBundle.getFileName().toString();
s3.putObject(o -> o.bucket(option.bucket())
.contentType("application/octet-stream")
.ifNoneMatch("*") // to avoid content overriding
.key(object).build(), pathToBundle);
return new Backup(pathToBundle.getFileName().toString(),
s3.putObject(option.bucket(), object, bundle, "application/octet-stream");
return new Backup(bundle.getFileName().toString(),
object, "s3",
from.toString(),
false,
Files.size(pathToBundle));
source.toString(),
subject, incremental, createdAt,
Files.size(bundle));
} catch (IOException e) {
if (i >= option.retryAmount()) {
throw new IllegalStateException("Cannot upload " + pathToBundle + " to S3");
if (attempt >= option.retryAmount()) {
throw new IllegalStateException("Cannot upload " + bundle + " to S3 after "
+ attempt + " retries", e);
}
var backoffSeconds = (long) Math.pow(2, attempt++);
log.error("Error uploading to S3, retrying in {}s ({}/{})",
backoffSeconds, attempt, option.retryAmount(), e);
Thread.sleep(backoffSeconds * 1000);
}
var nextTry = Math.pow(2, i++);
log.error("Error uploading S3. trying in {}s later. ({}/{})", nextTry, i, option.retryAmount(), e);
Thread.sleep((long) (nextTry * 1000));
} catch (Exception e) {
throw new IllegalStateException("Unrecoverable exception occurred when uploading " + pathToBundle, e);
} finally {
uploading.set(false);
}
} while (true);
}
@Override
@SneakyThrows
public void recoverBackup(Backup backup, Path restorePath) {
var resp = s3.getObject(b -> b.bucket(option.bucket()).key(backup.backupKey()));
public void recoverBackup(Backup backup, Path restorePath, boolean ignoreDeletions) {
var target = temporaryDownloadPath.resolve("s3_" + System.currentTimeMillis());
try (var fs = Files.newOutputStream(target, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE)) {
try {
try (var resp = s3.getObject(option.bucket(), backup.backupKey());
var fs = Files.newOutputStream(target, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE)) {
resp.transferTo(fs);
BundleReader.builder().build().extract(target, restorePath);
}
BundleReader.builder().ignoreDeletions(ignoreDeletions).build().extract(target, restorePath);
} finally {
Files.deleteIfExists(target);
}
@@ -93,15 +77,11 @@ public class S3BackupStrategy implements BackupStrategy {
@Override
@SneakyThrows
public void deleteBackup(Backup backup) {
s3.deleteObject(b -> b.bucket(option.bucket()).key(backup.backupKey()).build());
s3.deleteObject(option.bucket(), backup.backupKey());
}
@Override
public boolean isAvailable() {
if (uploading.get()) {
log.warn("Last S3 Backup is still uploading!");
return false;
}
return true;
}
+3 -3
View File
@@ -19,9 +19,9 @@
"justbackup.mixins.json"
],
"depends": {
"fabricloader": ">=0.16.14",
"minecraft": "~1.21.8",
"java": ">=21",
"fabricloader": ">=0.19.5",
"minecraft": "~26.3",
"java": ">=25",
"fabric-api": "*"
}
}
+1 -1
View File
@@ -1,7 +1,7 @@
{
"required": true,
"package": "io.ib67.sfcraft.mixin",
"compatibilityLevel": "JAVA_21",
"compatibilityLevel": "JAVA_25",
"mixins": [
"MixinMinecraftServer",
"MixinRegionBasedStorage"
@@ -0,0 +1,88 @@
package io.ib67.sfcraft;
import io.ib67.sfcraft.strategy.BackupChains;
import org.junit.jupiter.api.Test;
import java.util.List;
import static org.junit.jupiter.api.Assertions.*;
public class TestBackupChains {
private static Backup full(String key, String subject, long at) {
return new Backup(key, key, "local", "world", subject, false, at, 1);
}
private static Backup incr(String key, String subject, long at) {
return new Backup(key, key, "local", "world", subject, true, at, 1);
}
private static Backup legacy(String key) {
return new Backup(key, key, "local", "world", null, false, 0, 1);
}
@Test
public void fullBackupRestoresAlone() {
var full = full("f1", "world", 100);
var all = List.of(full, incr("i1", "world", 200));
assertEquals(List.of(full), BackupChains.resolveChain(all, full));
}
@Test
public void incrementalChainsBackToItsBase() {
var f1 = full("f1", "world", 100);
var i1a = incr("i1a", "world", 200);
var i1b = incr("i1b", "world", 300);
var f2 = full("f2", "world", 400);
var i2a = incr("i2a", "world", 500);
var all = List.of(i2a, f1, i1b, f2, i1a); // deliberately unordered
assertEquals(List.of(f1, i1a, i1b), BackupChains.resolveChain(all, i1b));
assertEquals(List.of(f2, i2a), BackupChains.resolveChain(all, i2a));
}
@Test
public void chainIgnoresOtherSubjects() {
var f1 = full("f1", "world", 100);
var otherFull = full("fo", "nether", 150);
var i1 = incr("i1", "world", 200);
assertEquals(List.of(f1, i1), BackupChains.resolveChain(List.of(f1, otherFull, i1), i1));
}
@Test
public void orphanIncrementalIsRejected() {
var i1 = incr("i1", "world", 200);
assertThrows(IllegalStateException.class,
() -> BackupChains.resolveChain(List.of(i1), i1));
}
@Test
public void legacyEntryRestoresStandalone() {
var old = legacy("old");
assertEquals(List.of(old), BackupChains.resolveChain(List.of(old), old));
}
@Test
public void rotationKeepsWholeGenerations() {
var f1 = full("f1", "world", 100);
var i1 = incr("i1", "world", 200);
var f2 = full("f2", "world", 300);
var i2 = incr("i2", "world", 400);
var f3 = full("f3", "world", 500);
var all = List.of(f1, i1, f2, i2, f3);
assertEquals(List.of(), BackupChains.rotationVictims(all, "world", 3));
assertEquals(List.of(f1, i1), BackupChains.rotationVictims(all, "world", 2));
assertEquals(List.of(f1, i1, f2, i2), BackupChains.rotationVictims(all, "world", 1));
}
@Test
public void rotationNeverTouchesLegacyOrOrphans() {
var orphan = incr("orphan", "world", 50);
var old = legacy("old");
var f1 = full("f1", "world", 100);
var f2 = full("f2", "world", 200);
var all = List.of(orphan, old, f1, f2);
assertEquals(List.of(f1), BackupChains.rotationVictims(all, "world", 1));
}
}
@@ -0,0 +1,59 @@
package io.ib67.sfcraft.s3;
import org.junit.jupiter.api.Test;
import java.net.URLEncoder;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.util.TreeMap;
import static org.junit.jupiter.api.Assertions.assertEquals;
/**
* Test vectors from
* https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
*/
class AwsV4SignerTest {
private static final String ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE";
private static final String SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";
@Test
void getObjectExample() {
var headers = new TreeMap<String, String>();
headers.put("host", "examplebucket.s3.amazonaws.com");
headers.put("range", "bytes=0-9");
headers.put("x-amz-content-sha256", AwsV4Signer.EMPTY_PAYLOAD_SHA256);
headers.put("x-amz-date", "20130524T000000Z");
var authorization = AwsV4Signer.authorization("GET", "/test.txt", "",
headers, AwsV4Signer.EMPTY_PAYLOAD_SHA256,
"20130524T000000Z", "us-east-1", "s3", ACCESS_KEY, SECRET_KEY);
assertEquals("AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,"
+ "SignedHeaders=host;range;x-amz-content-sha256;x-amz-date,"
+ "Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41",
authorization);
}
@Test
void putObjectExample() {
var payloadHash = AwsV4Signer.sha256Hex("Welcome to Amazon S3.".getBytes(StandardCharsets.UTF_8));
assertEquals("44ce7dd67c959e0d3524ffac1771dfbba87d2b6b4b4e99e42034a8b803f8b072", payloadHash);
var headers = new TreeMap<String, String>();
headers.put("date", "Fri, 24 May 2013 00:00:00 GMT");
headers.put("host", "examplebucket.s3.amazonaws.com");
headers.put("x-amz-content-sha256", payloadHash);
headers.put("x-amz-date", "20130524T000000Z");
headers.put("x-amz-storage-class", "REDUCED_REDUNDANCY");
var authorization = AwsV4Signer.authorization("PUT", "/test%24file.text", "",
headers, payloadHash,
"20130524T000000Z", "us-east-1", "s3", ACCESS_KEY, SECRET_KEY);
assertEquals("AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,"
+ "SignedHeaders=date;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class,"
+ "Signature=98ad721746da40c64f1a55b78f14c238d841ea1380cd77a1b5971af0ece108bd",
authorization);
}
}