added app resources for apps from appstores

docs/scarpet/api/Overview.md

@@ -9,11 +9,12 @@ These options affect directly how scarpet functions and can be triggered via `/c
  when loaded with the world (i.e. present in the world/scripts folder)
  - `scriptsAutoload`: when set to `false` will prevent apps loaded with the world to load automatically. You can still
  load them on demand via `/script load` command
- - `commandScriptACE`: command permission level that is used to trigger commands from scarpet scripts (regardless who triggeres
+ - `commandScriptACE`: command permission level that is used to trigger commands from scarpet scripts (regardless who triggers
  the code that calls the command). Defaults to `ops`, could be customized to any level via a numerical value (0, 1, 2, 3 or 4)
  - `scriptsOptimization`: when disabled, disables default app compile time optimizations. If your app behaves differently with
- and without optimizations, please file a bug report on the bug tracker and disable code optimizations
+ and without optimizations, please file a bug report on the bug tracker and disable code optimizations.
  - `scriptsDebugging`: Puts detailed information about apps loading, performance and runtime in system log.
+ - `scriptsAppStore`: location of the app store for downloadable scarpet apps - can be configured to point to other scarpet app store.
 
 ## App structure
 
@@ -103,6 +104,36 @@ predicate that is volatile and might change, the command might falsely do or do
 however player will always be able to type it in and either succeed, or fail, based on their current permissions.
 Custom permission applies to legacy commands with `'legacy_command_type_support'` as well
 as for the custom commands defined with `'commands'`, see below.
+*  `'resources'` - list of all downloadable resources when installing the app from an app store. List of resources needs to be 
+in a list and contain of map-like resources descriptors, looking like
+   ```
+   'resources' -> [
+        {
+            'source' -> 'https://raw.githubusercontent.com/gnembon/fabric-carpet/master/src/main/resources/assets/carpet/icon.png',
+            'type' -> 'url',
+            'target' -> 'foo/photos.zip/foo/cm.png',
+        },
+        {
+            'source' -> 'survival/README.md',
+            'type' -> 'store',
+            'target' -> 'survival_readme.md',
+            'shared' -> true,
+        },
+        {
+            'source' -> 'carpets.sc',
+            'type' -> 'app',
+            'target' -> 'apps/flying_carpets.sc',
+            'shared' -> true,
+        },
+    ]
+   ```
+   `source` and `type` indicate resource location: either an arbitrary url (type `'url'`), 
+   absolute location of a file in the app store (type `'store'`),
+or a relative location in the same folder as the app in question (type `'app'`). 
+`'target'` points to the path in app data, or shared app data folder
+if `'shared'` is specified and `true`. When re-downloading the app, all resources will be re-downloaded as well. 
+Currently, app resources
+are only downloaded when using `/carpet download` command.
 *   `'arguments'` - defines custom argument types for legacy commands with `'legacy_command_type_support'` as well
 as for the custom commands defined with `'commands'`, see below.
 *   `'commands'` - defines custom commands for the app to be executed with `/<app>` command, see below.

docs/scarpet/api/ScriptCommand.md

@@ -170,8 +170,9 @@ calling `gametick` should be balanced in each case separately
 
 # `/script download` command
 
-`/script download <global or local?> <path>` command allows to download and save scripts directly from [scarpet app store](https://www.github.com/gnembon/scarpet).
-You can choose whether to save the script in the global config scripts folder or the local per-world script folder, 
-(on servers it always saves in the scripts folder) and it will be saved there, e.g `/script download local survival/angel_block.sc`
-would save the `angel_block.sc` file [here](https://github.com/gnembon/scarpet/blob/master/programs/survival/angel_block.sc)
-to a folder called `appstore/survival` in the world's scripts folder.
+`/script download <path>` command allows downloading and running apps directly from an online app store (it's all free), 
+by default the [scarpet app store](https://www.github.com/gnembon/scarpet).
+Downloaded apps will be placed in the world's scripts folder automatically. Location of the app store is controlled
+with a global carpet setting of `/carpet scriptsAppStore`. Apps, if required, will also download all the resources they need
+to run it. Consecutive downloads of the same app will re-download its content and its resources, but will not remove anything
+that has been removed or renamed.

src/main/java/carpet/commands/ScriptCommand.java

@@ -119,8 +119,6 @@ public class ScriptCommand
     ) throws CommandSyntaxException {
         try {
             String previous = suggestionsBuilder.getRemaining();
-            List<String> suggestions = AppStoreManager.suggestionsFromPath(previous);
-            CarpetScriptServer.LOG.error(String.join(":", suggestions));
             AppStoreManager.suggestionsFromPath(previous).forEach(suggestionsBuilder::suggest);
             return suggestionsBuilder.buildFuture();
         }

src/main/java/carpet/script/CarpetScriptHost.java

@@ -48,6 +48,7 @@ import net.minecraft.util.math.BlockPos;
 import org.apache.commons.lang3.tuple.Pair;
 
 import java.math.BigInteger;
+import java.nio.file.Path;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.Comparator;
@@ -997,6 +998,12 @@ public class CarpetScriptHost extends ScriptHost
         return fdesc.listFolder(main);
     }
 
+    public boolean applyActionForResource(String path, boolean shared, Consumer<Path> action)
+    {
+        FileArgument fdesc = FileArgument.resourceFromPath(path, FileArgument.Reason.CREATE, shared);
+        return fdesc.findPathAndApply(main, action);
+    }
+
 
     public void tick()
     {

src/main/java/carpet/script/CarpetScriptServer.java

@@ -285,7 +285,8 @@ public class CarpetScriptServer
             return false;
         }
         //addEvents(source, name);
-        String action = reload?"reloaded":"loaded";
+        String action = (installer!=null)?(reload?"reinstalled":"installed"):(reload?"reloaded":"loaded");
+
         
         Boolean isCommandAdded = newHost.addAppCommands(s -> {
             if (!isRuleApp) Messenger.m(source, s);

src/main/java/carpet/script/argument/FileArgument.java

@@ -46,6 +46,7 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import java.util.function.Consumer;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 
@@ -79,7 +80,8 @@ public class FileArgument
         TEXT("text", ".txt"),
         NBT("nbt", ".nbt"),
         JSON("json", ".json"),
-        FOLDER("folder", "");
+        FOLDER("folder", ""),
+        ANY("any", "");
 
         private final String id;
         private final String extension;
@@ -118,11 +120,11 @@ public class FileArgument
     public static FileArgument from(List<Value> lv, boolean isFolder, Reason reason)
     {
         if (lv.size() < 2) throw new InternalExpressionException("File functions require path and type as first two arguments");
-        Pair<String, String> resource = recognizeResource(lv.get(0), isFolder);
         String origtype = lv.get(1).getString().toLowerCase(Locale.ROOT);
         boolean shared = origtype.startsWith("shared_");
         String typeString = shared ? origtype.substring(7) : origtype; //len(shared_)
         Type type = Type.of.get(typeString);
+        Pair<String, String> resource = recognizeResource(lv.get(0).getString(), isFolder, type);
         if (type==null)
             throw new InternalExpressionException("Unsupported file type: "+origtype);
         if (type==Type.FOLDER && !isFolder)
@@ -131,18 +133,26 @@ public class FileArgument
 
     }
 
-    public static Pair<String,String> recognizeResource(Value value, boolean isFolder)
+    public static FileArgument resourceFromPath(String path, Reason reason, boolean shared)
+    {
+        Pair<String, String> resource = recognizeResource(path, false, Type.ANY);
+        return  new FileArgument(resource.getLeft(), Type.ANY,resource.getRight(), false, shared, reason);
+    }
+
+    public static Pair<String,String> recognizeResource(String origfile, boolean isFolder, Type type)
     {
-        String origfile = value.getString();
         String[] pathElements = origfile.toLowerCase(Locale.ROOT).split("[/\\\\]+");
         List<String> path = new ArrayList<>();
         String zipPath = null;
-        for (String token : pathElements)
+        for (int i =0; i < pathElements.length; i++ )
         {
+            String token = pathElements[i];
             boolean isZip = token.endsWith(".zip");
             if (zipPath != null && isZip) throw new InternalExpressionException(token+" indicates zip access in an already zipped location "+zipPath);
             if (isZip) token = token.substring(0, token.length()-4);
-            token = token.replaceAll("[^A-Za-z0-9\\-+_]", "");
+            token = (type==Type.ANY && i == pathElements.length-1)? // sloppy really, but should work
+                    token.replaceAll("[^A-Za-z0-9\\-+_.]", ""):
+                    token.replaceAll("[^A-Za-z0-9\\-+_]", "");
             if (token.isEmpty()) continue;
             if (isZip) token = token + ".zip";
             path.add(token);
@@ -180,6 +190,7 @@ public class FileArgument
                 zipPath = resolve(getDescriptor(module, zipContainer));
                 if (!Files.exists(zipPath) && reason != Reason.CREATE) return null; // no zip file
                 try {
+                    if (!Files.exists(zipPath.getParent())) Files.createDirectories(zipPath.getParent());
                     zfs = FileSystems.newFileSystem(URI.create("jar:"+ zipPath.toUri().toString()), env);
                 }
                 catch (FileSystemNotFoundException | IOException fsnfe)
@@ -215,6 +226,23 @@ public class FileArgument
         throw new InternalExpressionException("Invalid file descriptor: "+res);
     }
 
+
+    public boolean findPathAndApply(Module module, Consumer<Path> action)
+    {
+        try { synchronized (writeIOSync)
+        {
+            Path dataFile = toPath(module);//, resourceName, supportedTypes.get(type), isShared);
+            if (dataFile == null) return false;
+            createPaths(dataFile);
+            action.accept(dataFile);
+        } }
+        finally
+        {
+            close();
+        }
+        return true;
+    }
+
     public Stream<Path> listFiles(Module module)
     {
         Path dir = toPath(module);

src/main/java/carpet/script/utils/AppStoreManager.java

@@ -2,10 +2,14 @@ package carpet.script.utils;
 
 import carpet.CarpetServer;
 import carpet.script.CarpetScriptHost;
+import carpet.script.CarpetScriptServer;
+import carpet.script.exception.InternalExpressionException;
+import carpet.script.value.MapValue;
 import carpet.script.value.Value;
 import carpet.settings.ParsedRule;
 import carpet.settings.Validator;
 import carpet.utils.Messenger;
+import com.google.common.collect.ImmutableList;
 import com.google.gson.JsonArray;
 import com.google.gson.JsonElement;
 import com.google.gson.JsonObject;
@@ -24,12 +28,12 @@ import java.io.InputStreamReader;
 import java.io.Reader;
 import java.io.StringWriter;
 import java.io.Writer;
-import java.net.HttpURLConnection;
 import java.net.URL;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.HashMap;
@@ -52,11 +56,6 @@ public class AppStoreManager
      */
     private static String scarpetRepoLink = "https://api.github.com/repos/gnembon/scarpet/contents/programs/";
 
-    public static void addResource(CarpetScriptHost carpetScriptHost, StoreNode storeSource, Value resource)
-    {
-        // todo
-        // locate online file, download to specified location
-    }
 
     public static class ScarpetAppStoreValidator extends Validator<String>
     {
@@ -134,8 +133,7 @@ public class AppStoreManager
             String response;
             try
             {
-                URL appURL = new URL(queryPath);
-                response = AppStoreManager.getStringFromStream(appURL.openStream());
+                response = AppStoreManager.getStringFromStream(queryPath);
             }
             catch (IOException e)
             {
@@ -150,7 +148,7 @@ public class AppStoreManager
                 {
                     children.put(name, folder(this, name));
                 }
-                else if (name.matches("(\\w+\\.scl?)"))
+                else// if (name.matches("(\\w+\\.scl?)"))
                 {
                     String value = jo.get("download_url").getAsString();
                     children.put(name, scriptFile(this, name, value));
@@ -176,11 +174,16 @@ public class AppStoreManager
         {
             if (isLeaf())
             {
-                return Collections.singletonList(getPath());
+                if (name.endsWith(".sc"))
+                    return Collections.singletonList(getPath());
+                return Collections.emptyList();
             }
             fillChildren();
             String prefix = getPath();
-            return children.values().stream().map(s -> prefix+s.pathElement().replaceAll("/$", "")).collect(Collectors.toList());
+            return children.values().stream().
+                    filter(n -> (!n.isLeaf() || n.name.endsWith(".sc"))).
+                    map(s -> prefix+s.pathElement().replaceAll("/$", "")).
+                    collect(Collectors.toList());
         }
 
         public StoreNode drillDown(String pathElement) throws IOException
@@ -220,8 +223,7 @@ public class AppStoreManager
 
 
 
-    /** A simple shorthand for calling the {@link AppStoreManager#getScriptCode} and {@link AppStoreManager#saveScriptToFile}
-     * methods to avoid repeating code and so it makes more sense what it's exactly doing.
+    /** Downloads script and saves it to appropriate place.
      *
      * @param path The user-inputted path to the script
      * @return {@code 1} if we succesfully saved the script, {@code 0} otherwise
@@ -229,9 +231,18 @@ public class AppStoreManager
 
     public static int downloadScript(ServerCommandSource source, String path)
     {
-        Triple<String,String, StoreNode> code = getScriptCode(path);
-        if (!saveScriptToFile(source, path, code.getLeft(), code.getMiddle(), false)) return 0;
-        boolean success = CarpetServer.scriptServer.addScriptHost(source, code.getLeft(), null, true, false, false, code.getRight());
+        Triple<String, String, StoreNode> nodeInfo = getFileNode(path);
+        String code;
+        try
+        {
+            code = getStringFromStream(nodeInfo.getMiddle());
+        }
+        catch (IOException e)
+        {
+            throw new CommandException(Messenger.c("rb Failed to obtain app file content: "+e.getMessage()));
+        }
+        if (!saveScriptToFile(source, path, nodeInfo.getLeft(), code, false)) return 0;
+        boolean success = CarpetServer.scriptServer.addScriptHost(source, nodeInfo.getLeft().replaceFirst("\\.sc$", ""), null, true, false, false, nodeInfo.getRight());
         return success?1:0;
     }
 
@@ -240,18 +251,22 @@ public class AppStoreManager
      * @param appPath The user inputted path to the scarpet script
      * @return Pair of app file name and content
      */
-    public static Triple<String,String, StoreNode> getScriptCode(String appPath)
+    public static Triple<String, String, StoreNode> getFileNode(String appPath)
+    {
+        return getFileNodeFrom(appStoreRoot, appPath);
+    }
+
+    public static Triple<String, String, StoreNode> getFileNodeFrom(StoreNode start, String appPath)
     {
         String[] path = appPath.split("/");
-        StoreNode appKiosk = appStoreRoot;
+        StoreNode appKiosk = start;
         try
         {
             for(String pathElement : Arrays.copyOfRange(path, 0, path.length-1))
                 appKiosk = appKiosk.drillDown(pathElement);
             String appName = path[path.length-1];
-            URL appURL = new URL(appKiosk.getValue(appName));
-            HttpURLConnection http = (HttpURLConnection) appURL.openConnection();
-            return Triple.of(appName, getStringFromStream((InputStream) http.getContent()), appKiosk);
+            appKiosk.getValue(appName);
+            return Triple.of(appName, appKiosk.getValue(appName), appKiosk);
         }
         catch (IOException e)
         {
@@ -259,6 +274,7 @@ public class AppStoreManager
         }
     }
 
+
     public static boolean saveScriptToFile(ServerCommandSource source, String path, String appFileName, String code, boolean globalSavePath){
         String  scriptPath;
         String location;
@@ -277,17 +293,16 @@ public class AppStoreManager
             Path scriptLocation = Paths.get(scriptPath);
             Files.createDirectories(scriptLocation.getParent());
             if(Files.exists(scriptLocation))
-                Messenger.m(source, String.format("gi Note: overwriting existing app '%s'", appFileName));
+                Messenger.m(source, String.format("gi Note: replaced existing app '%s'", appFileName));
             FileWriter fileWriter = new FileWriter(scriptPath);
             fileWriter.write(code);
             fileWriter.close();
         }
         catch (IOException e)
         {
-            Messenger.m(source, "r Error in downloading script: "+e.getMessage());
+            Messenger.m(source, "r Error in downloading app: "+e.getMessage());
             return false;
         }
-        Messenger.m(source, "gi Successfully created "+ appFileName + " in " + location);
         return true;
     }
 
@@ -298,8 +313,9 @@ public class AppStoreManager
      * @return the string input from the InputStream
      * @throws IOException if an I/O error occurs
      */
-    public static String getStringFromStream(InputStream inputStream) throws IOException
+    public static String getStringFromStream(String url) throws IOException
     {
+        InputStream inputStream = new URL(url).openStream();
         if (inputStream == null)
             throw new IOException("No app to be found on the appstore");
 
@@ -319,4 +335,55 @@ public class AppStoreManager
         }
         return stringWriter.toString();
     }
+
+    public static void writeUrlToFile(String url, Path destination) throws IOException
+    {
+        try (final InputStream in = new URL(url).openStream())
+        {
+            Files.copy(in, destination, StandardCopyOption.REPLACE_EXISTING);
+        }
+    }
+
+    public static void addResource(CarpetScriptHost carpetScriptHost, StoreNode storeSource, Value resource)
+    {
+        if (!(resource instanceof MapValue))
+            throw new InternalExpressionException("This is not a valid resource map: "+resource.getString());
+        Map<String, Value> resourceMap = ((MapValue) resource).getMap().entrySet().stream().collect(Collectors.toMap(e -> e.getKey().getString(), Map.Entry::getValue));
+        for (String key: ImmutableList.of("source", "target", "type"))
+            if (!resourceMap.containsKey(key)) throw new InternalExpressionException("Missing '"+key+"' field in resource descriptor: "+resource.getString());
+        String source = resourceMap.get("source").getString();
+        String target = resourceMap.get("target").getString();
+        boolean shared = resourceMap.getOrDefault("shared", Value.FALSE).getBoolean();
+        String type = resourceMap.get("type").getString();
+        String contentUrl;
+        if (type.equalsIgnoreCase("url"))
+        {
+            contentUrl = source;
+        }
+        else if (type.equalsIgnoreCase("store"))
+        {
+            Triple<String, String, StoreNode> nodeInfo = getFileNode(source);
+            contentUrl = nodeInfo.getMiddle();
+        }
+        else if (type.equalsIgnoreCase("app"))
+        {
+            Triple<String, String, StoreNode> nodeInfo = getFileNodeFrom(storeSource, source);
+            contentUrl = nodeInfo.getMiddle();
+        }
+        else
+        {
+            throw new InternalExpressionException("Incorrect source type, expecting 'url', 'store' or 'app', got "+type);
+        }
+        if (!carpetScriptHost.applyActionForResource(target, shared, p -> {
+            try {
+                writeUrlToFile(contentUrl, p);
+            } catch (IOException e) {
+                throw new InternalExpressionException("Unable to write resource "+target+": "+e.getMessage());
+            }
+        }))
+        {
+            throw new InternalExpressionException("Unable to write resource "+target);
+        }
+        CarpetScriptServer.LOG.info("Downloaded resource "+target+" from "+contentUrl);
+    }
 }